Bug Detection Blind Spots in AI Coding Harnesses (GStack and Beyond) | Towards Data Science

1. Introduction

a coding agent in action, you’ve probably noticed the default workflow: let the AI handle the routine bugs, but step in when the problem gets complicated. One-line fixes go to the agent; deep internals, numerical edge cases, and cross-file invariants stay with the senior engineer.

That made me wonder: what kinds of bugs do coding agents actually struggle with?

Over the past month, I ran 28 blind-scored debugging experiments on three real, recently fixed bugs from production open-source libraries: ky, immer, and decimal.js.

Ironically, the two bugs I expected to be the hardest turned out not to be a problem at all. One was buried deep inside Immer’s internals, and the other was a subtle numerical edge case in decimal.js. Across 16 attempts, the AI fixed both correctly every single time.

The third bug looked almost trivial: an HTTP client silently dropped a retry option. Yet it defeated every model and workflow I tested.

Across all 12 runs, the AI produced a “fix” that actually corrupts user data. By any reasonable standard, none of those runs were successful.

What concerns me isn’t just that the fixes were wrong—it’s that every single one passed the entire 84-test retry suite for the code under repair.

That’s the real failure mode. If your team merges AI-generated fixes because CI is green, this is exactly the kind of bug that slips through: it looks simple, the tests pass, and the implementation quietly corrupts user data when it collides with Ky’s existing option shape.

Here’s what these 28 runs revealed—and why I think they’re relevant to anyone building or deploying coding agents.

If you’re an engineer who relies on AI to write production code, a tech lead deciding where AI can be trusted, or a researcher interested in the real limits of code generation, these results highlight a failure mode that isn’t obvious from benchmark scores. The question isn’t whether AI can solve hard bugs. It’s whether it knows when it doesn’t have enough information to solve an easy one.

1. Difficulty didn’t predict failure—missing information did. When the correct fix could be inferred from the codebase and the bug report, the AI succeeded in all 16 attempts, including bugs buried in unfamiliar proxy internals and subtle numerical edge cases. But when the correct fix depended on an undocumented API contract, it failed in all 12 attempts—across Claude Haiku 4.5, Sonnet 5, and Opus 4.8, using three different agent workflows.
2. More process didn’t fix the problem. In one experiment, a reviewer agent correctly identified that the proposed patch would corrupt user data and explained exactly why. It then approved the change anyway, reasoning that the issue was unlikely and belonged to a pre-existing class of problems. The failure wasn’t in detection—it was in judgment. Even when the system recognized the risk, it lacked the decision-making needed to stop the bad fix from shipping.

2. Let’s start with the bugs that I tested

To perform this test, I select real bugs from real codebases, with real ground truth: the maintainer’s merged fix and the regression tests that shipped with it.

Three selection rules did the heavy lifting.

First, every bug was fixed upstream in July 2026, and it’s highly probable that the training cutoff for the latest Claude models (the 5 family) is before this period, so no model has seen the fix. Second, each fix shipped with regression tests, held out as a hidden grader the agent never sees. Third, I chose the three bugs to span difficulty, from a one-line fix to a two-file invariant repair.

Bug	Library	What breaks	Easy or Hard	Correct fix
ky #867	HTTP client built on fetch	numeric retry limit silently lost on .extend()	looks easy: one merge rule	expand the shorthand only at the options root
immer #1255	immutability layer behind Redux Toolkit	original state mutated after reverse() / sort()	two files of proxy internals	re-draft elements the reorder relocated
decimal.js #260	arbitrary-precision arithmetic	asin() returns wrong digits near x = 1	numerical analysis: catastrophic cancellation	reformulate 1 − x² as (1 − x)(1 + x)

2.1 The easy one: ky, PR #867

ky is a small HTTP client built by Sindre Sorhus. It wraps the browser’s fetch and adds the things every app ends up needing anyway: retries, timeouts, JSON handling. The normal way to use it is to build one base client with your shared settings, then specialize it per feature:

const api = ky.create({retry: 3}); // one client, retry up to 3 times
const users = api.extend({retry: {methods: ['get']}}); // same client, but only retry GETs

retry: 3 is shorthand for “retry failed requests up to 3 times.”

The bug: when the base client sets retry as a number, and the extension sets it as an object, the number silently vanishes. The extended client quietly falls back to the default of 2 retries. Nothing crashes. No warning. Your requests just retry fewer times than you configured, which is exactly the kind of bug nobody notices until a bad network day in production.

This is the ticket every triage guide would send straight to AI. Here is the exact bug report I gave every agent, word for word:

Bug: retry limit is ignored after .extend()

ky’s retry option accepts a number as shorthand for the retry limit (the docs say: “If retry is a number, it will be used as limit and other defaults will remain in place”). But when I set a numeric retry on a base instance and then extend it with an object, the limit is silently lost and falls back to the default (2):

import ky from 'ky';
const api = ky.create({retry: 3});
const extended = api.extend({retry: {methods: ['get']}});
// I expect `extended` to still retry up to 3 times, only narrowing the retriable methods.
// Instead it retries with the default limit of 2.

Setting both parent and child as objects works. Setting the number on the child works. Only number-on-parent + object-on-child loses the limit. Please fix it so the limit is preserved. You can check more on Figure 2.

2.2 The hard one: immer, PR #1255

Immer, the immutability engine behind Redux Toolkit, guarantees that the state you pass in is never mutated. You edit a proxy draft, and Immer produces a new state while leaving the original untouched—a key property for React.

With the optional array-methods plugin enabled, that guarantee broke. Calling reverse() or sort() on a draft array, then modifying an element, could mutate the caller’s original state.

The bug was subtle. Internally, Immer tracks draft objects by their array position. But reverse() and sort() reorder elements, so an original object can bypass the proxy check and be returned directly. Subsequent writes then mutate the original state instead of the draft.

The exact bug report every agent got:

Bug: mutating an element after reverse()/sort() mutates the original base state

immer promises never to modify the base state passed to produce. But with the array-methods plugin enabled, calling reverse() or sort() inside a recipe and then writing to an element leaks the write into the caller’s original array.

import {produce, enableArrayMethods, setAutoFreeze} from "immer"
enableArrayMethods()
setAutoFreeze(false)
const obj3 = {id: 3}
const base = [{id: 1}, {id: 2}, obj3]
const next = produce(base, d => {
d.reverse()
d[0].id = 99
})
// BUG: base is now [{id:1},{id:2},{id:99}] and obj3 is {id:99} — the base was mutated.
// Expected: base stays [{id:1},{id:2},{id:3}] and only `next` reflects the change.

The same recipe works correctly without the array-methods plugin. Please fix it so the base is never mutated. Check Figure 3 for more details.

2.3 The other hard one: decimal.js, PR #260

decimal.js exists because ordinary floating-point numbers can’t be trusted when precision matters—financial calculations, scientific computing, or anywhere the last digit counts. It provides arbitrary-precision arithmetic.

Its asin (inverse sine) implementation computed 1 - x² directly. For values of x close to 1, that’s numerically unstable: subtracting two nearly identical numbers causes catastrophic cancellation, where most significant digits disappear and the remaining digits are dominated by rounding error. The result was incorrect final digits precisely in the high-precision cases users cared about.

The interesting part was the fix. The reporter suggested increasing the internal precision, but that only moves the failure point—the same error reappears as x gets even closer to 1. The real solution was an algebraic rewrite: compute (1 - x)(1 + x) instead of 1 - x². The two expressions are mathematically equivalent, but the latter avoids catastrophic cancellation and preserves precision.

The exact bug report every agent got:

Bug: asin() loses precision for inputs very close to 1

Decimal.asin(x) returns a result whose last digit or two is wrong when x is very close to 1 (or -1), at higher precisions. The closer x gets to 1, the more the tail digits drift. Values well away from ±1 are fine.

const Decimal = require('./decimal.js');
Decimal.set({ precision: 30 });
// For x extremely close to 1, e.g. 0.99999999999999999, the final digits of
// asin(x) disagree with a high-precision reference (checked against mpmath).
console.log(new Decimal('0.99999999999999999').asin().toString());

Please make asin() accurate near ±1. Check Figure 4 for more details.

3. How I tested: models, workflows, and scoring

Every run followed the same shape: one AI agent (or team of agents), one fresh copy of the buggy repo, one symptom-only bug report, and a hidden grader the agent never sees.

3.1 The models

Three Claude tiers: Haiku 4.5 (the cheapest), Sonnet 5 (the middle), and Opus 4.8 (the most capable). Same bug reports, same repos, same rules.

3.2 Three workflows

This is where the exact prompts matter, so here they are.

Workflow 1, naive single agent. One agent gets the bug report above plus the repo and this instruction, verbatim:

Investigate, reproduce, and fix the bug. Then reply with a report containing exactly:

1. ROOT CAUSE: what actually caused the reported behavior, and the file(s)/lines involved.

2. DIFF: the final unified diff of your change.

3. TESTS: the result of running the test suite after your fix.

4. CONFIDENCE: low / medium / high that your fix is correct and complete, and one sentence why.

Workflow 2, gstack investigate (gstack an open-source suite of opinionated AI coding workflows that turns coding agents into a virtual software team for planning, building, reviewing, testing, and shipping software. Source: https://github.com/garrytan/gstack).

Same bug report, but the agent must follow five explicit steps and show each one. Step 3 is the one designed to catch exactly the trap in this study, quoted verbatim:

1. REPRODUCE — write a minimal repro and confirm the reported behavior.

2. ROOT CAUSE — trace the exact mechanism; name the file and lines.

3. IMPACT ENUMERATION — BEFORE writing any fix, enumerate everything your intended change will touch. List every code path and input that will reach the condition your fix keys on. Ask adversarially: what OTHER data or inputs could match that condition? What could my change affect that the bug report never mentioned? Write this list out in full.

4. IMPLEMENT — make the fix.

5. SELF-REVIEW — re-read your IMPACT ENUMERATION against your implementation. For each item, confirm the fix behaves correctly or adjust it. Then run the tests.

Workflow 3, parallel pipeline.

Four agents in sequence. Two independent diagnosers first, each told: “You are a read-only DIAGNOSTIC agent… Another engineer will implement the fix from your diagnosis, so be precise.” Then an implementer, given both diagnoses: “Two engineers independently diagnosed the bug; their diagnoses are below. Implement the fix, using their analysis. If they disagree, use your judgment.” Finally a reviewer with the power to change code, told: “Review their change critically before it ships — this is the last gate before merge… Does the fix have any UNINTENDED side effects? Think hard about everything the changed code path touches… What could be affected that the original bug report never mentioned? … If you find any problem, FIX it directly in the code yourself.”

3.3 Execution and scoring

Each run started from a fresh checkout of the repository at the pre-fix commit, fully isolated from every other run. Agents could run the library’s existing visible tests as often as they wanted: ky’s 84-test retry suite, Immer’s base test suite, or decimal.js’s 22,624-assertion suite. What they never saw were the regression tests the maintainer added with the actual fix.

After each run, I restored the original test files, added the held-out regression tests, ran the full suite, and scored only the code left on disk. The agent’s own claim of success did not count—and that distinction mattered: agents routinely wrote tests that passed their own fixes and then declared victory. A run was considered correct only if it passed the hidden tests capturing the maintainer’s final intended behavior, not merely the reported symptom.

I ran this 28 times as shown in the table below:

4. Results

4.1 The hard bugs completed 16 times out of 16

If raw algorithmic complexity were what caused AI debugging to fail, these were the bugs where I expected it to break down.

That was my hypothesis going into the experiments.

The Immer bug required reconstructing a structural-sharing invariant hidden deep inside its proxy implementation—something that’s challenging even for an engineer familiar with the codebase, let alone a model seeing it for the first time.

The decimal.js bug seemed even more deceptive. Before running the experiments, I verified that the obvious fix—the one suggested in the original bug report, simply increasing the working precision—passes all 22,624 visible assertions while still failing all four hidden edge-case tests near the numerical boundary. In other words, the most tempting solution appears completely correct unless you understand the underlying numerical analysis.

Instead, the results were the exact opposite of what I expected. Across every workflow and two model tiers, all 16 runs produced correct fixes. The consistency surprised me far more than the success itself.

For Immer, every run identified the same root cause: the proxy assumes an array element is unchanged if it matches the element at the same index in the original array. That assumption fails after reverse(), which reorders elements in place, causing moved objects to be returned directly instead of wrapped as drafts.

The agents produced two distinct but correct fixes. Five of the eight runs ensured any undrafted element was wrapped once the array had been reordered. The remaining three removed the index-based assumption entirely, checking whether an element came from the original array regardless of its current position.

The decimal.js results were equally notable. None of the eight runs took the tempting but incomplete “just increase the precision” approach, even though it passed the visible tests. Six of the eight runs independently rediscovered essentially the same reformulation later merged by the maintainer, with three citing the related acos fix (PR #217). The remaining two instead made the extra precision adaptive, adding more guard digits as the input approaches 1—an alternative that also passes the hidden tests.

These experiments suggest that difficult reasoning, unfamiliar code, and subtle mathematics were not the main bottlenecks. When the information needed to derive the correct fix was already present in the codebase or bug report, the models consistently found it.

On bugs whose root cause can be uncovered by reading and reasoning about the code, current models performed better than I expected. I underestimated them.

Both bugs also contained the clues needed to derive the correct fix. In decimal.js, the sibling acos implementation already used the cancellation-free formulation. In Immer, the plugin sets a flag indicating the array has been reordered. The necessary information was already in the codebase, and every run found and used it.

That makes what happened on the easy bug even stranger.

4.2. Then a one-line bug beat everything

Back to Ky.

By the usual intuition about AI debugging, this should have been the easiest bug—simpler than either Immer or decimal.js. At first, it looked that way.

The bug was straightforward. If retry is first set to a number and later overridden with a retry object, the numeric limit is silently lost. All 12 runs identified the root cause: Ky’s generic deep-merge logic doesn’t understand its own numeric retry shorthand. Merging { retry: 3 } with a retry object simply discards the 3.

The obvious fix was to normalize numeric values into { limit: 3 } before merging. It fixed the reported bug and passed all 84 visible retry tests.

But it was still wrong.

The same merge function also processes the user’s json payload. If that payload happens to contain a retry field, the naive fix silently rewrites user data, turning a number into a retry configuration object. The maintainer’s fix avoids this by applying the conversion only to the top-level retry option, where retry unambiguously refers to Ky’s configuration rather than arbitrary user data.

This is a fair test, not a gotcha, because the pull request contains both versions of the fix. The contributor initially made the same naive change. During review, someone realized it could corrupt user payloads, so a second commit narrowed the fix and added a regression test to ensure nested user data was never rewritten.

In other words, a human made this exact mistake in production code, and human review caught it. I used that regression test as my second hidden test.

All 12 runs reproduced the original mistake. None arrived at the reviewed fix. Every run passed the visible tests and the hidden test for the reported bug, but all 12 failed the payload-corruption test. They fixed the reported bug while breaking behavior that had previously worked. That distinction matters.

What tripped up the agents wasn’t the code itself but a fact about the world outside it: users can put arbitrary keys in their payloads. You can infer that from the codebase—one of the twelve agents actually did—but it isn’t stated in the bug report or near the function being fixed. Reaching the correct solution requires reasoning about how real users might use the library, not just the code in front of you.

That’s what makes the contrast with Immer and decimal.js interesting. In those cases, the information needed to solve the bug was buried somewhere in the repository, but it was discoverable—and all 16 runs found it.

This doesn’t seem to be an isolated pattern. A recent benchmark of precise code edits found frontier models passing unit tests more than 76% of the time while matching the maintainer’s edit less than 45% of the time. My standard is actually looser—several fixes I counted as correct differ from the maintainer’s patch—but the gap points to the same issue I saw with ky: green tests show a fix works for the cases you tested. They don’t prove the fix is actually correct.

4.3. The reviewer who found the bug and approved it anyway

In one run of the parallel pipeline, where a reviewer agent audits the implementer’s fix before it lands, the reviewer did what reviewers are for. It traced the fix and wrote out the failure precisely, in its own words: “the fix keys on the string ‘retry’ at every nesting depth… deepMerge({json:{retry:3}}, {json:{retry:{foo:1}}}) → {json:{retry:{limit:3,foo:1}}} (user request-body corruption).”

Then it approved the merge.

Its reasoning, paraphrased: the deep-merge function already couples to option names at all depths, so this is a pre-existing class of problem. A colliding key is unlikely in practice. A clean fix needs a broader refactor. Ship it with a noted follow-up. Every sentence is defensible. I still think the verdict is wrong: it’s a shipped data-corruption bug that a maintainer rejected in the real PR.

This single run reframes the entire failure. Across 12 runs, the missing contract surfaced exactly once—and the process still failed. The breakdown wasn’t detection; it was the ship decision. The reviewer weighed a credible data-corruption finding against scope and “you aren’t gonna need it,” and chose to ship.

The fix isn’t a smarter model or more agents. It’s a rule: any review that identifies a potential unintended side effect or data corruption blocks the merge. No weighing likelihood, no discretion. Under that rule, this failure becomes a catch. One run in twelve is enough—because the process only needs one reviewer to stop a bad change. Judgment is where the process failed; a gate removes that decision.

5. Conclusion

5.1 It wasn’t the model, and it wasn’t the method

Three obvious objections remain: maybe the models were too weak, maybe a single-agent workflow was the problem, or maybe a more rigorous review process would have caught the bug. The results argue against all three.

1. model strength. If this were simply a capability gap, the strongest models should avoid mistakes the cheapest ones make. They didn’t. Claude Haiku 4.5, Sonnet 5, and Opus 4.8 all fell into the same trap as single agents: 7 runs, 7 corrupting fixes. The cheapest model and the most capable one produced the same failure. That surprised me more than the Immer result.
2. workflow. I held Opus constant and tested two more structured processes. The first required explicit investigation: reproduce the bug, isolate the root cause, enumerate potential side effects (“What else could this affect that the report didn’t mention?”), then implement and self-review. Three runs, three corrupting fixes.

One run came remarkably close. It noticed that a nested user key named retry could incorrectly gain a limit, checked ky’s option types, found no such field, and dismissed the risk. It searched the library’s vocabulary. The correct answer lived in users’ data—a space no artifact ever mentioned. An investigation is only as good as the assumptions that bound it.

The second workflow—a pipeline with two independent diagnosers, an implementer, and a reviewer empowered to rewrite the patch—also failed. Both runs shipped the corruption, including one where the reviewer explicitly identified the failure mode and approved the fix anyway.

The problem wasn’t the model. It wasn’t the workflow. Across models and increasingly elaborate processes, the failure remained the same.

5.2. Important finding: Sort tickets by information, not difficulty

Stop sorting bugs by how hard they look. Ask a more useful question:

Is everything needed to produce the correct fix visible in the code and the ticket, or does correctness depend on how the system is actually used?

If the answer is “it’s all in the repo”—an invariant, an algorithm, a race condition—the results suggest AI is a much better debugger than its reputation implies. I trust it far more on these bugs than I did a month ago.

But when the fix depends on an unstated contract—who calls the code, what data flows through it, what behavior callers rely on—the picture changes completely. No model or agent setup solved that problem reliably. Worse, the bad fixes still came with green CI.

That leaves two places with real leverage. First, the ticket: every contract you make explicit gives the agent information it cannot infer from the code. One extra sentence can be worth more than a model upgrade.

Second, the review gate: if an AI reviewer flags a potential side effect or data-corruption risk, that finding should block the merge automatically. In the one case where the reviewer caught the problem, the workflow ignored it. I wouldn’t design a system that depends on someone making the right judgment call next time.

6. Limitations

The biggest limitation is the hard/easy labeling. I assigned those labels by intuition before running the experiment rather than using an independent measure. And “discoverable from the code” may partly reflect patterns the models had already seen, not just information available in the repository. A stronger design would have engineers, blind to the hypothesis, rate difficulty beforehand.

The unstated-contract result also rests on a single bug (ky PR #867). Running it across three model tiers and three workflows makes the result notable—12/12 failures versus 16/16 successes on the other bugs—but it is still one case study, not a general law.

There is also an asymmetry in scoring. For Immer and decimal.js, the hidden regression test already fails before the fix. For ky, the reported bug is already covered; the decisive hidden test instead detects payload corruption introduced by the agent’s fix. Likewise, the “visible suite” for ky refers only to the 84 retry tests covering the affected code, not the full suite, which has roughly 47 unrelated failures on Node 22.

Finally, the sample sizes are small (one to three seeds per configuration), so the results should be treated as directional rather than definitive. “Structured investigation” refers to my own prompting method, not a commercial tool. Haiku was not tested on the two hard bugs, leaving its capability floor unknown. And because all three bugs come from well-tested JavaScript/TypeScript projects, the findings may not generalize to weaker test ecosystems. Although I selected post-cutoff fixes to reduce contamination, the models may still have encountered similar bugs during training.

7. Sources and references

Tested libraries:

• ky: HTTP client built on fetch, by Sindre Sorhus
• immer: the immutability library behind Redux Toolkit
• decimal.js: arbitrary-precision arithmetic for JavaScript

Bug reports and fixes used as ground truth:

• ky PR #867: the easy bug, numeric retry limit dropped on extend; its two-commit history contains both the naive fix and the correction that scoped it away from user data
• immer PR #1255: the hard bug, base state mutated after reverse/sort under the array-methods plugin
• decimal.js PR #260: the hard bug, asin catastrophic cancellation near x = 1, fixed by reformulation rather than precision bumping
• decimal.js issue #249: the original user report for the asin bug, including the reporter’s own precision-bump suggestion the maintainer rejected
• decimal.js PR #217: the sibling acos fix that several runs cited by name — the in-repo signpost for the correct asin fix

Related research:

• Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?: recent research finding frontier models pass unit tests far more often than they produce precise edits, consistent with the failure mode shown here