One Capital Letter Was Silently Breaking My AI Support Bot, and It Wasn't in the New Model | Towards Data Science

Picture a support inbox for a bank. Every message that comes in needs to be sorted into a category, a lost card, a refund request, a wrong charge, and sent to the right team.

Now picture that sorting job handed to an AI model instead of a person. The model reads the message and hands back a short note in a fixed format, a little like a form with the same boxes every time, so the rest of the program can read it automatically and decide what to do next. No human has to interpret free text.

That fixed format is usually a small block of computer readable text called JSON, short for JavaScript Object Notation. Think of it as labeled boxes on a form. One box called intent holds the category. Another called priority holds how urgent it is.

The program reading the model's answer does not understand English. It looks for those exact boxes, spelled exactly the way it expects, every single time. If a box goes missing, or a label is spelled slightly differently than the program expects, the program has no way to notice on its own. It just quietly stops working for that one message, while everything on the surface still looks fine.

AI companies release new model versions constantly, and deciding which LLM to use for a given job usually comes down to one number from the AI testing teams already run, how often it picks the right category. That single score can go up while something else, the exact shape of the reply, quietly gets worse, and a rising average has no way to warn you.

With plain prompting, you simply write:

"Return your answer as JSON."

The model may still return:

That extra sentence can break code that expects JSON only. The model may also leave out a field or use a value the program does not expect.

Structured Outputs is a stricter feature that makes the model follow a predefined JSON structure, such as requiring intent, priority, and needs_human. It can prevent many formatting problems, but the application still needs to check whether the values and decision are correct.

I ran a real LLM regression test on one small, real application, instead of trusting the accuracy number alone. I built a support triage assistant, gave it 47 real customer messages from a public banking dataset, and ran the exact same messages through three real versions of an OpenAI model, an older one, the one I am treating as the model currently in production, and a newer candidate being considered as a replacement.

I expected the newer model to choose the correct category more often, but I also worried that it might occasionally ignore the exact format the program requires. Instead, the newer model followed the format every time. The production model made the formatting mistake. It did so quietly, on every refund question in the sample, spelling one label Request_refund with a capital R instead of the lowercase request_refund the rest of the system expects.

A human reading the reply would call it correct. A program matching labels exactly would silently drop every one of those tickets.

That is the problem this article is about: a model can sound correct to a person and still be wrong for the software that uses its answer.

What this project builds, and why it uses Weave

Before writing any code, it helps to have one clear picture of what gets built and how its pieces fit together.

This project does five things:

1. Weave records what happens when the application runs: the question, the instructions, the model, the response, and the timing.

2. The instructions given to the model are saved with a version number, so older and newer instructions can be compared.

3. The real customer questions are saved as a test dataset, so every model answers the same examples.

4. A strict checker tests each response for exact requirements, such as valid JSON, required fields, allowed labels, and the correct category.

5. A second AI model reads each response and gives it a quality score, more like a human reviewer would.

The strict checker looks for exact machine requirements. The second AI judge evaluates the answer more like a reader. Using both helps reveal problems that either checker might miss.

Each of those ideas gets explained properly as it comes up. For now, start with Weave itself, since everything else in this article is recorded inside it.

Weave is a tool from Weights & Biases (W&B) for watching what an AI application actually does while it runs. Add one line, @weave.op(), above any Python function, and every single call to that function gets saved automatically, the exact text that went in, the exact text that came back, and how long it took.

Weave calls one of these saved records a trace, and it stores every trace in a project you can open and browse in a web page, the same way a photo app keeps a timeline of every photo you take.

A trace is not only useful for debugging a broken run after the fact. Once an application has been answering real questions for a while, its saved traces are also a ready made source of real examples, which matters later in this article, since the same 47 real questions that trace the application also become the dataset it gets tested against.

The application itself is deliberately small, one function, triage_message(text, model, prompt_ref), that reads one real customer message and asks a model to answer with a JSON object shaped like this:

Four boxes, every time. intent names the category. priority is low, medium, or high. needs_human is true or false, and it decides whether the message gets escalated to a person instead of handled automatically. reply is the short message the customer actually sees.

Only two things change across the rest of this article: which model answers, and which version of the instructions or the grading rules is active. The application logic itself never changes, which is what makes the comparisons later in this article fair.

One choice about how the model gets asked matters enough to explain now. The request to OpenAI uses plain prompted JSON, meaning the model is simply told in words to reply in this shape. It does not use OpenAI's stricter Structured Outputs feature, the one already mentioned above, which can force a model's answer into a fixed shape by construction.

That is deliberate, not an oversight. Using the strict feature here would have hidden some of the very failures this article is built to look for, an invalid response, extra text wrapped around the JSON, or a mislabeled field. Later in the article, once you have seen what actually broke, there is an honest look at exactly which of those failures the strict feature would and would not have caught.

The real customer messages come from BANKING77, a public dataset from a 2020 research paper by Iñigo Casanueva and coauthors at PolyAI (CC BY 4.0 license). It contains 13,083 real banking customer service questions, each labeled by hand with one of 77 fine grained categories, a card that never arrived, a refund that never showed up, a payment the customer does not recognize, and so on.

Fine grained means many of those 77 categories sound close enough to genuinely confuse a model, which is exactly the property that makes this dataset useful here. A model that can only tell the easy cases apart is not being tested very hard.

Setup

This project was written and run with Python 3.11 and Weave:

You need an OpenAI application programming interface (API) key, and a free Weights & Biases account for Weave. Run wandb login once in the activated environment, or set a WANDB_API_KEY environment variable. Save an OPENAI_API_KEY the same way, either as an environment variable or in a .env file next to the script below.

One small version note. This project was run against weave==0.52.40. Weave printed a notice on every run saying that exact version had been recalled over a technical issue and recommending an upgrade. The recall did not change anything in the results here, but install the current release instead of pinning an old one, pip install -U weave, unless you have a specific reason not to.

The complete script

Everything in this article, the traced application, the two versions of its instructions, the dataset, the strict rule based grader, the AI grader, and the model comparison, lives in one script. Save it as banking77_regression.py:

Run the steps in order, each one building on the outputs of the last:

Each command does one job:

1. fetch downloads the BANKING77 test questions used in the article.

2. smoke runs the first prompt on six questions so we can catch obvious problems before the full evaluation.

3. dataset saves the improved prompt and publishes the reusable Weave Dataset.

4. evaluate runs all three models against the same 47 questions and records the outputs and scores.

5. judge_check compares the AI judge with the strict checker.

6. refine_judge publishes a clearer judging rubric and runs the evaluation again.

7. judge_check_v2 checks whether the revised judge now agrees with the strict checker.

8. contract compares the production stand in and the candidate on exact output requirements.

9. sorted_diffs sorts model score differences so the largest changes are easy to inspect first.

The rest of this article explains what those runs produced, in plain terms, using the real output saved along the way.

Why the instructions needed a second version

The smoke step exists for a reason worth explaining before anything else. Before trusting one set of instructions with 47 real customer messages across three models, run it on a small handful first and actually read what comes back.

That first attempt, PROMPT_V1, told the model the JSON shape and the allowed category labels, but it never told the model the rule for deciding needs_human and priority. It left that judgment call entirely up to the model.

On six smoke test examples, the JSON formatting and the category choice were already perfect, 6 out of 6. The escalation decision was correct on only 2 out of 6. The failures were not random guesses either.

On the message "I still have not received my new card, I ordered over a week ago," the correct category, a card that has not arrived, is meant to be handled routinely under the rule this project defines. The model answered that it needed a human right away and marked it medium priority.

That is a perfectly reasonable read of the words, the message does sound a little frustrated. It is also the wrong answer for a program that needs one fixed rule applied the same way every time, not a rule that shifts depending on tone.

PROMPT_V2 fixes this by spelling the rule out directly. It names exactly which categories require a person and high priority, and states that every other category must use low or medium, never high, no matter how the message sounds.

Rerun on the same six examples, the escalation decision was correct on 6 out of 6, with the JSON formatting and category choice unchanged. Both versions of the instructions stayed saved in Weave under the same name, a small prompt registry that a reader can open and compare side by side, not a claim to take on faith.

Turning real production traces into an LLM eval dataset

A Weave Dataset is a saved, versioned list of rows, and once it exists, a Weave Evaluation can run any application against every row and grade what comes back. This project's dataset holds 47 real BANKING77 questions across 20 of the dataset's 77 real categories.

Those 20 were not picked at random. They form five groups of categories that sound close enough to genuinely confuse a model, card problems, transfer problems, refund problems, unrecognized payments, and identity checks, plus a few standalone security and billing categories.

23 of the 47 questions are meant to escalate to a person under the fixed rule, and 24 are not, a deliberately even split. Each row carries the real question text, the real correct category, whether it should escalate, and the full list of 77 valid category labels the model is allowed to choose from.

Two different graders, checking two different things

Every answer in this project gets graded twice, by two very different kinds of checker, and the difference between them matters for everything that follows. One checker, contract_scorer, follows a fixed rule with no room for interpretation, the same way a form processing machine either finds a barcode in the right spot or does not.

It parses the raw text as JSON, tries a couple of common fallback methods if the first attempt fails, and then checks a short list of yes or no questions.

Are all four fields present? Is the category one of the 77 real allowed labels? Does it match the correct answer? Is the priority an allowed value? Does the escalation decision match the rule?

None of that requires judgment. A field is either there or it is not.

The second checker is a graded AI judge, a separate model call that reads the customer's message, the correct answer, the escalation rule, and the first model's raw response, then hands back a score from 0 to 10 with a short explanation, closer to a second person reading the reply and forming an opinion.

Its scoring rules were written only after reading real responses from the smoke test, not guessed in advance, and they follow the same priorities as the strict checker on purpose.

Broken output or a disallowed category caps the score at 2. A wrong category or a broken escalation rule caps it at 5. Only a response that is valid, correctly labeled, and correctly escalated gets judged on how good the actual reply text is.

The judge runs on a separate model, gpt-4.1, one that is not any of the three models being compared, so it is never grading its own family's work.

Comparing three real LLM model versions

A Weave Evaluation ties the dataset, the application, and both graders together in one run. Three model choices stand in for a real AI model selection decision a team might face:

• gpt-4o-mini, standing in for an older model still running in some legacy code path.

• gpt-4.1-mini, treated here as the model currently in production.

• gpt-5-mini, the newer model a team is considering deploying in its place.

Running the full comparison, using the corrected grading rules from the next section, produced this real result, computed from all 47 questions per model:

A number close to 1.000 means nearly every one of the 47 answers passed that check.

Every one of these runs is visible and comparable in Weave's own dashboard.

Checking whether the AI judge can actually be trusted

Before trusting any score an AI judge hands out, it helps to check its work against something that cannot be argued with, which is exactly what the strict rule based checker is for. The first version of the judge's scoring rules, run against all three models, disagreed with the strict checker 4 times on the older model, 3 times on production, and 6 times on the candidate.

Every one of those disagreements had the same shape. The judge scored a response as only partly correct even though the strict checker said the category and the escalation decision were both right.

Reading the judge's own written explanations showed exactly why. The judge had read the rule "not escalate, and not use high priority" as if it meant one single specific priority value was required, and it was marking a response wrong just for choosing medium instead of low, even though the rule never asked for one specific value between the two.

That is a real, fixable misreading, not a vague sense that something was off.

The corrected version of the rules, JUDGE_PROMPT_V2, states plainly that low and medium are both correct for a routine case, and neither counts as a mismatch. Rerunning the same 47 questions per model against the corrected judge dropped the disagreements to zero for the older model and the candidate model.

Production still showed 4 disagreements afterward, and it would have been easy to assume the judge simply needed one more fix. It did not. Reading those four cases one by one turned up something else entirely, which is the actual point of the next section.

Sorting by the size of the disagreement instead of reading every answer

Forty seven questions across three models adds up to 141 separate graded answers, more than anyone wants to read line by line. Sorting by how far apart two models' scores are turns that pile into a short list, worth starting with the biggest disagreements and working down from there.

Sorting the older and candidate models this way puts several perfect swings at the top, the older model scoring 3 out of 10, the candidate scoring 10 out of 10, on the exact same question.

One of them shows the pattern clearly. For the message "How do I locate my card?", the correct category is a card that has not arrived yet. The older model, gpt-4o-mini, answered with a different, real category about linking a card, a genuinely plausible misreading if you are not holding the full list of 77 labels in front of you, the word "locate" does sound a little like a linking question out of context.

The newer model, gpt-5-mini, answered correctly. Nothing about this pair has anything to do with formatting. Both answers were valid JSON with all four fields present.

This difference is about which model actually understood the message correctly, on a dataset built specifically to include categories that sound alike.

What actually happened when production was tested against the candidate

This is the comparison the whole project was really built for. Treat gpt-4.1-mini as the model currently in production, and gpt-5-mini as the candidate being considered to replace it, then check every measurement, not only the overall judge score.

The candidate matched or beat production on every single one, including every formatting check. Neither model ever produced invalid JSON or left out a field, both were perfect there.

But the row for allowed category labels tells a different story. Production scored 0.936. Both other models scored a perfect 1.000. Production is the one with a real formatting problem here, not the candidate.

The cause is specific, and it repeats identically across every refund question in the sample. On three separate real customer messages, all asking about a refund, gpt-4.1-mini answered with the category spelled "Request_refund", a capital R.

The real BANKING77 label, and the value written in every row of the dataset, is lowercase, request_refund. A program checking that label the way real software actually does, an exact match, would silently fail to route every single one of these tickets, even though the reply text underneath reads just fine.

The AI judge gave that response a 9, and its own written explanation said plainly, "the correct intent (case sensitivity is not penalized)," naming the exact thing it was choosing to ignore. The strict checker disagreed, correctly, because "Request_refund" simply is not one of the 77 real category labels at all, and an exact match check is precisely what routing code in a real system actually runs.

The same capital R habit showed up on two other real refund questions in the sample, not only this one, and both gpt-4o-mini and gpt-5-mini wrote the correct lowercase label on all three.

This one specific habit is a real, already shipping formatting bug in the model currently in production. The candidate did not introduce it. It was only visible at all because something else existed to compare it against.

This deserves to be said plainly, since the honest result matters more than a tidy one. This project did not find the pattern it set out looking for. The candidate never broke a rule production was following correctly.

The project still earned its cost, because a test built to catch a new problem caught an old one instead, on a bug a person skimming the reply would never notice, since the reply itself reads as completely correct.

One more real case is worth including precisely because it complicates the story instead of wrapping it up neatly. For the message "Where can I view my PIN?", the correct category should have triggered escalation.

All three models, including the newer one, answered with a different category about changing a PIN, and marked it as not needing a person, a reasonable sounding guess that misses the point. It is the only question in the whole sample where the newer model got both the category and the escalation decision wrong at once.

Reading the message again, it genuinely reads more like a request to view or change a PIN than a report of one being blocked, which is worth treating as a possible labeling question in the original dataset, not only a shared model mistake.

A similar case turned up inside the four leftover judge disagreements on production. One customer described a declined card purchase, and the dataset's own answer for that question was a category about a declined transfer, while gpt-4.1-mini answered with a different, real, defensible category about a declined card payment.

Public datasets are built by people, and their labels are not beyond question. An honest project says so when it finds a case like that, instead of quietly counting it as one more model mistake.

The full loop, and what it does not prove

Put together end to end, this project is one repeatable loop for regression testing an LLM before it replaces one already in production. Trace a small real application with Weave. Fix its instructions once a real smoke test finds a real gap.

Turn real traces into a dataset. Build two graders, one strict and one that reads for meaning, and check them against each other. Run a full comparison across three models.

Sort the results by how much they disagree instead of reading every row. Finally, run the one comparison a real deployment decision actually depends on, the model already live against the one being considered to replace it.

One honest question remains open, and it is worth sitting with rather than resolving too neatly. The AI judge read straight past the capital letter difference in Request_refund because it was grading for meaning, and the strict checker caught it because it was not. That gap, a judge that reads more kindly than the exact rule a real system depends on, is close to unavoidable for any grader built to read like a person.

If a project only had an AI judge, with no strict rule based checker running alongside it, how would anyone ever catch a bug like this one, an answer that looks obviously correct and is silently, mechanically wrong underneath?

What this specific project did prove is narrower than a verdict on which model is better in general, and more useful because of it. On one small application, across 47 real customer messages, the newer model never lost to the one already running in production, on formatting or on accuracy.

The most useful thing this project found was not really about the future model being considered at all. It was about the one already live, and the only reason to see it was building something to compare it against.

Sources

• Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić, Efficient Intent Detection with Dual Sentence Encoders, Proceedings of the 2nd Workshop on Natural Language Processing (NLP) for Conversational AI, Association for Computational Linguistics (ACL), 2020. Introduces the real BANKING77 dataset used throughout this article.

• Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto, Positive-Congruent Training: Towards Regression-Free Model Updates, Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Introduces the negative flip, the finding that motivated this project's original hypothesis.

• OpenAI, Introducing Structured Outputs in the API, official product announcement. The source for this article's opening claim about the unreliability of plain prompted JSON.

• Weights & Biases, Store and track versions of prompts, Weave documentation.

• Weights & Biases, Build an evaluation, Weave documentation.

• Weights & Biases, Track application versions with models, Weave documentation.