Prompt evals: how Rechat's Lucy stalled, and how to know a prompt change helped

89% of organizations have observability on their agents. 52% run offline evals on a test set. Only 37% run evals on real traffic. The numbers come from LangChain’s State of Agent Engineering survey, published in June 2026. Put in standup terms: almost everyone can see what the model is doing, and fewer than half can say whether today’s prompt is better than yesterday’s.

In the previous part we built a system prompt in four blocks. This time we look at what happens after you change one line of it. The best-documented case I know on the subject is Lucy.

Lucy ran on prompts, and it worked

Rechat is a SaaS for real estate agents: contracts, property search, meetings, marketing materials. Lucy is their assistant, a chat interface that replaces all the clicks and forms. Hamel Husain, who led CodeSearchNet at GitHub (one of the precursors of Copilot), worked with them and published the case on March 29, 2024, in the post Your AI Product Needs Evals.

At first everything moved fast. Prompt engineering alone delivered improvement after improvement. New feature? Another paragraph in the prompt, a few questions in the chat, looks good, deploy. Every team that built an assistant two years ago knows this rhythm.

When the curve went flat

As Lucy gained more capabilities, performance stalled. Hamel describes three symptoms, and I think most of you will recognize at least two:

• Every fix for one problem created a new one. Whack-a-mole, in his words.
• There was no way to know how the system did across all tasks at once, other than vibe checks.
• The prompts grew into long blobs trying to cover every edge case.

From the outside this doesn’t look like an incident. No outage, no 500s in the logs, no alert. Just a different complaint every week, and every week someone fixes it. That’s the most dangerous kind of regression, because it has no timestamp.

The first fix: one more line in the prompt

This is the intuitive fix, and it’s the wrong one. Lucy showed the user an internal ID from the DB? Add “never display IDs”. Property search returns junk on a vague question? Add an example. Each line like that gets tested on the case that caused it, and only on that case.

Wait, why is that a problem? Because a prompt has no scope. A line you wrote for property search also changes how the model phrases a contract summary. Without a fixed set that runs on every change, you see exactly one cell in the table: the one you looked at. Everything else shifts quietly.

The real cause: nothing to run

Hamel opens the post with a sentence I come back to in every conversation on this:

I’ve found that unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems.

— Hamel Husain, Your AI Product Needs Evals

The prompt’s wording wasn’t the problem. Measurement was missing. Every decision about the prompt rested on five questions someone typed that morning. And that’s exactly the situation LangChain’s survey describes two years later: teams see traces, but they don’t judge them.

What Rechat built, in three levels

Roughly, three layers, from cheap to expensive.

Level 1, unit tests. Assertions in code, no model. No UUID in the response. Search never runs with an empty query. The number of properties in the answer matches what the tool returned. They run on every change, and results are kept over time so you see a trend, not just pass/fail. Some of the test cases they generated with an LLM, per feature.

Level 2, traces and LLM-as-judge. Log every conversation, read them yourself, and write a judge for the things code can’t check. Hamel insists on one point: measure how often the judge agrees with a human, and don’t trust it before that.

Level 3, A/B. Only once the first two levels are stable do you test on real users.

My take: I don’t trust a judge that hasn’t been checked against at least 50 human-graded answers. An uncalibrated judge tends to like long, polite answers, which is the opposite of what a real estate agent needs in the middle of the day.

The grader was also written by a person, and it can be wrong. In January 2026 Anthropic described how Opus 4.5 “failed” a τ2-bench task because it found a loophole in the airline’s policy and gave the customer a better solution than the eval expected. When the eval says fail, read the transcript before you touch the prompt.

What to check this week

Okay, enough story. This is the minimum to avoid becoming Lucy, using promptfoo (open source, runs locally, has a GitHub Action for CI):

1. A golden set of 30–50 cases. From the logs, not from your head. Every user complaint goes into the set before you fix it.
2. Code first. Whatever a regex or a JSON schema can check doesn’t go to the judge. Cheap, deterministic, doesn’t argue.
3. A judge with a narrow rubric. One sentence that checks one thing. Not “is the answer good”.
4. Pairwise comparison. The current prompt against the candidate, on the same inputs. A 7 versus a 7.5 doesn’t say much. “Which of the two is better” does.
5. Regression on real traffic. Once a week, pull 20 traces from production (Braintrust, LangSmith, or just a log) and add whatever looks odd to the set.

description: prompt regression, current vs candidate
prompts:
  - file://prompts/current.txt
  - file://prompts/candidate.txt
providers:
  - id: anthropic:messages:claude-sonnet-5-5   # replace with the model ID you run in production
    config:
      temperature: 0
defaultTest:
  options:
    provider: anthropic:messages:claude-opus-5-5   # the judge: a stronger model than the one under test
  assert:
    - type: not-regex
      value: '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
    - type: javascript
      value: 'output.length < 1500'
    - type: llm-rubric
      value: Does not mention a property, price or address that did not appear in the context
    - type: select-best
      value: Which answer moves the agent to the next task in fewer messages
tests: file://golden.csv

Both prompt files use {{question}}, and the CSV holds a single column with that name:

question
Find me a 4-room apartment in Ramat Gan under 3 million
Schedule a meeting with yesterday's buyer
How much commission do I get on the Bialik Street deal

npx promptfoo@latest eval -c promptfooconfig.yaml --repeat 3 --no-cache -o results.json
npx promptfoo@latest view

The --repeat 3 isn’t decoration. The same input doesn’t return the same answer, and a test that passes 2 out of 3 times is flaky, not green. select-best is the pairwise comparison: the judge sees both versions side by side and picks one. If the candidate doesn’t win or at least tie in most cases, it doesn’t ship.

And if you only have time for one item, do the first. Lucy didn’t stall because the prompt was bad. It stalled because every fix was tested on one case, and a complaint that never became a test comes back in the next version.