Harness engineering: what broke when OpenAI let Codex write a million lines

01 · Late August 2025: an empty repo and three engineers

Late August 2025. An internal team at OpenAI pushes the first commit to an empty repo. Codex CLI with GPT-5 writes the scaffold: folder structure, CI, formatting rules, package manager, framework. Codex even wrote AGENTS.md, the file that tells the agent how to work in the repo. The team had one rule: no human writes a line of code. Not in the app, not in the tests, not in CI, not in the docs.

Five months later the repo holds about a million lines and around 1,500 merged PRs. The team grew from three engineers to seven. Throughput was 3.5 PRs per engineer per day, and it went up as the team grew, not down. They estimate it took a tenth of the time it would have taken by hand. Ryan Lopopolo published this in February 2026, and everyone has been talking about harness engineering since.

But this isn’t a clean success story. The part worth learning from is what broke along the way.

02 · A few months in: Codex copies itself

Codex learns from the repo it works in. If a pattern exists in the repo, it will replicate it, good ones and bad ones alike. In a repo where agents wrote every line, there’s no old, carefully written code holding the style together. Every shortcut that gets in once becomes a template the next agent copies, and then the one after that.

Birgitta Böckeler of ThoughtWorks wrote on martinfowler.com about the same thing at other teams. Unconstrained coding assistants forget conventions, repeat the same mistakes, and slowly wear down the consistency of the codebase. This isn’t an OpenAI-specific bug. It’s what happens anywhere you let an agent write a lot of code.

03 · From the outside, everything is green

This is the annoying part. The code compiles, the tests pass, and the PR chart looks great. No alert fires when the architecture slowly falls apart. Habitat Thinking, who document Böckeler’s idea, put it in two sentences: “The code continues to compile and pass tests. The degradation is quiet.”

The outside numbers tell the same story. The DORA report found that more AI adoption goes with higher software delivery throughput, and also with more instability. Apiiro counted more than 10,000 new security findings per month in AI-written code by June 2025, 10x the December 2024 level. Speed goes up, quality erodes, and the dashboard only shows the speed.

Caveat: These are OpenAI’s numbers on their own internal product, with access to their best model and no paying customer. The “tenth of the time” is their own estimate, with no control group. I take the mechanisms from the post. I wouldn’t put the ratios in a slide for management.

04 · The first fix: cleanup Fridays

The team’s first response was entirely human. Every Friday, 20% of the week, they manually cleaned up what they called AI slop. It didn’t hold. At 3.5 PRs per engineer per day, one cleanup day a week is a bucket under a burst pipe.

The second fix was to write more instructions. One big AGENTS.md with all the rules. That failed too, and they described exactly why: the file eats the context, when everything is marked important nothing is, and it goes stale faster than anyone updates it.

Wait, why would that ever work? Augment put it well: telling an agent “follow our coding standards” is probabilistic compliance. Wiring in a linter that blocks the PR is deterministic enforcement. OpenAI’s first two fixes stayed on the probabilistic side.

05 · The real cause: what the agent can’t see doesn’t exist

The problem wasn’t the model. The knowledge of how the code should look lived in places the agent can’t read, like engineers’ heads or Slack threads. And the checks in the repo caught functional breakage, not architectural drift. If the agent can’t see something or feel it in its loop, then as far as it’s concerned, it doesn’t exist.

So they moved the rules into the repo. AGENTS.md shrank to about 100 lines that act as a map to a docs/ folder, and docs/ became the source of truth. Custom linters, written by Codex itself, enforce the boundaries between layers, and their error messages include a fix instruction. The agent reads the error and fixes it on its own. On top of that, background Codex tasks scan the repo against a list of principles stored in the repo and open small refactor PRs. Garbage collection instead of Fridays.

They described the shift in the job like this:

what changes when a software engineering team’s primary job is no longer to write code, but to design environments, specify intent, and build feedback loops that allow Codex agents to do reliable work. — OpenAI, Harness engineering: leveraging Codex in an agent-first world

The rule behind it is credited to Mitchell Hashimoto (HashiCorp, Terraform), who wrote in February 2026 that every agent mistake gets a permanent fix in the environment, so the same mistake doesn’t come back. Addy Osmani credits the name to Viv Trivedy. The fact that people are still arguing about who coined the name is a good sign, not a bad one.

06 · What to check in your repo this week

You don’t need a million lines to hit the same wall. An agent opening PRs for a few weeks is enough. Here’s what to check:

• wc -l AGENTS.md CLAUDE.md. If it’s over 150 lines, split it into files under docs/ and keep only a map in the main file.
• Go through your last 20 review comments on agent PRs. Any comment that shows up twice is a check you haven’t written yet.
• Every linter or CI error message needs a FIX line and a WHY line pointing to a doc. The agent reads the message, so instructions inside it work better than a line in the prompt.
• Run a weekly scheduled job that asks the agent to find deviations from docs/principles.md and open up to three small PRs. No more, or you’ve just moved Friday into review.

This is the simplest boundary linter there is. It blocks direct DB access outside the allowed layer and tells the agent what to do instead:

import pathlib, re, sys

IMPORT_DB = re.compile(r'^\s*(from|import)\s+app\.db\b', re.M)
ALLOWED = ('app/repo/', 'app/db/')

bad = [p for p in pathlib.Path('app').rglob('*.py')
       if not str(p).startswith(ALLOWED) and IMPORT_DB.search(p.read_text())]

for p in bad:
    print(f'{p}: direct import of app.db outside the repo layer.\n'
          '  FIX: add a function in app/repo/ and call it from here.\n'
          '  WHY: docs/architecture.md#data-access')
sys.exit(1 if bad else 0)

Wire it into pre-commit and CI, and see how many PRs it catches in the first week. The interesting part of this file isn’t the regex. It’s the three lines that tell the agent what to do after it gets blocked.