Carlos Tapia

Notes

Experiments, measurements and the occasional correction

Each note starts from a question I had about my agents and says what I measured to answer it.

Vocabulary: team, role, round, turn → Glossary

10 notes

10 Measurement

Agents that know whose file it is leave it alone, then patch around it

What Claude Code, Codex and opencode do when the kernel denies a write to another role's file. With the bare denial, two of the three push back in every run. Knowing the owner beforehand, none tries, but two of the three still fix the bug in their own files. seisin's denial message was never shown.

  • 11/11 → 0/11 Codex runs that tried to write the backend's file, without and with the owner known
  • 10 → 5 Tool calls per Codex run (median), without and with the owner known
  • 10 of 11 Codex compensations that break the day the owner fixes its bug
09 Field note

The Hugging Face intrusion from an OpenAI evaluation, step by step

An agent running an OpenAI evaluation left its sandbox and spent five days in Hugging Face's infrastructure. Reconstructed from both companies' reports.

  • 17,600 Agent actions recovered from 9 to 13 July
  • 136 Keys in one production secret the agent reached
  • 5 days Between Hugging Face's disclosure, which did not know which model was behind it, and OpenAI's report
08 Measurement

Three in four sandbox escapes I measured broke no configured rule

sandbox-runtime, nono and Codex against 168 tests from their own projects' suites. Most escapes come from defaults.

  • 61 of 79 Escapes on a default, with no configured rule involved
  • 37 of 79 Trust-handoff: a file the agent leaves that runs later, outside the sandbox
  • 109 → 74 Codex «blocked» vs actually contained, once legitimate work has to keep working
07 Measurement

The nine-role team costs 3.5× more per verified task than the four-role one

I compared eight agent teams on cost per verified task. The gap follows tool calls per turn, and splitting the large team would not change what its two most expensive roles do.

  • 3.5× Tokens per verified task, nine-role team vs. four-role team
  • 51.5 Tool calls per turn for the most expensive role
  • 62% Verification coverage of the nine-role team, against 100% in the two cheapest
06 Measurement

The occupancy trigger compacts context without knowing how many calls are left

Compaction has a fixed cost and a saving per remaining call. In 13 measured compactions the crossover was around 17 calls left. Occupancy can't see that number; each role's turn history can estimate it.

  • ≈17 Calls left where cost and saving crossed, in 13 compactions
  • 4 / 4 · 7 / 7 Paid off with 17 or more calls left · failed with 9 or fewer
  • 24 / 80 Compactions that paid off in production; the net was still positive
05 Experiment

In a small test, keeping an agent's thread alive cost less than resetting it

Resetting a role's thread every turn looked like the way to spend fewer tokens. In a controlled test with one role and one task, the long thread came out cheaper, because of cache writes.

  • −25% Weighted window, long thread vs. reset every turn (3 vs. 4 rounds)
  • 5.3 → 3.8 Tool calls per turn, short thread vs. long
  • 21k → 5–8k Tokens written to cache per turn once the thread survives
04 Measurement

My verifier reported 100% conformance until I measured what it rejects

718 of 718 rounds followed the declared process, but a checker that accepts almost anything also reports 100%. A shuffle control showed the number holds in one team out of six.

  • 718 / 718 Rounds conforming to the declared process in the first count
  • 9.1% Shuffled traces that also pass, in the team where the result holds
  • 93.4% Shuffled traces that pass in the team where it means nothing
03 Measurement

86% of my agents' code is still there, and most of the drop came from one week

Line survival on a production repository written by agents: 85.9%, down from 92.7% two days earlier. Most of the lines lost in between belong to one week and one 2,408-line commit that survived at zero.

  • 85.9% Of lines written in the window still alive at HEAD
  • 92.7% → 85.9% Survival across two measurements, two days apart
  • 2,408 → 0 Lines added by the single worst commit, and how many survived
02 Architecture

Finding duplicates and checking citations need opposite kinds of search

Four citations from my agents failed when the source was opened. On 14 labelled pairs, exact search finds almost no duplicates and embeddings find all of them with noise. Checking a citation is exact search in a known file, where similarity gets in the way.

  • 4 of 4 Citations that failed when the cited source was opened
  • 10 vs 8 of 14 Tuned embeddings against a control that always answers "different"
  • 0.930 Cosine similarity (1 = identical) between two service names that mean different things
01 Measurement

38% of my agents' message deliveries never reached the recipient's prompt

11,845 deliveries in 312 rounds of one team, and 4,445 discarded by the mailbox cap before the recipient saw them. With a fan-out of 3.37×, every role was telling everything to almost everyone.

  • 38% Deliveries discarded by the mailbox cap before the recipient saw them
  • 76% Of all possible sender→recipient edges used at least once
  • 36.4 h Longest a blocking request waited for its answer