Finding duplicates and checking citations need opposite kinds of search

One night, four claims my agents made citing a source fell apart when the source was opened:

TABLE 1 · An ADR (architecture decision record) is the short document where an architecture decision is written down with its reason.

The agents had two ways to search: exact, with grep or a full-text index that keeps identifiers as written, and semantic, with embeddings, which return whatever is closest in meaning. I used the semantic one to detect duplicate problem reports, and I wanted to know which is good for what.

Finding duplicates

The documented pattern for code is agentic search with grep over embeddings (Agentic Search Over Vector Embeddings); its own catalogue lists more tokens and limited semantic understanding as the cost. Code has exact identifiers. My case was prose: two reports from different roles about the same problem may not share a word.

TABLE 2 · 14 labelled pairs: 6 duplicates and 8 different problems. For the tuned threshold only the total was recorded.

Exact search is almost never wrong when it says “same”, but it almost never says it. Embeddings on a loose threshold find every duplicate and flag everything else too. The tuned threshold was picked on these same 14 pairs, and it beats the null control by 2 pairs of 14.

Checking a citation

Verifying “does this ADR say X?” is exact search in a known file. A semantic engine returns what most resembles the claim, including a passage that says the opposite, because it shares the nouns. On this corpus, pairs that were different problems reached 0.77 cosine similarity. Two internal service names that differ by one word came out at 0.930: embeddings blur exactly the identifier that separates them.

So I split the jobs. Semantic search stays for duplicates. The citation check, which only confirmed that the cited document could be found, was to be extended to confirm with exact search that the document says what is attributed to it.

What checking costs

Agentic search, where the agent iterates with grep until it hits the line, spends tool calls:

one agentic search  ≈  5–10 tool calls × ~15 s   ≈  75–150 s
a median turn       ≈  372 s
                    →  20% to 40% of a turn, per search

The 5–10 calls are an estimate from the literature; the 15 seconds and the 372-second median turn are from my teams. The agents run on a subscription with a usage allowance per 5-hour window, and tool calls eat into it. Checking a citation happens now and then and can pay for agentic search. Duplicate detection runs every time a report is opened, so it goes through the index.