My verifier reported 100% conformance until I measured what it rejects

Each of my agent teams declares a process: which roles play, in which stages, in what order. In the product team the stages are decide → look → build → verify. The trace of a round is the sequence of roles that actually played. I checked whether the traces follow the declared process, and all 718 rounds of the four teams I checked passed.

That number is fitness: how much of the log the model can reproduce. I used it to claim that the team follows its process, which is a statement about the model, and the conformance checking literature separates those two uses on its first page:

“fitness is certainly the most used metric as it states how well an observed behavior fits the defined process model, i.e. to evaluate the quality of the executed work. When making statements about the quality of the defined process model process analysts prefer the metrics precision, generalization and simplicity.”

The control

My checker accepts any in-order subsequence of the stages. To see how much that lets through, I shuffled each trace (same roles, same length) and scored it again. If shuffled traces pass as often as real ones, 100% says nothing.

TABLE 1 · Shuffled: the same trace in random order. Fully random: roles drawn from the team’s declared set, same length. Run a day after the first count, adding lab and landing; the product team had 335 rounds by then, one more than the 334 in the 718.

Where 100% means something

In the product team real order passes 100% and shuffled passes 9.1%: the model rejects nine of every ten alternative orderings of the same roles, over 335 rounds of 6.17 turns each.

In platform 35.4% pass: with 2.52 turns per round and three stages there are few orderings to tell apart.

In management 67.7% pass because the manager role is declared in two stages, deciding at the start of the round and verifying at the end. The checker’s first version gave each role a single stage, read the manager’s return as going backwards and flagged 71% of that team’s rounds as deviating. Fixing that bug is what made the model accept most orderings.

In hotfix 93.4% pass. 87.5% of its rounds have a single turn, and a one-turn trace conforms by definition. Lab (4 rounds) and landing (1 round of one turn) are the same case.

Since this control, I don’t report a conformance figure without its shuffled rate next to it, and below about two turns per round I don’t report it at all: there is no order to check.