Task specifications
Required outcomes and prohibited side effects, written down per task.
Approach
Define the work, then build the test.
Every task begins with an explicit definition of successful work. We then build the environment, reference solutions and checks needed to evaluate that definition.
Practices
Marked as they stand today on the accounts-payable prototype: 8 in place, 4 planned.
In place used on the prototype now Planned not yet done
Reference solutions
A scripted solution for every task, proving it is solvable under the policy.
Synthetic records
Generated from a seed. No customer data is needed for the prototype.
Grader testing
Scripted shortcut strategies that must fail. Both grader bugs found so far have a regression strategy.
Surface variation per seed
New names, IDs and amounts for each seed.
Decision variation
Queue order and the mix of traps change per seed. Varying the evidence a decision turns on is planned.
Alternative solutions
Four valid alternatives must pass every run: investigating first, reversing the order of actions, adding notes, and splitting payments.
Reproducibility
Every task is regenerated exactly from its seed.
Versioned releases
Numbered environment versions, so results can be tied to an exact task set.
Recorded model executions
Executions by real models, recorded and reported with the fields listed on the environment page.
Practitioner review
Domain review by a practitioner, where the workflow needs it.
Public and private separation
The public sample is for inspection and harness testing. Evaluation cases are built privately and never published.
How we test a grader before trusting it
A grader is software, and it has bugs. We treat it as the thing under test before we use it to test anything else.
Write the definition of done
Required outcomes and prohibited side effects, each tied to the policy rule it comes from.
Prove it is solvable
A scripted reference solution must pass every check on every seed.
Attack it
Strategies that do not do the work must fail on every seed: doing nothing, holding everything, paying everything, stacking hold codes, over-reaching.
Accept other valid work
Correct solutions reached another way must pass: investigating first, a different order, extra notes, split payments.
Check the side effects
Diff every record against the starting snapshot. Payments must go to verified accounts.
Keep every bug
Each grader bug found so far became a regression strategy that runs from then on.
Report per check
Publish which checks failed, not just the pass rate, so near misses are visible.
The failure modes we design against
- 01 Task + policyWhat correct work is 5
- 02 Agent actsTool calls on the records
- 03 Final stateWhat actually changed 3
- 04 GraderChecks against the policy 14
- 05 ScorePass or fail 2
- 1Accepts wrong work
- 2Partial credit hides failure
- 3Side effects unmeasured
- 4Rejects valid work
- 5Tasks memorised
-
The grader accepts wrong work
In placeWhat it looks like
An agent resolves every item, then writes off the balances it has just put on hold. Nothing is paid twice, so a check that looks only at payments is satisfied. Our first grader accepted this on three of the five task families.
How we test for it
Six scripted shortcut strategies run on every seed and must fail. Each grader bug found so far became a permanent regression strategy.
-
Partial credit hides failed work
In placeWhat it looks like
The six scripted shortcut strategies satisfy 50–93% of the individual checks they face. Scored as an average, “resolve correctly, then write off” would earn 85%.
How we test for it
A run passes only when every check passes. Per-check results are reported, so a near miss is visible but not rewarded.
-
Side effects go unmeasured
In placeWhat it looks like
An agent fixes the item in front of it and quietly changes another record: a different invoice, a vendor’s bank details, a bank-feed line.
How we test for it
Every record is snapshotted before the run. Any change outside the queue items fails the run, as does any payment to an unverified account.
-
The grader rejects valid work
In placeWhat it looks like
A correct resolution reached in a different order, or explained in different words, fails a check that is stricter than the policy.
How we test for it
Four valid alternative solutions must pass every run. Holds carry one of four policy codes, so a correct hold cannot fail on its wording.
-
Tasks can be memorised
Partly in placeWhat it looks like
A task set reused across training and evaluation stops measuring the skill and starts measuring recall.
How we test for it
Each seed generates new names, IDs and amounts, and private cases are kept apart from public samples. Varying the decisions themselves is planned.
See it applied.
The accounts-payable prototype shows these practices on a real task: the specification, the reference solution, the checks and the shortcut strategies that fail them.