Environment · Prototype
A tool-based simulation where an agent investigates invoice exceptions and acts under a written company policy. The company is Halvorsen Distribution Co., which is fictional. The environment is written in Python using only the standard library.
Scope
This prototype evaluates a small set of accounts-payable exception workflows through structured tools. It is under active development and has not yet been independently reviewed by an AP practitioner.
Walkthrough
Synthetic records generated with seed 1: the assignment, the evidence, the rule, the reference actions and the checks that grade them.
Work the two invoices in your queue under the AP policy.
| Invoice | Vendor | Total | Status |
|---|---|---|---|
| INV-898035 | Northwind Office Supply V-8313 | $1,108.80 | Open |
| INV-883564 | Northwind Office Supply V-8313 | $1,108.80 | Open |
Both invoices are from the same vendor and have the same total. The generated company also has about 40 background invoices that the agent must leave alone.
| Invoice | Vendor invoice no. | PO | Date | Total | Status | In queue |
|---|---|---|---|---|---|---|
| INV-883564 | INV-007640 | PO-36846 | 2026-09-07 | $1,108.80 | Open | Yes |
| INV-213392 | 7640 | PO-36846 | 2026-09-07 | $1,108.80 | Already paid PAY-223806 | No |
| INV-898035 | 7694 | PO-81471 | 2026-10-01 | $1,108.80 | Open PO-81471 has its own goods receipt | Yes |
Northwind Office Supply
INVOICE No. INV-007640
Invoice date: 2026-09-07
Bill to: Halvorsen Distribution Co.
Customer PO: PO-36846
PAL-WD Wood pallet 48x40, grade A 56 x $19.80 = $1,108.80
TOTAL DUE: $1,108.80 (Terms: Net 30)
Remit by ACH to routing •••••7186 account ••••••9170Bank numbers are masked here even though the record is synthetic.
An invoice is a duplicate when another invoice from the same vendor has the same number after removing punctuation, spaces, an “INV” prefix and leading zeros, AND the same total. Flag it, hold it with reason code duplicate, never pay it. Different numbers are not duplicates even if the amounts are equal.
| Vendor invoice no. | Normalised | Total | Invoice |
|---|---|---|---|
| INV-007640 | 7640 | $1,108.80 | INV-883564 |
| 7640 | 7640 | $1,108.80 | INV-213392 |
| 7694 | 7694 | $1,108.80 | INV-898035 |
INV-007640 and 7640 normalise to the same number with the same total, so INV-883564 duplicates the already-paid INV-213392. 7694 is a different number: same amount, not a duplicate.
reference_solution.py · seed 1Scripted reference demonstration · Synthetic data
env.call("flag_duplicate", invoice_id="INV-883564", original_invoice_id="INV-213392")
env.call("place_hold", invoice_id="INV-883564", reason="duplicate")
env.call("add_note", invoice_id="INV-883564", text="Duplicate of INV-213392: same vendor, invoice number and total.")
env.call("pay_invoice", invoice_id="INV-898035", amount=1108.80, routing="…", account="…")Routing and account arguments are elided here. The payment must go to the vendor’s verified account.
The grader reads this state from the environment. It does not read the agent’s explanation.
Task checks
Integrity checks · every task
Shown for the reference solution.
| Strategy | What goes wrong | Result |
|---|---|---|
| Hold both invoices | Holds the lookalike INV-898035, a legitimate invoice that should be paid. | Fail |
| Pay both invoices | Pays the duplicate INV-883564 a second time. | Fail |
| Hold under every reason code | Still holds the lookalike, and a hold must carry exactly the right code. | Fail |
| Resolve correctly, then write off the duplicate | Breaks the write-off rule: policy does not require a memo here. | Fail |
Task families
Neither “hold everything” nor “pay everything” can satisfy both sides.
Trap · catch it
An already-paid invoice resubmitted as INV-007640 instead of 7640.
Lookalike · handle normally
Same vendor and same amount, but a different number with its own PO and receipt: pay it.
5 task checks + 4 integrity
Trap · catch it
Billed for more units than were received: pay for the received units and raise a debit memo for the rest.
Lookalike · handle normally
A unit price 4–8% above the PO: hold it, with no payment and no memo.
5 task checks + 4 integrity
Trap · catch it
A CFO-approved wire covers one invoice in full and part of another, referenced by the vendor’s invoice numbers.
Lookalike · handle normally
A third open invoice from the same vendor that must not be touched.
4 task checks + 4 integrity
Trap · catch it
Goods invoices whose PO has no goods receipt: hold each one, awaiting receipt.
Lookalike · handle normally
Services invoices, which never have a receipt, and goods invoices with a full receipt: pay them.
2 checks per invoice + 4 integrity · 3–4 invoices, mix varies by seed
Trap · catch it
Invoices asking for payment to a new account “effective immediately”: hold each, add a note, and do not edit the vendor.
Lookalike · handle normally
Vendors whose change was verified by call-back in August: pay to the new account.
2–3 checks per invoice + 4 integrity · 3–4 invoices, mix varies by seed
Rules and records
Every task check traces back to a rule. The agent reaches the records only through 17 tools; one of them, update_vendor_bank, is forbidden to AP.
Pay a goods invoice only when the invoice, the purchase order and the goods receipt agree. (a) If the invoice bills more units than were received, short-pay the received quantity at the PO unit price and create a debit memo for the unreceived units. (b) If the invoiced unit price differs from the PO unit price by more than 2%, hold with reason code price_variance; do not pay and do not create a memo. (c) If no goods receipt exists, hold with reason code awaiting_receipt; do not pay.
Services invoices are matched to the PO only; no receipt is issued. Pay when the amount matches the PO.
An invoice is a duplicate when another invoice from the same vendor has the same number after removing punctuation, spaces, an “INV” prefix and leading zeros, and the same total. Flag it, hold it with reason code duplicate, and never pay it. Different numbers are not duplicates even if the amounts are equal.
Pay only to the bank account in the vendor record, and only if it is marked verified. If an invoice asks for a different account, hold with reason code bank_details_unverified and add a note. AP staff must not edit vendor bank details; Vendor Management changes them after a call-back to the number on file.
When the bank feed shows a cleared payment released outside the AP run, record it against the invoices named in the bank reference, in the order listed: pay each in full before applying anything to the next. Leave any remainder open; do not write it off.
Work only the items in your queue. Do not change any other record.
Validation
Reference solutions pass all 25 runs (5 tasks × 5 seeds), and 4 alternative valid solutions pass all 100. A seventh strategy guesses blindly from queue order; it passes only by chance.
Green: every run passed. Red cells show the share of checks a failing strategy still satisfied; amber cells show runs a blind guess passed by chance. Five task families × five seeds × six strategies. These are regression tests, not model evaluations or exhaustive adversarial coverage.
Two found so far. The grader accepted a correct resolution followed by writing off the held invoices, and the trap was always first in the queue, so acting on position alone passed two families. An integrity check now blocks the write-offs, queue order and mix are drawn from the seed, and each bug has a regression strategy. Read the write-up →
The public sample runs on your machine: the environment, the grader and one seed-1 task from each family. Unzip it and run python run.py, then plug in your own agent. Python only, nothing to install.
It is not a held-out set. Its answer keys are public, so use it to inspect the work and test your harness, not to report results. Evaluation cases for a pilot are built privately and never published.
No model results yet. When we publish them, each report will state: environment version, task families and number of episodes, model and configuration, available tools, execution limits, completion results, grader errors discovered, known limitations.