What went in
152 documents built around a deliberate trap.
The test set came in 5 document conditions, from clean PDFs to faxes. 152 of its documents were built around a trap: a total that is not the amount owed, a bill sent twice, a due date that is missing. Below is a real page for each document condition and for each of the 15 kinds of trap. Every business on them is invented.
Document conditions
5 ways the same invoice arrives.
A vendor's PDF is the easy case. The rest were rendered as scans, phone photos, faxes and photocopies, so there is no text to read, only pixels.
Clean PDF
326 documents in the test set · this one is a gutter supply invoice
The PDF exactly as the vendor's billing software made it, with every character still real text.
Traps
15 kinds of trap in 5 places on the page.
Each trap hides where a real one would: in the totals, the line table, the dates, the letterhead, or in a second copy of a bill you already have. Pick a part of the invoice to see the traps printed there.
What is owed
40 documents · 3 kinds of trap
Total is not the amount owed
The invoice totals $658.22, a $656.90 payment is already applied, and $1.32 is due.
On 20 documents in the test set
See how the test scored on the accuracy case study, and every document it got wrong on every miss and why.