Where Models Get Taiwan's Labor and Administrative Rules Wrong
Pass/fail benchmarks show that models miss local statutory math, but not whether they are running an outdated Taiwan regulation, importing a US formula, or failing at multi-step arithmetic. Encoding both rules and counter-hypotheses as code separates those failures.
Related datasetLaborCalc-TWAdminCalc-TW
Executable error diagnosis · hypotheses.pyn = 6,360
Rest-day overtime (overtime_pay)159 matched (89 + 70)
- Taiwan statutory rule
- 4/3 → 5/3 → 8/3
- Wrong rule executed
- Flat 2× (89) · US 1.5× (70)
Business ID (ban_checksum)173 matched (31.0%)
- Current (since 2023)
- Divisor ÷ 5 rule
- Deprecated rule
- Pre-2023 divisor ÷ 10
- Matched a coded wrong rule
- 38%
- 1,296 / 3,400 wrong
- Unexplained calculation failures
- 56%
- 1,891 / 3,400 wrong
- AaStatutory rule
- AaSubstituted wrong rule