Articles

TaiwanEval's announcements, release notes, method explainers, field notes from running evaluations, and perspectives.

RSS feed

Where Models Get Taiwan's Labor and Administrative Rules Wrong

Pass/fail benchmarks show that models miss local statutory math, but not whether they are running an outdated Taiwan regulation, importing a US formula, or failing at multi-step arithmetic. Encoding both rules and counter-hypotheses as code separates those failures.

Related datasetLaborCalc-TWAdminCalc-TW

Executable error diagnosis · hypotheses.pyn = 6,360

Rest-day overtime (overtime_pay)159 matched (89 + 70)
Taiwan statutory rule
4/3 → 5/3 → 8/3
Wrong rule executed
Flat 2× (89) · US 1.5× (70)
Business ID (ban_checksum)173 matched (31.0%)
Current (since 2023)
Divisor ÷ 5 rule
Deprecated rule
Pre-2023 divisor ÷ 10
Matched a coded wrong rule
38%
1,296 / 3,400 wrong
Unexplained calculation failures
56%
1,891 / 3,400 wrong
  • Statutory rule
  • Substituted wrong rule
items.jsonl

1 article.