Teams deploying language models into local workflows often assume that strong general reasoning and fluent Traditional Chinese carry over to everyday statutory arithmetic—or that a static pass/fail question bank is enough to verify compliance.
Evidence from TaiwanEval's first two deterministic calculation benchmarks, LaborCalc-TW and AdminCalc-TW, challenges both assumptions:
- Nine of twelve tested models lose to a naive shortcut on Taiwan labor law. In LaborCalc-TW (
2026Q4), 9 of 12 models score below a naive reference baseline (0.626) that simply divides elapsed days by 365 for annual leave and applies only the first-tier4/3overtime rate to every hour [1][2]. Even more striking, small cloud models that answer nearly nine in ten questions correctly on statutory annual leave—such as Beam 1.0 (0.916) and Claude Haiku 5.5 (0.895)—drop to0.516and0.305when calculating overtime pay [1][10][12]. - Wrong answers are often systematic executions of an outdated or foreign rule, not random slips. In AdminCalc-TW (
admincalc-2026Q4), models reach87.1%accuracy on Minguo calendar conversion but fall to38.0%on Taiwan's 8-digit business ID check digit [3][4]. Among the 558 wrong check-digit answers, 31.0% (173) equal the exact output of Taiwan's pre-2023 divisor-10 rule [4][5][14].
TaiwanEval builds these benchmarks like school math word problems: we extract the governing formula from official rules, encode it into a dual-checked programmatic specification in Python and JavaScript, and expand that specification into varied workplace and administrative scenarios. That format tests four capabilities at once: knowing Taiwan's institutional rules, modeling a scenario into the right formula, performing the calculation, and following the instruction to output only a single number. Because every item has one exact integer answer, encoding outdated Taiwan laws and foreign formulas into code lets us compare wrong outputs directly—separating rule mix-ups from calculation breakdowns and failed answer formatting.
Why Pass/Fail Benchmarks Hide the Cost of Local Rule Failures
General benchmarks test broad reasoning, open-domain knowledge, or competition mathematics, where an answer is simply right or wrong. Production workflows in a specific jurisdiction depend on deterministic arithmetic governed by local rules: piecewise rate tables, statutory rounding conventions, eligibility thresholds, local calendars, and regulatory revisions:
- Statutory specificity and piecewise structure. Under Taiwan's Labor Standards Act (勞動基準法), statutory annual leave (特休) and overtime pay (加班費) follow tiered seniority schedules, day-type distinctions, and hour brackets [1][10][12]. A monthly-salaried worker on a rest day is paid under Taiwan's piecewise
4/3,5/3, and8/3brackets—not a flat US-style1.5xtime-and-a-half rate or a flat2xrate [1][2][10][11]. - Regulatory revisions that outlive pretraining corpora. Administrative standards change over time. Taiwan's Ministry of Finance revised the Unified Business Number (統一編號) check-digit divisor from 10 to 5—announced on 22 December 2021 with a deadline of 31 March 2023 for public and private check-digit programs to be updated [5][14].
- Local calendar and numeral conventions. Administrative workflows convert Minguo years (民國紀年) to and from the Gregorian (AD) calendar across varied Traditional Chinese phrasing and numeral formats [3][6][15].
When an evaluation only records pass/fail accuracy on a static question bank, every failure looks identical. A model that applies a deprecated pre-2023 Taiwan rule, a model that transplants a US overtime formula, a model that loses track of multi-step arithmetic, and a cloud API whose content filter blocks the prompt all receive the same 0. For an engineering team, confusing those failure modes has a direct cost: you cannot tell whether to fix the system with deterministic tool routing, updated statutory context, or a different serving provider.
Resolution: Rules as Code and Executable Error Hypotheses
To separate those failure modes without relying on an LLM judge, TaiwanEval builds calculation benchmarks using software verification practices, pairing forward rule verification with backward error diagnosis [9]:
Forward Verification: From Statute to Dual Implementations
Each rule family begins with the official source text—Taiwan's Labor Standards Act for LaborCalc-TW, and Ministry of Finance and Ministry of the Interior notices for AdminCalc-TW [5][6][10][12][13][14][15]. Before generating any questions, we write a declarative specification of the inputs, branches, boundaries, and rounding rules, then implement the calculation independently in both Python and JavaScript without automated transpilation. A rule is accepted only when both implementations return identical answers across hand-verified golden cases and randomized inputs, and a named reviewer signs off on the cryptographic hashes of the specification, code, templates, and sampler [8].
From those signed rules, a sampler generates parameter combinations covering every statutory branch and tier boundary, rendered into varied Traditional Chinese prompts (including paired questions that ask the same calculation in different words, plus a held-out private split to guard against data contamination) [1][6]. LaborCalc-TW also includes 10 zero-answer control questions per family (such as a weekday with zero overtime hours), which are reported separately and excluded from accuracy and error attribution [1][2].
Backward Diagnosis: Executable Hypotheses and Naive Baselines
Every wrong answer on a non-control question stays in the denominator and is classified into one of four mutually exclusive categories (hypothesis-matched + unexplained + non-integer + unscored = wrong) [2][4]:
- Hypothesis-matched: A wrong integer answer that equals the output of a coded alternative rule—such as the pre-2023 divisor-10 check-digit rule, US-style
1.5xovertime, or calendar-year leave proration. - Unexplained: A wrong integer answer that matches no coded hypothesis, indicating an arithmetic or branching breakdown.
- Non-integer: A response from which no integer answer could be extracted.
- Unscored: No answer was returned due to a timeout, API error, or provider content-filter rejection (counted as wrong, never dropped from the denominator).
To verify the scoring pipeline itself and give the leaderboard a practical reference bar, each benchmark release runs three synthetic solvers: an answer-key solver (oracle, 1.000) that submits the ground-truth answer to confirm the ceiling, a random guesser (0.003 in LaborCalc-TW; 0.000 in AdminCalc-TW) to confirm the floor, and a naive reference solver that encodes the single most common shortcut in each rule family [2][3]:
- In LaborCalc-TW (
0.626overall), it calculates annual leave (0.832) by dividing elapsed days by 365 (ignoring leap years and Civil Code Article 121 month-end rules), and calculates overtime pay (0.421) by applying only the first-tier4/3rate to all weekday and rest-day hours while paying a flat day's wage on holidays [2][10][13]. - In AdminCalc-TW (
0.265overall), it validates business IDs (0.53) under the deprecated pre-2023 divisor-10 rule and converts Minguo years (0.00) using an off-by-one1912offset instead of1911[3][14][15].
Why We Selected Cost-Effective Open-Weight and Small Frontier Models
In enterprise back-office workflows—such as payroll validation, leave accounting, or invoice check-digit verification—calling top-tier flagship APIs on every record is rarely practical on unit cost, latency, or data-residency grounds. Across LaborCalc-TW (2026Q4) (400 public questions; 380 scored non-control items per model across 12 models, or 2,280 evaluations per family) and AdminCalc-TW (admincalc-2026Q4) (200 public questions; 100 scored items per family across 9 models, or 900 evaluations per family), we focused on the cost-effective deployment options that teams actually weigh against one another [1][3]:
- Self-hostable open-weight models across parameter scales (
8Bto27B):Qwen3.8-27B,gpt-oss-20b, andGemma 4 12B, representing general open-weight models that an enterprise can self-host on a single workstation or GPU. - Taiwan-tuned open-weight models (
8Bto12B):TAIDE 12BandBreeze 2 8B, locally fine-tuned on Taiwanese corpora and widely considered by domestic teams that want a self-hosted model tailored to Taiwan. - Small and flash-tier frontier models:
GPT-6 Luna,Claude Haiku 5.5,DeepSeek-V4.1-Flash,GLM-5.3-Flash,Qwen3.8-Flash,Beam 1.0, andFlint 1.0—lightweight, low-cost API models run at their highest reasoning setting to test what small commercial endpoints deliver at comparable unit economics.
What Executable Error Attribution Reveals
Because each model configuration ran once per rule family, small score differences (such as 0.934 vs. 0.932 at the top of LaborCalc-TW) are not meaningful [1]. Across the 6,360 non-control evaluations, four findings stand out [2][4].
1. From Seniority Tier Matching to Multi-Bracket Overtime Math
Table 1 and Figure 2 show the 12 evaluated models on LaborCalc-TW (2026Q4), where the overall score is the unweighted mean of annual leave and overtime pay accuracy [1][7].
| Model | Overall | Annual leave | Overtime pay |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 0.934 | 0.958 | 0.911 |
| GPT-6 Luna | 0.932 | 0.942 | 0.921 |
| Beam 1.0 | 0.716 | 0.916 | 0.516 |
| Claude Haiku 5.5 | 0.600 | 0.895 | 0.305 |
| GLM-5.3-Flash | 0.592 | 0.674 | 0.511 |
| Qwen3.8-Flash | 0.545 | 0.616 | 0.474 |
| Qwen3.8-27B | 0.166 | 0.184 | 0.147 |
| TAIDE 12B | 0.134 | 0.258 | 0.011 |
| gpt-oss-20b | 0.074 | 0.147 | 0.000 |
| Gemma 4 12B | 0.061 | 0.084 | 0.037 |
| Breeze 2 8B | 0.037 | 0.068 | 0.005 |
| Flint 1.0 | 0.037 | 0.053 | 0.021 |
Nine of the 12 models fall below the 0.626 naive baseline [1][2]. Comparing the two tasks side by side in Figure 3 shows why: models that can recall Taiwan's Labor Standards Act Article 38 seniority tiers from memory and map a worker's tenure to a fixed number of annual-leave days (such as 3 days at 6 months, 7 days at 1 year, and 10 days at 2 years) struggle as soon as the rule requires multi-bracket overtime arithmetic [10][12]. Beam 1.0 drops by 40 percentage points from annual leave (0.916) to overtime pay (0.516), and Claude Haiku 5.5 drops by 59 percentage points (0.895 to 0.305) [1].
Across the 2,280 non-control evaluations per family, executable error attribution pinpoints the specific rule substitutions behind those misses:
- Overtime pay (
1,547wrong answers;32.1%accuracy):592match a coded wrong-rule hypothesis,769are unexplained integer errors,166are non-integer responses, and20are unscored [2]. Within the wrong answers, 89 apply a flat2xdouble-pay multiplier and 70 import a US-style1.5xovertime formula (including 47 using the 240-hour monthly divisor) [11] instead of Taiwan's4/3and5/3brackets [2][10]. - Annual leave (
1,179wrong answers;48.3%accuracy):284match a coded hypothesis,872(74%of wrong answers) are unexplained,22are non-integer, and1is unscored [2]. Matched errors show models substituting calendar-year proration (62) [12], dividing elapsed days by 365 while ignoring Civil Code Article 121 month-end rules (61) [13], shifting tier boundaries by one day (51) [12], and skipping the six-month seniority tier (45) [2][12].
2. Pre-2023 Regulatory Staleness and Provider Filtering in Administrative Rules
Table 2 and Figure 4 show the 9 evaluated models on AdminCalc-TW (admincalc-2026Q4), where the naive baseline scores 0.265 [3].
| Model | Overall | Business ID check digit | Minguo year conversion |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 0.855 | 0.71 | 1.00 |
| GLM-5.3-Flash | 0.750 | 0.50 | 1.00 |
| Beam 1.0 | 0.680 | 0.36 | 1.00 |
| gpt-oss-20b | 0.660 | 0.34 | 0.98 |
| Qwen3.8-Flash | 0.645 | 0.32 | 0.97 |
| Qwen3.8-27B | 0.620 | 0.27 | 0.97 |
| Flint 1.0 | 0.555 | 0.29 | 0.82 |
| TAIDE 12B | 0.535 | 0.28 | 0.79 |
| Breeze 2 8B | 0.330 | 0.35 | 0.31 |
Six of the nine models achieve 0.97 to 1.00 on Minguo calendar conversion (87.1% overall accuracy), where the rule is a single-step offset of 1911 [3][4]. On the 8-digit business ID check digit, however, accuracy spans only 0.27 to 0.71 (38.0% overall) [3][4]. Because roughly half of the check-digit questions yield a different answer under the pre-2023 divisor-10 rule than under the current divisor-5 rule, a model that only knows the older rule would be expected to score around 0.50 even with flawless arithmetic [6].
Across the 900 evaluations per family, error attribution separates two distinct deployment risks:
- Pre-2023 check-digit logic drives nearly a third of all business ID errors (
558wrong answers;38.0%accuracy):377wrong answers (67.6%of errors) match an explicit counter-hypothesis,178(31.9%) are unexplained,3are non-integer, and0are unscored [4]. Specifically, 173 wrong answers (31.0%) match the deprecated pre-2023 divisor-10 rule [14], and 109 (19.5%) omit the tens-plus-units digit cross-sum on each weighted product, with the remaining matches split across seventh-digit (7) edge cases (25,24, and22items) and inverted valid/invalid flags (24items) [4][14]. - Calendar boundary slips and nuanced cloud content-filtering behavior (
116wrong answers;87.1%accuracy): On Minguo year conversion,43wrong answers match a hypothesis—led by off-by-one1912offsets (22), reversed addition/subtraction (9),1910offsets (7), unshifted years (4), and treating Minguo year 1 as year 0 (1)—while72are unexplained and1is unscored [4]. Taiwanese developers rarely overlook the risk of Mainland cloud censorship, yet the empirical results reveal a more nuanced picture than blanket assumptions suggest: even whenQwen3.8-27BandQwen3.8-Flashare called directly through Alibaba Cloud Bailian's API across Minguo year prompts explicitly naming the Republic of China, content filtering is far less sweeping than commonly feared—blocking only a single item (0050, converting 「中華民國100年」 to the Gregorian calendar, onQwen3.8-27B), which our scoring rules retain in the denominator and count as wrong in its0.97score alongside items0022and0034[3][4][6]. Meanwhile,DeepSeek-V4.1-Flashserved via a non-Mainland provider (OpenCode API) exhibits virtually zero censorship on institutional Taiwan tasks and scores a perfect1.00on Minguo years [1][3]—the exact opposite of the August 2025 conclusions ofhsiaoa/ai-taiwan-sovereignty-benchmarkon political sovereignty prompts [16]. That does not mean censorship issues inDeepSeek-V4.1-Flashdo not exist; rather, model refusal and filtering vary sharply by question type (political sovereignty vs. institutional rule execution) and serving provider, making censorship a complex, deployment-dependent variable.
3. Model-by-Model Analysis: What Origin, Scale, and Local Fine-Tuning Actually Buy
Comparing individual models across the two benchmarks overturns several common procurement assumptions:
DeepSeek-V4.1-Flashexcels on both Taiwan benchmarks despite being a Mainland Chinese model. A common assumption is that Mainland Chinese models will confuse Taiwan's regulations with PRC rules. Instead,DeepSeek-V4.1-Flashtops both leaderboards (0.934on LaborCalc-TW and0.855on AdminCalc-TW) [1][3]. On LaborCalc-TW, it answers100%(130/130) of non-boundary annual-leave questions and95%(76/80) of rest-day overtime questions—including19/20on 9-to-12-hour rest-day shifts where Taiwan's third-tier8/3rate applies [1]. On AdminCalc-TW, it scores1.00on Minguo years and0.71on business ID check digits, outperforming the second-place model (GLM-5.3-Flashat0.50) by 21 percentage points [3].GPT-6 LunabeatsClaude Haiku 5.5by over 60 percentage points on overtime pay (0.921vs.0.305). Both models are small commercial endpoints and perform similarly on statutory annual-leave tier matching (0.942vs.0.895) [1]. On overtime pay,Claude Haiku 5.5still answers40/60(66.7%) of weekday overtime items (whereGPT-6 Lunagets60/60), but collapses to18/130(13.8%) across rest-day (5/80) and holiday (13/50) overtime—whereGPT-6 Lunascores115/130(88.5%) [1].Beam 1.0exhibits the same pattern: a perfect60/60on weekday overtime followed by28/80on rest days and10/50on holidays [1].- Taiwan-tuned
TAIDE 12BandBreeze 2 8Bfail on both statutory knowledge and arithmetic. Neither model's local-corpus fine-tuning translates into rule execution. On LaborCalc-TW,TAIDE 12Bscores0.258on annual leave and0.011(2/190) on overtime pay (0.134overall), whileBreeze 2 8Bscores0.068and0.005(1/190,0.037overall)—and even on the 10 zero-overtime control questions (where the correct answer is0),TAIDE 12Bmisses half (5/10) andBreeze 2 8Bmisses six (4/10) [1]. Even on single-step Minguo year conversion (ROC year + 1911), where every other cloud and open-weight model exceptFlint 1.0scores0.97–1.00,TAIDE 12Bdrops to0.79andBreeze 2 8Bfalls to0.31[3]. GLM-5.3-Flashtrades statutory tier recall for steadier multi-step math, while self-hosted open weights hit a hard ceiling without tools.GLM-5.3-FlashtrailsClaude Haiku 5.5by 22 points on annual-leave seniority tiers (0.674vs.0.895), yet beatsHaiku 5.5by over 20 points on overtime calculation (0.511vs.0.305) and finishes second overall on AdminCalc-TW (0.750) [1][3]. Meanwhile, mid-sized open weights (Qwen3.8-27Bandgpt-oss-20b) score0.97and0.98on single-step Minguo year conversion, but collapse on LaborCalc-TW overtime pay (0.147and0.000) [1][3].- One Civil Code edge case trips even the top reasoning models. On the 10 annual-leave questions testing month-end hire dates (such as January 31) where the anniversary month has no corresponding day under Civil Code Article 121, even
DeepSeek-V4.1-Flash,GPT-6 Luna,Beam 1.0, andClaude Haiku 5.5all score5/10(50%) [1][13].
4. Cross-Dataset Error Synthesis: Wrong Rules (38%) vs. Calculation Collapse (56%)
Table 3 and Figure 5 combine all 3,400 wrong answers across the 6,360 non-control evaluations in both benchmarks [2][4].
| Benchmark & rule family | Evaluated items | Wrong | Accuracy | Hypothesis-matched | Unexplained | Non-integer | Unscored |
|---|---|---|---|---|---|---|---|
| LaborCalc-TW: overtime pay | 2,280 | 1,547 | 32.1% | 592 | 769 | 166 | 20 |
| LaborCalc-TW: annual leave | 2,280 | 1,179 | 48.3% | 284 | 872 | 22 | 1 |
| AdminCalc-TW: business ID check digit | 900 | 558 | 38.0% | 377 | 178 | 3 | 0 |
| AdminCalc-TW: Minguo year conversion | 900 | 116 | 87.1% | 43 | 72 | 0 | 1 |
| Total | 6,360 | 3,400 | — | 1,296 (38%) | 1,891 (56%) | 191 | 22 |
Unexplained integer errors total 1,641 in LaborCalc-TW (769 + 872) and 250 in AdminCalc-TW (178 + 72), or 1,891 (56%) of all 3,400 wrong answers, while 1,296 (38%) match an explicit counter-hypothesis [2][4].
Conclusion: Fluency Is Not Compliance
Replacing pass/fail accuracy with executable rules and counter-hypotheses changes how engineering teams should evaluate and deploy language models in Taiwan:
- Local language fine-tuning does not buy statutory competence—reasoning discipline does. Training a small model on Taiwanese text helps it write natural Traditional Chinese, but
TAIDE 12B(0.134) andBreeze 2 8B(0.037) fail on both statutory leave tiers and basic Minguo year arithmetic, whileDeepSeek-V4.1-Flash(0.934and0.855) andGPT-6 Luna(0.932) lead the field through stronger instruction following and multi-step arithmetic [1][3]. - Delegate deterministic rules to code, not token prediction. More than half of all errors across both benchmarks (
56%, or1,891of3,400wrong answers) are unexplained arithmetic or branching collapses, and cost-effective models likeClaude Haiku 5.5(0.895to0.305) andBeam 1.0(0.916to0.516) drop by 40 to 59 percentage points as soon as a rule moves from statutory seniority-tier matching to multi-bracket overtime calculation [1][2][4]. When a workflow is governed by a statutory formula with one exact answer, the model should extract the structured parameters from prose while a verified rule engine computes the number. - Use executable counter-hypotheses to audit which world model your system is running. While numerical agreement with a counter-hypothesis does not prove a model's internal reasoning trace [6], knowing that
38%of all errors (1,296) match executable alternative rules tells you immediately whether a failure stems from stale pretraining data (such as the pre-2023 business ID revision), imported US overtime defaults, or an upstream API content filter blocking phrases like 「中華民國100年」 [2][4][5].
LaborCalc-TW and AdminCalc-TW test whether models compute the right integer when the prompt explicitly names the Taiwan rule. In real deployments, failure often starts one step earlier: when a user writes in Taiwan Traditional Chinese without naming a location, does the model assume Taiwan's jurisdiction by default? Alongside expanding LaborCalc-TW, we are preparing a dedicated Default-Taiwan (預設台灣) diagnostic to measure that implicit default directly.
In the meantime, you can inspect the full results and rule specifications on the LaborCalc-TW and AdminCalc-TW pages, test scenarios in the interactive statutory calculator, or audit the code and data directly in the public repositories (laborcalc-tw on GitHub, LaborCalc-TW on Hugging Face, admincalc-tw on GitHub, and AdminCalc-TW on Hugging Face).
Sources
- [1]
laborcalc-tw: index/2026Q4/index.json - [2]
laborcalc-tw: eval/scores/2026Q4/items.jsonl - [3]
admincalc-tw: index/admincalc-2026Q4/site-data/results.json - [4]
admincalc-tw: eval/scores/admincalc-2026Q4/items.jsonl - [5]
admincalc-tw: silver/families/ban_checksum/spec.yaml - [6]
admincalc-tw: README - [7]
laborcalc-tw: figures/scores.svg - [8]
laborcalc-tw: gold/releases/<release>/public/*.manifest.json;admincalc-tw: gold/releases/<release>/public/*.manifest.json - [9]
laborcalc-tw: figures/pipeline.svg;laborcalc-tw: figures/scores.svg;admincalc-tw: figures/pipeline.svg;admincalc-tw: figures/scores.svg - [10] Taiwan Labor Standards Act (勞動基準法), Articles 24 (
4/3and5/3weekday/rest-day overtime brackets) and 39 (holiday double pay); Council of Labor Affairs Interpretation Tai-90-Lao-Dong-2-0026202 (240-hour monthly divisor for regular hourly pay); Ministry of Labor Overtime Pay Diagram (8/3bracket for rest-day hours 9–12) and Monthly Overtime Pay Calculator - [11] U.S. Fair Labor Standards Act of 1938, 29 U.S.C. § 207(a)(1) (
1.5xtime-and-a-half overtime rate beyond 40 hours per week) (U.S. GovInfo statutory text); U.S. Department of Labor, Wage and Hour Division — Overtime Pay - [12] Taiwan Labor Standards Act (勞動基準法), Article 38, Paragraph 1 (statutory annual-leave seniority tiers: 3 days at 6 months, 7/10/14/15 days up to 30 days); Enforcement Rules of the Labor Standards Act (勞動基準法施行細則), Article 5, Paragraph 1 (seniority counted from hire date) and Article 24, Paragraph 2 (anniversary-year vs. calendar-year leave periods)
- [13] Taiwan Civil Code (民法), Article 121 (month-end period rule when the final month lacks a corresponding day); Ministry of Labor Statutory Annual Leave Calculator
- [14] Fiscal Information Agency, Ministry of Finance — Unified Business Number (統一編號) Check-Digit Logic Revision Notice (published 2021-12-22; divisor revised from 10 to 5 with a 2023-03-31 transition deadline) (attachment PDF)
- [15] Department of Household Registration, Ministry of the Interior — Minguo and Gregorian Year/Age Comparison Table (年歲對照表)
- [16]
hsiaoa/ai-taiwan-sovereignty-benchmark(Taiwan Sovereignty Benchmark, August 2025)