Teams deploying language models into local workflows often assume that strong general reasoning and fluent Traditional Chinese carry over to everyday statutory arithmetic—or that a static pass/fail question bank is enough to verify compliance.

Evidence from TaiwanEval's first two deterministic calculation benchmarks, LaborCalc-TW and AdminCalc-TW, challenges both assumptions:

  • Nine of twelve tested models lose to a naive shortcut on Taiwan labor law. In LaborCalc-TW (2026Q4), 9 of 12 models score below a naive reference baseline (0.626) that simply divides elapsed days by 365 for annual leave and applies only the first-tier 4/3 overtime rate to every hour [1][2]. Even more striking, small cloud models that answer nearly nine in ten questions correctly on statutory annual leave—such as Beam 1.0 (0.916) and Claude Haiku 5.5 (0.895)—drop to 0.516 and 0.305 when calculating overtime pay [1][10][12].
  • Wrong answers are often systematic executions of an outdated or foreign rule, not random slips. In AdminCalc-TW (admincalc-2026Q4), models reach 87.1% accuracy on Minguo calendar conversion but fall to 38.0% on Taiwan's 8-digit business ID check digit [3][4]. Among the 558 wrong check-digit answers, 31.0% (173) equal the exact output of Taiwan's pre-2023 divisor-10 rule [4][5][14].

TaiwanEval builds these benchmarks like school math word problems: we extract the governing formula from official rules, encode it into a dual-checked programmatic specification in Python and JavaScript, and expand that specification into varied workplace and administrative scenarios. That format tests four capabilities at once: knowing Taiwan's institutional rules, modeling a scenario into the right formula, performing the calculation, and following the instruction to output only a single number. Because every item has one exact integer answer, encoding outdated Taiwan laws and foreign formulas into code lets us compare wrong outputs directly—separating rule mix-ups from calculation breakdowns and failed answer formatting.


Why Pass/Fail Benchmarks Hide the Cost of Local Rule Failures

General benchmarks test broad reasoning, open-domain knowledge, or competition mathematics, where an answer is simply right or wrong. Production workflows in a specific jurisdiction depend on deterministic arithmetic governed by local rules: piecewise rate tables, statutory rounding conventions, eligibility thresholds, local calendars, and regulatory revisions:

  • Statutory specificity and piecewise structure. Under Taiwan's Labor Standards Act (勞動基準法), statutory annual leave (特休) and overtime pay (加班費) follow tiered seniority schedules, day-type distinctions, and hour brackets [1][10][12]. A monthly-salaried worker on a rest day is paid under Taiwan's piecewise 4/3, 5/3, and 8/3 brackets—not a flat US-style 1.5x time-and-a-half rate or a flat 2x rate [1][2][10][11].
  • Regulatory revisions that outlive pretraining corpora. Administrative standards change over time. Taiwan's Ministry of Finance revised the Unified Business Number (統一編號) check-digit divisor from 10 to 5—announced on 22 December 2021 with a deadline of 31 March 2023 for public and private check-digit programs to be updated [5][14].
  • Local calendar and numeral conventions. Administrative workflows convert Minguo years (民國紀年) to and from the Gregorian (AD) calendar across varied Traditional Chinese phrasing and numeral formats [3][6][15].

When an evaluation only records pass/fail accuracy on a static question bank, every failure looks identical. A model that applies a deprecated pre-2023 Taiwan rule, a model that transplants a US overtime formula, a model that loses track of multi-step arithmetic, and a cloud API whose content filter blocks the prompt all receive the same 0. For an engineering team, confusing those failure modes has a direct cost: you cannot tell whether to fix the system with deterministic tool routing, updated statutory context, or a different serving provider.


Resolution: Rules as Code and Executable Error Hypotheses

To separate those failure modes without relying on an LLM judge, TaiwanEval builds calculation benchmarks using software verification practices, pairing forward rule verification with backward error diagnosis [9]:

Four-stage diagram showing how TaiwanEval builds and diagnoses a rule benchmark: official statute text is encoded into dual-checked Python and JavaScript rules with human sign-off, sampled across every branch into natural-language questions, and scored against executable wrong-rule hypotheses.
Figure 1 From statute text to deterministic error attribution: both ground-truth rules and counter-hypotheses are executable code [9]. Figure source

Forward Verification: From Statute to Dual Implementations

Each rule family begins with the official source text—Taiwan's Labor Standards Act for LaborCalc-TW, and Ministry of Finance and Ministry of the Interior notices for AdminCalc-TW [5][6][10][12][13][14][15]. Before generating any questions, we write a declarative specification of the inputs, branches, boundaries, and rounding rules, then implement the calculation independently in both Python and JavaScript without automated transpilation. A rule is accepted only when both implementations return identical answers across hand-verified golden cases and randomized inputs, and a named reviewer signs off on the cryptographic hashes of the specification, code, templates, and sampler [8].

From those signed rules, a sampler generates parameter combinations covering every statutory branch and tier boundary, rendered into varied Traditional Chinese prompts (including paired questions that ask the same calculation in different words, plus a held-out private split to guard against data contamination) [1][6]. LaborCalc-TW also includes 10 zero-answer control questions per family (such as a weekday with zero overtime hours), which are reported separately and excluded from accuracy and error attribution [1][2].

Backward Diagnosis: Executable Hypotheses and Naive Baselines

Every wrong answer on a non-control question stays in the denominator and is classified into one of four mutually exclusive categories (hypothesis-matched + unexplained + non-integer + unscored = wrong) [2][4]:

  • Hypothesis-matched: A wrong integer answer that equals the output of a coded alternative rule—such as the pre-2023 divisor-10 check-digit rule, US-style 1.5x overtime, or calendar-year leave proration.
  • Unexplained: A wrong integer answer that matches no coded hypothesis, indicating an arithmetic or branching breakdown.
  • Non-integer: A response from which no integer answer could be extracted.
  • Unscored: No answer was returned due to a timeout, API error, or provider content-filter rejection (counted as wrong, never dropped from the denominator).

To verify the scoring pipeline itself and give the leaderboard a practical reference bar, each benchmark release runs three synthetic solvers: an answer-key solver (oracle, 1.000) that submits the ground-truth answer to confirm the ceiling, a random guesser (0.003 in LaborCalc-TW; 0.000 in AdminCalc-TW) to confirm the floor, and a naive reference solver that encodes the single most common shortcut in each rule family [2][3]:

  • In LaborCalc-TW (0.626 overall), it calculates annual leave (0.832) by dividing elapsed days by 365 (ignoring leap years and Civil Code Article 121 month-end rules), and calculates overtime pay (0.421) by applying only the first-tier 4/3 rate to all weekday and rest-day hours while paying a flat day's wage on holidays [2][10][13].
  • In AdminCalc-TW (0.265 overall), it validates business IDs (0.53) under the deprecated pre-2023 divisor-10 rule and converts Minguo years (0.00) using an off-by-one 1912 offset instead of 1911 [3][14][15].

Why We Selected Cost-Effective Open-Weight and Small Frontier Models

In enterprise back-office workflows—such as payroll validation, leave accounting, or invoice check-digit verification—calling top-tier flagship APIs on every record is rarely practical on unit cost, latency, or data-residency grounds. Across LaborCalc-TW (2026Q4) (400 public questions; 380 scored non-control items per model across 12 models, or 2,280 evaluations per family) and AdminCalc-TW (admincalc-2026Q4) (200 public questions; 100 scored items per family across 9 models, or 900 evaluations per family), we focused on the cost-effective deployment options that teams actually weigh against one another [1][3]:

  1. Self-hostable open-weight models across parameter scales (8B to 27B): Qwen3.8-27B, gpt-oss-20b, and Gemma 4 12B, representing general open-weight models that an enterprise can self-host on a single workstation or GPU.
  2. Taiwan-tuned open-weight models (8B to 12B): TAIDE 12B and Breeze 2 8B, locally fine-tuned on Taiwanese corpora and widely considered by domestic teams that want a self-hosted model tailored to Taiwan.
  3. Small and flash-tier frontier models: GPT-6 Luna, Claude Haiku 5.5, DeepSeek-V4.1-Flash, GLM-5.3-Flash, Qwen3.8-Flash, Beam 1.0, and Flint 1.0—lightweight, low-cost API models run at their highest reasoning setting to test what small commercial endpoints deliver at comparable unit economics.

What Executable Error Attribution Reveals

Because each model configuration ran once per rule family, small score differences (such as 0.934 vs. 0.932 at the top of LaborCalc-TW) are not meaningful [1]. Across the 6,360 non-control evaluations, four findings stand out [2][4].

1. From Seniority Tier Matching to Multi-Bracket Overtime Math

Table 1 and Figure 2 show the 12 evaluated models on LaborCalc-TW (2026Q4), where the overall score is the unweighted mean of annual leave and overtime pay accuracy [1][7].

Model Overall Annual leave Overtime pay
DeepSeek-V4.1-Flash 0.934 0.958 0.911
GPT-6 Luna 0.932 0.942 0.921
Beam 1.0 0.716 0.916 0.516
Claude Haiku 5.5 0.600 0.895 0.305
GLM-5.3-Flash 0.592 0.674 0.511
Qwen3.8-Flash 0.545 0.616 0.474
Qwen3.8-27B 0.166 0.184 0.147
TAIDE 12B 0.134 0.258 0.011
gpt-oss-20b 0.074 0.147 0.000
Gemma 4 12B 0.061 0.084 0.037
Breeze 2 8B 0.037 0.068 0.005
Flint 1.0 0.037 0.053 0.021
Horizontal bar chart of LaborCalc-TW 2026Q4 overall accuracy across 12 models compared against the 0.626 naive baseline. DeepSeek-V4.1-Flash (0.934), GPT-6 Luna (0.932), and Beam 1.0 (0.716) clear the baseline, while 9 of 12 models score below it.
Figure 2 LaborCalc-TW 2026Q4 overall accuracy by model against the 0.626 naive shortcut baseline [1][7]. Figure source

Nine of the 12 models fall below the 0.626 naive baseline [1][2]. Comparing the two tasks side by side in Figure 3 shows why: models that can recall Taiwan's Labor Standards Act Article 38 seniority tiers from memory and map a worker's tenure to a fixed number of annual-leave days (such as 3 days at 6 months, 7 days at 1 year, and 10 days at 2 years) struggle as soon as the rule requires multi-bracket overtime arithmetic [10][12]. Beam 1.0 drops by 40 percentage points from annual leave (0.916) to overtime pay (0.516), and Claude Haiku 5.5 drops by 59 percentage points (0.895 to 0.305) [1].

Paired bar chart comparing LaborCalc-TW accuracy on statutory annual-leave tier matching versus multi-bracket overtime pay calculation across 12 models, highlighting the 40-point drop for Beam 1.0 and 59-point drop for Claude Haiku 5.5.
Figure 3 The complexity cliff in LaborCalc-TW: models handle statutory annual-leave seniority tiers far better than piecewise overtime arithmetic [1]. Figure source

Across the 2,280 non-control evaluations per family, executable error attribution pinpoints the specific rule substitutions behind those misses:

  • Overtime pay (1,547 wrong answers; 32.1% accuracy): 592 match a coded wrong-rule hypothesis, 769 are unexplained integer errors, 166 are non-integer responses, and 20 are unscored [2]. Within the wrong answers, 89 apply a flat 2x double-pay multiplier and 70 import a US-style 1.5x overtime formula (including 47 using the 240-hour monthly divisor) [11] instead of Taiwan's 4/3 and 5/3 brackets [2][10].
  • Annual leave (1,179 wrong answers; 48.3% accuracy): 284 match a coded hypothesis, 872 (74% of wrong answers) are unexplained, 22 are non-integer, and 1 is unscored [2]. Matched errors show models substituting calendar-year proration (62) [12], dividing elapsed days by 365 while ignoring Civil Code Article 121 month-end rules (61) [13], shifting tier boundaries by one day (51) [12], and skipping the six-month seniority tier (45) [2][12].

2. Pre-2023 Regulatory Staleness and Provider Filtering in Administrative Rules

Table 2 and Figure 4 show the 9 evaluated models on AdminCalc-TW (admincalc-2026Q4), where the naive baseline scores 0.265 [3].

Model Overall Business ID check digit Minguo year conversion
DeepSeek-V4.1-Flash 0.855 0.71 1.00
GLM-5.3-Flash 0.750 0.50 1.00
Beam 1.0 0.680 0.36 1.00
gpt-oss-20b 0.660 0.34 0.98
Qwen3.8-Flash 0.645 0.32 0.97
Qwen3.8-27B 0.620 0.27 0.97
Flint 1.0 0.555 0.29 0.82
TAIDE 12B 0.535 0.28 0.79
Breeze 2 8B 0.330 0.35 0.31
Paired horizontal bar chart of AdminCalc-TW accuracy across 9 models comparing Minguo year conversion (0.31 to 1.00) against the business ID check digit (0.27 to 0.71).
Figure 4 AdminCalc-TW accuracy by rule family: single-step Minguo year conversion vs. 8-digit business ID check-digit arithmetic [3]. Figure source

Six of the nine models achieve 0.97 to 1.00 on Minguo calendar conversion (87.1% overall accuracy), where the rule is a single-step offset of 1911 [3][4]. On the 8-digit business ID check digit, however, accuracy spans only 0.27 to 0.71 (38.0% overall) [3][4]. Because roughly half of the check-digit questions yield a different answer under the pre-2023 divisor-10 rule than under the current divisor-5 rule, a model that only knows the older rule would be expected to score around 0.50 even with flawless arithmetic [6].

Across the 900 evaluations per family, error attribution separates two distinct deployment risks:

  • Pre-2023 check-digit logic drives nearly a third of all business ID errors (558 wrong answers; 38.0% accuracy): 377 wrong answers (67.6% of errors) match an explicit counter-hypothesis, 178 (31.9%) are unexplained, 3 are non-integer, and 0 are unscored [4]. Specifically, 173 wrong answers (31.0%) match the deprecated pre-2023 divisor-10 rule [14], and 109 (19.5%) omit the tens-plus-units digit cross-sum on each weighted product, with the remaining matches split across seventh-digit (7) edge cases (25, 24, and 22 items) and inverted valid/invalid flags (24 items) [4][14].
  • Calendar boundary slips and nuanced cloud content-filtering behavior (116 wrong answers; 87.1% accuracy): On Minguo year conversion, 43 wrong answers match a hypothesis—led by off-by-one 1912 offsets (22), reversed addition/subtraction (9), 1910 offsets (7), unshifted years (4), and treating Minguo year 1 as year 0 (1)—while 72 are unexplained and 1 is unscored [4]. Taiwanese developers rarely overlook the risk of Mainland cloud censorship, yet the empirical results reveal a more nuanced picture than blanket assumptions suggest: even when Qwen3.8-27B and Qwen3.8-Flash are called directly through Alibaba Cloud Bailian's API across Minguo year prompts explicitly naming the Republic of China, content filtering is far less sweeping than commonly feared—blocking only a single item (0050, converting 「中華民國100年」 to the Gregorian calendar, on Qwen3.8-27B), which our scoring rules retain in the denominator and count as wrong in its 0.97 score alongside items 0022 and 0034 [3][4][6]. Meanwhile, DeepSeek-V4.1-Flash served via a non-Mainland provider (OpenCode API) exhibits virtually zero censorship on institutional Taiwan tasks and scores a perfect 1.00 on Minguo years [1][3]—the exact opposite of the August 2025 conclusions of hsiaoa/ai-taiwan-sovereignty-benchmark on political sovereignty prompts [16]. That does not mean censorship issues in DeepSeek-V4.1-Flash do not exist; rather, model refusal and filtering vary sharply by question type (political sovereignty vs. institutional rule execution) and serving provider, making censorship a complex, deployment-dependent variable.

3. Model-by-Model Analysis: What Origin, Scale, and Local Fine-Tuning Actually Buy

Comparing individual models across the two benchmarks overturns several common procurement assumptions:

  • DeepSeek-V4.1-Flash excels on both Taiwan benchmarks despite being a Mainland Chinese model. A common assumption is that Mainland Chinese models will confuse Taiwan's regulations with PRC rules. Instead, DeepSeek-V4.1-Flash tops both leaderboards (0.934 on LaborCalc-TW and 0.855 on AdminCalc-TW) [1][3]. On LaborCalc-TW, it answers 100% (130/130) of non-boundary annual-leave questions and 95% (76/80) of rest-day overtime questions—including 19/20 on 9-to-12-hour rest-day shifts where Taiwan's third-tier 8/3 rate applies [1]. On AdminCalc-TW, it scores 1.00 on Minguo years and 0.71 on business ID check digits, outperforming the second-place model (GLM-5.3-Flash at 0.50) by 21 percentage points [3].
  • GPT-6 Luna beats Claude Haiku 5.5 by over 60 percentage points on overtime pay (0.921 vs. 0.305). Both models are small commercial endpoints and perform similarly on statutory annual-leave tier matching (0.942 vs. 0.895) [1]. On overtime pay, Claude Haiku 5.5 still answers 40/60 (66.7%) of weekday overtime items (where GPT-6 Luna gets 60/60), but collapses to 18/130 (13.8%) across rest-day (5/80) and holiday (13/50) overtime—where GPT-6 Luna scores 115/130 (88.5%) [1]. Beam 1.0 exhibits the same pattern: a perfect 60/60 on weekday overtime followed by 28/80 on rest days and 10/50 on holidays [1].
  • Taiwan-tuned TAIDE 12B and Breeze 2 8B fail on both statutory knowledge and arithmetic. Neither model's local-corpus fine-tuning translates into rule execution. On LaborCalc-TW, TAIDE 12B scores 0.258 on annual leave and 0.011 (2/190) on overtime pay (0.134 overall), while Breeze 2 8B scores 0.068 and 0.005 (1/190, 0.037 overall)—and even on the 10 zero-overtime control questions (where the correct answer is 0), TAIDE 12B misses half (5/10) and Breeze 2 8B misses six (4/10) [1]. Even on single-step Minguo year conversion (ROC year + 1911), where every other cloud and open-weight model except Flint 1.0 scores 0.97–1.00, TAIDE 12B drops to 0.79 and Breeze 2 8B falls to 0.31 [3].
  • GLM-5.3-Flash trades statutory tier recall for steadier multi-step math, while self-hosted open weights hit a hard ceiling without tools. GLM-5.3-Flash trails Claude Haiku 5.5 by 22 points on annual-leave seniority tiers (0.674 vs. 0.895), yet beats Haiku 5.5 by over 20 points on overtime calculation (0.511 vs. 0.305) and finishes second overall on AdminCalc-TW (0.750) [1][3]. Meanwhile, mid-sized open weights (Qwen3.8-27B and gpt-oss-20b) score 0.97 and 0.98 on single-step Minguo year conversion, but collapse on LaborCalc-TW overtime pay (0.147 and 0.000) [1][3].
  • One Civil Code edge case trips even the top reasoning models. On the 10 annual-leave questions testing month-end hire dates (such as January 31) where the anniversary month has no corresponding day under Civil Code Article 121, even DeepSeek-V4.1-Flash, GPT-6 Luna, Beam 1.0, and Claude Haiku 5.5 all score 5/10 (50%) [1][13].

4. Cross-Dataset Error Synthesis: Wrong Rules (38%) vs. Calculation Collapse (56%)

Table 3 and Figure 5 combine all 3,400 wrong answers across the 6,360 non-control evaluations in both benchmarks [2][4].

Benchmark & rule family Evaluated items Wrong Accuracy Hypothesis-matched Unexplained Non-integer Unscored
LaborCalc-TW: overtime pay 2,280 1,547 32.1% 592 769 166 20
LaborCalc-TW: annual leave 2,280 1,179 48.3% 284 872 22 1
AdminCalc-TW: business ID check digit 900 558 38.0% 377 178 3 0
AdminCalc-TW: Minguo year conversion 900 116 87.1% 43 72 0 1
Total 6,360 3,400 — 1,296 (38%) 1,891 (56%) 191 22
Stacked bar chart and ranked bar chart of 3,400 wrong answers across LaborCalc-TW and AdminCalc-TW: 1,296 (38%) match a known wrong rule, 1,891 (56%) are unexplained integer errors, 191 are non-integer, and 22 are unscored, led by the pre-2023 divisor-10 business ID rule (173) and missing digit cross-sums (109).
Figure 5 Where 3,400 wrong answers go across both benchmarks, and the top alternative rules matched by wrong integer outputs [2][4]. Figure source

Unexplained integer errors total 1,641 in LaborCalc-TW (769 + 872) and 250 in AdminCalc-TW (178 + 72), or 1,891 (56%) of all 3,400 wrong answers, while 1,296 (38%) match an explicit counter-hypothesis [2][4].


Conclusion: Fluency Is Not Compliance

Replacing pass/fail accuracy with executable rules and counter-hypotheses changes how engineering teams should evaluate and deploy language models in Taiwan:

  1. Local language fine-tuning does not buy statutory competence—reasoning discipline does. Training a small model on Taiwanese text helps it write natural Traditional Chinese, but TAIDE 12B (0.134) and Breeze 2 8B (0.037) fail on both statutory leave tiers and basic Minguo year arithmetic, while DeepSeek-V4.1-Flash (0.934 and 0.855) and GPT-6 Luna (0.932) lead the field through stronger instruction following and multi-step arithmetic [1][3].
  2. Delegate deterministic rules to code, not token prediction. More than half of all errors across both benchmarks (56%, or 1,891 of 3,400 wrong answers) are unexplained arithmetic or branching collapses, and cost-effective models like Claude Haiku 5.5 (0.895 to 0.305) and Beam 1.0 (0.916 to 0.516) drop by 40 to 59 percentage points as soon as a rule moves from statutory seniority-tier matching to multi-bracket overtime calculation [1][2][4]. When a workflow is governed by a statutory formula with one exact answer, the model should extract the structured parameters from prose while a verified rule engine computes the number.
  3. Use executable counter-hypotheses to audit which world model your system is running. While numerical agreement with a counter-hypothesis does not prove a model's internal reasoning trace [6], knowing that 38% of all errors (1,296) match executable alternative rules tells you immediately whether a failure stems from stale pretraining data (such as the pre-2023 business ID revision), imported US overtime defaults, or an upstream API content filter blocking phrases like 「中華民國100年」 [2][4][5].

LaborCalc-TW and AdminCalc-TW test whether models compute the right integer when the prompt explicitly names the Taiwan rule. In real deployments, failure often starts one step earlier: when a user writes in Taiwan Traditional Chinese without naming a location, does the model assume Taiwan's jurisdiction by default? Alongside expanding LaborCalc-TW, we are preparing a dedicated Default-Taiwan (預設台灣) diagnostic to measure that implicit default directly.

In the meantime, you can inspect the full results and rule specifications on the LaborCalc-TW and AdminCalc-TW pages, test scenarios in the interactive statutory calculator, or audit the code and data directly in the public repositories (laborcalc-tw on GitHub, LaborCalc-TW on Hugging Face, admincalc-tw on GitHub, and AdminCalc-TW on Hugging Face).


Sources