# Where Models Get Taiwan's Labor and Administrative Rules Wrong

Pass/fail benchmarks show that models miss local statutory math, but not whether they are running an outdated Taiwan regulation, importing a US formula, or failing at multi-step arithmetic. Encoding both rules and counter-hypotheses as code separates those failures.

Calvin Ku · 2026-11-17 · Method explainers
https://taiwaneval.org/en/articles/where-models-get-taiwan-rules-wrong

## Key takeaways

- In `LaborCalc-TW` (`2026Q4`), **9 of 12 cost-effective open-weight and small frontier models score below a naive shortcut baseline (0.626)**, and mid-tier models drop by 40 to 59 percentage points when moving from statutory annual-leave tier matching to multi-bracket overtime calculation.
- In `AdminCalc-TW` (`admincalc-2026Q4`), **31.0% of the 558 wrong business ID check-digit answers (173) equal the output of Taiwan's pre-2023 divisor-10 rule** rather than random arithmetic errors.
- **Origin and local text fine-tuning do not predict statutory accuracy**: `DeepSeek-V4.1-Flash` leads both Taiwan benchmarks (`0.934` and `0.855`), `GPT-6 Luna` beats `Claude Haiku 5.5` by over 60 points on overtime pay (`0.921` vs. `0.305`), and Taiwan-tuned `TAIDE 12B` and `Breeze 2 8B` struggle on both knowledge and reasoning.

Teams deploying language models into local workflows often assume that strong general reasoning and fluent Traditional Chinese carry over to everyday statutory arithmetic—or that a static pass/fail question bank is enough to verify compliance.

Evidence from TaiwanEval's first two deterministic calculation benchmarks, **LaborCalc-TW** and **AdminCalc-TW**, challenges both assumptions:

- **Nine of twelve tested models lose to a naive shortcut on Taiwan labor law.** In LaborCalc-TW (`2026Q4`), **9 of 12 models score below a naive reference baseline (`0.626`)** that simply divides elapsed days by 365 for annual leave and applies only the first-tier `4/3` overtime rate to every hour [1][2]. Even more striking, small cloud models that answer nearly nine in ten questions correctly on statutory annual leave—such as Beam 1.0 (`0.916`) and Claude Haiku 5.5 (`0.895`)—drop to `0.516` and `0.305` when calculating overtime pay [1][10][12].
- **Wrong answers are often systematic executions of an outdated or foreign rule, not random slips.** In AdminCalc-TW (`admincalc-2026Q4`), models reach `87.1%` accuracy on Minguo calendar conversion but fall to `38.0%` on Taiwan's 8-digit business ID check digit [3][4]. Among the 558 wrong check-digit answers, **31.0% (173) equal the exact output of Taiwan's pre-2023 divisor-10 rule** [4][5][14].

> **Evidence**: Across 6,360 non-control evaluations in LaborCalc-TW and AdminCalc-TW, 3,400 answers are wrong: 1,296 (38%) match an executable alternative rule, while 1,891 (56%) are unexplained integer errors. ([Dataset: LaborCalc-TW](https://taiwaneval.org/en/datasets/laborcalc-tw))



TaiwanEval builds these benchmarks like school math word problems: we extract the governing formula from official rules, encode it into a dual-checked programmatic specification in Python and JavaScript, and expand that specification into varied workplace and administrative scenarios. That format tests four capabilities at once: knowing Taiwan's institutional rules, modeling a scenario into the right formula, performing the calculation, and following the instruction to output only a single number. Because every item has one exact integer answer, encoding outdated Taiwan laws and foreign formulas into code lets us compare wrong outputs directly—separating rule mix-ups from calculation breakdowns and failed answer formatting.

---

## Why Pass/Fail Benchmarks Hide the Cost of Local Rule Failures

General benchmarks test broad reasoning, open-domain knowledge, or competition mathematics, where an answer is simply right or wrong. Production workflows in a specific jurisdiction depend on deterministic arithmetic governed by local rules: piecewise rate tables, statutory rounding conventions, eligibility thresholds, local calendars, and regulatory revisions:

- **Statutory specificity and piecewise structure.** Under Taiwan's Labor Standards Act (勞動基準法), statutory annual leave (特休) and overtime pay (加班費) follow tiered seniority schedules, day-type distinctions, and hour brackets [1][10][12]. A monthly-salaried worker on a rest day is paid under Taiwan's piecewise `4/3`, `5/3`, and `8/3` brackets—not a flat US-style `1.5x` time-and-a-half rate or a flat `2x` rate [1][2][10][11].
- **Regulatory revisions that outlive pretraining corpora.** Administrative standards change over time. Taiwan's Ministry of Finance revised the Unified Business Number (統一編號) check-digit divisor from 10 to 5—announced on 22 December 2021 with a deadline of 31 March 2023 for public and private check-digit programs to be updated [5][14].
- **Local calendar and numeral conventions.** Administrative workflows convert Minguo years (民國紀年) to and from the Gregorian (AD) calendar across varied Traditional Chinese phrasing and numeral formats [3][6][15].

When an evaluation only records pass/fail accuracy on a static question bank, every failure looks identical. A model that applies a deprecated pre-2023 Taiwan rule, a model that transplants a US overtime formula, a model that loses track of multi-step arithmetic, and a cloud API whose content filter blocks the prompt all receive the same `0`. For an engineering team, confusing those failure modes has a direct cost: you cannot tell whether to fix the system with deterministic tool routing, updated statutory context, or a different serving provider.

---

## Resolution: Rules as Code and Executable Error Hypotheses

To separate those failure modes without relying on an LLM judge, TaiwanEval builds calculation benchmarks using software verification practices, pairing forward rule verification with backward error diagnosis [9]:

![Four-stage diagram showing how TaiwanEval builds and diagnoses a rule benchmark: official statute text is encoded into dual-checked Python and JavaScript rules with human sign-off, sampled across every branch into natural-language questions, and scored against executable wrong-rule hypotheses.](https://taiwaneval.org/images/articles/where-models-get-taiwan-rules-wrong/laborcalc-tw-pipeline.svg)

**Figure 1. From statute text to deterministic error attribution: both ground-truth rules and counter-hypotheses are executable code [9].**

[Figure source](https://github.com/taiwaneval/laborcalc-tw/blob/main/figures/pipeline.svg)



### Forward Verification: From Statute to Dual Implementations

Each rule family begins with the official source text—Taiwan's Labor Standards Act for LaborCalc-TW, and Ministry of Finance and Ministry of the Interior notices for AdminCalc-TW [5][6][10][12][13][14][15]. Before generating any questions, we write a declarative specification of the inputs, branches, boundaries, and rounding rules, then implement the calculation independently in both Python and JavaScript without automated transpilation. A rule is accepted only when both implementations return identical answers across hand-verified golden cases and randomized inputs, and a named reviewer signs off on the cryptographic hashes of the specification, code, templates, and sampler [8].

From those signed rules, a sampler generates parameter combinations covering every statutory branch and tier boundary, rendered into varied Traditional Chinese prompts (including paired questions that ask the same calculation in different words, plus a held-out private split to guard against data contamination) [1][6]. LaborCalc-TW also includes 10 zero-answer control questions per family (such as a weekday with zero overtime hours), which are reported separately and excluded from accuracy and error attribution [1][2].

### Backward Diagnosis: Executable Hypotheses and Naive Baselines

Every wrong answer on a non-control question stays in the denominator and is classified into one of four mutually exclusive categories (`hypothesis-matched + unexplained + non-integer + unscored = wrong`) [2][4]:

- **Hypothesis-matched:** A wrong integer answer that equals the output of a coded alternative rule—such as the pre-2023 divisor-10 check-digit rule, US-style `1.5x` overtime, or calendar-year leave proration.
- **Unexplained:** A wrong integer answer that matches no coded hypothesis, indicating an arithmetic or branching breakdown.
- **Non-integer:** A response from which no integer answer could be extracted.
- **Unscored:** No answer was returned due to a timeout, API error, or provider content-filter rejection (counted as wrong, never dropped from the denominator).

To verify the scoring pipeline itself and give the leaderboard a practical reference bar, each benchmark release runs three synthetic solvers: an answer-key solver (`oracle`, `1.000`) that submits the ground-truth answer to confirm the ceiling, a random guesser (`0.003` in LaborCalc-TW; `0.000` in AdminCalc-TW) to confirm the floor, and a **naive reference solver** that encodes the single most common shortcut in each rule family [2][3]:
- In **LaborCalc-TW (`0.626` overall)**, it calculates annual leave (`0.832`) by dividing elapsed days by 365 (ignoring leap years and Civil Code Article 121 month-end rules), and calculates overtime pay (`0.421`) by applying only the first-tier `4/3` rate to all weekday and rest-day hours while paying a flat day's wage on holidays [2][10][13].
- In **AdminCalc-TW (`0.265` overall)**, it validates business IDs (`0.53`) under the deprecated pre-2023 divisor-10 rule and converts Minguo years (`0.00`) using an off-by-one `1912` offset instead of `1911` [3][14][15].

### Why We Selected Cost-Effective Open-Weight and Small Frontier Models

In enterprise back-office workflows—such as payroll validation, leave accounting, or invoice check-digit verification—calling top-tier flagship APIs on every record is rarely practical on unit cost, latency, or data-residency grounds. Across **LaborCalc-TW (`2026Q4`)** (400 public questions; 380 scored non-control items per model across 12 models, or 2,280 evaluations per family) and **AdminCalc-TW (`admincalc-2026Q4`)** (200 public questions; 100 scored items per family across 9 models, or 900 evaluations per family), we focused on the **cost-effective deployment options** that teams actually weigh against one another [1][3]:

1. **Self-hostable open-weight models across parameter scales (`8B` to `27B`):** `Qwen3.8-27B`, `gpt-oss-20b`, and `Gemma 4 12B`, representing general open-weight models that an enterprise can self-host on a single workstation or GPU.
2. **Taiwan-tuned open-weight models (`8B` to `12B`):** `TAIDE 12B` and `Breeze 2 8B`, locally fine-tuned on Taiwanese corpora and widely considered by domestic teams that want a self-hosted model tailored to Taiwan.
3. **Small and flash-tier frontier models:** `GPT-6 Luna`, `Claude Haiku 5.5`, `DeepSeek-V4.1-Flash`, `GLM-5.3-Flash`, `Qwen3.8-Flash`, `Beam 1.0`, and `Flint 1.0`—lightweight, low-cost API models run at their highest reasoning setting to test what small commercial endpoints deliver at comparable unit economics.

---

## What Executable Error Attribution Reveals

Because each model configuration ran once per rule family, small score differences (such as `0.934` vs. `0.932` at the top of LaborCalc-TW) are not meaningful [1]. Across the 6,360 non-control evaluations, four findings stand out [2][4].

### 1. From Seniority Tier Matching to Multi-Bracket Overtime Math

Table 1 and Figure 2 show the 12 evaluated models on LaborCalc-TW (`2026Q4`), where the overall score is the unweighted mean of annual leave and overtime pay accuracy [1][7].

| Model | Overall | Annual leave | Overtime pay |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 0.934 | 0.958 | 0.911 |
| GPT-6 Luna | 0.932 | 0.942 | 0.921 |
| Beam 1.0 | 0.716 | 0.916 | 0.516 |
| Claude Haiku 5.5 | 0.600 | 0.895 | 0.305 |
| GLM-5.3-Flash | 0.592 | 0.674 | 0.511 |
| Qwen3.8-Flash | 0.545 | 0.616 | 0.474 |
| Qwen3.8-27B | 0.166 | 0.184 | 0.147 |
| TAIDE 12B | 0.134 | 0.258 | 0.011 |
| gpt-oss-20b | 0.074 | 0.147 | 0.000 |
| Gemma 4 12B | 0.061 | 0.084 | 0.037 |
| Breeze 2 8B | 0.037 | 0.068 | 0.005 |
| Flint 1.0 | 0.037 | 0.053 | 0.021 |

![Horizontal bar chart of LaborCalc-TW 2026Q4 overall accuracy across 12 models compared against the 0.626 naive baseline. DeepSeek-V4.1-Flash (0.934), GPT-6 Luna (0.932), and Beam 1.0 (0.716) clear the baseline, while 9 of 12 models score below it.](https://taiwaneval.org/images/articles/where-models-get-taiwan-rules-wrong/laborcalc-tw-scores.svg)

**Figure 2. LaborCalc-TW 2026Q4 overall accuracy by model against the 0.626 naive shortcut baseline [1][7].**

[Figure source](https://github.com/taiwaneval/laborcalc-tw/blob/main/figures/scores.svg)



Nine of the 12 models fall below the `0.626` naive baseline [1][2]. Comparing the two tasks side by side in Figure 3 shows why: models that can recall Taiwan's Labor Standards Act Article 38 seniority tiers from memory and map a worker's tenure to a fixed number of annual-leave days (such as 3 days at 6 months, 7 days at 1 year, and 10 days at 2 years) struggle as soon as the rule requires multi-bracket overtime arithmetic [10][12]. Beam 1.0 drops by 40 percentage points from annual leave (`0.916`) to overtime pay (`0.516`), and Claude Haiku 5.5 drops by 59 percentage points (`0.895` to `0.305`) [1].

![Paired bar chart comparing LaborCalc-TW accuracy on statutory annual-leave tier matching versus multi-bracket overtime pay calculation across 12 models, highlighting the 40-point drop for Beam 1.0 and 59-point drop for Claude Haiku 5.5.](https://taiwaneval.org/images/articles/where-models-get-taiwan-rules-wrong/laborcalc-tw-family-gap.svg)

**Figure 3. The complexity cliff in LaborCalc-TW: models handle statutory annual-leave seniority tiers far better than piecewise overtime arithmetic [1].**

[Figure source](https://github.com/taiwaneval/laborcalc-tw/blob/v2026Q4/index/2026Q4/index.json)



Across the 2,280 non-control evaluations per family, executable error attribution pinpoints the specific rule substitutions behind those misses:

- **Overtime pay (`1,547` wrong answers; `32.1%` accuracy):** `592` match a coded wrong-rule hypothesis, `769` are unexplained integer errors, `166` are non-integer responses, and `20` are unscored [2]. Within the wrong answers, **89 apply a flat `2x` double-pay multiplier** and **70 import a US-style `1.5x` overtime formula** (including **47** using the 240-hour monthly divisor) [11] instead of Taiwan's `4/3` and `5/3` brackets [2][10].
- **Annual leave (`1,179` wrong answers; `48.3%` accuracy):** `284` match a coded hypothesis, `872` (`74%` of wrong answers) are unexplained, `22` are non-integer, and `1` is unscored [2]. Matched errors show models substituting **calendar-year proration (`62`)** [12], **dividing elapsed days by 365 while ignoring Civil Code Article 121 month-end rules (`61`)** [13], **shifting tier boundaries by one day (`51`)** [12], and **skipping the six-month seniority tier (`45`)** [2][12].

### 2. Pre-2023 Regulatory Staleness and Provider Filtering in Administrative Rules

Table 2 and Figure 4 show the 9 evaluated models on AdminCalc-TW (`admincalc-2026Q4`), where the naive baseline scores `0.265` [3].

| Model | Overall | Business ID check digit | Minguo year conversion |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 0.855 | 0.71 | 1.00 |
| GLM-5.3-Flash | 0.750 | 0.50 | 1.00 |
| Beam 1.0 | 0.680 | 0.36 | 1.00 |
| gpt-oss-20b | 0.660 | 0.34 | 0.98 |
| Qwen3.8-Flash | 0.645 | 0.32 | 0.97 |
| Qwen3.8-27B | 0.620 | 0.27 | 0.97 |
| Flint 1.0 | 0.555 | 0.29 | 0.82 |
| TAIDE 12B | 0.535 | 0.28 | 0.79 |
| Breeze 2 8B | 0.330 | 0.35 | 0.31 |

![Paired horizontal bar chart of AdminCalc-TW accuracy across 9 models comparing Minguo year conversion (0.31 to 1.00) against the business ID check digit (0.27 to 0.71).](https://taiwaneval.org/images/articles/where-models-get-taiwan-rules-wrong/admincalc-tw-scores.svg)

**Figure 4. AdminCalc-TW accuracy by rule family: single-step Minguo year conversion vs. 8-digit business ID check-digit arithmetic [3].**

[Figure source](https://github.com/taiwaneval/admincalc-tw/blob/v2026Q4/index/admincalc-2026Q4/site-data/results.json)



Six of the nine models achieve `0.97` to `1.00` on Minguo calendar conversion (`87.1%` overall accuracy), where the rule is a single-step offset of `1911` [3][4]. On the 8-digit business ID check digit, however, accuracy spans only `0.27` to `0.71` (`38.0%` overall) [3][4]. Because roughly half of the check-digit questions yield a different answer under the pre-2023 divisor-10 rule than under the current divisor-5 rule, a model that only knows the older rule would be expected to score around `0.50` even with flawless arithmetic [6].

Across the 900 evaluations per family, error attribution separates two distinct deployment risks:

- **Pre-2023 check-digit logic drives nearly a third of all business ID errors (`558` wrong answers; `38.0%` accuracy):** `377` wrong answers (`67.6%` of errors) match an explicit counter-hypothesis, `178` (`31.9%`) are unexplained, `3` are non-integer, and `0` are unscored [4]. Specifically, **173 wrong answers (`31.0%`) match the deprecated pre-2023 divisor-10 rule** [14], and **109 (`19.5%`) omit the tens-plus-units digit cross-sum** on each weighted product, with the remaining matches split across seventh-digit (`7`) edge cases (`25`, `24`, and `22` items) and inverted valid/invalid flags (`24` items) [4][14].
- **Calendar boundary slips and nuanced cloud content-filtering behavior (`116` wrong answers; `87.1%` accuracy):** On Minguo year conversion, `43` wrong answers match a hypothesis—led by off-by-one `1912` offsets (`22`), reversed addition/subtraction (`9`), `1910` offsets (`7`), unshifted years (`4`), and treating Minguo year 1 as year 0 (`1`)—while `72` are unexplained and `1` is unscored [4]. Taiwanese developers rarely overlook the risk of Mainland cloud censorship, yet the empirical results reveal a more nuanced picture than blanket assumptions suggest: even when `Qwen3.8-27B` and `Qwen3.8-Flash` are called directly through Alibaba Cloud Bailian's API across Minguo year prompts explicitly naming the Republic of China, content filtering is far less sweeping than commonly feared—blocking only a single item (`0050`, converting 「中華民國100年」 to the Gregorian calendar, on `Qwen3.8-27B`), which our scoring rules retain in the denominator and count as wrong in its `0.97` score alongside items `0022` and `0034` [3][4][6]. Meanwhile, `DeepSeek-V4.1-Flash` served via a non-Mainland provider (`OpenCode API`) exhibits virtually zero censorship on institutional Taiwan tasks and scores a perfect `1.00` on Minguo years [1][3]—the exact opposite of the August 2025 conclusions of [`hsiaoa/ai-taiwan-sovereignty-benchmark`](https://github.com/hsiaoa/ai-taiwan-sovereignty-benchmark) on political sovereignty prompts [16]. That does not mean censorship issues in `DeepSeek-V4.1-Flash` do not exist; rather, model refusal and filtering vary sharply by question type (political sovereignty vs. institutional rule execution) and serving provider, making censorship a complex, deployment-dependent variable.

### 3. Model-by-Model Analysis: What Origin, Scale, and Local Fine-Tuning Actually Buy

Comparing individual models across the two benchmarks overturns several common procurement assumptions:

- **`DeepSeek-V4.1-Flash` excels on both Taiwan benchmarks despite being a Mainland Chinese model.** A common assumption is that Mainland Chinese models will confuse Taiwan's regulations with PRC rules. Instead, `DeepSeek-V4.1-Flash` tops both leaderboards (`0.934` on LaborCalc-TW and `0.855` on AdminCalc-TW) [1][3]. On LaborCalc-TW, it answers `100%` (`130/130`) of non-boundary annual-leave questions and `95%` (`76/80`) of rest-day overtime questions—including `19/20` on 9-to-12-hour rest-day shifts where Taiwan's third-tier `8/3` rate applies [1]. On AdminCalc-TW, it scores `1.00` on Minguo years and `0.71` on business ID check digits, outperforming the second-place model (`GLM-5.3-Flash` at `0.50`) by 21 percentage points [3].
- **`GPT-6 Luna` beats `Claude Haiku 5.5` by over 60 percentage points on overtime pay (`0.921` vs. `0.305`).** Both models are small commercial endpoints and perform similarly on statutory annual-leave tier matching (`0.942` vs. `0.895`) [1]. On overtime pay, `Claude Haiku 5.5` still answers `40/60` (`66.7%`) of weekday overtime items (where `GPT-6 Luna` gets `60/60`), but collapses to **`18/130` (`13.8%`)** across rest-day (`5/80`) and holiday (`13/50`) overtime—where `GPT-6 Luna` scores **`115/130` (`88.5%`)** [1]. `Beam 1.0` exhibits the same pattern: a perfect `60/60` on weekday overtime followed by `28/80` on rest days and `10/50` on holidays [1].
- **Taiwan-tuned `TAIDE 12B` and `Breeze 2 8B` fail on both statutory knowledge and arithmetic.** Neither model's local-corpus fine-tuning translates into rule execution. On LaborCalc-TW, `TAIDE 12B` scores `0.258` on annual leave and `0.011` (`2/190`) on overtime pay (`0.134` overall), while `Breeze 2 8B` scores `0.068` and `0.005` (`1/190`, `0.037` overall)—and even on the 10 zero-overtime control questions (where the correct answer is `0`), `TAIDE 12B` misses half (`5/10`) and `Breeze 2 8B` misses six (`4/10`) [1]. Even on single-step Minguo year conversion (`ROC year + 1911`), where every other cloud and open-weight model except `Flint 1.0` scores `0.97–1.00`, `TAIDE 12B` drops to `0.79` and `Breeze 2 8B` falls to `0.31` [3].
- **`GLM-5.3-Flash` trades statutory tier recall for steadier multi-step math, while self-hosted open weights hit a hard ceiling without tools.** `GLM-5.3-Flash` trails `Claude Haiku 5.5` by 22 points on annual-leave seniority tiers (`0.674` vs. `0.895`), yet beats `Haiku 5.5` by over 20 points on overtime calculation (`0.511` vs. `0.305`) and finishes second overall on AdminCalc-TW (`0.750`) [1][3]. Meanwhile, mid-sized open weights (`Qwen3.8-27B` and `gpt-oss-20b`) score `0.97` and `0.98` on single-step Minguo year conversion, but collapse on LaborCalc-TW overtime pay (`0.147` and `0.000`) [1][3].
- **One Civil Code edge case trips even the top reasoning models.** On the 10 annual-leave questions testing month-end hire dates (such as January 31) where the anniversary month has no corresponding day under Civil Code Article 121, even `DeepSeek-V4.1-Flash`, `GPT-6 Luna`, `Beam 1.0`, and `Claude Haiku 5.5` all score `5/10` (`50%`) [1][13].

### 4. Cross-Dataset Error Synthesis: Wrong Rules (38%) vs. Calculation Collapse (56%)

Table 3 and Figure 5 combine all 3,400 wrong answers across the 6,360 non-control evaluations in both benchmarks [2][4].

| Benchmark & rule family | Evaluated items | Wrong | Accuracy | Hypothesis-matched | Unexplained | Non-integer | Unscored |
|---|---|---|---|---|---|---|---|
| LaborCalc-TW: overtime pay | 2,280 | 1,547 | 32.1% | 592 | 769 | 166 | 20 |
| LaborCalc-TW: annual leave | 2,280 | 1,179 | 48.3% | 284 | 872 | 22 | 1 |
| AdminCalc-TW: business ID check digit | 900 | 558 | 38.0% | 377 | 178 | 3 | 0 |
| AdminCalc-TW: Minguo year conversion | 900 | 116 | 87.1% | 43 | 72 | 0 | 1 |
| **Total** | **6,360** | **3,400** | — | **1,296 (38%)** | **1,891 (56%)** | **191** | **22** |

![Stacked bar chart and ranked bar chart of 3,400 wrong answers across LaborCalc-TW and AdminCalc-TW: 1,296 (38%) match a known wrong rule, 1,891 (56%) are unexplained integer errors, 191 are non-integer, and 22 are unscored, led by the pre-2023 divisor-10 business ID rule (173) and missing digit cross-sums (109).](https://taiwaneval.org/images/articles/where-models-get-taiwan-rules-wrong/error-attribution-breakdown.svg)

**Figure 5. Where 3,400 wrong answers go across both benchmarks, and the top alternative rules matched by wrong integer outputs [2][4].**

[Figure source](https://github.com/taiwaneval/laborcalc-tw/blob/v2026Q4/eval/scores/2026Q4/items.jsonl)



Unexplained integer errors total `1,641` in LaborCalc-TW (`769 + 872`) and `250` in AdminCalc-TW (`178 + 72`), or `1,891` (`56%`) of all `3,400` wrong answers, while `1,296` (`38%`) match an explicit counter-hypothesis [2][4].

---

## Conclusion: Fluency Is Not Compliance

Replacing pass/fail accuracy with executable rules and counter-hypotheses changes how engineering teams should evaluate and deploy language models in Taiwan:

1. **Local language fine-tuning does not buy statutory competence—reasoning discipline does.** Training a small model on Taiwanese text helps it write natural Traditional Chinese, but `TAIDE 12B` (`0.134`) and `Breeze 2 8B` (`0.037`) fail on both statutory leave tiers and basic Minguo year arithmetic, while `DeepSeek-V4.1-Flash` (`0.934` and `0.855`) and `GPT-6 Luna` (`0.932`) lead the field through stronger instruction following and multi-step arithmetic [1][3].
2. **Delegate deterministic rules to code, not token prediction.** More than half of all errors across both benchmarks (`56%`, or `1,891` of `3,400` wrong answers) are unexplained arithmetic or branching collapses, and cost-effective models like `Claude Haiku 5.5` (`0.895` to `0.305`) and `Beam 1.0` (`0.916` to `0.516`) drop by 40 to 59 percentage points as soon as a rule moves from statutory seniority-tier matching to multi-bracket overtime calculation [1][2][4]. When a workflow is governed by a statutory formula with one exact answer, the model should extract the structured parameters from prose while a verified rule engine computes the number.
3. **Use executable counter-hypotheses to audit which world model your system is running.** While numerical agreement with a counter-hypothesis does not prove a model's internal reasoning trace [6], knowing that `38%` of all errors (`1,296`) match executable alternative rules tells you immediately whether a failure stems from stale pretraining data (such as the pre-2023 business ID revision), imported US overtime defaults, or an upstream API content filter blocking phrases like 「中華民國100年」 [2][4][5].

LaborCalc-TW and AdminCalc-TW test whether models compute the right integer *when the prompt explicitly names the Taiwan rule*. In real deployments, failure often starts one step earlier: when a user writes in Taiwan Traditional Chinese without naming a location, does the model assume Taiwan's jurisdiction by default? Alongside expanding LaborCalc-TW, we are preparing a dedicated **Default-Taiwan (預設台灣)** diagnostic to measure that implicit default directly.

In the meantime, you can inspect the full results and rule specifications on the [LaborCalc-TW](https://taiwaneval.org/en/datasets/laborcalc-tw) and [AdminCalc-TW](https://taiwaneval.org/en/datasets/admincalc-tw) pages, test scenarios in the [interactive statutory calculator](https://taiwaneval.org/en/datasets/laborcalc-tw/calculator), or audit the code and data directly in the public repositories ([laborcalc-tw on GitHub](https://github.com/taiwaneval/laborcalc-tw), [LaborCalc-TW on Hugging Face](https://huggingface.co/datasets/taiwaneval/LaborCalc-TW), [admincalc-tw on GitHub](https://github.com/taiwaneval/admincalc-tw), and [AdminCalc-TW on Hugging Face](https://huggingface.co/datasets/taiwaneval/AdminCalc-TW)).

---

## Sources

- [1] [`laborcalc-tw: index/2026Q4/index.json`](https://github.com/taiwaneval/laborcalc-tw/blob/v2026Q4/index/2026Q4/index.json)
- [2] [`laborcalc-tw: eval/scores/2026Q4/items.jsonl`](https://github.com/taiwaneval/laborcalc-tw/blob/v2026Q4/eval/scores/2026Q4/items.jsonl)
- [3] [`admincalc-tw: index/admincalc-2026Q4/site-data/results.json`](https://github.com/taiwaneval/admincalc-tw/blob/v2026Q4/index/admincalc-2026Q4/site-data/results.json)
- [4] [`admincalc-tw: eval/scores/admincalc-2026Q4/items.jsonl`](https://github.com/taiwaneval/admincalc-tw/blob/v2026Q4/eval/scores/admincalc-2026Q4/items.jsonl)
- [5] [`admincalc-tw: silver/families/ban_checksum/spec.yaml`](https://github.com/taiwaneval/admincalc-tw/blob/v2026Q4/silver/families/ban_checksum/spec.yaml)
- [6] [`admincalc-tw: README`](https://github.com/taiwaneval/admincalc-tw/blob/v2026Q4/README.md)
- [7] [`laborcalc-tw: figures/scores.svg`](https://github.com/taiwaneval/laborcalc-tw/blob/main/figures/scores.svg)
- [8] [`laborcalc-tw: gold/releases/<release>/public/*.manifest.json`](https://github.com/taiwaneval/laborcalc-tw/tree/v2026Q4/gold/releases/2026Q4/public); [`admincalc-tw: gold/releases/<release>/public/*.manifest.json`](https://github.com/taiwaneval/admincalc-tw/tree/v2026Q4/gold/releases/admincalc-2026Q4/public)
- [9] [`laborcalc-tw: figures/pipeline.svg`](https://github.com/taiwaneval/laborcalc-tw/blob/main/figures/pipeline.svg); [`laborcalc-tw: figures/scores.svg`](https://github.com/taiwaneval/laborcalc-tw/blob/main/figures/scores.svg); [`admincalc-tw: figures/pipeline.svg`](https://github.com/taiwaneval/admincalc-tw/blob/main/figures/pipeline.svg); [`admincalc-tw: figures/scores.svg`](https://github.com/taiwaneval/admincalc-tw/blob/main/figures/scores.svg)
- [10] [Taiwan Labor Standards Act (勞動基準法), Articles 24 (`4/3` and `5/3` weekday/rest-day overtime brackets)](https://law.moj.gov.tw/LawClass/LawSingle.aspx?pcode=N0030001&flno=24) and [39 (holiday double pay)](https://law.moj.gov.tw/LawClass/LawSingle.aspx?pcode=N0030001&flno=39); [Council of Labor Affairs Interpretation Tai-90-Lao-Dong-2-0026202 (240-hour monthly divisor for regular hourly pay)](https://www.mol.gov.tw/topic/3067/5991/6010/19252/); [Ministry of Labor Overtime Pay Diagram (`8/3` bracket for rest-day hours 9–12)](https://www.mol.gov.tw/1607/28162/28166/28180/81521/81529/post) and [Monthly Overtime Pay Calculator](https://calcr2.mol.gov.tw/Monthly)
- [11] [U.S. Fair Labor Standards Act of 1938, 29 U.S.C. § 207(a)(1) (`1.5x` time-and-a-half overtime rate beyond 40 hours per week)](https://www.law.cornell.edu/uscode/text/29/207) ([U.S. GovInfo statutory text](https://www.govinfo.gov/app/details/USCODE-2023-title29/USCODE-2023-title29-chap8-sec207)); [U.S. Department of Labor, Wage and Hour Division — Overtime Pay](https://www.dol.gov/agencies/whd/overtime)
- [12] [Taiwan Labor Standards Act (勞動基準法), Article 38, Paragraph 1 (statutory annual-leave seniority tiers: 3 days at 6 months, 7/10/14/15 days up to 30 days)](https://law.moj.gov.tw/LawClass/LawSingle.aspx?pcode=N0030001&flno=38); [Enforcement Rules of the Labor Standards Act (勞動基準法施行細則), Article 5, Paragraph 1 (seniority counted from hire date)](https://law.moj.gov.tw/LawClass/LawSingle.aspx?pcode=N0030002&flno=5) and [Article 24, Paragraph 2 (anniversary-year vs. calendar-year leave periods)](https://law.moj.gov.tw/LawClass/LawSingle.aspx?pcode=N0030002&flno=24)
- [13] [Taiwan Civil Code (民法), Article 121 (month-end period rule when the final month lacks a corresponding day)](https://law.moj.gov.tw/LawClass/LawSingle.aspx?pcode=B0000001&flno=121); [Ministry of Labor Statutory Annual Leave Calculator](https://calcr2.mol.gov.tw/RestDays)
- [14] [Fiscal Information Agency, Ministry of Finance — Unified Business Number (統一編號) Check-Digit Logic Revision Notice (published 2021-12-22; divisor revised from 10 to 5 with a 2023-03-31 transition deadline)](https://www.fia.gov.tw/singlehtml/3?cntId=c4d9cff38c8642ef8872774ee9987283) ([attachment PDF](https://www.fia.gov.tw/download/ff9c37611e9e46dab952676d24dc0b67))
- [15] [Department of Household Registration, Ministry of the Interior — Minguo and Gregorian Year/Age Comparison Table (年歲對照表)](https://www.ris.gov.tw/app/portal/219)
- [16] [`hsiaoa/ai-taiwan-sovereignty-benchmark` (Taiwan Sovereignty Benchmark, August 2025)](https://github.com/hsiaoa/ai-taiwan-sovereignty-benchmark)
