Institutions & ProceduresRelease 2026Q4Released

LaborCalc-TW

Can AI get Taiwan's overtime pay and annual leave right?

Taiwan's Labor Standards Act sets exact formulas for overtime rates and for annual leave by length of service. LaborCalc-TW turns them into 400 questions and checks whether language models reach the statutory number.

public questions
400
kinds of calculation: Annual leave days, Overtime pay
2
statutory situations
20
model configurations tested
12

Questions, rule code and per-question results are public: Code on GitHubDataset on Hugging Face

What a question looks like

Each question describes an everyday situation at work. The model must work out one definite number under the statute.

From the question to the statutory answer

Both questions come from the public set. The working beside each is the rule code's own.

From the question to the statutory answer

Overtime pay · Public item 0015

黃欣怡任職的公司有48名員工,黃欣怡月薪七萬六千二百元。某個休息日(週六),黃欣怡工作了12小時,當天搭計程車回家花了350元。依勞動基準法,雇主至少應給付黃欣怡多少加班費?

In English Huang Hsin-yi works at a company with 48 employees, on a monthly salary of NT$76,200. On a rest day (a Saturday) she worked 12 hours and took a taxi home for NT$350. Under the Labor Standards Act, how much overtime pay must her employer pay at least?

  • What the calculation needs
  • Details that do not change the statutory amount
  1. Work out the hourly wage

    76,200 ÷ 240 = 317.5

  2. Apply the rate for each stretch of hours

    First 2 hours × 4/3846.67
    Hours 3 to 8 × 5/33,175
    From the 9th hour × 8/33,386.67
  3. Add up, then round up to the whole dollar

    846.67 + 3,175 + 3,386.67 = 7,408.33

Statutory answerNT$7,409

Annual leave days · Public item 0001

王淑芬於2010/10/7到職,服務年資持續未中斷。截至2022年12月25日,依勞動基準法規定其得享有之年度休假日數為幾日?

In English Wang Shu-fen started on 7 October 2010 and has worked without a break since. As of 25 December 2022, how many days of annual leave is she entitled to under the Labor Standards Act?

  • What the calculation needs
  1. Work out completed service

    2010-10-07 to 2022-10-07 is 12 completed years; the next anniversary is 2023-10-07

  2. Ten years or more: one more day for each year

    10 years16 days
    11 years17 days
    12 years18 days
    13 years19 days

Statutory answer18days

The whole question set

The 400 questions are spread over the statute's tiers and time bands, and each of the 20 statutory situations has questions of its own. 15 other cases are outside the scope and are not asked; the methodology page records each one with its reason.

How the questions are spread

Each calculation's questions follow the structure of its article.

How the questions are spread
  1. 0 days

    Under six monthscontrol · 10 questions

  2. 3 days

    Six months20 questions

  3. 7 days

    1 year20 questions

  4. 10 days

    2 years20 questions

  5. 14 days

    3–4 years20 questions

  6. 15 days

    5–9 years20 questions

  7. 16–29 days

    10–23 yearsone more day a year · 20 questions

  8. 30 days

    24 years or moreceiling · 10 questions

Under Article 38, Paragraph 1 of the Labor Standards Act. Column height is the statutory number of leave days; the count below is that tier's questions. Another 60 questions test the edges of a tier: On the anniversary (25), The day before (25), Month-end start (10). A month-end start is one whose anniversary month has no matching day.

Hour 1Hour 2Hour 3Hour 4Hour 5Hour 6Hour 7Hour 8Hour 9Hour 10Hour 11Hour 12Weekday× 4/330 questions× 5/330 questionsQuestions go up to hour 4Rest day× 4/330 questions× 5/330 questions× 8/320 questionsNational holidayUp to 8 hours: one extra day's wage20 questions× 4/315 questions× 5/315 questions

Under Articles 24 and 39 of the Labor Standards Act. Each cell gives the rate, and how many questions have that day's hours ending in it; on a weekday, hours count from the end of normal working time. Another 10 control questions have no overtime; their statutory answer is 0 and they are not scored.

Questions and statutory answers are generated by code. A named person signs off each rule before it is used, and scoring involves no human judgement.

  1. Statute
  2. Rule code
  3. Generated questions
  4. Named sign-off
  5. Answer comparison
See the cases left out, and why

Models tested and how they did

12 model configurations answered the same question set. Each one's scores, test settings and results by situation are below.

Share answered correctly, by model

Column height is the share of scored questions answered correctly: 190 scored questions per calculation, 1 run(s) each.

  1. DeepSeek-V4.1-FlashDeepSeekModel ID deepseek-v4.1-flashTotal parameters 552BQuantization apiRan on opencodeAnswer mode Highest reasoning setting
  2. GPT-6 LunaOpenAIModel ID gpt-6-lunaQuantization apiRan on opencodeAnswer mode Highest reasoning setting
  3. Beam 1.0ReflectionModel ID beam-1.0Total parameters 501BQuantization apiRan on opencode-freeAnswer mode Highest reasoning setting
  4. Claude Haiku 5.5AnthropicModel ID claude-haiku-5.5Quantization apiRan on openrouterAnswer mode Highest reasoning setting
  5. GLM-5.3-FlashZhipu AIModel ID glm-5.3-flashTotal parameters 320BQuantization apiRan on opencodeAnswer mode Highest reasoning setting
  6. Qwen3.8-FlashAlibabaModel ID qwen3.8-flashQuantization apiRan on opencodeAnswer mode Highest reasoning setting
  7. Qwen3.8-27BAlibabaModel ID qwen3.8-27bTotal parameters 27.8BQuantization optiq4-mlxRan on mac-studio-applem1ultraAnswer mode Highest reasoning setting
  8. TAIDE 12BTAIDEModel ID taide-12b-2602Total parameters 12.4BQuantization q4km-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)
  9. gpt-oss-20bOpenAIModel ID gpt-oss-20bTotal parameters 20.9BQuantization mxfp4Ran on openrouterAnswer mode Highest reasoning setting
  10. Gemma 4 12BGoogleModel ID gemma-4-12b-itTotal parameters 11.9BQuantization fp8Ran on colab-l4Answer mode Highest reasoning setting
  11. Breeze 2 8BMediaTekModel ID breeze2-8bTotal parameters 8.37BQuantization q4ks-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)
  12. Flint 1.0RecursiveModel ID flint-1.0Total parameters 39.8BQuantization apiRan on openrouter-freeAnswer mode Highest reasoning setting

Parameters and overall score

Across: a model's total parameters, on a logarithmic scale. Up: overall score. The dashed line joins the configurations that no other one beats at the same or a smaller size.

DenseMixture of expertsPareto frontier

0%25%50%75%100%5B10B20B50B100B200B500B1000BTotal parametersOverallBreeze 2 8B: 8.37B parameters, overall 3.7%Breeze 2 8BGemma 4 12B: 11.9B parameters, overall 6.1%Gemma 4 12BTAIDE 12B: 12.4B parameters, overall 13.4%TAIDE 12Bgpt-oss-20b: 20.9B parameters, overall 7.4%gpt-oss-20bQwen3.8-27B: 27.8B parameters, overall 16.6%Qwen3.8-27BFlint 1.0: 39.8B parameters, overall 3.7%Flint 1.0GLM-5.3-Flash: 320B parameters, overall 59.2%GLM-5.3-FlashBeam 1.0: 501B parameters, overall 71.6%Beam 1.0DeepSeek-V4.1-Flash: 552B parameters, overall 93.4%DeepSeek-V4.1-Flash

Not plotted: GPT-6 Luna, Claude Haiku 5.5, Qwen3.8-Flash, which have no published parameter count.

Parameter counts are from each model's public model card: Breeze 2 8B, Gemma 4 12B, TAIDE 12B, gpt-oss-20b, Qwen3.8-27B, Flint 1.0, GLM-5.3-Flash, Beam 1.0, DeepSeek-V4.1-Flash.

A report card for each model

See how each configuration did in every statutory situation. Choosing a column above opens its card.

CorrectWrongOne square is one question

DeepSeek-V4.1-Flash

DeepSeekModel ID deepseek-v4.1-flashTotal parameters 552BQuantization apiRan on opencodeAnswer mode Highest reasoning setting

93.4%Overall: the mean of the two shares

Annual leave days
95.8%
Overtime pay
91.1%

Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.

Annual leave days182 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay173 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

GPT-6 Luna

OpenAIModel ID gpt-6-lunaQuantization apiRan on opencodeAnswer mode Highest reasoning setting

93.2%Overall: the mean of the two shares

Annual leave days
94.2%
Overtime pay
92.1%

Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.

Annual leave days179 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay175 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Beam 1.0

ReflectionModel ID beam-1.0Total parameters 501BQuantization apiRan on opencode-freeAnswer mode Highest reasoning setting

71.6%Overall: the mean of the two shares

Annual leave days
91.6%
Overtime pay
51.6%

Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.

Annual leave days174 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay98 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Claude Haiku 5.5

AnthropicModel ID claude-haiku-5.5Quantization apiRan on openrouterAnswer mode Highest reasoning setting

60.0%Overall: the mean of the two shares

Annual leave days
89.5%
Overtime pay
30.5%

Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.

Annual leave days170 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay58 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

GLM-5.3-Flash

Zhipu AIModel ID glm-5.3-flashTotal parameters 320BQuantization apiRan on opencodeAnswer mode Highest reasoning setting

59.2%Overall: the mean of the two shares

Annual leave days
67.4%
Overtime pay
51.1%

Control questions (statutory answer 0): Annual leave days 9/10, Overtime pay 10/10; not counted in the score.

Annual leave days128 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay97 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Qwen3.8-Flash

AlibabaModel ID qwen3.8-flashQuantization apiRan on opencodeAnswer mode Highest reasoning setting

54.5%Overall: the mean of the two shares

Annual leave days
61.6%
Overtime pay
47.4%

Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.

Annual leave days117 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay90 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Qwen3.8-27B

AlibabaModel ID qwen3.8-27bTotal parameters 27.8BQuantization optiq4-mlxRan on mac-studio-applem1ultraAnswer mode Highest reasoning setting

16.6%Overall: the mean of the two shares

Annual leave days
18.4%
Overtime pay
14.7%

Control questions (statutory answer 0): Annual leave days 5/10, Overtime pay 10/10; not counted in the score.

Annual leave days35 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay28 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

TAIDE 12B

TAIDEModel ID taide-12b-2602Total parameters 12.4BQuantization q4km-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)

13.4%Overall: the mean of the two shares

Annual leave days
25.8%
Overtime pay
1.1%

Control questions (statutory answer 0): Annual leave days 4/10, Overtime pay 5/10; not counted in the score.

Annual leave days49 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay2 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

gpt-oss-20b

OpenAIModel ID gpt-oss-20bTotal parameters 20.9BQuantization mxfp4Ran on openrouterAnswer mode Highest reasoning setting

7.4%Overall: the mean of the two shares

Annual leave days
14.7%
Overtime pay
0.0%

Control questions (statutory answer 0): Annual leave days 6/10, Overtime pay 10/10; not counted in the score.

Annual leave days28 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay0 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Gemma 4 12B

GoogleModel ID gemma-4-12b-itTotal parameters 11.9BQuantization fp8Ran on colab-l4Answer mode Highest reasoning setting

6.1%Overall: the mean of the two shares

Annual leave days
8.4%
Overtime pay
3.7%

Control questions (statutory answer 0): Annual leave days 3/10, Overtime pay 10/10; not counted in the score.

Annual leave days16 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay7 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Breeze 2 8B

MediaTekModel ID breeze2-8bTotal parameters 8.37BQuantization q4ks-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)

3.7%Overall: the mean of the two shares

Annual leave days
6.8%
Overtime pay
0.5%

Control questions (statutory answer 0): Annual leave days 1/10, Overtime pay 4/10; not counted in the score.

Annual leave days13 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay1 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12

Flint 1.0

RecursiveModel ID flint-1.0Total parameters 39.8BQuantization apiRan on openrouter-freeAnswer mode Highest reasoning setting

3.7%Overall: the mean of the two shares

Annual leave days
5.3%
Overtime pay
2.1%

Control questions (statutory answer 0): Annual leave days 9/10, Overtime pay 10/10; not counted in the score.

Annual leave days10 of 190
  • Six months
  • 1 year
  • 2 years
  • 3–4 years
  • 5–9 years
  • 10–23 years
  • 24 years or more
  • On the anniversary
  • The day before
  • Month-end start
Overtime pay4 of 190
Weekday
  • Hours 1–2
  • Hours 3–4
Rest day
  • Hours 1–2
  • Hours 3–8
  • Hours 9–12
National holiday
  • Hours 1–8
  • Hours 9–10
  • Hours 11–12
1 / 12

Articles

Articles that draw on LaborCalc-TW's questions or results.

  1. Where Models Get Taiwan's Labor and Administrative Rules Wrong

    Pass/fail benchmarks show that models miss local statutory math, but not whether they are running an outdated Taiwan regulation, importing a US formula, or failing at multi-step arithmetic. Encoding both rules and counter-hypotheses as code separates those failures.

All articles

Where it sits in TaiwanEval

TaiwanEval sorts the ways AI goes wrong in Taiwan into three kinds. LaborCalc-TW is the first public dataset under “Institutions & Procedures”.

Institutions & Procedures

Faced with a Taiwan regulation or application procedure, can the AI accurately cite the responsible Taiwan agency's rules, or does it force another country's system onto Taiwan?

AdminCalc-TWadmincalc-2026Q4 · Annual leave days, Overtime pay · 200 questions

Language & Terminology

Does the AI truly understand everyday Taiwan usage and specialist terms across fields, or does it stop at converting Simplified Chinese to Traditional Chinese word for word?

No dataset yet

Cross-context Confusion

In everyday Taiwan situations, does the AI treat apps, payment methods or common knowledge from China or elsewhere as if they applied equally in Taiwan?

No dataset yet

Planned next

These question families have been chosen but not started. Neither their order nor their dates are set.

  • Traditional and Simplified Chinese conversion
  • Questions on statute text

Content updated