Can AI get Taiwan's overtime pay and annual leave right?
Taiwan's Labor Standards Act sets exact formulas for overtime rates and for annual leave by length of service. LaborCalc-TW turns them into 400 questions and checks whether language models reach the statutory number.
public questions
400
kinds of calculation: Annual leave days, Overtime pay
In English Huang Hsin-yi works at a company with 48 employees, on a monthly salary of NT$76,200. On a rest day (a Saturday) she worked 12 hours and took a taxi home for NT$350. Under the Labor Standards Act, how much overtime pay must her employer pay at least?
In English Wang Shu-fen started on 7 October 2010 and has worked without a break since. As of 25 December 2022, how many days of annual leave is she entitled to under the Labor Standards Act?
AaWhat the calculation needs
Work out completed service
2010-10-07 to 2022-10-07 is 12 completed years; the next anniversary is 2023-10-07
Ten years or more: one more day for each year
10 years16 days
11 years17 days
12 years18 days
13 years19 days
Statutory answer18days
The whole question set
The 400 questions are spread over the statute's tiers and time bands, and each of the 20 statutory situations has questions of its own. 15 other cases are outside the scope and are not asked; the methodology page records each one with its reason.
How the questions are spread
Each calculation's questions follow the structure of its article.
0 days
Under six monthscontrol · 10 questions
3 days
Six months20 questions
7 days
1 year20 questions
10 days
2 years20 questions
14 days
3–4 years20 questions
15 days
5–9 years20 questions
16–29 days
10–23 yearsone more day a year · 20 questions
30 days
24 years or moreceiling · 10 questions
Under Article 38, Paragraph 1 of the Labor Standards Act. Column height is the statutory number of leave days; the count below is that tier's questions. Another 60 questions test the edges of a tier: On the anniversary (25), The day before (25), Month-end start (10). A month-end start is one whose anniversary month has no matching day.
Hour 1Hour 2Hour 3Hour 4Hour 5Hour 6Hour 7Hour 8Hour 9Hour 10Hour 11Hour 12Weekday× 4/330 questions× 5/330 questionsQuestions go up to hour 4Rest day× 4/330 questions× 5/330 questions× 8/320 questionsNational holidayUp to 8 hours: one extra day's wage20 questions× 4/315 questions× 5/315 questions
Under Articles 24 and 39 of the Labor Standards Act. Each cell gives the rate, and how many questions have that day's hours ending in it; on a weekday, hours count from the end of normal working time. Another 10 control questions have no overtime; their statutory answer is 0 and they are not scored.
Questions and statutory answers are generated by code. A named person signs off each rule before it is used, and scoring involves no human judgement.
12 model configurations answered the same question set. Each one's scores, test settings and results by situation are below.
Share answered correctly, by model
Column height is the share of scored questions answered correctly: 190 scored questions per calculation, 1 run(s) each.
0%25%50%75%100%
Setup
DeepSeek-V4.1-FlashDeepSeekModel ID deepseek-v4.1-flashTotal parameters 552BQuantization apiRan on opencodeAnswer mode Highest reasoning setting
GPT-6 LunaOpenAIModel ID gpt-6-lunaQuantization apiRan on opencodeAnswer mode Highest reasoning setting
Beam 1.0ReflectionModel ID beam-1.0Total parameters 501BQuantization apiRan on opencode-freeAnswer mode Highest reasoning setting
Claude Haiku 5.5AnthropicModel ID claude-haiku-5.5Quantization apiRan on openrouterAnswer mode Highest reasoning setting
GLM-5.3-FlashZhipu AIModel ID glm-5.3-flashTotal parameters 320BQuantization apiRan on opencodeAnswer mode Highest reasoning setting
Qwen3.8-FlashAlibabaModel ID qwen3.8-flashQuantization apiRan on opencodeAnswer mode Highest reasoning setting
Qwen3.8-27BAlibabaModel ID qwen3.8-27bTotal parameters 27.8BQuantization optiq4-mlxRan on mac-studio-applem1ultraAnswer mode Highest reasoning setting
TAIDE 12BTAIDEModel ID taide-12b-2602Total parameters 12.4BQuantization q4km-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)
gpt-oss-20bOpenAIModel ID gpt-oss-20bTotal parameters 20.9BQuantization mxfp4Ran on openrouterAnswer mode Highest reasoning setting
Gemma 4 12BGoogleModel ID gemma-4-12b-itTotal parameters 11.9BQuantization fp8Ran on colab-l4Answer mode Highest reasoning setting
Breeze 2 8BMediaTekModel ID breeze2-8bTotal parameters 8.37BQuantization q4ks-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)
Flint 1.0RecursiveModel ID flint-1.0Total parameters 39.8BQuantization apiRan on openrouter-freeAnswer mode Highest reasoning setting
Parameters and overall score
Across: a model's total parameters, on a logarithmic scale. Up: overall score. The dashed line joins the configurations that no other one beats at the same or a smaller size.
DenseMixture of expertsPareto frontier
Not plotted: GPT-6 Luna, Claude Haiku 5.5, Qwen3.8-Flash, which have no published parameter count.
See how each configuration did in every statutory situation. Choosing a column above opens its card.
CorrectWrongOne square is one question
DeepSeek-V4.1-Flash
DeepSeekModel ID deepseek-v4.1-flashTotal parameters 552BQuantization apiRan on opencodeAnswer mode Highest reasoning setting
93.4%Overall: the mean of the two shares
Annual leave days
95.8%
Overtime pay
91.1%
Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.
Annual leave days182 of 190
Six months20/20
1 year20/20
2 years20/20
3–4 years20/20
5–9 years20/20
10–23 years20/20
24 years or more10/10
On the anniversary25/25
The day before22/25
Month-end start5/10
Overtime pay173 of 190
Weekday
Hours 1–230/30
Hours 3–430/30
Rest day
Hours 1–228/30
Hours 3–829/30
Hours 9–1219/20
National holiday
Hours 1–87/20
Hours 9–1015/15
Hours 11–1215/15
GPT-6 Luna
OpenAIModel ID gpt-6-lunaQuantization apiRan on opencodeAnswer mode Highest reasoning setting
93.2%Overall: the mean of the two shares
Annual leave days
94.2%
Overtime pay
92.1%
Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.
Annual leave days179 of 190
Six months20/20
1 year19/20
2 years20/20
3–4 years20/20
5–9 years20/20
10–23 years19/20
24 years or more9/10
On the anniversary24/25
The day before23/25
Month-end start5/10
Overtime pay175 of 190
Weekday
Hours 1–230/30
Hours 3–430/30
Rest day
Hours 1–228/30
Hours 3–829/30
Hours 9–1210/20
National holiday
Hours 1–818/20
Hours 9–1015/15
Hours 11–1215/15
Beam 1.0
ReflectionModel ID beam-1.0Total parameters 501BQuantization apiRan on opencode-freeAnswer mode Highest reasoning setting
71.6%Overall: the mean of the two shares
Annual leave days
91.6%
Overtime pay
51.6%
Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.
Annual leave days174 of 190
Six months20/20
1 year20/20
2 years20/20
3–4 years20/20
5–9 years19/20
10–23 years16/20
24 years or more10/10
On the anniversary23/25
The day before21/25
Month-end start5/10
Overtime pay98 of 190
Weekday
Hours 1–230/30
Hours 3–430/30
Rest day
Hours 1–24/30
Hours 3–818/30
Hours 9–126/20
National holiday
Hours 1–83/20
Hours 9–104/15
Hours 11–123/15
Claude Haiku 5.5
AnthropicModel ID claude-haiku-5.5Quantization apiRan on openrouterAnswer mode Highest reasoning setting
60.0%Overall: the mean of the two shares
Annual leave days
89.5%
Overtime pay
30.5%
Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.
Annual leave days170 of 190
Six months20/20
1 year20/20
2 years19/20
3–4 years20/20
5–9 years19/20
10–23 years13/20
24 years or more10/10
On the anniversary23/25
The day before21/25
Month-end start5/10
Overtime pay58 of 190
Weekday
Hours 1–217/30
Hours 3–423/30
Rest day
Hours 1–24/30
Hours 3–81/30
Hours 9–120/20
National holiday
Hours 1–86/20
Hours 9–103/15
Hours 11–124/15
GLM-5.3-Flash
Zhipu AIModel ID glm-5.3-flashTotal parameters 320BQuantization apiRan on opencodeAnswer mode Highest reasoning setting
59.2%Overall: the mean of the two shares
Annual leave days
67.4%
Overtime pay
51.1%
Control questions (statutory answer 0): Annual leave days 9/10, Overtime pay 10/10; not counted in the score.
Annual leave days128 of 190
Six months20/20
1 year16/20
2 years17/20
3–4 years13/20
5–9 years15/20
10–23 years9/20
24 years or more6/10
On the anniversary16/25
The day before10/25
Month-end start6/10
Overtime pay97 of 190
Weekday
Hours 1–224/30
Hours 3–429/30
Rest day
Hours 1–210/30
Hours 3–89/30
Hours 9–124/20
National holiday
Hours 1–812/20
Hours 9–105/15
Hours 11–124/15
Qwen3.8-Flash
AlibabaModel ID qwen3.8-flashQuantization apiRan on opencodeAnswer mode Highest reasoning setting
54.5%Overall: the mean of the two shares
Annual leave days
61.6%
Overtime pay
47.4%
Control questions (statutory answer 0): Annual leave days 10/10, Overtime pay 10/10; not counted in the score.
Annual leave days117 of 190
Six months10/20
1 year16/20
2 years15/20
3–4 years18/20
5–9 years17/20
10–23 years2/20
24 years or more9/10
On the anniversary12/25
The day before14/25
Month-end start4/10
Overtime pay90 of 190
Weekday
Hours 1–224/30
Hours 3–425/30
Rest day
Hours 1–211/30
Hours 3–814/30
Hours 9–123/20
National holiday
Hours 1–84/20
Hours 9–105/15
Hours 11–124/15
Qwen3.8-27B
AlibabaModel ID qwen3.8-27bTotal parameters 27.8BQuantization optiq4-mlxRan on mac-studio-applem1ultraAnswer mode Highest reasoning setting
16.6%Overall: the mean of the two shares
Annual leave days
18.4%
Overtime pay
14.7%
Control questions (statutory answer 0): Annual leave days 5/10, Overtime pay 10/10; not counted in the score.
Annual leave days35 of 190
Six months2/20
1 year9/20
2 years1/20
3–4 years2/20
5–9 years3/20
10–23 years2/20
24 years or more4/10
On the anniversary5/25
The day before5/25
Month-end start2/10
Overtime pay28 of 190
Weekday
Hours 1–211/30
Hours 3–47/30
Rest day
Hours 1–25/30
Hours 3–83/30
Hours 9–120/20
National holiday
Hours 1–81/20
Hours 9–101/15
Hours 11–120/15
TAIDE 12B
TAIDEModel ID taide-12b-2602Total parameters 12.4BQuantization q4km-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)
13.4%Overall: the mean of the two shares
Annual leave days
25.8%
Overtime pay
1.1%
Control questions (statutory answer 0): Annual leave days 4/10, Overtime pay 5/10; not counted in the score.
Annual leave days49 of 190
Six months3/20
1 year11/20
2 years5/20
3–4 years9/20
5–9 years6/20
10–23 years1/20
24 years or more4/10
On the anniversary6/25
The day before3/25
Month-end start1/10
Overtime pay2 of 190
Weekday
Hours 1–21/30
Hours 3–40/30
Rest day
Hours 1–20/30
Hours 3–80/30
Hours 9–120/20
National holiday
Hours 1–81/20
Hours 9–100/15
Hours 11–120/15
gpt-oss-20b
OpenAIModel ID gpt-oss-20bTotal parameters 20.9BQuantization mxfp4Ran on openrouterAnswer mode Highest reasoning setting
7.4%Overall: the mean of the two shares
Annual leave days
14.7%
Overtime pay
0.0%
Control questions (statutory answer 0): Annual leave days 6/10, Overtime pay 10/10; not counted in the score.
Annual leave days28 of 190
Six months0/20
1 year4/20
2 years4/20
3–4 years2/20
5–9 years5/20
10–23 years0/20
24 years or more0/10
On the anniversary2/25
The day before8/25
Month-end start3/10
Overtime pay0 of 190
Weekday
Hours 1–20/30
Hours 3–40/30
Rest day
Hours 1–20/30
Hours 3–80/30
Hours 9–120/20
National holiday
Hours 1–80/20
Hours 9–100/15
Hours 11–120/15
Gemma 4 12B
GoogleModel ID gemma-4-12b-itTotal parameters 11.9BQuantization fp8Ran on colab-l4Answer mode Highest reasoning setting
6.1%Overall: the mean of the two shares
Annual leave days
8.4%
Overtime pay
3.7%
Control questions (statutory answer 0): Annual leave days 3/10, Overtime pay 10/10; not counted in the score.
Annual leave days16 of 190
Six months1/20
1 year7/20
2 years0/20
3–4 years1/20
5–9 years0/20
10–23 years1/20
24 years or more1/10
On the anniversary2/25
The day before1/25
Month-end start2/10
Overtime pay7 of 190
Weekday
Hours 1–22/30
Hours 3–40/30
Rest day
Hours 1–22/30
Hours 3–81/30
Hours 9–120/20
National holiday
Hours 1–82/20
Hours 9–100/15
Hours 11–120/15
Breeze 2 8B
MediaTekModel ID breeze2-8bTotal parameters 8.37BQuantization q4ks-ggufRan on mac-studio-applem1ultraAnswer mode Step-by-step answer (the model has no reasoning setting)
3.7%Overall: the mean of the two shares
Annual leave days
6.8%
Overtime pay
0.5%
Control questions (statutory answer 0): Annual leave days 1/10, Overtime pay 4/10; not counted in the score.
Annual leave days13 of 190
Six months0/20
1 year4/20
2 years2/20
3–4 years0/20
5–9 years0/20
10–23 years0/20
24 years or more0/10
On the anniversary2/25
The day before2/25
Month-end start3/10
Overtime pay1 of 190
Weekday
Hours 1–20/30
Hours 3–40/30
Rest day
Hours 1–21/30
Hours 3–80/30
Hours 9–120/20
National holiday
Hours 1–80/20
Hours 9–100/15
Hours 11–120/15
Flint 1.0
RecursiveModel ID flint-1.0Total parameters 39.8BQuantization apiRan on openrouter-freeAnswer mode Highest reasoning setting
3.7%Overall: the mean of the two shares
Annual leave days
5.3%
Overtime pay
2.1%
Control questions (statutory answer 0): Annual leave days 9/10, Overtime pay 10/10; not counted in the score.
Annual leave days10 of 190
Six months1/20
1 year0/20
2 years0/20
3–4 years2/20
5–9 years0/20
10–23 years1/20
24 years or more1/10
On the anniversary0/25
The day before3/25
Month-end start2/10
Overtime pay4 of 190
Weekday
Hours 1–20/30
Hours 3–43/30
Rest day
Hours 1–20/30
Hours 3–81/30
Hours 9–120/20
National holiday
Hours 1–80/20
Hours 9–100/15
Hours 11–120/15
1 / 12
Articles
Articles that draw on LaborCalc-TW's questions or results.
Pass/fail benchmarks show that models miss local statutory math, but not whether they are running an outdated Taiwan regulation, importing a US formula, or failing at multi-step arithmetic. Encoding both rules and counter-hypotheses as code separates those failures.
TaiwanEval sorts the ways AI goes wrong in Taiwan into three kinds. LaborCalc-TW is the first public dataset under “Institutions & Procedures”.
Institutions & Procedures
Faced with a Taiwan regulation or application procedure, can the AI accurately cite the responsible Taiwan agency's rules, or does it force another country's system onto Taiwan?
Does the AI truly understand everyday Taiwan usage and specialist terms across fields, or does it stop at converting Simplified Chinese to Traditional Chinese word for word?
No dataset yet
Cross-context Confusion
In everyday Taiwan situations, does the AI treat apps, payment methods or common knowledge from China or elsewhere as if they applied equally in Taiwan?
No dataset yet
Planned next
These question families have been chosen but not started. Neither their order nor their dates are set.