Checkable evidence of how AI really performs in Taiwan.

TaiwanEval tests where AI goes wrong on Taiwan's institutions, language and everyday context, and publishes what was tested, how, and where the limits are.

Why TaiwanEval is needed

A high score and fluent Chinese do not mean a model gets Taiwan right.

  • It scores well on general benchmarksThis does not tell you whether it gets a Taiwan rule wrong.
  • It writes fluent Traditional ChineseThis does not tell you whether it is describing Taiwan's system.

TaiwanEval evaluates where AI goes wrong when it is used in Taiwan, and records the systems, versions, settings and limits behind each result, so results can be reproduced and challenged.

A LaborCalc-TW question

An employee paid NT$36,000 a month works 8 hours on a rest day. Under the Labor Standards Act, how much overtime pay is owed?

By the statuteNT$1,900

Regular hourly wage = 36,000 ÷ 240 = NT$150. First 2 hours × 4/3: 400; Hours 3 to 8 × 5/3: 1,500.

Labor Standards Act 第 24 條第 2 項

Each model's answer will be published with the first release of results.

Three kinds of Taiwan-specific failure

When AI fails in Taiwan, the cause is often more than language.

Institutions & Procedures

Faced with a Taiwan regulation or application procedure, can the AI accurately cite the responsible Taiwan agency's rules, or does it force another country's system onto Taiwan?

Example question
Where do I apply for parental-leave allowance?
Example of a wrong answer
You can apply through the Working Family and Student Financial Assistance Agency, once you confirm the income and working-hour thresholds.

That is a Hong Kong agency, not a Taiwan one. The question was about Taiwan, and the AI answered with a real Hong Kong institution.

DatasetAdminCalc-TWLaborCalc-TW

Language & Terminology

Does the AI truly understand everyday Taiwan usage and specialist terms across fields, or does it stop at converting Simplified Chinese to Traditional Chinese word for word?

Example question
How do I link my EasyCard invoice carrier to my mobile barcode?
Example of a wrong answer
Add the EasyCard to a mobile wallet to finish linking the carrier.

"Linking a carrier" means connecting an electronic-invoice carrier, but the AI read it as a payment-card wallet setup. Fluent wording, wrong meaning.

DatasetNo dataset yet

Cross-context Confusion

In everyday Taiwan situations, does the AI treat apps, payment methods or common knowledge from China or elsewhere as if they applied equally in Taiwan?

Example question
What are the common payment methods at a bubble-tea shop?
Example of a wrong answer
Common payment methods include cash, Alipay, or WeChat Pay.

Taiwan bubble-tea shops commonly take cash, LINE Pay, JKoPay or EasyWallet. The AI treated payment habits common in China as if they were universal.

DatasetNo dataset yet

  • The part of the answer that is wrong

Datasets

Newest datasetRelease admincalc-2026Q4Released

AdminCalc-TW: can AI get Taiwan's business ID check digits and Minguo calendar years right?

AdminCalc-TW draws questions from the Ministry of Finance's business ID check-digit specification and the Ministry of the Interior's Minguo-to-AD year table. Each question has one exact integer answer verified by rule code.

public questions
200
reference cases
40
official sources
2

Business ID check digits, Minguo calendar year conversion

Articles

  1. Where Models Get Taiwan's Labor and Administrative Rules Wrong

    Pass/fail benchmarks show that models miss local statutory math, but not whether they are running an outdated Taiwan regulation, importing a US formula, or failing at multi-step arithmetic. Encoding both rules and counter-hypotheses as code separates those failures.

All articles