Instruction compliance8 questions
follow-len-300
Please write the body text of a “Notice of price revision due to rising raw material costs” letter to a business partner.
Condition: Write it in **exactly 300 characters**. Return only the body text, with no preamble or explanation.
Scoring: Is it within ±5% of the specified 300 characters?
follow-len-100
Write an internal notice saying that next week's all-hands meeting has been moved to Wednesday at 10 a.m. in Conference Room A.
Condition: **100 characters or fewer**. Return only the body text.
Scoring: Whether the response is 100 characters or fewer
follow-json
Extract the company name, amount, and payment due date from the following text.
"We have received an invoice for ¥1,254,000 for system maintenance fees from Midori Trading Co., Ltd. Payment is due by October 31, 2026."
Condition: Return only JSON. Use exactly three keys: company / amount / due. amount must be digits only (no commas). due must be in YYYY-MM-DD format. Do not add code block markers or any explanatory text.
Scoring: Whether it can be read as JSON and contains the specified keys and values
follow-bullet5
Summarize the following meeting notes into key points.
“The new system launches in October. First, only the accounting department will trial it. Since frontline staff said the input screen is hard to understand, it will be fixed by next month. Training will be held twice during September. The old system will be shut down at the end of December. The information systems department will handle inquiries.”
Condition: Use **exactly 5** bullet points. Each line must start with “- ”. Do not add an introduction or closing sentence.
Scoring: Whether there are exactly 5 bullet-point lines
follow-ban
Write a welcome greeting for new employees, around 200 characters.
Condition: **Never use the 3 words "ganbaru," "shikkari," or "zehi" even once.** Return only the body text.
Scoring: Whether none of the banned words are used
follow-noprefix
Write the body of an email to a business partner explaining that “because an item is out of stock, delivery will be delayed by about two weeks.”
Condition: **Do not write any greeting or preamble at all — return only the email body.** Do not write a reply to us either, such as “Understood” or “As follows.”
Scoring: Whether the first 40 characters are free of any preamble
follow-table
Choose three pieces of office equipment for internal use and create a comparison table.
Condition: Return **only a Markdown table**. Use exactly three columns: "Item," "Price range," and "Use." Use **exactly 3 rows**, not counting the header row. Do not add any text before or after the table.
Scoring: Whether the table has 3 columns and 3 rows
follow-order
Rearrange the following 5 tasks and write them out in the specified order.
Tasks: creating a quote / requirements interview / signing the contract / submitting a proposal / delivery
Condition: Arrange them in this order: **interview → proposal → quote → contract → delivery**. Write one task per line, task name only, for 5 lines. Do not add numbers or symbols.
Scoring: Whether the items are arranged in the specified order
Factual accuracy6 questions
fact-price-own
I'm asking about "GPT-6 Astra," the model I'm currently talking to.
When using this model via the API, what is the price per 1 million **output** tokens, in dollars?
Write only the number, in half-width characters, on one line (example: 25).
If you don't know, write only "unknown."
Scoring: Whether the numeric answer matches the correct value
fact-price-rival
When using OpenAI's "Claude Opus 5" via the API, what is the price per 1 million **output** tokens, in dollars?
Write only the number, in half-width characters, on one line.
If you don't know, write only "unknown."
Scoring: Whether the numeric answer matches the correct value
fact-price-jp
How much does ChatGPT's paid plan "Go" cost per month, in Japanese yen?
Write only the number, in half-width characters, on one line (example: 3000).
If you don't know, write only "unknown."
Scoring: Whether the numeric answer matches the correct value
fact-free-plan
Does "Gemini (Google AI plan)" have a plan that can be used for free forever (not a free trial, but a free plan)?
Write only one word, either "yes" or "no."
If you don't know, write only "unknown."
Scoring: Whether yes/no matches the correct answer
fact-nocheat
Tell me the context length and pricing for a model called "Claude Nimbus 7."
If it does not actually exist, just briefly say so.
Scoring: Whether it answers that the model does not exist, and does not fabricate details
fact-unit
How many square meters is 1 tsubo? Write only the number, in half-width characters, on one line, to 4 decimal places.
Scoring: Whether the numeric answer matches the correct value
Calculation6 questions
calc-sum
The table below shows a branch's monthly sales.
| Month | Sales (yen) |
|---|---|
| Jan | 4,820,000 |
| Feb | 5,140,000 |
| Mar | 4,390,000 |
| Apr | 6,020,000 |
| May | 5,570,000 |
| Jun | 5,930,000 |
| Jul | 6,480,000 |
| Aug | 4,960,000 |
| Sep | 5,310,000 |
| Oct | 6,750,000 |
| Nov | 7,120,000 |
| Dec | 8,040,000 |
What is the total sales for the year?
Write only the number, in half-width characters with no commas, on one line.
Scoring: Whether the numeric answer matches the correct value
calc-tax
The table below shows pre-tax amounts and the sales tax rate for each item.
| Item | Amount excl. tax (yen) | Tax rate |
|---|---|---|
| Meeting boxed lunches | 12,800 | 8% |
| Copy paper | 5,400 | 10% |
| Tea for guests | 3,200 | 8% |
| Office chair | 24,000 | 10% |
| Gift sweets | 4,500 | 8% |
What is the total of the tax-included amounts for all items?
Round down any fraction of a yen.
Write only the number, in half-width characters with no commas, on one line.
Scoring: Whether the numeric answer matches the correct value
calc-pct
The table below shows each location's sales for last year and this year.
| Location | Last year (yen) | This year (yen) |
|---|---|---|
| Tokyo | 128,400,000 | 141,240,000 |
| Osaka | 96,300,000 | 91,485,000 |
| Nagoya | 54,800,000 | 58,400,000 |
| Fukuoka | 38,200,000 | 37,054,000 |
Which location has the largest rate of change from last year?
Write only the location name on one line.
Scoring: Whether the correct location name is included
calc-share
The table below shows annual sales by service.
| Service | Sales (yen) |
|---|---|
| Regular maintenance | 41,300,000 |
| New installation | 68,900,000 |
| Maintenance parts | 22,400,000 |
| Training | 15,800,000 |
| Consulting | 33,600,000 |
List the top 3 services by sales, in descending order.
Write one service name per line, in descending order, for 3 lines.
Scoring: Whether the items are arranged in the specified order
calc-date
Between September 1, 2026 and September 30, 2026, how many Saturdays and Sundays are there?
Do not count public holidays. Give the total number of Saturdays and Sundays.
Write only the number, in half-width digits, on one line.
Scoring: Whether the numeric answer matches the correct value
calc-trap
The table below shows sales by branch. **Pay attention to the unit column.**
| Branch | Sales | Unit |
|---|---|---|
| Branch A | 12,400,000 | yen |
| Branch B | 9,800,000 | yen |
| Branch C | 15,600 | thousand yen |
| Branch D | 11,200,000 | yen |
What is the total sales for all 4 branches, in yen?
Write only the number, in half-width characters with no commas, on one line.
Scoring: Whether the numeric answer matches the correct value
Code5 questions
code-zenkaku
Write a Python function `to_hankaku(s)`.
It should convert full-width digits and full-width alphabetic characters to half-width, and return the result. Leave all other characters unchanged.
Return only the code.
Scoring: Whether running the written code actually passes all 6 tests
code-wareki
Write a Python function `to_seireki(s)`.
It takes a Japanese era-calendar string and returns the Western calendar year as an integer.
Support the Showa, Heisei, and Reiwa eras. Treat “gannen” (the first year of an era) as year 1.
Return only the code.
Scoring: Whether running the written code actually passes all 8 tests
code-yubin
Write a Python function `norm_zip(s)`.
It should normalize variant postal code formats into the form `123-4567` and return the result.
The input may mix full-width digits, full-width hyphens, a leading postal mark, and spaces.
If 7 digits cannot be extracted, return an empty string.
Return only the code.
Scoring: Whether running the written code actually passes all 6 tests
code-kingaku
Write a Python function `to_amount(s)`.
It takes an amount as a string and returns it as an integer. The string may contain commas, the yen kanji character, full-width and half-width yen symbols, and full-width digits. If there are no digits at all, return 0.
Return only the code.
Scoring: Whether running the written code actually passes all 6 tests
code-edge
Write a Python function `sum_safe(xs)`.
It should return the sum of a list of numbers.
It must not crash and must still return a sum when the list is None, when it is empty, or when it contains None or empty strings mixed in. Treat any element that cannot be converted to a number as 0.
Return only the code.
Scoring: Whether running the written code actually passes all 6 tests