We send exactly the same task prompt and the same materials to the 4 AIs on the same day. We open a new chat every time so earlier answers don't carry over. The sending order is rotated in each run. We also record the seconds from sending until the answer stops.

Why do you send the same task 3 times?

Because AI answers vary even for the same question. From a single try, you can't tell whether a result reflects real ability or chance. We send each task 3 times, and the share of tasks whose conclusion matched is reported as the consistency rate.

Who does the scoring?

Scoring is done only by a program. We do not let AI score the answers. The correct answers are set when the tasks are created, and whether an answer is right is decided by whether numbers match, whether required words are present, whether the format fits, whether URLs actually open and the result of running formulas. Matters of taste are not scored.

Is the data in the tasks from real companies?

No. We create new data for a fictional company every time. We do not use real companies, real people or text from other companies. We share the materials we used as they are, so you can repeat the same test.

How do you count cases where the AI couldn't answer?

Refusals and timeouts stay in the base and are counted as wrong. If we hit the plan's usage limit or a human verification screen appears, we remove that trial from the base and treat it as not measured. Operating errors on AInformation's side are also removed from the base, and we do not report them as the AI's fault.

Do you use the free or paid versions?

We use each AI's paid plan through the browser interface. We don't use the API. For each run, we record the plan name, the model name shown on screen and the time we checked it.

Does this ranking show which AI is the smartest?

No. We measure only 5 things on a fixed set of work tasks: accuracy, time to answer, consistency rate, real-source rate and instruction compliance. We do not measure writing style, readability or behavior in long conversations. If the tasks change, the ranking changes too.

Standardize inconsistent spellings of client names (4 AIs, 6 tasks, 3 runs each)

Name: Standardize inconsistent spellings of client names (4 AIs, 6 tasks, 3 runs each)
Creator: AInformation
Published: 2026-10-19
License: https://creativecommons.org/licenses/by/4.0/

RESULT

What this run found

The line below is generated directly from the scoring results. The page adds no words of its own.

Data cleanup and name matching: 6 tasks, 3 trials each Six tasks were given three times each for a total of 18 trials. ChatGPT had the highest accuracy rate, passing 15 of 18 trials (83.3%); Gemini was second at 77.8% 72 valid trials. Results from a single round of testing. Rankings may change on a different day or with different tasks. See scores by AI

SCORES

Scores by AI

Sorted by accuracy, highest first. The base is the number of trials in which that metric could be measured. “—” means not measured; it does not mean 0%.

Standardize inconsistent spellings of client names (4 AIs, 6 tasks, 3 runs each): scores by AI
AI	AccuracyCorrect ÷ valid trials	ConsistencySame conclusion in all 3 trials	Real-source rateShare of URLs that opened	Instruction complianceMet length and format rules	Time to answerMedian / range
1ChatGPT	83.3% 15 of 18 trials	100% out of 6 tasks	—	83.3%	38.7 s 33–54.6 seconds
2Gemini	77.8% 14 of 18 trials	88.9% out of 6 tasks	—	83.3%	63.5 s 19.1–134.9 seconds
3Claude	66.7% 12 of 18 trials	100% out of 6 tasks	—	33.3%	20.2 s 16.9–32.2 seconds
4Copilot	66.7% 12 of 18 trials	100% out of 6 tasks	—	85.7%	17.1 s 13.9–17.4 seconds

The real-source rate is measured only in runs that included a task asking for URLs along with the answer. For ChatGPT, Gemini, Claude and Copilot, “—” means this run did not include that kind of task. It does not mean 0%.

Trials with no answer
Copilot … Timed out: 4 times
Refusals and timeouts stay in the base and count as incorrect. Plan limits, human verification screens and operating errors on AInformation's side are excluded from the base.

TASKS

Results by task

We sent each task to each of the 4 AIs 3 times. ○ means correct and × means incorrect. The numbers below are the seconds each trial took. Open each section to read the full task prompt, the answer key set in advance and why each verdict was given.

Easy

The client names in the following transaction list are written inconsistently. How many companies are there once you gro… Scoring type: contains_all — Whether all the required words and phrases are included

ChatGPT
- ChatGPT, trial 1: Correct 33 s
- ChatGPT, trial 2: Correct 36.3 s
- ChatGPT, trial 3: Correct 35.6 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 127.5 s
- Gemini, trial 2: Correct 61.6 s
- Gemini, trial 3: Correct 52.4 s
3 of 3 trials correct
Claude
- Claude, trial 1: Correct 17.5 s
- Claude, trial 2: Correct 17.1 s
- Claude, trial 3: Correct 17 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 17.4 s
- Copilot, trial 2: Correct 17.1 s
- Copilot, trial 3: Correct 13.9 s
3 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (413 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

The client names in the following transaction list are written inconsistently. How many companies are there once you group the identical ones together? Rules - Give the answer in exactly one line, in the form "N companies" - Do not write the company names or any explanation ---- Transaction list start ---- ID,Client name,Rep,Amount A001,Sample Transport Co., Ltd.,Takahashi,730000 A011,(Ltd.) Sample Foods,Takahashi,563000 A007,SAMPLE INDUSTRY (LTD.),Ito,761000 A010,Sample Foods Co., Ltd.,Tanaka,231000 A009,SAMPLE FOODS (Ltd.),Tanaka,799000 A003,Sample Trading (Ltd.),Takahashi,250000 A002,(Ltd.) Sample Transport,Ito,483000 A004,SAMPLE TRADING (LTD.),Kato,502000 A008,Sample Industry (Ltd.),Tanaka,474000 A006,(Ltd.) Sample Trading,Kobayashi,641000 A005,Sample Trading Co., Ltd. ,Nakamura,314000 ---- Transaction list end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Auto-graded, contains_all) Number of companies after grouping = 4, or 4 companies, or 4 entries

Scoring detailsWe show, trial by trial, why each verdict was given (12 of 12 trials correct)

ChatGPT, trial 1: Correct: The one item was present (33 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: The one item was present (36.3 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: The one item was present (35.6 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: The one item was present (127.5 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: The one item was present (61.6 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: The one item was present (52.4 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: The one item was present (17.5 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: The one item was present (17.1 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: The one item was present (17 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: The one item was present (17.4 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Correct: The one item was present (17.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Correct: The one item was present (13.9 seconds) · Full text of this answer (Japanese)

Easy

Group the client names in the following transaction list and list every company name. Rules - One line per company. Star… Scoring type: set_match — How many of the required items were missed. Counts both omissions and made-up additions

ChatGPT
- ChatGPT, trial 1: Correct 36.3 s
- ChatGPT, trial 2: Correct 36.4 s
- ChatGPT, trial 3: Correct 38.7 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 50.1 s
- Gemini, trial 2: Correct 26.1 s
- Gemini, trial 3: Correct 89.1 s
3 of 3 trials correct
Claude
- Claude, trial 1: Correct 20.2 s
- Claude, trial 2: Correct 23.9 s
- Claude, trial 3: Correct 32.2 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 17.2 s
- Copilot, trial 2: Correct 17.1 s
- Copilot, trial 3: Correct 17 s
3 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (417 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

Group the client names in the following transaction list and list every company name. Rules - One line per company. Start each line with "- " - Group identical companies into a single line. Do not write amounts or counts ---- Transaction list start ---- ID,Client name,Rep,Amount A001,Sample Transport Co., Ltd.,Takahashi,730000 A011,(Ltd.) Sample Foods,Takahashi,563000 A007,SAMPLE INDUSTRY (LTD.),Ito,761000 A010,Sample Foods Co., Ltd.,Tanaka,231000 A009,SAMPLE FOODS (Ltd.),Tanaka,799000 A003,Sample Trading (Ltd.),Takahashi,250000 A002,(Ltd.) Sample Transport,Ito,483000 A004,SAMPLE TRADING (LTD.),Kato,502000 A008,Sample Industry (Ltd.),Tanaka,474000 A006,(Ltd.) Sample Trading,Kobayashi,641000 A005,Sample Trading Co., Ltd. ,Nakamura,314000 ---- Transaction list end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Auto-graded, set_match) Sample Trading [Sample Trading] / Sample Foods [Sample Foods] / Sample Industry [Sample Industry] / Sample Transport [Sample Transport] - up to 0 extra entries allowed

Scoring detailsWe show, trial by trial, why each verdict was given (12 of 12 trials correct)

ChatGPT, trial 1: Correct: All 4 items matched, 0 fabricated items (36.3 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All 4 items matched, 0 fabricated items (36.4 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All 4 items matched, 0 fabricated items (38.7 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: All 4 items matched, 0 fabricated items (50.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: All 4 items matched, 0 fabricated items (26.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: All 4 items matched, 0 fabricated items (89.1 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: All 4 items matched, 0 fabricated items (20.2 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: All 4 items matched, 0 fabricated items (23.9 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: All 4 items matched, 0 fabricated items (32.2 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All 4 items matched, 0 fabricated items (17.2 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Correct: All 4 items matched, 0 fabricated items (17.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Correct: All 4 items matched, 0 fabricated items (17 seconds) · Full text of this answer (Japanese)

Medium

For the following transaction list, give the number of entries per company after grouping identical companies together. … Scoring type: exact_number — Whether the number matches the correct answer. The rounding tolerance is set in advance

ChatGPT
- ChatGPT, trial 1: Correct 39.7 s
- ChatGPT, trial 2: Correct 38.8 s
- ChatGPT, trial 3: Correct 35.8 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 19.3 s
- Gemini, trial 2: Correct 113.4 s
- Gemini, trial 3: Incorrect 19.1 s
2 of 3 trials correct
Claude
- Claude, trial 1: Correct 20.4 s
- Claude, trial 2: Correct 16.9 s
- Claude, trial 3: Correct 19.9 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 17.4 s
- Copilot, trial 2: Incorrect Timed out
- Copilot, trial 3: Incorrect Timed out
1 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (417 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

For the following transaction list, give the number of entries per company after grouping identical companies together. Rules - One line per company, in the form "Company name: N entries" - Sort from the highest count to the lowest. Do not write any explanation ---- Transaction list start ---- ID,Client name,Rep,Amount A001,Sample Transport Co., Ltd.,Takahashi,730000 A011,(Ltd.) Sample Foods,Takahashi,563000 A007,SAMPLE INDUSTRY (LTD.),Ito,761000 A010,Sample Foods Co., Ltd.,Tanaka,231000 A009,SAMPLE FOODS (Ltd.),Tanaka,799000 A003,Sample Trading (Ltd.),Takahashi,250000 A002,(Ltd.) Sample Transport,Ito,483000 A004,SAMPLE TRADING (LTD.),Kato,502000 A008,Sample Industry (Ltd.),Tanaka,474000 A006,(Ltd.) Sample Trading,Kobayashi,641000 A005,Sample Trading Co., Ltd. ,Nakamura,314000 ---- Transaction list end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Auto-graded, exact_number) Sample Trading = 4 (margin ±0) / Sample Foods = 3 (margin ±0) / Sample Industry = 2 (margin ±0) / Sample Transport = 2 (margin ±0)

Scoring detailsWe show, trial by trial, why each verdict was given (9 of 12 trials correct)

ChatGPT, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (39.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (38.8 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (35.8 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (19.3 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (113.4 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Incorrect: Numbers were wrong: Sample Trading × (correct answer 4, no row for Sample Trading) / Sample Foods × (correct answer 3, no row for Sample Foods) / Sample Industry × (correct answer 2, no row for Sample Industry) / Sample Transport × (correct answer 2, no row for Sample Transport) (19.1 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (20.4 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (16.9 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (19.9 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2) (17.4 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Incorrect: Timed out (the answer did not finish before the time limit) (301.6 seconds)
Copilot, trial 3: Incorrect: Timed out (the answer did not finish before the time limit) (301 seconds)

Medium

For the following transaction list, give the total amount per company after grouping identical companies together. Rules… Scoring type: exact_number — Whether the number matches the correct answer. The rounding tolerance is set in advance

ChatGPT
- ChatGPT, trial 1: Correct 39.4 s
- ChatGPT, trial 2: Correct 38.7 s
- ChatGPT, trial 3: Correct 38.9 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 65.4 s
- Gemini, trial 2: Correct 31.2 s
- Gemini, trial 3: Correct 76.9 s
3 of 3 trials correct
Claude
- Claude, trial 1: Correct 26.1 s
- Claude, trial 2: Correct 26.2 s
- Claude, trial 3: Correct 23.2 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 16.9 s
- Copilot, trial 2: Incorrect Timed out
- Copilot, trial 3: Correct 16.9 s
2 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (431 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

For the following transaction list, give the total amount per company after grouping identical companies together. Rules - One line per company, in the form "Company name: N yen" - Sort from the highest amount to the lowest. Write numbers using half-width digits. Do not write any explanation ---- Transaction list start ---- ID,Client name,Rep,Amount A001,Sample Transport Co., Ltd.,Takahashi,730000 A011,(Ltd.) Sample Foods,Takahashi,563000 A007,SAMPLE INDUSTRY (LTD.),Ito,761000 A010,Sample Foods Co., Ltd.,Tanaka,231000 A009,SAMPLE FOODS (Ltd.),Tanaka,799000 A003,Sample Trading (Ltd.),Takahashi,250000 A002,(Ltd.) Sample Transport,Ito,483000 A004,SAMPLE TRADING (LTD.),Kato,502000 A008,Sample Industry (Ltd.),Tanaka,474000 A006,(Ltd.) Sample Trading,Kobayashi,641000 A005,Sample Trading Co., Ltd. ,Nakamura,314000 ---- Transaction list end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Auto-graded, exact_number) Sample Trading = 1707000 (margin ±0) / Sample Foods = 1593000 (margin ±0) / Sample Industry = 1235000 (margin ±0) / Sample Transport = 1213000 (margin ±0)

Scoring detailsWe show, trial by trial, why each verdict was given (11 of 12 trials correct)

ChatGPT, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (39.4 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (38.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (38.9 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (65.4 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (31.2 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (76.9 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (26.1 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (26.2 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (23.2 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (16.9 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Incorrect: Timed out (the answer did not finish before the time limit) (301.1 seconds)
Copilot, trial 3: Correct: All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000) (16.9 seconds) · Full text of this answer (Japanese)

Hard

Group the client names in the following transaction list into a markdown table. Rules - Exactly 3 columns, in the order:… Scoring type: exact_number — Whether the number matches the correct answer. The rounding tolerance is set in advance

ChatGPT
- ChatGPT, trial 1: Incorrect 38.9 s
- ChatGPT, trial 2: Incorrect 38.7 s
- ChatGPT, trial 3: Incorrect 38.9 s
0 of 3 trials correct
Gemini
- Gemini, trial 1: Incorrect 119.5 s
- Gemini, trial 2: Incorrect 80 s
- Gemini, trial 3: Incorrect 134.9 s
0 of 3 trials correct
Claude
- Claude, trial 1: Incorrect 17 s
- Claude, trial 2: Incorrect 17 s
- Claude, trial 3: Incorrect 20.1 s
0 of 3 trials correct
Copilot
- Copilot, trial 1: Incorrect 17.1 s
- Copilot, trial 2: Incorrect 17 s
- Copilot, trial 3: Incorrect Timed out
0 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (449 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

Group the client names in the following transaction list into a markdown table. Rules - Exactly 3 columns, in the order: Company name / Count / Total amount - Group identical companies into a single row. Sort from the highest amount to the lowest - Output only the table. Do not write any preamble or explanation ---- Transaction list start ---- ID,Client name,Rep,Amount A001,Sample Transport Co., Ltd.,Takahashi,730000 A011,(Ltd.) Sample Foods,Takahashi,563000 A007,SAMPLE INDUSTRY (LTD.),Ito,761000 A010,Sample Foods Co., Ltd.,Tanaka,231000 A009,SAMPLE FOODS (Ltd.),Tanaka,799000 A003,Sample Trading (Ltd.),Takahashi,250000 A002,(Ltd.) Sample Transport,Ito,483000 A004,SAMPLE TRADING (LTD.),Kato,502000 A008,Sample Industry (Ltd.),Tanaka,474000 A006,(Ltd.) Sample Trading,Kobayashi,641000 A005,Sample Trading Co., Ltd. ,Nakamura,314000 ---- Transaction list end ----

Answer key set in advanceWe set it when we created the task. 0 of the 4 AIs got it right at least once

(Auto-graded, exact_number) Sample Trading = 1707000 (margin ±0) / Sample Foods = 1593000 (margin ±0) / Sample Industry = 1235000 (margin ±0) / Sample Transport = 1213000 (margin ±0)

Scoring detailsWe show, trial by trial, why each verdict was given (0 of 12 trials correct)

ChatGPT, trial 1: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (38.9 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (38.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (38.9 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (119.5 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (80 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (134.9 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (17 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (17 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (20.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (17.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Incorrect: Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0) (17 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Incorrect: Timed out (the answer did not finish before the time limit) (301.8 seconds)

Hard

Group the client names in the following transaction list and output JSON only. Rules - Output JSON only. Do not write an… Scoring type: set_match — How many of the required items were missed. Counts both omissions and made-up additions

ChatGPT
- ChatGPT, trial 1: Correct 54.6 s
- ChatGPT, trial 2: Correct 38.7 s
- ChatGPT, trial 3: Correct 42.2 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 43.4 s
- Gemini, trial 2: Correct 116.5 s
- Gemini, trial 3: Correct 19.1 s
3 of 3 trials correct
Claude
- Claude, trial 1: Incorrect 26.1 s
- Claude, trial 2: Incorrect 20.2 s
- Claude, trial 3: Incorrect 23.1 s
0 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 17.1 s
- Copilot, trial 2: Correct 14.4 s
- Copilot, trial 3: Correct 17 s
3 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (499 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

Group the client names in the following transaction list and output JSON only. Rules - Output JSON only. Do not write any explanation before or after it - Use the form {"companies": [{"name": "Company name", "count": count, "total": total amount}]} - Group identical companies into a single entry. Enter numbers as numeric values (no comma separators) ---- Transaction list start ---- ID,Client name,Rep,Amount A001,Sample Transport Co., Ltd.,Takahashi,730000 A011,(Ltd.) Sample Foods,Takahashi,563000 A007,SAMPLE INDUSTRY (LTD.),Ito,761000 A010,Sample Foods Co., Ltd.,Tanaka,231000 A009,SAMPLE FOODS (Ltd.),Tanaka,799000 A003,Sample Trading (Ltd.),Takahashi,250000 A002,(Ltd.) Sample Transport,Ito,483000 A004,SAMPLE TRADING (LTD.),Kato,502000 A008,Sample Industry (Ltd.),Tanaka,474000 A006,(Ltd.) Sample Trading,Kobayashi,641000 A005,Sample Trading Co., Ltd. ,Nakamura,314000 ---- Transaction list end ----

Answer key set in advanceWe set it when we created the task. 3 of the 4 AIs got it right at least once

(Auto-graded, set_match) Sample Trading [Sample Trading] / Sample Foods [Sample Foods] / Sample Industry [Sample Industry] / Sample Transport [Sample Transport] - up to 0 extra entries allowed

Scoring detailsWe show, trial by trial, why each verdict was given (9 of 12 trials correct)

ChatGPT, trial 1: Correct: All 4 items matched, 0 fabricated items (54.6 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All 4 items matched, 0 fabricated items (38.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All 4 items matched, 0 fabricated items (42.2 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: All 4 items matched, 0 fabricated items (43.4 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: All 4 items matched, 0 fabricated items (116.5 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: All 4 items matched, 0 fabricated items (19.1 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Incorrect: Could not be read as JSON (did not follow the format instructions) (26.1 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Incorrect: Could not be read as JSON (did not follow the format instructions) (20.2 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Incorrect: Could not be read as JSON (did not follow the format instructions) (23.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All 4 items matched, 0 fabricated items (17.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Correct: All 4 items matched, 0 fabricated items (14.4 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Correct: All 4 items matched, 0 fabricated items (17 seconds) · Full text of this answer (Japanese)

ANSWERS

Full answers from the 4 AIs

These are the answers we scored, posted as is without edits. That way, anyone can recount the results the same way we did.

ChatGPT

18 answers15 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Correct

t2, trial 2

Full text file (Japanese) · Correct

t2, trial 3

Full text file (Japanese) · Correct

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Correct

t3, trial 3

Full text file (Japanese) · Correct

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Correct

t6, trial 2

Full text file (Japanese) · Correct

t6, trial 3

Full text file (Japanese) · Correct
Gemini

18 answers14 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Correct

t2, trial 2

Full text file (Japanese) · Correct

t2, trial 3

Full text file (Japanese) · Correct

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Correct

t3, trial 3

Full text file (Japanese) · Incorrect

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Correct

t6, trial 2

Full text file (Japanese) · Correct

t6, trial 3

Full text file (Japanese) · Correct
Claude

18 answers12 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Correct

t2, trial 2

Full text file (Japanese) · Correct

t2, trial 3

Full text file (Japanese) · Correct

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Correct

t3, trial 3

Full text file (Japanese) · Correct

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Incorrect

t6, trial 2

Full text file (Japanese) · Incorrect

t6, trial 3

Full text file (Japanese) · Incorrect
Copilot

14 answers12 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Correct

t2, trial 2

Full text file (Japanese) · Correct

t2, trial 3

Full text file (Japanese) · Correct

t3, trial 1

Full text file (Japanese) · Correct

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Correct

t6, trial 2

Full text file (Japanese) · Correct

t6, trial 3

Full text file (Japanese) · Correct

MATERIALS

Materials used in the tasks

A list of transactions with four fictional client companies, with names written inconsistently using (KK), the kanji abbreviation for incorporated, half-width katakana, spaces, and so on. The correct answer was fixed when the list was created. We recreated them for this run on October 18, 2026. Download them to repeat the same test yourself.

meibo_clean.csv 202B (Japanese)
meibo_dirty.csv 544B (Japanese)

METHOD

How we measure

We list this run's conditions exactly as they were, so anyone can repeat it.

AIs measured and their model names at the time

AI services change under the hood without notice. We record the model name shown on screen and the time we checked it.

AIs measured and model names
AI	Plan	Model name shown on screen	Time checked
ChatGPT	ChatGPT Pro	GPT-6.1 Sol Light	2026-10-18 20:21
Gemini	Google AI Pro	Flash	2026-10-18 20:21
Claude	Claude Max	Opus 4.8 high	2026-10-18 20:21
Copilot	Microsoft 365 Personal		2026-10-18 20:21

What we did in this run

We recreated the materials for the tasks specifically for this run. The data are for a fictional company and include no real companies or people.
We created 6 tasks and set the answer keys at that point.
We sent the same prompt and the same materials to ChatGPT, Gemini, Claude and Copilot.
We sent each task 3 times, in a new chat each time.
We rotated the sending order from run to run. This keeps the time of day from skewing the ranking.
We waited up to 300 seconds for each answer, and recorded anything longer as a timeout.
Scoring: Machine scoring only (not graded by AI).
We sent 72 trials in total, and 72 of them were valid.

How we count trials with no answer

Refused or timed out
We keep it in the base and count it as wrong. Because at work, that is the same as getting no answer.
Hit the plan's usage limit
We remove it from the base. We treat that trial as not measured.
A human verification screen appeared
We stop there. We neither click through it automatically nor try to get around it.
Operating error on AInformation's side
We remove it from the base. We do not report it as the AI's fault.

LIMITS

Limits of this benchmark

So that no one judges by the numbers alone, we list what we could not measure.

This is a single run on a single day. Rankings can change with the day or the task
The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier
Measurements use only the default model and default mode (features such as Deep Research are not used)
The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company
Only tasks that can be scored by machine are covered. Writing quality and readability are not measured
Measurements are taken from the same screen a person would use in a browser. These are not results from the API
For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is

DATA

Published data

In the CSV, 1 row is 1 trial. The figures on this page match a recount of that CSV. The license is CC BY 4.0.

Latest version (CSV) (Japanese) 1 row = 1 trial, 17KB
Latest version (JSON) 44KB
v1 (version at publication) (CSV) (Japanese) 1 row = 1 trial, 17KB
v1 (version at publication) (JSON) 44KB

The cumulative rankings are at /en/data/research/bench/leaderboard.json. We do not delete versions (such as v1). That way, links from people who cited them keep working.

FAQ

How do you measure?: We send exactly the same task prompt and the same materials to the 4 AIs on the same day. We open a new chat every time so earlier answers don't carry over. The sending order is rotated in each run. We also record the seconds from sending until the answer stops.
Why do you send the same task 3 times?: Because AI answers vary even for the same question. From a single try, you can't tell whether a result reflects real ability or chance. We send each task 3 times, and the share of tasks whose conclusion matched is reported as the consistency rate.
Who does the scoring?: Scoring is done only by a program. We do not let AI score the answers. The correct answers are set when the tasks are created, and whether an answer is right is decided by whether numbers match, whether required words are present, whether the format fits, whether URLs actually open and the result of running formulas. Matters of taste are not scored.
Is the data in the tasks from real companies?: No. We create new data for a fictional company every time. We do not use real companies, real people or text from other companies. We share the materials we used as they are, so you can repeat the same test.
How do you count cases where the AI couldn't answer?: Refusals and timeouts stay in the base and are counted as wrong. If we hit the plan's usage limit or a human verification screen appears, we remove that trial from the base and treat it as not measured. Operating errors on AInformation's side are also removed from the base, and we do not report them as the AI's fault.
Do you use the free or paid versions?: We use each AI's paid plan through the browser interface. We don't use the API. For each run, we record the plan name, the model name shown on screen and the time we checked it.
Does this ranking show which AI is the smartest?: No. We measure only 5 things on a fixed set of work tasks: accuracy, time to answer, consistency rate, real-source rate and instruction compliance. We do not measure writing style, readability or behavior in long conversations. If the tasks change, the ranking changes too.
Can I cite these results?: Yes. The license is CC BY 4.0. As the source, please write “Source: AInformation, AI Practical Work Benchmark” with the URL of the page you used and the measurement date. The results are also available as JSON and CSV. With the CSV (1 row per trial), you can recount the figures we publish.

CITATION

Cite this benchmark

Source: AInformation, “AI Practical Work Benchmark: Standardize inconsistent spellings of client names (4 AIs, 6 tasks, 3 runs each)” (https://ainformation.jp/en/ai-surveys/ai-work-benchmark/company-name-cleanup-2026-10-19). Measured on October 19, 2026. 6 tasks, 3 trials each, 72 valid trials

When you cite a figure, please include the base too (how many of how many trials). Please cite AInformation as the source, along with the URL of the page you used and the measurement date.

What this run found

Scores by AI

Results by task

Full answers from the 4 AIs

Materials used in the tasks

How we measure

AIs measured and their model names at the time

What we did in this run

How we count trials with no answer

Limits of this benchmark

Published data

FAQ

Cite this benchmark

Related pages