We send exactly the same task prompt and the same materials to the 4 AIs on the same day. We open a new chat every time so earlier answers don't carry over. The sending order is rotated in each run. We also record the seconds from sending until the answer stops.

Why do you send the same task 3 times?

Because AI answers vary even for the same question. From a single try, you can't tell whether a result reflects real ability or chance. We send each task 3 times, and the share of tasks whose conclusion matched is reported as the consistency rate.

Who does the scoring?

Scoring is done only by a program. We do not let AI score the answers. The correct answers are set when the tasks are created, and whether an answer is right is decided by whether numbers match, whether required words are present, whether the format fits, whether URLs actually open and the result of running formulas. Matters of taste are not scored.

Is the data in the tasks from real companies?

No. We create new data for a fictional company every time. We do not use real companies, real people or text from other companies. We share the materials we used as they are, so you can repeat the same test.

How do you count cases where the AI couldn't answer?

Refusals and timeouts stay in the base and are counted as wrong. If we hit the plan's usage limit or a human verification screen appears, we remove that trial from the base and treat it as not measured. Operating errors on AInformation's side are also removed from the base, and we do not report them as the AI's fault.

Do you use the free or paid versions?

We use each AI's paid plan through the browser interface. We don't use the API. For each run, we record the plan name, the model name shown on screen and the time we checked it.

Does this ranking show which AI is the smartest?

No. We measure only 5 things on a fixed set of work tasks: accuracy, time to answer, consistency rate, real-source rate and instruction compliance. We do not measure writing style, readability or behavior in long conversations. If the tasks change, the ranking changes too.

Extracting decisions and action items from meeting minutes (4 AIs, 6 tasks, 3 runs each)

Name: Extracting decisions and action items from meeting minutes (4 AIs, 6 tasks, 3 runs each)
Creator: AInformation
Published: 2026-10-29
License: https://creativecommons.org/licenses/by/4.0/

RESULT

What this run found

The line below is generated directly from the scoring results. The page adds no words of its own.

Meeting minutes and summaries: 6 tasks, 3 trials each We gave each AI 6 tasks, asked 3 times each, and the highest accuracy went to ChatGPT at 83.3% (15/18 trials), with Gemini second at 55.6% 72 valid trials. Results from a single round of testing. Rankings may change on a different day or with different tasks. See scores by AI

SCORES

Scores by AI

Sorted by accuracy, highest first. The base is the number of trials in which that metric could be measured. “—” means not measured; it does not mean 0%.

Extracting decisions and action items from meeting minutes (4 AIs, 6 tasks, 3 runs each): scores by AI
AI	AccuracyCorrect ÷ valid trials	ConsistencySame conclusion in all 3 trials	Real-source rateShare of URLs that opened	Instruction complianceMet length and format rules	Time to answerMedian / range
1ChatGPT	83.3% 15 of 18 trials	100% out of 6 tasks	—	83.3%	25.1 s 20.2–32.8 seconds
2Claude	55.6% 10 of 18 trials	88.9% out of 6 tasks	—	22.2%	16.9 s 13.8–26.1 seconds
3Gemini	55.6% 10 of 18 trials	83.3% out of 6 tasks	—	83.3%	19.1 s 19–19.1 seconds
4Copilot	55.6% 10 of 18 trials	66.7% out of 6 tasks	—	56.2%	18.6 s 13.8–23.1 seconds

The real-source rate is measured only in runs that included a task asking for URLs along with the answer. For ChatGPT, Claude, Gemini and Copilot, “—” means this run did not include that kind of task. It does not mean 0%.

Trials with no answer
Copilot … Timed out: 2 times
Refusals and timeouts stay in the base and count as incorrect. Plan limits, human verification screens and operating errors on AInformation's side are excluded from the base.

TASKS

Results by task

We sent each task to each of the 4 AIs 3 times. ○ means correct and × means incorrect. The numbers below are the seconds each trial took. Open each section to read the full task prompt, the answer key set in advance and why each verdict was given.

Easy

From the meeting minutes below, list only the names of “people who took away an action item.” Rules - One line per perso… Scoring type: set_match — How many of the required items were missed. Counts both omissions and made-up additions

ChatGPT
- ChatGPT, trial 1: Correct 26.7 s
- ChatGPT, trial 2: Correct 29.8 s
- ChatGPT, trial 3: Correct 20.2 s
3 of 3 trials correct
Claude
- Claude, trial 1: Correct 13.9 s
- Claude, trial 2: Correct 13.9 s
- Claude, trial 3: Correct 14 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 19.1 s
- Gemini, trial 2: Correct 19.1 s
- Gemini, trial 3: Correct 19 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Incorrect Timed out
- Copilot, trial 2: Correct 17.1 s
- Copilot, trial 3: Correct 16.9 s
2 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (1,340 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

From the meeting minutes below, list only the names of “people who took away an action item.” Rules - One line per person. Start each line with a bullet ( · ). Write only the name, not the content - Count only “action items” that were assigned by name (e.g., “Mr./Ms. XX, please...”) - If the same person has multiple action items, combine them into one line - Do not list attendees who have no action items - No preamble and no comments ---- Meeting minutes start ---- ■ Product Planning and Sales Regular Meeting (October 29, 2026, 10:00–11:30, Meeting Room B) Attendees: Suzuki, Takahashi, Nakamura, Tanaka, Ito, Yamamoto Minutes taker: Yamamoto * These minutes are a fictional document created by AInformation to test AI. They have no connection to any real company or person. ■ Discussion Ito: In last month's internal survey, 71% of respondents wanted to keep working from home. The results have already been shared with everyone. Yamamoto: Mr. Tanaka, please check the cost of the accessories and share it by December 20. Tanaka: We will not exhibit at the Nagoya trade show. Instead, we will hold our own online business meeting on October 20. Nakamura: We will not decide today on adding another logistics warehouse. We'll check candidate site rents first and bring it back next time. Suzuki: We finished submitting the new catalog on November 20. The cover photo has also been replaced. Yamamoto: Mr. Tanaka, please create a monthly complaint tally format by November 25. Ito: Warehouse operations are fine for now. Shipping is expected to peak in the third week of October. Suzuki: We will not decide today on next fiscal year's organization. It's not on today's agenda, so we'll bring it back next time. Takahashi: We decided to move forward with opening a store in Fukuoka. The condition is that the rent for the property near the station drops to ¥1,000,000. If it doesn't drop, we'll pass. Nakamura: We will not decide today on making novelty items either. The budget hasn't been set, so it's on hold. Yamamoto: Mr. Tanaka, please arrange the gift photo shoot. Please finish shooting by December 20. Tanaka: We'll narrow the year-end gift down to one item: an assortment set of Product B (eco bag) for ¥5,240. The two-set option will be shelved this time. Ito: Last week there were 8 complaints, 3 more than the previous month. The cause was stitching coming loose in 9 cases, with the rest being crushed boxes during delivery. Tanaka: A client asked if we could shorten the delivery time from the current 7 days to 5 days. The factory told us that would be difficult. Tanaka: We'll set 3 priorities for the second half: turning around Product E (insulated bottle), raising the e-commerce ratio to 39%, and cutting logistics costs by 0.7 points as a share of sales. Suzuki: For Product A (tumbler), let's start selling it as a set with accessories from October. We'll include a replacement gasket and a dedicated brush, with no price change. Please keep the accessory cost within ¥195 per unit. Yamamoto: Copy paper was ordered last week. It will arrive within this month. Yamamoto: Mr. Suzuki, please start recruiting short-term part-time staff on December 5. Yamamoto: E-commerce sales in December were ¥31,100,000, up 16% from the same month the previous year. ■ Next meeting: November 15, 10:00, Meeting Room B End ---- Meeting minutes end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Machine-graded set_match) Tanaka [Tanaka] / Suzuki [Suzuki] Up to 0 extra entries allowed

Scoring detailsWe show, trial by trial, why each verdict was given (11 of 12 trials correct)

ChatGPT, trial 1: Correct: Both items matched, 0 extra items (26.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: Both items matched, 0 extra items (29.8 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: Both items matched, 0 extra items (20.2 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: Both items matched, 0 extra items (13.9 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: Both items matched, 0 extra items (13.9 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: Both items matched, 0 extra items (14 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: Both items matched, 0 extra items (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: Both items matched, 0 extra items (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: Both items matched, 0 extra items (19 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Incorrect: Timed out (the answer did not finish before the time limit) (302.2 seconds)
Copilot, trial 2: Correct: Both items matched, 0 extra items (17.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Correct: Both items matched, 0 extra items (16.9 seconds) · Full text of this answer (Japanese)

Easy

From the meeting minutes below, extract only the decisions that include a monetary amount (yen). Rules - One item per li… Scoring type: set_match — How many of the required items were missed. Counts both omissions and made-up additions

ChatGPT
- ChatGPT, trial 1: Correct 23.7 s
- ChatGPT, trial 2: Correct 23.9 s
- ChatGPT, trial 3: Correct 23.6 s
3 of 3 trials correct
Claude
- Claude, trial 1: Incorrect 13.8 s
- Claude, trial 2: Incorrect 14 s
- Claude, trial 3: Incorrect 17 s
0 of 3 trials correct
Gemini
- Gemini, trial 1: Incorrect 19.1 s
- Gemini, trial 2: Incorrect 19 s
- Gemini, trial 3: Incorrect 19.1 s
0 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 20 s
- Copilot, trial 2: Incorrect Timed out
- Copilot, trial 3: Correct 20.1 s
2 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (1,408 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

From the meeting minutes below, extract only the decisions that include a monetary amount (yen). Rules - One item per line. Start each line with a sequential number: Amount 1: Amount 2: ... - Do not list decisions that don't include an amount - A “decision” is a policy decided at this meeting. Do not include the following 3 types - “Action items” assigned by name (e.g., “Mr./Ms. XX, please...”) - Items marked as “not decided today” - Reports or shared figures about things already completed, or the date of the next meeting - Write amounts exactly as they appear in the minutes - No preamble and no comments ---- Meeting minutes start ---- ■ Product Planning and Sales Regular Meeting (October 29, 2026, 10:00–11:30, Meeting Room B) Attendees: Suzuki, Takahashi, Nakamura, Tanaka, Ito, Yamamoto Minutes taker: Yamamoto * These minutes are a fictional document created by AInformation to test AI. They have no connection to any real company or person. ■ Discussion Ito: In last month's internal survey, 71% of respondents wanted to keep working from home. The results have already been shared with everyone. Yamamoto: Mr. Tanaka, please check the cost of the accessories and share it by December 20. Tanaka: We will not exhibit at the Nagoya trade show. Instead, we will hold our own online business meeting on October 20. Nakamura: We will not decide today on adding another logistics warehouse. We'll check candidate site rents first and bring it back next time. Suzuki: We finished submitting the new catalog on November 20. The cover photo has also been replaced. Yamamoto: Mr. Tanaka, please create a monthly complaint tally format by November 25. Ito: Warehouse operations are fine for now. Shipping is expected to peak in the third week of October. Suzuki: We will not decide today on next fiscal year's organization. It's not on today's agenda, so we'll bring it back next time. Takahashi: We decided to move forward with opening a store in Fukuoka. The condition is that the rent for the property near the station drops to ¥1,000,000. If it doesn't drop, we'll pass. Nakamura: We will not decide today on making novelty items either. The budget hasn't been set, so it's on hold. Yamamoto: Mr. Tanaka, please arrange the gift photo shoot. Please finish shooting by December 20. Tanaka: We'll narrow the year-end gift down to one item: an assortment set of Product B (eco bag) for ¥5,240. The two-set option will be shelved this time. Ito: Last week there were 8 complaints, 3 more than the previous month. The cause was stitching coming loose in 9 cases, with the rest being crushed boxes during delivery. Tanaka: A client asked if we could shorten the delivery time from the current 7 days to 5 days. The factory told us that would be difficult. Tanaka: We'll set 3 priorities for the second half: turning around Product E (insulated bottle), raising the e-commerce ratio to 39%, and cutting logistics costs by 0.7 points as a share of sales. Suzuki: For Product A (tumbler), let's start selling it as a set with accessories from October. We'll include a replacement gasket and a dedicated brush, with no price change. Please keep the accessory cost within ¥195 per unit. Yamamoto: Copy paper was ordered last week. It will arrive within this month. Yamamoto: Mr. Suzuki, please start recruiting short-term part-time staff on December 5. Yamamoto: E-commerce sales in December were ¥31,100,000, up 16% from the same month the previous year. ■ Next meeting: November 15, 10:00, Meeting Room B End ---- Meeting minutes end ----

Answer key set in advanceWe set it when we created the task. 2 of the 4 AIs got it right at least once

(Machine-graded set_match) Set sales [set + ¥195] / Gift [gift + ¥5,240] / New store [Fukuoka + ¥1,000,000] Up to 0 extra entries allowed

Scoring detailsWe show, trial by trial, why each verdict was given (5 of 12 trials correct)

ChatGPT, trial 1: Correct: All 3 items matched, 0 fabricated items (23.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All 3 items matched, 0 fabricated items (23.9 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All 3 items matched, 0 fabricated items (23.6 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Incorrect: 3 of 3 matched, 1 extra entry added (13.8 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Incorrect: 3 of 3 matched, 1 extra entry added (14 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Incorrect: 3 of 3 matched, 1 extra entry added (17 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Incorrect: 0 of 3 matched, missed; set sales/gift/new store; 4 extra entries added (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Incorrect: 0 of 3 matched, missed; set sales/gift/new store; 4 extra entries added (19 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Incorrect: 0 of 3 matched, missed; set sales/gift/new store; 4 extra entries added (19.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All 3 items matched, 0 fabricated items (20 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Incorrect: Timed out (the answer did not finish before the time limit) (300.2 seconds)
Copilot, trial 3: Correct: All 3 items matched, 0 fabricated items (20.1 seconds) · Full text of this answer (Japanese)

Medium

Summarize the following meeting minutes in 200 characters or fewer, then write only two lines of counts at the end. Rule… Scoring type: exact_number — Whether the number matches the correct answer. The rounding tolerance is set in advance

ChatGPT
- ChatGPT, trial 1: Correct 26.8 s
- ChatGPT, trial 2: Correct 26.5 s
- ChatGPT, trial 3: Correct 29.7 s
3 of 3 trials correct
Claude
- Claude, trial 1: Correct 23.1 s
- Claude, trial 2: Correct 23.1 s
- Claude, trial 3: Correct 26.1 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 19.1 s
- Gemini, trial 2: Incorrect 19.1 s
- Gemini, trial 3: Incorrect 19 s
1 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 20.1 s
- Copilot, trial 2: Correct 20.1 s
- Copilot, trial 3: Incorrect 13.8 s
2 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (1,384 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

Summarize the following meeting minutes in 200 characters or fewer, then write only two lines of counts at the end. Rules - The summary must be 200 characters or fewer - After the summary, write two lines, in this order: Decisions: N and To-dos: N - Write the counts as half-width digits - "Decisions" means the policies decided at this meeting. Do not include the following three types - A "to-do" assigned by calling on a specific person by name (e.g., "(Name)-san, please...") - Items for which someone said "We won't decide this today" - Reports on things already finished, shared figures, or the date of the next meeting ---- Meeting minutes start ---- Product Planning and Sales Regular Meeting (October 29, 2026, 10:00 a.m.–11:30 a.m., Meeting Room B) Attendees: Suzuki, Takahashi, Nakamura, Tanaka, Ito, Yamamoto Minutes taker: Yamamoto *These minutes are a fictional document created by AInformation to test AI. They have no connection to any real company or person. Discussion Ito: In last month's internal survey, 71% of respondents said they want to keep working from home. The results have already been shared with everyone. Yamamoto: Tanaka-san, please check the cost of the accessories and share it by December 20. Tanaka: We will not exhibit at the Nagoya trade show. Instead, we will hold our own online business meeting on October 20. Nakamura: We won't decide today on adding another logistics warehouse. We'll check candidate site rents and bring it back next time. Suzuki: The new catalog has already gone to print as of November 20. The cover photo has also been replaced. Yamamoto: Tanaka-san, please create a monthly complaint tally format by November 25. Ito: Warehouse operations are fine for now. We expect shipments to be concentrated in the third week of October. Suzuki: We won't decide on next fiscal year's organization today. It's not on today's agenda, so we'll bring it back next time. Takahashi: We have decided to open a store in Fukuoka. The condition is that the rent for the property near the station drops to ¥1,000,000. If it doesn't drop, we'll drop the plan. Nakamura: We also won't decide on making novelty items today. The budget hasn't been set, so it's on hold. Yamamoto: Tanaka-san, please arrange the gift photo shoot. Finish the shoot by December 20. Tanaka: We will narrow the year-end gift down to one item: an assortment of Product B (eco bag), priced at ¥5,240. The two-pack option will be shelved this time. Ito: There were 8 complaints last week, 3 more than the previous month. The cause was stitching that came loose in 9 cases, with the rest being crushed boxes during shipping. Tanaka: A client asked if we could shorten the delivery time from the current 7 days to 5 days. The factory has told us this would be difficult. Tanaka: We will focus on three priorities for the second half: turning around Product E (thermal bottle), raising the e-commerce ratio to 39%, and cutting logistics costs by 0.7 points as a share of sales. Suzuki: For Product A (tumbler), let's start selling it as a set with accessories from October. We'll include a replacement gasket and a dedicated brush, and keep the price unchanged. Please keep the accessory cost at ¥195 or less per unit. Yamamoto: Copy paper was ordered last week. It will arrive within this month. Yamamoto: Suzuki-san, please start recruiting short-term part-time staff on December 5. Yamamoto: December e-commerce sales were ¥31.1 million, up 16% from the same month the previous year. Next meeting: November 15, 10:00 a.m., Meeting Room B End ---- Meeting minutes end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Automated scoring: exact_number) Number of decisions = 5 (margin ±0) / Number of to-dos = 4 (margin ±0)

Scoring detailsWe show, trial by trial, why each verdict was given (9 of 12 trials correct)

ChatGPT, trial 1: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (26.8 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (26.5 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (29.7 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (23.1 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (23.1 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (26.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Incorrect: A number is wrong: number of decisions - incorrect (correct answer: 5, answer given: 10.0) / number of to-dos - correct (correct answer: 4) (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Incorrect: A number is wrong: number of decisions - incorrect (correct answer: 5, answer given: 71.0) / number of to-dos - correct (correct answer: 4) (19 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (20.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Correct: All numbers match: number of decisions - correct (correct answer: 5) / number of to-dos - correct (correct answer: 4) (20.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Incorrect: Numbers are wrong: number of decisions - incorrect (correct answer: 5, no "Decisions" line) / number of to-dos - incorrect (correct answer: 4, no "To-dos" line) (13.8 seconds) · Full text of this answer (Japanese)

Medium

Read the following meeting minutes and answer four questions. Rules - The answer must be exactly four lines, in this ord… Scoring type: contains_all — Whether all the required words and phrases are included

ChatGPT
- ChatGPT, trial 1: Correct 26.4 s
- ChatGPT, trial 2: Correct 23.5 s
- ChatGPT, trial 3: Correct 23.5 s
3 of 3 trials correct
Claude
- Claude, trial 1: Correct 16.9 s
- Claude, trial 2: Correct 16.9 s
- Claude, trial 3: Correct 16.9 s
3 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 19 s
- Gemini, trial 2: Correct 19.1 s
- Gemini, trial 3: Correct 19.1 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 23.1 s
- Copilot, trial 2: Correct 20 s
- Copilot, trial 3: Incorrect 13.9 s
2 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (1,423 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

Read the following meeting minutes and answer four questions. Rules - The answer must be exactly four lines, in this order: date of next meeting / number of decisions / number of to-dos / number of items left undecided - Format each line as: Next: .../ Decisions: N/ To-dos: N/ Pending: N - Write the counts as half-width digits. Do not add explanations - "Decisions" means the policies decided at this meeting. Do not include the following three types - A "to-do" assigned by calling on a specific person by name (e.g., "(Name)-san, please...") - Items for which someone said "We won't decide this today" - Reports on things already finished, shared figures, or the date of the next meeting ---- Meeting minutes start ---- Product Planning and Sales Regular Meeting (October 29, 2026, 10:00 a.m.–11:30 a.m., Meeting Room B) Attendees: Suzuki, Takahashi, Nakamura, Tanaka, Ito, Yamamoto Minutes taker: Yamamoto *These minutes are a fictional document created by AInformation to test AI. They have no connection to any real company or person. Discussion Ito: In last month's internal survey, 71% of respondents said they want to keep working from home. The results have already been shared with everyone. Yamamoto: Tanaka-san, please check the cost of the accessories and share it by December 20. Tanaka: We will not exhibit at the Nagoya trade show. Instead, we will hold our own online business meeting on October 20. Nakamura: We won't decide today on adding another logistics warehouse. We'll check candidate site rents and bring it back next time. Suzuki: The new catalog has already gone to print as of November 20. The cover photo has also been replaced. Yamamoto: Tanaka-san, please create a monthly complaint tally format by November 25. Ito: Warehouse operations are fine for now. We expect shipments to be concentrated in the third week of October. Suzuki: We won't decide on next fiscal year's organization today. It's not on today's agenda, so we'll bring it back next time. Takahashi: We have decided to open a store in Fukuoka. The condition is that the rent for the property near the station drops to ¥1,000,000. If it doesn't drop, we'll drop the plan. Nakamura: We also won't decide on making novelty items today. The budget hasn't been set, so it's on hold. Yamamoto: Tanaka-san, please arrange the gift photo shoot. Finish the shoot by December 20. Tanaka: We will narrow the year-end gift down to one item: an assortment of Product B (eco bag), priced at ¥5,240. The two-pack option will be shelved this time. Ito: There were 8 complaints last week, 3 more than the previous month. The cause was stitching that came loose in 9 cases, with the rest being crushed boxes during shipping. Tanaka: A client asked if we could shorten the delivery time from the current 7 days to 5 days. The factory has told us this would be difficult. Tanaka: We will focus on three priorities for the second half: turning around Product E (thermal bottle), raising the e-commerce ratio to 39%, and cutting logistics costs by 0.7 points as a share of sales. Suzuki: For Product A (tumbler), let's start selling it as a set with accessories from October. We'll include a replacement gasket and a dedicated brush, and keep the price unchanged. Please keep the accessory cost at ¥195 or less per unit. Yamamoto: Copy paper was ordered last week. It will arrive within this month. Yamamoto: Suzuki-san, please start recruiting short-term part-time staff on December 5. Yamamoto: December e-commerce sales were ¥31.1 million, up 16% from the same month the previous year. Next meeting: November 15, 10:00 a.m., Meeting Room B End ---- Meeting minutes end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Automated scoring: contains_all) Date of next meeting = November 15 / Number of decisions = 5 or 5 items / Number of to-dos = 4 or 4 items / Number of pending items = 3 or 3 items

Scoring detailsWe show, trial by trial, why each verdict was given (11 of 12 trials correct)

ChatGPT, trial 1: Correct: All 4 items present (26.4 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: All 4 items present (23.5 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: All 4 items present (23.5 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Correct: All 4 items present (16.9 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Correct: All 4 items present (16.9 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: All 4 items present (16.9 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: All 4 items present (19 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: All 4 items present (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: All 4 items present (19.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: All 4 items present (23.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Correct: All 4 items present (20 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Incorrect: 0 of 4 items present - missing date of next meeting / number of decisions / number of to-dos / number of pending items (13.9 seconds) · Full text of this answer (Japanese)

Hard

From the meeting minutes below, pull out who must do what and by when, and present it as a markdown table. Rules - Use e… Scoring type: set_match — How many of the required items were missed. Counts both omissions and made-up additions

ChatGPT
- ChatGPT, trial 1: Incorrect 20.5 s
- ChatGPT, trial 2: Incorrect 23.5 s
- ChatGPT, trial 3: Incorrect 29.6 s
0 of 3 trials correct
Claude
- Claude, trial 1: Incorrect 13.9 s
- Claude, trial 2: Incorrect 13.8 s
- Claude, trial 3: Incorrect 13.9 s
0 of 3 trials correct
Gemini
- Gemini, trial 1: Incorrect 19.1 s
- Gemini, trial 2: Incorrect 19 s
- Gemini, trial 3: Incorrect 19 s
0 of 3 trials correct
Copilot
- Copilot, trial 1: Incorrect 17.1 s
- Copilot, trial 2: Incorrect 17.2 s
- Copilot, trial 3: Incorrect 13.9 s
0 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (1,356 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

From the meeting minutes below, pull out who must do what and by when, and present it as a markdown table. Rules - Use exactly 3 columns, in this order: Owner / Task / Deadline - Only include “tasks” that were assigned by name (e.g., “Mr./Ms. X, please... ”) - Do not include decisions, status reports, or items where someone said “we won't decide this today” - Write the deadline exactly as it appears in the minutes - Output only the table. No preamble, no comments ---- Meeting minutes start ---- ■ Product planning and sales regular meeting (October 29, 2026, 10:00–11:30, Meeting Room B) Attendees: Suzuki, Takahashi, Nakamura, Tanaka, Ito, Yamamoto Minutes: Yamamoto * These minutes are a fictional document created by AInformation to test AI tools. They have no connection to any real company or person. ■ Discussion Ito: In last month's internal survey, 71% of staff said they want to keep working from home. The results have already been shared with everyone. Yamamoto: Tanaka, please check the cost of the accessories and share it by December 20. Tanaka: We will not exhibit at the Nagoya trade show. Instead, we will hold our own online business meeting on October 20. Nakamura: We won't decide today on adding another logistics warehouse. We'll research candidate site rents and bring it back next time. Suzuki: The new catalog was sent to print on November 20. The cover photo has also been replaced. Yamamoto: Tanaka, please create a monthly complaint-tally template by November 25. Ito: Warehouse operations are fine for now. We expect shipments to peak in the third week of October. Suzuki: We won't decide today on next term's organization. It's not on today's agenda, so we'll carry it over to next time. Takahashi: We have decided to open a store in Fukuoka. The condition is that the rent for the property near the station drops to ¥1,000,000. If it doesn't drop, we will shelve it. Nakamura: We also won't decide today on producing novelties. The budget hasn't been set, so it's on hold. Yamamoto: Tanaka, please arrange the gift photo shoot. Finish shooting by December 20. Tanaka: We'll narrow the year-end gift down to one item: an assortment of Product B (eco bag), priced at ¥5,240. The two-bag set idea is shelved for now. Ito: There were 8 complaints last week, 3 more than the previous month. Loose stitching caused 9 of them; the rest were due to crushed boxes during shipping. Tanaka: A client asked whether we could shorten the lead time from the current 7 days to 5 days. The factory has told us that's difficult. Tanaka: We'll focus on 3 priorities for the second half: turning around Product E (insulated bottle), raising the e-commerce ratio to 39%, and cutting logistics cost as a share of sales by 0.7 points. Suzuki: For Product A (tumbler), let's start selling it as a set with accessories from October. We'll include a replacement gasket and a dedicated brush, and keep the price unchanged. Please keep the accessory cost at ¥195 or less per unit. Yamamoto: Copy paper was ordered last week. It will arrive within this month. Yamamoto: Suzuki, please start recruiting short-term part-time staff on December 5. Yamamoto: E-commerce sales in December were ¥31,100,000, up 16% from the same month last year. ■ Next meeting: November 15, 10:00, Meeting Room B End ---- Meeting minutes end ----

Answer key set in advanceWe set it when we created the task. 0 of the 4 AIs got it right at least once

(Machine-scored set_match) Tanaka [Tanaka + photo shoot + December 20] / Suzuki [Suzuki + part-time staff + December 5] / Tanaka [Tanaka + cost + December 20] / Tanaka [Tanaka + tally + November 25] — up to 0 extra items allowed

Scoring detailsWe show, trial by trial, why each verdict was given (0 of 12 trials correct)

ChatGPT, trial 1: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (20.5 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (23.5 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (29.6 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (13.9 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (13.8 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (13.9 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (19 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (19 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (17.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (17.2 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Incorrect: 0 of 4 matched, missing Tanaka / Suzuki / Tanaka / Tanaka (13.9 seconds) · Full text of this answer (Japanese)

Hard

Read the meeting minutes below and output JSON only. Rules - Output JSON only. No explanatory text before or after - Use… Scoring type: set_match — How many of the required items were missed. Counts both omissions and made-up additions

ChatGPT
- ChatGPT, trial 1: Correct 23.7 s
- ChatGPT, trial 2: Correct 32.8 s
- ChatGPT, trial 3: Correct 26.5 s
3 of 3 trials correct
Claude
- Claude, trial 1: Incorrect 20.1 s
- Claude, trial 2: Incorrect 20 s
- Claude, trial 3: Correct 20.1 s
1 of 3 trials correct
Gemini
- Gemini, trial 1: Correct 19.1 s
- Gemini, trial 2: Correct 19 s
- Gemini, trial 3: Correct 19 s
3 of 3 trials correct
Copilot
- Copilot, trial 1: Correct 20.2 s
- Copilot, trial 2: Correct 20.1 s
- Copilot, trial 3: Incorrect 13.9 s
2 of 3 trials correct

Full task promptWe sent the original Japanese version of this prompt (1,463 Japanese characters) unchanged to all 4 AIs. The text shown here is our translation.

Read the meeting minutes below and output JSON only. Rules - Output JSON only. No explanatory text before or after - Use this shape: {"decided": ["…"], "todo": ["…"], "pending": ["…"]} - decided = policies decided at this meeting. todo = tasks assigned by name (e.g., “Mr./Ms. X, please...”). pending = items where someone said “we won't decide this today” - Do not include reports of things already done, numbers that were simply shared, or the date of the next meeting, in any of these lists - Each todo item should be in the form Owner: what to do (deadline) - Keep every number that appears in the minutes (amounts, headcounts, dates, percentages) ---- Meeting minutes start ---- ■ Product planning and sales regular meeting (October 29, 2026, 10:00–11:30, Meeting Room B) Attendees: Suzuki, Takahashi, Nakamura, Tanaka, Ito, Yamamoto Minutes: Yamamoto * These minutes are a fictional document created by AInformation to test AI tools. They have no connection to any real company or person. ■ Discussion Ito: In last month's internal survey, 71% of staff said they want to keep working from home. The results have already been shared with everyone. Yamamoto: Tanaka, please check the cost of the accessories and share it by December 20. Tanaka: We will not exhibit at the Nagoya trade show. Instead, we will hold our own online business meeting on October 20. Nakamura: We won't decide today on adding another logistics warehouse. We'll research candidate site rents and bring it back next time. Suzuki: The new catalog was sent to print on November 20. The cover photo has also been replaced. Yamamoto: Tanaka, please create a monthly complaint-tally template by November 25. Ito: Warehouse operations are fine for now. We expect shipments to peak in the third week of October. Suzuki: We won't decide today on next term's organization. It's not on today's agenda, so we'll carry it over to next time. Takahashi: We have decided to open a store in Fukuoka. The condition is that the rent for the property near the station drops to ¥1,000,000. If it doesn't drop, we will shelve it. Nakamura: We also won't decide today on producing novelties. The budget hasn't been set, so it's on hold. Yamamoto: Tanaka, please arrange the gift photo shoot. Finish shooting by December 20. Tanaka: We'll narrow the year-end gift down to one item: an assortment of Product B (eco bag), priced at ¥5,240. The two-bag set idea is shelved for now. Ito: There were 8 complaints last week, 3 more than the previous month. Loose stitching caused 9 of them; the rest were due to crushed boxes during shipping. Tanaka: A client asked whether we could shorten the lead time from the current 7 days to 5 days. The factory has told us that's difficult. Tanaka: We'll focus on 3 priorities for the second half: turning around Product E (insulated bottle), raising the e-commerce ratio to 39%, and cutting logistics cost as a share of sales by 0.7 points. Suzuki: For Product A (tumbler), let's start selling it as a set with accessories from October. We'll include a replacement gasket and a dedicated brush, and keep the price unchanged. Please keep the accessory cost at ¥195 or less per unit. Yamamoto: Copy paper was ordered last week. It will arrive within this month. Yamamoto: Suzuki, please start recruiting short-term part-time staff on December 5. Yamamoto: E-commerce sales in December were ¥31,100,000, up 16% from the same month last year. ■ Next meeting: November 15, 10:00, Meeting Room B End ---- Meeting minutes end ----

Answer key set in advanceWe set it when we created the task. 4 of the 4 AIs got it right at least once

(Machine-scored set_match, checked category by category) decided: set sale / gift / priorities / store opening / trade show; todo: Tanaka / Suzuki / Tanaka / Tanaka; pending: logistics warehouse / novelties / next term's organization

Scoring detailsWe show, trial by trial, why each verdict was given (9 of 12 trials correct)

ChatGPT, trial 1: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (23.7 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 2: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (32.8 seconds) · Full text of this answer (Japanese)
ChatGPT, trial 3: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (26.5 seconds) · Full text of this answer (Japanese)
Claude, trial 1: Incorrect: Mismatched in every category (decided 0/5, todo 0/4, pending 0/3) (20.1 seconds) · Full text of this answer (Japanese)
Claude, trial 2: Incorrect: Mismatched in every category (decided 0/5, todo 0/4, pending 0/3) (20 seconds) · Full text of this answer (Japanese)
Claude, trial 3: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (20.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 1: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (19.1 seconds) · Full text of this answer (Japanese)
Gemini, trial 2: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (19 seconds) · Full text of this answer (Japanese)
Gemini, trial 3: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (19 seconds) · Full text of this answer (Japanese)
Copilot, trial 1: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (20.2 seconds) · Full text of this answer (Japanese)
Copilot, trial 2: Correct: Matched in every category (decided 5/5, todo 4/4, pending 3/3) (20.1 seconds) · Full text of this answer (Japanese)
Copilot, trial 3: Incorrect: Mismatched in every category (decided 0/5, todo 0/4, pending 0/3) (13.9 seconds) · Full text of this answer (Japanese)

ANSWERS

Full answers from the 4 AIs

These are the answers we scored, posted as is without edits. That way, anyone can recount the results the same way we did.

ChatGPT

18 answers15 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Correct

t2, trial 2

Full text file (Japanese) · Correct

t2, trial 3

Full text file (Japanese) · Correct

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Correct

t3, trial 3

Full text file (Japanese) · Correct

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Correct

t6, trial 2

Full text file (Japanese) · Correct

t6, trial 3

Full text file (Japanese) · Correct
Claude

18 answers10 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Incorrect

t2, trial 2

Full text file (Japanese) · Incorrect

t2, trial 3

Full text file (Japanese) · Incorrect

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Correct

t3, trial 3

Full text file (Japanese) · Correct

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Incorrect

t6, trial 2

Full text file (Japanese) · Incorrect

t6, trial 3

Full text file (Japanese) · Correct
Gemini

18 answers10 of them correct

t1, trial 1

Full text file (Japanese) · Correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Incorrect

t2, trial 2

Full text file (Japanese) · Incorrect

t2, trial 3

Full text file (Japanese) · Incorrect

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Incorrect

t3, trial 3

Full text file (Japanese) · Incorrect

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Correct

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Correct

t6, trial 2

Full text file (Japanese) · Correct

t6, trial 3

Full text file (Japanese) · Correct
Copilot

16 answers10 of them correct

t1, trial 2

Full text file (Japanese) · Correct

t1, trial 3

Full text file (Japanese) · Correct

t2, trial 1

Full text file (Japanese) · Correct

t2, trial 3

Full text file (Japanese) · Correct

t3, trial 1

Full text file (Japanese) · Correct

t3, trial 2

Full text file (Japanese) · Correct

t3, trial 3

Full text file (Japanese) · Incorrect

t4, trial 1

Full text file (Japanese) · Correct

t4, trial 2

Full text file (Japanese) · Correct

t4, trial 3

Full text file (Japanese) · Incorrect

t5, trial 1

Full text file (Japanese) · Incorrect

t5, trial 2

Full text file (Japanese) · Incorrect

t5, trial 3

Full text file (Japanese) · Incorrect

t6, trial 1

Full text file (Japanese) · Correct

t6, trial 2

Full text file (Japanese) · Correct

t6, trial 3

Full text file (Japanese) · Incorrect

MATERIALS

Materials used in the tasks

Meeting minutes from a fictional company, Sample Trading Co. A mix of decisions, action items, deferred items, and small talk. Recreated for each run We recreated them for this run on October 28, 2026. Download them to repeat the same test yourself.

kaigi_kiroku.txt 3KB (Japanese)
kaigi_kotae.txt 1KB (Japanese)

METHOD

How we measure

We list this run's conditions exactly as they were, so anyone can repeat it.

AIs measured and their model names at the time

AI services change under the hood without notice. We record the model name shown on screen and the time we checked it.

AIs measured and model names
AI	Plan	Model name shown on screen	Time checked
ChatGPT	ChatGPT Pro	GPT-6.1 Sol Light	2026-10-28 19:49
Gemini	Google AI Pro	Flash-Lite	2026-10-28 19:49
Claude	Claude Max	Opus 5.5 Medium	2026-10-28 19:49
Copilot	Microsoft 365 Personal		2026-10-28 19:49

What we did in this run

We recreated the materials for the tasks specifically for this run. The data are for a fictional company and include no real companies or people.
We created 6 tasks and set the answer keys at that point.
We sent the same prompt and the same materials to ChatGPT, Gemini, Claude and Copilot.
We sent each task 3 times, in a new chat each time.
We rotated the sending order from run to run. This keeps the time of day from skewing the ranking.
We waited up to 300 seconds for each answer, and recorded anything longer as a timeout.
Scoring: Machine scoring only (not graded by AI).
We sent 72 trials in total, and 72 of them were valid.

How we count trials with no answer

Refused or timed out
We keep it in the base and count it as wrong. Because at work, that is the same as getting no answer.
Hit the plan's usage limit
We remove it from the base. We treat that trial as not measured.
A human verification screen appeared
We stop there. We neither click through it automatically nor try to get around it.
Operating error on AInformation's side
We remove it from the base. We do not report it as the AI's fault.

LIMITS

Limits of this benchmark

So that no one judges by the numbers alone, we list what we could not measure.

This is a single run on a single day. Rankings can change with the day or the task
The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier
Measurements use only the default model and default mode (features such as Deep Research are not used)
The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company
Only tasks that can be scored by machine are covered. Writing quality and readability are not measured
Measurements are taken from the same screen a person would use in a browser. These are not results from the API
For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is

DATA

Published data

In the CSV, 1 row is 1 trial. The figures on this page match a recount of that CSV. The license is CC BY 4.0.

Latest version (CSV) (Japanese) 1 row = 1 trial, 13KB
Latest version (JSON) 57KB
v1 (version at publication) (CSV) (Japanese) 1 row = 1 trial, 13KB
v1 (version at publication) (JSON) 57KB

The cumulative rankings are at /en/data/research/bench/leaderboard.json. We do not delete versions (such as v1). That way, links from people who cited them keep working.

FAQ

How do you measure?: We send exactly the same task prompt and the same materials to the 4 AIs on the same day. We open a new chat every time so earlier answers don't carry over. The sending order is rotated in each run. We also record the seconds from sending until the answer stops.
Why do you send the same task 3 times?: Because AI answers vary even for the same question. From a single try, you can't tell whether a result reflects real ability or chance. We send each task 3 times, and the share of tasks whose conclusion matched is reported as the consistency rate.
Who does the scoring?: Scoring is done only by a program. We do not let AI score the answers. The correct answers are set when the tasks are created, and whether an answer is right is decided by whether numbers match, whether required words are present, whether the format fits, whether URLs actually open and the result of running formulas. Matters of taste are not scored.
Is the data in the tasks from real companies?: No. We create new data for a fictional company every time. We do not use real companies, real people or text from other companies. We share the materials we used as they are, so you can repeat the same test.
How do you count cases where the AI couldn't answer?: Refusals and timeouts stay in the base and are counted as wrong. If we hit the plan's usage limit or a human verification screen appears, we remove that trial from the base and treat it as not measured. Operating errors on AInformation's side are also removed from the base, and we do not report them as the AI's fault.
Do you use the free or paid versions?: We use each AI's paid plan through the browser interface. We don't use the API. For each run, we record the plan name, the model name shown on screen and the time we checked it.
Does this ranking show which AI is the smartest?: No. We measure only 5 things on a fixed set of work tasks: accuracy, time to answer, consistency rate, real-source rate and instruction compliance. We do not measure writing style, readability or behavior in long conversations. If the tasks change, the ranking changes too.
Can I cite these results?: Yes. The license is CC BY 4.0. As the source, please write “Source: AInformation, AI Practical Work Benchmark” with the URL of the page you used and the measurement date. The results are also available as JSON and CSV. With the CSV (1 row per trial), you can recount the figures we publish.

CITATION

Cite this benchmark

Source: AInformation, “AI Practical Work Benchmark: Extracting decisions and action items from meeting minutes (4 AIs, 6 tasks, 3 runs each)” (https://ainformation.jp/en/ai-surveys/ai-work-benchmark/meeting-minutes-2026-10-29). Measured on October 29, 2026. 6 tasks, 3 trials each, 72 valid trials

When you cite a figure, please include the base too (how many of how many trials). Please cite AInformation as the source, along with the URL of the page you used and the measurement date.

What this run found

Scores by AI

Results by task

Full answers from the 4 AIs

Materials used in the tasks

How we measure

AIs measured and their model names at the time

What we did in this run

How we count trials with no answer

Limits of this benchmark

Published data

FAQ

Cite this benchmark

Related pages