{"bench_id":"b010-meibo","slug":"meibo-2026-10-19","title":"Standardize inconsistent spellings of client names (4 AIs, 6 tasks, 3 runs each)","field":"Data cleanup and name matching","field_key":"meibo","round":1,"run_date":"2026-10-19","timezone":"Asia/Tokyo","version":"v1","conductor":"AInformation","funder":"AInformation (self-funded)","sponsor":null,"method":{"tasks_n":6,"repeats":3,"ais_n":4,"trials_total":72,"trials_valid":72,"excluded":[],"new_chat_each_time":true,"order_rotated":true,"order_by_repeat":{"1":["ChatGPT","Gemini","Claude","Copilot"],"2":["Gemini","Claude","Copilot","ChatGPT"],"3":["Claude","Copilot","ChatGPT","Gemini"]},"timeout_seconds":300,"scoring":"Machine scoring only (not graded by AI)","materials_generated_at":"2026-10-18","materials_note":"A list of transactions with four fictional client companies, with names written inconsistently using (KK), the kanji abbreviation for incorporated, half-width katakana, spaces, and so on. The correct answer was fixed when the list was created.","counted_outcomes":["scored","refused","timeout"],"excluded_outcomes":["limit","our_error","human_check"],"answer_cleanup":"When reading the answer off the screen, some AIs put a line meant to be read aloud, or a mid-thought summary, in the same box as the answer. Before scoring, we remove the part that appears at the start of the answer and then appears again, identical, later on. All four AIs are treated the same way. The uncleaned answers are kept as-is in answers/","csv_url":"/data/research/bench/meibo-2026-10-19/latest.csv"},"ais":[{"name":"ChatGPT","plan":"ChatGPT Pro","model_shown":"GPT-6.1 Sol Light","checked_at":"2026-10-18 20:21"},{"name":"Gemini","plan":"Google AI Pro","model_shown":"Flash","checked_at":"2026-10-18 20:21"},{"name":"Claude","plan":"Claude Max","model_shown":"Opus 4.8 high","checked_at":"2026-10-18 20:21"},{"name":"Copilot","plan":"Microsoft 365 Personal","model_shown":null,"checked_at":"2026-10-18 20:21"}],"summary":{"headline":"Six tasks were given three times each for a total of 18 trials. ChatGPT had the highest accuracy rate, passing 15 of 18 trials (83.3%); Gemini was second at 77.8%","winner":"ChatGPT","note":"Results from a single round of testing. Rankings may change on a different day or with different tasks"},"scores":[{"ai":"ChatGPT","correct":15,"trials":18,"accuracy":0.83330000000000004067857162226573564112186431884765625,"accuracy_display":83.2999999999999971578290569595992565155029296875,"seconds_median":38.7000000000000028421709430404007434844970703125,"seconds_min":33,"seconds_max":54.60000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.83330000000000004067857162226573564112186431884765625,"instruction_follow_display":83.2999999999999971578290569595992565155029296875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Gemini","correct":14,"trials":18,"accuracy":0.77780000000000004689582056016661226749420166015625,"accuracy_display":77.7999999999999971578290569595992565155029296875,"seconds_median":63.5,"seconds_min":19.10000000000000142108547152020037174224853515625,"seconds_max":134.900000000000005684341886080801486968994140625,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":0.83330000000000004067857162226573564112186431884765625,"instruction_follow_display":83.2999999999999971578290569595992565155029296875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Claude","correct":12,"trials":18,"accuracy":0.66669999999999995932142837773426435887813568115234375,"accuracy_display":66.7000000000000028421709430404007434844970703125,"seconds_median":20.199999999999999289457264239899814128875732421875,"seconds_min":16.89999999999999857891452847979962825775146484375,"seconds_max":32.2000000000000028421709430404007434844970703125,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.333299999999999985167420391007908619940280914306640625,"instruction_follow_display":33.2999999999999971578290569595992565155029296875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Copilot","correct":12,"trials":18,"accuracy":0.66669999999999995932142837773426435887813568115234375,"accuracy_display":66.7000000000000028421709430404007434844970703125,"seconds_median":17.10000000000000142108547152020037174224853515625,"seconds_min":13.9000000000000003552713678800500929355621337890625,"seconds_max":17.39999999999999857891452847979962825775146484375,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.85709999999999997299937604111619293689727783203125,"instruction_follow_display":85.7000000000000028421709430404007434844970703125,"failures":{"refused":0,"timeout":4,"limit":0,"our_error":0,"human_check":0}}],"tasks":[{"task_id":"t1","level":"易","prompt":"The client names in the following transaction list are written inconsistently. How many companies are there once you group the identical ones together?\n\nRules\n- Give the answer in exactly one line, in the form \"N companies\"\n- Do not write the company names or any explanation\n\n---- Transaction list start ----\nID,Client name,Rep,Amount\nA001,Sample Transport Co., Ltd.,Takahashi,730000\nA011,(Ltd.) Sample Foods,Takahashi,563000\nA007,SAMPLE INDUSTRY (LTD.),Ito,761000\nA010,Sample Foods Co., Ltd.,Tanaka,231000\nA009,SAMPLE FOODS (Ltd.),Tanaka,799000\nA003,Sample Trading (Ltd.),Takahashi,250000\nA002,(Ltd.) Sample Transport,Ito,483000\nA004,SAMPLE TRADING (LTD.),Kato,502000\nA008,Sample Industry (Ltd.),Tanaka,474000\nA006,(Ltd.) Sample Trading,Kobayashi,641000\nA005,Sample Trading Co., Ltd. ,Nakamura,314000\n---- Transaction list end ----","scorer":"contains_all","answer_key":"(Auto-graded, contains_all) Number of companies after grouping = 4, or 4 companies, or 4 entries","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":33,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":36.2999999999999971578290569595992565155029296875,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.60000000000000142108547152020037174224853515625,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":127.5,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":61.60000000000000142108547152020037174224853515625,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":52.39999999999999857891452847979962825775146484375,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.5,"chars":124,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":64,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":62,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.39999999999999857891452847979962825775146484375,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_copilot_3.txt","shot":null}]}]},{"task_id":"t2","level":"易","prompt":"Group the client names in the following transaction list and list every company name.\n\nRules\n- One line per company. Start each line with \"- \"\n- Group identical companies into a single line. Do not write amounts or counts\n\n---- Transaction list start ----\nID,Client name,Rep,Amount\nA001,Sample Transport Co., Ltd.,Takahashi,730000\nA011,(Ltd.) Sample Foods,Takahashi,563000\nA007,SAMPLE INDUSTRY (LTD.),Ito,761000\nA010,Sample Foods Co., Ltd.,Tanaka,231000\nA009,SAMPLE FOODS (Ltd.),Tanaka,799000\nA003,Sample Trading (Ltd.),Takahashi,250000\nA002,(Ltd.) Sample Transport,Ito,483000\nA004,SAMPLE TRADING (LTD.),Kato,502000\nA008,Sample Industry (Ltd.),Tanaka,474000\nA006,(Ltd.) Sample Trading,Kobayashi,641000\nA005,Sample Trading Co., Ltd. ,Nakamura,314000\n---- Transaction list end ----","scorer":"set_match","answer_key":"(Auto-graded, set_match) Sample Trading [Sample Trading] / Sample Foods [Sample Foods] / Sample Industry [Sample Industry] / Sample Transport [Sample Transport] - up to 0 extra entries allowed","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":36.2999999999999971578290569595992565155029296875,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":36.39999999999999857891452847979962825775146484375,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":50.10000000000000142108547152020037174224853515625,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":26.10000000000000142108547152020037174224853515625,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":89.099999999999994315658113919198513031005859375,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.199999999999999289457264239899814128875732421875,"chars":263,"why":"All 4 items matched, 0 fabricated items","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":23.89999999999999857891452847979962825775146484375,"chars":298,"why":"All 4 items matched, 0 fabricated items","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":32.2000000000000028421709430404007434844970703125,"chars":229,"why":"All 4 items matched, 0 fabricated items","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.199999999999999289457264239899814128875732421875,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":47,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_copilot_3.txt","shot":null}]}]},{"task_id":"t3","level":"中","prompt":"For the following transaction list, give the number of entries per company after grouping identical companies together.\n\nRules\n- One line per company, in the form \"Company name: N entries\"\n- Sort from the highest count to the lowest. Do not write any explanation\n\n---- Transaction list start ----\nID,Client name,Rep,Amount\nA001,Sample Transport Co., Ltd.,Takahashi,730000\nA011,(Ltd.) Sample Foods,Takahashi,563000\nA007,SAMPLE INDUSTRY (LTD.),Ito,761000\nA010,Sample Foods Co., Ltd.,Tanaka,231000\nA009,SAMPLE FOODS (Ltd.),Tanaka,799000\nA003,Sample Trading (Ltd.),Takahashi,250000\nA002,(Ltd.) Sample Transport,Ito,483000\nA004,SAMPLE TRADING (LTD.),Kato,502000\nA008,Sample Industry (Ltd.),Tanaka,474000\nA006,(Ltd.) Sample Trading,Kobayashi,641000\nA005,Sample Trading Co., Ltd. ,Nakamura,314000\n---- Transaction list end ----","scorer":"exact_number","answer_key":"(Auto-graded, exact_number) Sample Trading = 4 (margin ±0) / Sample Foods = 3 (margin ±0) / Sample Industry = 2 (margin ±0) / Sample Transport = 2 (margin ±0)","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":39.7000000000000028421709430404007434844970703125,"chars":59,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.7999999999999971578290569595992565155029296875,"chars":59,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.7999999999999971578290569595992565155029296875,"chars":59,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":43,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":113.400000000000005684341886080801486968994140625,"chars":43,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":27,"why":"Numbers were wrong: Sample Trading × (correct answer 4, no row for Sample Trading) / Sample Foods × (correct answer 3, no row for Sample Foods) / Sample Industry × (correct answer 2, no row for Sample Industry) / Sample Transport × (correct answer 2, no row for Sample Transport)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.39999999999999857891452847979962825775146484375,"chars":121,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":119,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.89999999999999857891452847979962825775146484375,"chars":197,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.39999999999999857891452847979962825775146484375,"chars":43,"why":"All numbers matched: Sample Trading ○ (correct answer 4) / Sample Foods ○ (correct answer 3) / Sample Industry ○ (correct answer 2) / Sample Transport ○ (correct answer 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_03_t3.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":301.6000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":301,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]}]},{"task_id":"t4","level":"中","prompt":"For the following transaction list, give the total amount per company after grouping identical companies together.\n\nRules\n- One line per company, in the form \"Company name: N yen\"\n- Sort from the highest amount to the lowest. Write numbers using half-width digits. Do not write any explanation\n\n---- Transaction list start ----\nID,Client name,Rep,Amount\nA001,Sample Transport Co., Ltd.,Takahashi,730000\nA011,(Ltd.) Sample Foods,Takahashi,563000\nA007,SAMPLE INDUSTRY (LTD.),Ito,761000\nA010,Sample Foods Co., Ltd.,Tanaka,231000\nA009,SAMPLE FOODS (Ltd.),Tanaka,799000\nA003,Sample Trading (Ltd.),Takahashi,250000\nA002,(Ltd.) Sample Transport,Ito,483000\nA004,SAMPLE TRADING (LTD.),Kato,502000\nA008,Sample Industry (Ltd.),Tanaka,474000\nA006,(Ltd.) Sample Trading,Kobayashi,641000\nA005,Sample Trading Co., Ltd. ,Nakamura,314000\n---- Transaction list end ----","scorer":"exact_number","answer_key":"(Auto-graded, exact_number) Sample Trading = 1707000 (margin ±0) / Sample Foods = 1593000 (margin ±0) / Sample Industry = 1235000 (margin ±0) / Sample Transport = 1213000 (margin ±0)","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":39.39999999999999857891452847979962825775146484375,"chars":91,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":83,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":83,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":65.400000000000005684341886080801486968994140625,"chars":67,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":31.199999999999999289457264239899814128875732421875,"chars":67,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":76.900000000000005684341886080801486968994140625,"chars":67,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":26.10000000000000142108547152020037174224853515625,"chars":165,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":26.199999999999999289457264239899814128875732421875,"chars":161,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":23.199999999999999289457264239899814128875732421875,"chars":153,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":67,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_04_t4.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":301.1000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":67,"why":"All numbers matched: Sample Trading ○ (correct answer 1707000) / Sample Foods ○ (correct answer 1593000) / Sample Industry ○ (correct answer 1235000) / Sample Transport ○ (correct answer 1213000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_copilot_3.txt","shot":null}]}]},{"task_id":"t5","level":"難","prompt":"Group the client names in the following transaction list into a markdown table.\n\nRules\n- Exactly 3 columns, in the order: Company name / Count / Total amount\n- Group identical companies into a single row. Sort from the highest amount to the lowest\n- Output only the table. Do not write any preamble or explanation\n\n---- Transaction list start ----\nID,Client name,Rep,Amount\nA001,Sample Transport Co., Ltd.,Takahashi,730000\nA011,(Ltd.) Sample Foods,Takahashi,563000\nA007,SAMPLE INDUSTRY (LTD.),Ito,761000\nA010,Sample Foods Co., Ltd.,Tanaka,231000\nA009,SAMPLE FOODS (Ltd.),Tanaka,799000\nA003,Sample Trading (Ltd.),Takahashi,250000\nA002,(Ltd.) Sample Transport,Ito,483000\nA004,SAMPLE TRADING (LTD.),Kato,502000\nA008,Sample Industry (Ltd.),Tanaka,474000\nA006,(Ltd.) Sample Trading,Kobayashi,641000\nA005,Sample Trading Co., Ltd. ,Nakamura,314000\n---- Transaction list end ----","scorer":"exact_number","answer_key":"(Auto-graded, exact_number) Sample Trading = 1707000 (margin ±0) / Sample Foods = 1593000 (margin ±0) / Sample Industry = 1235000 (margin ±0) / Sample Transport = 1213000 (margin ±0)","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":107,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":107,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":107,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":119.5,"chars":103,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":80,"chars":87,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":134.900000000000005684341886080801486968994140625,"chars":103,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17,"chars":172,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17,"chars":170,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":20.10000000000000142108547152020037174224853515625,"chars":188,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":95,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17,"chars":95,"why":"Numbers were wrong: Sample Trading × (correct answer 1707000, answer given 41707000.0) / Sample Foods × (correct answer 1593000, answer given 31593000.0) / Sample Industry × (correct answer 1235000, answer given 21235000.0) / Sample Transport × (correct answer 1213000, answer given 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":301.80000000000001136868377216160297393798828125,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]}]},{"task_id":"t6","level":"難","prompt":"Group the client names in the following transaction list and output JSON only.\n\nRules\n- Output JSON only. Do not write any explanation before or after it\n- Use the form {\"companies\": [{\"name\": \"Company name\", \"count\": count, \"total\": total amount}]}\n- Group identical companies into a single entry. Enter numbers as numeric values (no comma separators)\n\n---- Transaction list start ----\nID,Client name,Rep,Amount\nA001,Sample Transport Co., Ltd.,Takahashi,730000\nA011,(Ltd.) Sample Foods,Takahashi,563000\nA007,SAMPLE INDUSTRY (LTD.),Ito,761000\nA010,Sample Foods Co., Ltd.,Tanaka,231000\nA009,SAMPLE FOODS (Ltd.),Tanaka,799000\nA003,Sample Trading (Ltd.),Takahashi,250000\nA002,(Ltd.) Sample Transport,Ito,483000\nA004,SAMPLE TRADING (LTD.),Kato,502000\nA008,Sample Industry (Ltd.),Tanaka,474000\nA006,(Ltd.) Sample Trading,Kobayashi,641000\nA005,Sample Trading Co., Ltd. ,Nakamura,314000\n---- Transaction list end ----","scorer":"set_match","answer_key":"(Auto-graded, set_match) Sample Trading [Sample Trading] / Sample Foods [Sample Foods] / Sample Industry [Sample Industry] / Sample Transport [Sample Transport] - up to 0 extra entries allowed","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":54.60000000000000142108547152020037174224853515625,"chars":207,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":207,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":42.2000000000000028421709430404007434844970703125,"chars":207,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":43.39999999999999857891452847979962825775146484375,"chars":357,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":116.5,"chars":357,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":357,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":26.10000000000000142108547152020037174224853515625,"chars":449,"why":"Could not be read as JSON (did not follow the format instructions)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_06_t6.png"},{"n":2,"ok":false,"outcome":"scored","seconds":20.199999999999999289457264239899814128875732421875,"chars":441,"why":"Could not be read as JSON (did not follow the format instructions)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":23.10000000000000142108547152020037174224853515625,"chars":439,"why":"Could not be read as JSON (did not follow the format instructions)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":207,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.4000000000000003552713678800500929355621337890625,"chars":207,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":191,"why":"All 4 items matched, 0 fabricated items","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_copilot_3.txt","shot":null}]}]}],"limits":["This is a single run on a single day. Rankings can change with the day or the task","The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier","Measurements use only the default model and default mode (features such as Deep Research are not used)","The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company","Only tasks that can be scored by machine are covered. Writing quality and readability are not measured","Measurements are taken from the same screen a person would use in a browser. These are not results from the API","For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is"],"license":"CC BY 4.0","attribution":"Please cite AInformation as the source, along with the URL of the page you used and the measurement date.","page_url":"https://ainformation.jp/research/bench/meibo-2026-10-19"}