{"bench_id":"b011-gazou","slug":"gazou-2026-10-24","title":"Reading numbers from images of receipts, charts, and tables (4 AIs, 6 tasks, 3 runs each)","field":"Reading images and photos","field_key":"gazou","round":2,"run_date":"2026-10-24","timezone":"Asia/Tokyo","version":"v1","conductor":"AInformation","funder":"AInformation (self-funded)","sponsor":null,"method":{"tasks_n":6,"repeats":3,"ais_n":4,"trials_total":72,"trials_valid":72,"excluded":[],"new_chat_each_time":true,"order_rotated":true,"order_by_repeat":{"1":["ChatGPT","Gemini","Claude","Copilot"],"2":["Gemini","Claude","Copilot","ChatGPT"],"3":["Claude","Copilot","ChatGPT","Gemini"]},"timeout_seconds":300,"scoring":"Machine scoring only (not graded by AI)","materials_generated_at":"2026-10-23","materials_note":"Original images we created ourselves: a receipt photo, a bar chart image, and a slightly tilted table image. The original numbers were set when we created them","counted_outcomes":["scored","refused","timeout"],"excluded_outcomes":["limit","our_error","human_check"],"answer_cleanup":"When reading the answer off the screen, some AIs put a line meant to be read aloud, or a mid-thought summary, in the same box as the answer. Before scoring, we remove the part that appears at the start of the answer and then appears again, identical, later on. All four AIs are treated the same way. The uncleaned answers are kept as-is in answers/","csv_url":"/data/research/bench/gazou-2026-10-24/latest.csv"},"ais":[{"name":"ChatGPT","plan":"ChatGPT Pro","model_shown":"GPT-6.1 Sol Light","checked_at":"2026-10-23 20:56"},{"name":"Gemini","plan":"Google AI Pro","model_shown":"Flash","checked_at":"2026-10-23 20:56"},{"name":"Claude","plan":"Claude Max","model_shown":"Opus 4.8 high","checked_at":"2026-10-23 20:56"},{"name":"Copilot","plan":"Microsoft 365 Personal","model_shown":null,"checked_at":"2026-10-23 20:56"}],"summary":{"headline":"We gave 6 tasks to AIs, with 3 trials each. ChatGPT had the highest accuracy, passing 14 of 18 trials (77.8%); second was Gemini at 66.7%","winner":"ChatGPT","note":"Results from a single round of testing. Rankings may change on a different day or with different tasks"},"scores":[{"ai":"ChatGPT","correct":14,"trials":18,"accuracy":0.77780000000000004689582056016661226749420166015625,"accuracy_display":77.7999999999999971578290569595992565155029296875,"seconds_median":36,"seconds_min":23.5,"seconds_max":51.60000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":1,"limit":0,"our_error":0,"human_check":0}},{"ai":"Gemini","correct":12,"trials":18,"accuracy":0.66669999999999995932142837773426435887813568115234375,"accuracy_display":66.7000000000000028421709430404007434844970703125,"seconds_median":183.69999999999998863131622783839702606201171875,"seconds_min":89.099999999999994315658113919198513031005859375,"seconds_max":254.30000000000001136868377216160297393798828125,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":3,"limit":0,"our_error":0,"human_check":0}},{"ai":"Claude","correct":10,"trials":18,"accuracy":0.55559999999999998276933865781757049262523651123046875,"accuracy_display":55.60000000000000142108547152020037174224853515625,"seconds_median":13.9000000000000003552713678800500929355621337890625,"seconds_min":13.800000000000000710542735760100185871124267578125,"seconds_max":20.10000000000000142108547152020037174224853515625,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":0.1666999999999999870770039933631778694689273834228515625,"instruction_follow_display":16.699999999999999289457264239899814128875732421875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Copilot","correct":12,"trials":18,"accuracy":0.66669999999999995932142837773426435887813568115234375,"accuracy_display":66.7000000000000028421709430404007434844970703125,"seconds_median":14.199999999999999289457264239899814128875732421875,"seconds_min":13.800000000000000710542735760100185871124267578125,"seconds_max":17.10000000000000142108547152020037174224853515625,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":1,"limit":0,"our_error":0,"human_check":0}}],"tasks":[{"task_id":"t1","level":"易","prompt":"In the attached bar chart image, what is the value of the tallest bar?\n\nRules\n- Answer in exactly one line. Write only the number, using half-width characters\n- Do not write the month name or any explanation","scorer":"exact_number","answer_key":"(Automated scoring, exact_number) Value of the tallest bar = 9700 (margin ±0)","materials":["/data/research/bench/gazou-2026-10-24/materials/chart.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":35.89999999999999857891452847979962825775146484375,"chars":4,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/chatgpt_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":51.60000000000000142108547152020037174224853515625,"chars":4,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.7999999999999971578290569595992565155029296875,"chars":4,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":183.69999999999998863131622783839702606201171875,"chars":5,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_gemini_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/gemini_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":95.2999999999999971578290569595992565155029296875,"chars":5,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":183.599999999999994315658113919198513031005859375,"chars":5,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":36,"why":"Numbers do not match: value of the tallest bar incorrect (correct answer: 9700, given answer: 97009700.0)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_claude_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/claude_01_t1.png"},{"n":2,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":36,"why":"Numbers do not match: value of the tallest bar incorrect (correct answer: 9700, given answer: 97009700.0)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":36,"why":"Numbers do not match: value of the tallest bar incorrect (correct answer: 9700, given answer: 97009700.0)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":4,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_copilot_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/copilot_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":4,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":4,"why":"All numbers match: value of the tallest bar correct (correct answer: 9700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t1_copilot_3.txt","shot":null}]}]},{"task_id":"t2","level":"易","prompt":"Read the total amount from the attached receipt image.\n\nRules\n- Answer in exactly one line, in the form \"Total: X\"\n- Write numbers using half-width characters. Do not write any explanation","scorer":"exact_number","answer_key":"(Automated scoring, exact_number) Total = 1440 (margin ±0)","materials":["/data/research/bench/gazou-2026-10-24/materials/receipt.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":23.5,"chars":8,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/chatgpt_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":35.89999999999999857891452847979962825775146484375,"chars":8,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":39.10000000000000142108547152020037174224853515625,"chars":8,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":214.099999999999994315658113919198513031005859375,"chars":10,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_gemini_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/gemini_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":214,"chars":10,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":183.900000000000005684341886080801486968994140625,"chars":10,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":42,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_claude_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/claude_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":42,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":42,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":8,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_copilot_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/copilot_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":8,"why":"All numbers match: total correct (correct answer: 1440)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t2_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":301.6000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]}]},{"task_id":"t3","level":"中","prompt":"In the attached table image (slightly tilted), calculate the monthly total by adding up the 5 products for each month.\n\nRules\n- One line per month, in the form \"Month: XXX\", listed from top to bottom\n- Write numbers using half-width characters. Do not show any calculation steps","scorer":"exact_number","answer_key":"(Automated scoring, exact_number) 2025-07 = 58700 (margin ±0) / 2025-08 = 59500 (margin ±0) / 2025-09 = 51200 (margin ±0) / 2025-10 = 43500 (margin ±0)","materials":["/data/research/bench/gazou-2026-10-24/materials/table_photo.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":36.10000000000000142108547152020037174224853515625,"chars":44,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/chatgpt_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":35.89999999999999857891452847979962825775146484375,"chars":40,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.7000000000000028421709430404007434844970703125,"chars":40,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":181.80000000000001136868377216160297393798828125,"chars":44,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_gemini_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/gemini_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":254.30000000000001136868377216160297393798828125,"chars":44,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":302.5,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17,"chars":203,"why":"Numbers do not match: 2025-07 incorrect (correct answer: 58700, no row for July) / 2025-08 incorrect (correct answer: 59500, no row for August) / 2025-09 incorrect (correct answer: 51200, no row for September) / 2025-10 incorrect (correct answer: 43500, no row for October)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_claude_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/claude_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.60000000000000142108547152020037174224853515625,"chars":157,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":20.10000000000000142108547152020037174224853515625,"chars":198,"why":"Numbers do not match: 2025-07 incorrect (correct answer: 58700, no row for July) / 2025-08 incorrect (correct answer: 59500, no row for August) / 2025-09 incorrect (correct answer: 51200, no row for September) / 2025-10 incorrect (correct answer: 43500, no row for October)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":14,"chars":40,"why":"Numbers do not match: 2025-07 incorrect (correct answer: 58700, given answer: 7.0) / 2025-08 incorrect (correct answer: 59500, given answer: 7.0) / 2025-09 incorrect (correct answer: 51200, given answer: 7.0) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_copilot_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/copilot_03_t3.png"},{"n":2,"ok":false,"outcome":"scored","seconds":14,"chars":40,"why":"Numbers do not match: 2025-07 incorrect (correct answer: 58700, given answer: 7.0) / 2025-08 incorrect (correct answer: 59500, given answer: 7.0) / 2025-09 incorrect (correct answer: 51200, given answer: 7.0) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":43,"why":"All numbers match: 2025-07 correct (correct answer: 58700) / 2025-08 correct (correct answer: 59500) / 2025-09 correct (correct answer: 51200) / 2025-10 correct (correct answer: 43500)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t3_copilot_3.txt","shot":null}]}]},{"task_id":"t4","level":"中","prompt":"What do you get when you add up all the values in the attached bar chart image?\n\nRules\n- Answer in exactly one line, in the form \"Total: XXX\"\n- Write numbers using half-width characters. Do not show any calculation steps","scorer":"exact_number","answer_key":"(Automated scoring, exact_number) Total for the 6 months = 54800 (margin ±0)","materials":["/data/research/bench/gazou-2026-10-24/materials/chart.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":false,"outcome":"timeout","seconds":302.30000000000001136868377216160297393798828125,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":"/data/research/bench/gazou-2026-10-24/shots/chatgpt_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":36,"chars":9,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.89999999999999857891452847979962825775146484375,"chars":9,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":232.599999999999994315658113919198513031005859375,"chars":10,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_gemini_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/gemini_04_t4.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":302.5,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":89.099999999999994315658113919198513031005859375,"chars":10,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":126,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_claude_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/claude_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":124,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":130,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":9,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":9,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":9,"why":"All numbers match: total for the 6 months correct (correct answer: 54800)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t4_copilot_3.txt","shot":null}]}]},{"task_id":"t5","level":"難","prompt":"Read the attached receipt image and output only JSON.\n\nRules\n- Output only JSON. Do not write any explanation before or after it\n- Use the format {\"items\": [{\"name\": \"item name\", \"price\": amount}], \"subtotal\": subtotal, \"tax\": sales tax, \"total\": total}\n- Enter numbers as numeric values (not as strings; do not include yen signs or commas)","scorer":"exact_number","answer_key":"(Automated scoring, exact_number) Subtotal = 1334 (margin ±0) / Sales tax = 106 (margin ±0) / Total = 1440 (margin ±0)","materials":["/data/research/bench/gazou-2026-10-24/materials/receipt.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":39.2000000000000028421709430404007434844970703125,"chars":199,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/chatgpt_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":39,"chars":199,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":35.89999999999999857891452847979962825775146484375,"chars":199,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":183.599999999999994315658113919198513031005859375,"chars":355,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_gemini_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/gemini_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":156.099999999999994315658113919198513031005859375,"chars":355,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":137.900000000000005684341886080801486968994140625,"chars":265,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":441,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_claude_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/claude_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17,"chars":449,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":17,"chars":527,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":199,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_copilot_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/copilot_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17,"chars":199,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":199,"why":"Numbers do not match: subtotal incorrect (correct answer: 1334, no subtotal row) / sales tax incorrect (correct answer: 106, no tax row) / total incorrect (correct answer: 1440, no total row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t5_copilot_3.txt","shot":null}]}]},{"task_id":"t6","level":"難","prompt":"What do you get when you add up all the values in the attached table image (slightly tilted)?\n\nRules\n- Answer in exactly one line, in the form \"Total: XXX\"\n- Do not add the month column. Write numbers using half-width characters. Do not show any calculation steps","scorer":"exact_number","answer_key":"(Automated scoring, exact_number) Total of all values in the table = 212900 (margin ±0)","materials":["/data/research/bench/gazou-2026-10-24/materials/table_photo.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":45.2999999999999971578290569595992565155029296875,"chars":10,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/chatgpt_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":10,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":10,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"timeout","seconds":302.3999999999999772626324556767940521240234375,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":"/data/research/bench/gazou-2026-10-24/shots/gemini_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":220.30000000000001136868377216160297393798828125,"chars":11,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":232.099999999999994315658113919198513031005859375,"chars":11,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17,"chars":144,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_claude_1.txt","shot":"/data/research/bench/gazou-2026-10-24/shots/claude_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17,"chars":70,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":76,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":10,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":17,"chars":10,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":10,"why":"All numbers match: total of all values in the table correct (correct answer: 212900)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-10-24/answers/t6_copilot_3.txt","shot":null}]}]}],"limits":["This is a single run on a single day. Rankings can change with the day or the task","The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier","Measurements use only the default model and default mode (features such as Deep Research are not used)","The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company","Only tasks that can be scored by machine are covered. Writing quality and readability are not measured","Measurements are taken from the same screen a person would use in a browser. These are not results from the API","For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is"],"license":"CC BY 4.0","attribution":"Please cite AInformation as the source, along with the URL of the page you used and the measurement date.","page_url":"https://ainformation.jp/research/bench/gazou-2026-10-24"}