{"bench_id":"b013-gazou","slug":"gazou-2026-11-03","title":"Reading numbers from images of receipts, charts, and tables (4 AIs, 6 tasks, 3 runs each)","field":"Reading images and photos","field_key":"gazou","round":1,"run_date":"2026-11-03","timezone":"Asia/Tokyo","version":"v1","conductor":"AInformation","funder":"AInformation (self-funded)","sponsor":null,"method":{"tasks_n":6,"repeats":3,"ais_n":4,"trials_total":72,"trials_valid":72,"excluded":[],"new_chat_each_time":true,"order_rotated":true,"order_by_repeat":{"1":["ChatGPT","Gemini","Claude","Copilot"],"2":["Gemini","Claude","Copilot","ChatGPT"],"3":["Claude","Copilot","ChatGPT","Gemini"]},"timeout_seconds":300,"scoring":"Machine scoring only (not graded by AI)","materials_generated_at":"2026-11-02","materials_note":"Original images we created ourselves: a receipt photo, a bar chart image, and a slightly tilted table image. The original numbers were set when we created them","counted_outcomes":["scored","refused","timeout"],"excluded_outcomes":["limit","our_error","human_check"],"answer_cleanup":"When reading the answer off the screen, some AIs put a line meant to be read aloud, or a mid-thought summary, in the same box as the answer. Before scoring, we remove the part that appears at the start of the answer and then appears again, identical, later on. All four AIs are treated the same way. The uncleaned answers are kept as-is in answers/","csv_url":"/data/research/bench/gazou-2026-11-03/latest.csv"},"ais":[{"name":"ChatGPT","plan":"ChatGPT Pro","model_shown":"GPT-6.1 Sol Medium","checked_at":"2026-11-02 21:03"},{"name":"Gemini","plan":"Google AI Pro","model_shown":"Flash","checked_at":"2026-11-02 21:03"},{"name":"Claude","plan":"Claude Max","model_shown":"Opus 4.8 High","checked_at":"2026-11-02 21:03"},{"name":"Copilot","plan":"Microsoft 365 Personal","model_shown":null,"checked_at":"2026-11-02 21:03"}],"summary":{"headline":"We gave 6 tasks to AIs, 3 times each; Claude had the highest accuracy at 100.0% (18/18 trials), with ChatGPT second at 72.2%","winner":"Claude","note":"Results from a single round of testing. Rankings may change on a different day or with different tasks"},"scores":[{"ai":"ChatGPT","correct":13,"trials":18,"accuracy":0.72219999999999995310417943983338773250579833984375,"accuracy_display":72.2000000000000028421709430404007434844970703125,"seconds_median":25.39999999999999857891452847979962825775146484375,"seconds_min":20.39999999999999857891452847979962825775146484375,"seconds_max":48.2999999999999971578290569595992565155029296875,"consistency":0.83330000000000004067857162226573564112186431884765625,"consistency_display":83.2999999999999971578290569595992565155029296875,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":4,"limit":0,"our_error":0,"human_check":0}},{"ai":"Gemini","correct":7,"trials":18,"accuracy":0.388900000000000023447910280083306133747100830078125,"accuracy_display":38.89999999999999857891452847979962825775146484375,"seconds_median":69.2000000000000028421709430404007434844970703125,"seconds_min":19,"seconds_max":204.900000000000005684341886080801486968994140625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.625,"instruction_follow_display":62.5,"failures":{"refused":0,"timeout":10,"limit":0,"our_error":0,"human_check":0}},{"ai":"Claude","correct":18,"trials":18,"accuracy":1,"accuracy_display":100,"seconds_median":13.9000000000000003552713678800500929355621337890625,"seconds_min":13.800000000000000710542735760100185871124267578125,"seconds_max":17.10000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.1666999999999999870770039933631778694689273834228515625,"instruction_follow_display":16.699999999999999289457264239899814128875732421875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Copilot","correct":0,"trials":18,"accuracy":0,"accuracy_display":0,"seconds_median":14,"seconds_min":13.800000000000000710542735760100185871124267578125,"seconds_max":17.10000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.5,"instruction_follow_display":50,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}}],"tasks":[{"task_id":"t1","level":"易","prompt":"Read 3 numbers from the attached receipt image.\n\nRules\n- The answer must be exactly 3 lines, in this order: Subtotal: N / Consumption tax: N / Total: N\n- Write numbers as plain digits. No yen sign or commas needed. Do not add any explanation","scorer":"exact_number","answer_key":"(Machine-graded exact_number) Subtotal = 2,316 (margin ±0) / Consumption tax = 185 (margin ±0) / Total = 2,501 (margin ±0)","materials":["/data/research/bench/gazou-2026-11-03/materials/receipt.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":35.7999999999999971578290569595992565155029296875,"chars":26,"why":"All figures match: Subtotal ○ (correct answer 2,316) / Consumption tax ○ (correct answer 185) / Total ○ (correct answer 2,501)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/chatgpt_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":29.699999999999999289457264239899814128875732421875,"chars":26,"why":"All figures match: Subtotal ○ (correct answer 2,316) / Consumption tax ○ (correct answer 185) / Total ○ (correct answer 2,501)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":20.5,"chars":26,"why":"All figures match: Subtotal ○ (correct answer 2,316) / Consumption tax ○ (correct answer 185) / Total ○ (correct answer 2,501)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":26,"why":"Figures are wrong: Subtotal × (correct answer 2,316; no “Subtotal” row) / Consumption tax × (correct answer 185; no “Tax” row) / Total × (correct answer 2,501; no “Total” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_gemini_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/gemini_01_t1.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":302.6000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":302.19999999999998863131622783839702606201171875,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":60,"why":"All figures match: Subtotal ○ (correct answer 2,316) / Consumption tax ○ (correct answer 185) / Total ○ (correct answer 2,501)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_claude_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/claude_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":60,"why":"All figures match: Subtotal ○ (correct answer 2,316) / Consumption tax ○ (correct answer 185) / Total ○ (correct answer 2,501)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":92,"why":"All figures match: Subtotal ○ (correct answer 2,316) / Consumption tax ○ (correct answer 185) / Total ○ (correct answer 2,501)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":14,"chars":46,"why":"Figures are wrong: Subtotal × (correct answer 2,316; no “Subtotal” row) / Consumption tax × (correct answer 185; no “Tax” row) / Total × (correct answer 2,501; no “Total” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_copilot_1.txt","shot":null},{"n":2,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"Figures are wrong: Subtotal × (correct answer 2,316; no “Subtotal” row) / Consumption tax × (correct answer 185; no “Tax” row) / Total × (correct answer 2,501; no “Total” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"Figures are wrong: Subtotal × (correct answer 2,316; no “Subtotal” row) / Consumption tax × (correct answer 185; no “Tax” row) / Total × (correct answer 2,501; no “Total” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t1_copilot_3.txt","shot":null}]}]},{"task_id":"t2","level":"易","prompt":"Read the change amount from the attached receipt image.\n\nRules\n- The answer must be exactly 1 line, in the form Change: N\n- Write numbers as plain digits. Do not add any explanation","scorer":"exact_number","answer_key":"(Machine-graded exact_number) Change = 499 (margin ±0)","materials":["/data/research/bench/gazou-2026-11-03/materials/receipt.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":24.10000000000000142108547152020037174224853515625,"chars":8,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/chatgpt_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":20.39999999999999857891452847979962825775146484375,"chars":8,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":20.39999999999999857891452847979962825775146484375,"chars":8,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":59.7000000000000028421709430404007434844970703125,"chars":23,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_gemini_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/gemini_02_t2.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":302.30000000000001136868377216160297393798828125,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":302.1000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":42,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_claude_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/claude_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":42,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":42,"why":"All figures match: Change ○ (correct answer 499)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":46,"why":"Figures are wrong: Change × (correct answer 499; no “Change” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_copilot_1.txt","shot":null},{"n":2,"ok":false,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":46,"why":"Figures are wrong: Change × (correct answer 499; no “Change” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":46,"why":"Figures are wrong: Change × (correct answer 499; no “Change” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t2_copilot_3.txt","shot":null}]}]},{"task_id":"t3","level":"中","prompt":"List every item purchased, based on the attached receipt image.\n\nRules\n- One item per line. Start each line with a bullet ( · ). Write the item name and the price\n- Do not include the subtotal, sales tax, total, amount received, or change lines","scorer":"set_match","answer_key":"(Machine-graded set_match) Milk 1000ml [Milk +314] / Yogurt 400g [Yogurt +184] / Udon 3 servings [Udon +184] / Chicken breast 2 pieces [Chicken breast +454] / Detergent refill pack [Detergent +496] / Coffee 200g [Coffee +684] — up to 0 extra items allowed","materials":["/data/research/bench/gazou-2026-11-03/materials/receipt.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":27.199999999999999289457264239899814128875732421875,"chars":89,"why":"All 6 items match, 0 extra items added","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/chatgpt_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":32.7999999999999971578290569595992565155029296875,"chars":89,"why":"All 6 items match, 0 extra items added","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":300.6000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":189.80000000000001136868377216160297393798828125,"chars":89,"why":"All 6 items match, 0 extra items added","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_gemini_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/gemini_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19,"chars":89,"why":"All 6 items match, 0 extra items added","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":302.1000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17,"chars":198,"why":"All 6 items match, 0 extra items added","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_claude_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/claude_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":149,"why":"All 6 items match, 0 extra items added","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":163,"why":"All 6 items match, 0 extra items added","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":46,"why":"0 of 6 items match; missing Milk 1000ml / Yogurt 400g / Udon 3 servings / Chicken breast 2 pieces / Detergent refill pack / Coffee 200g; 1 extra item added","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_copilot_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/copilot_03_t3.png"},{"n":2,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"0 of 6 items match; missing Milk 1000ml / Yogurt 400g / Udon 3 servings / Chicken breast 2 pieces / Detergent refill pack / Coffee 200g; 1 extra item added","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"0 of 6 items match; missing Milk 1000ml / Yogurt 400g / Udon 3 servings / Chicken breast 2 pieces / Detergent refill pack / Coffee 200g; 1 extra item added","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t3_copilot_3.txt","shot":null}]}]},{"task_id":"t4","level":"中","prompt":"Looking at the attached bar chart image, which month has the tallest bar?\n\nRules\n- The answer must be exactly 1 line: just the month name (for example, January)\n- Do not include numeric values or any explanation","scorer":"contains_all","answer_key":"(Machine-graded contains_all) Highest month = June","materials":["/data/research/bench/gazou-2026-11-03/materials/chart.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":26.699999999999999289457264239899814128875732421875,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/chatgpt_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":48.2999999999999971578290569595992565155029296875,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":300.80000000000001136868377216160297393798828125,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":61.7000000000000028421709430404007434844970703125,"chars":2,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_gemini_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/gemini_04_t4.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":302.19999999999998863131622783839702606201171875,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":76.7000000000000028421709430404007434844970703125,"chars":15,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":30,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_claude_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/claude_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":30,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":30,"why":"The one item was present","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"0 of 1 item present; missing “highest month”","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_copilot_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/copilot_04_t4.png"},{"n":2,"ok":false,"outcome":"scored","seconds":14,"chars":46,"why":"0 of 1 item present; missing “highest month”","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":14,"chars":46,"why":"0 of 1 item present; missing “highest month”","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t4_copilot_3.txt","shot":null}]}]},{"task_id":"t5","level":"難","prompt":"From the attached bar chart image, read off every monthly value.\n\nRules\n- One line per month, in the format Month: Value, listed from left to right\n- Write numbers using standard digits. Do not add any explanation","scorer":"exact_number","answer_key":"(Machine-graded exact_number) January = 5,500 (margin ±0) / February = 5,500 (margin ±0) / March = 8,200 (margin ±0) / April = 12,300 (margin ±0) / May = 17,800 (margin ±0) / June = 24,700 (margin ±0)","materials":["/data/research/bench/gazou-2026-11-03/materials/chart.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.5,"chars":56,"why":"All figures match: January ○ (correct answer 5,500) / February ○ (correct answer 5,500) / March ○ (correct answer 8,200) / April ○ (correct answer 12,300) / May ○ (correct answer 17,800) / June ○ (correct answer 24,700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/chatgpt_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":26.60000000000000142108547152020037174224853515625,"chars":56,"why":"All figures match: January ○ (correct answer 5,500) / February ○ (correct answer 5,500) / March ○ (correct answer 8,200) / April ○ (correct answer 12,300) / May ○ (correct answer 17,800) / June ○ (correct answer 24,700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":300.80000000000001136868377216160297393798828125,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":204.900000000000005684341886080801486968994140625,"chars":62,"why":"All figures match: January ○ (correct answer 5,500) / February ○ (correct answer 5,500) / March ○ (correct answer 8,200) / April ○ (correct answer 12,300) / May ○ (correct answer 17,800) / June ○ (correct answer 24,700)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_gemini_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/gemini_05_t5.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":302.1000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":302.30000000000001136868377216160297393798828125,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":90,"why":"All figures match: January ○ (correct answer 5,500) / February ○ (correct answer 5,500) / March ○ (correct answer 8,200) / April ○ (correct answer 12,300) / May ○ (correct answer 17,800) / June ○ (correct answer 24,700)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_claude_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/claude_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":90,"why":"All figures match: January ○ (correct answer 5,500) / February ○ (correct answer 5,500) / March ○ (correct answer 8,200) / April ○ (correct answer 12,300) / May ○ (correct answer 17,800) / June ○ (correct answer 24,700)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":98,"why":"All figures match: January ○ (correct answer 5,500) / February ○ (correct answer 5,500) / March ○ (correct answer 8,200) / April ○ (correct answer 12,300) / May ○ (correct answer 17,800) / June ○ (correct answer 24,700)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":14,"chars":46,"why":"Figures are wrong: January × (correct answer 5,500; no “January” row) / February × (correct answer 5,500; no “February” row) / March × (correct answer 8,200; no “March” row) / April × (correct answer 12,300; no “April” row) / May × (correct answer 17,800; no “May” row) / June × (correct answer 24,700; no “June” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_copilot_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/copilot_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"Figures are wrong: January × (correct answer 5,500; no “January” row) / February × (correct answer 5,500; no “February” row) / March × (correct answer 8,200; no “March” row) / April × (correct answer 12,300; no “April” row) / May × (correct answer 17,800; no “May” row) / June × (correct answer 24,700; no “June” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":46,"why":"Figures are wrong: January × (correct answer 5,500; no “January” row) / February × (correct answer 5,500; no “February” row) / March × (correct answer 8,200; no “March” row) / April × (correct answer 12,300; no “April” row) / May × (correct answer 17,800; no “May” row) / June × (correct answer 24,700; no “June” row)","instruction_follow":false,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t5_copilot_3.txt","shot":null}]}]},{"task_id":"t6","level":"難","prompt":"Read all the values in the Product A column from the attached table image (which is slightly tilted).\n\nRules\n- One value per line, in the form Month: N, from top to bottom\n- Write numbers as plain digits. Do not add any explanation","scorer":"exact_number","answer_key":"(Machine-graded exact_number) 2025-07 = 32,900 (margin ±0) / 2025-08 = 35,600 (margin ±0) / 2025-09 = 23,300 (margin ±0) / 2025-10 = 11,000 (margin ±0)","materials":["/data/research/bench/gazou-2026-11-03/materials/table_photo.png"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":20.60000000000000142108547152020037174224853515625,"chars":40,"why":"Figures are wrong: 2025-07 × (correct answer 32,900; no “2025-07” row) / 2025-08 × (correct answer 35,600; no “2025-08” row) / 2025-09 × (correct answer 23,300; no “2025-09” row) / 2025-10 × (correct answer 11,000; no “2025-10” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_chatgpt_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/chatgpt_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":20.5,"chars":59,"why":"All figures match: 2025-07 ○ (correct answer 32,900) / 2025-08 ○ (correct answer 35,600) / 2025-09 ○ (correct answer 23,300) / 2025-10 ○ (correct answer 11,000)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":301.19999999999998863131622783839702606201171875,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":134.900000000000005684341886080801486968994140625,"chars":63,"why":"All figures match: 2025-07 ○ (correct answer 32,900) / 2025-08 ○ (correct answer 35,600) / 2025-09 ○ (correct answer 23,300) / 2025-10 ○ (correct answer 11,000)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_gemini_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/gemini_06_t6.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":302.1000000000000227373675443232059478759765625,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":302.5,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":124,"why":"All figures match: 2025-07 ○ (correct answer 32,900) / 2025-08 ○ (correct answer 35,600) / 2025-09 ○ (correct answer 23,300) / 2025-10 ○ (correct answer 11,000)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_claude_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/claude_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17,"chars":126,"why":"All figures match: 2025-07 ○ (correct answer 32,900) / 2025-08 ○ (correct answer 35,600) / 2025-09 ○ (correct answer 23,300) / 2025-10 ○ (correct answer 11,000)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":128,"why":"All figures match: 2025-07 ○ (correct answer 32,900) / 2025-08 ○ (correct answer 35,600) / 2025-09 ○ (correct answer 23,300) / 2025-10 ○ (correct answer 11,000)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":14,"chars":46,"why":"Figures are wrong: 2025-07 × (correct answer 32,900; no “2025-07” row) / 2025-08 × (correct answer 35,600; no “2025-08” row) / 2025-09 × (correct answer 23,300; no “2025-09” row) / 2025-10 × (correct answer 11,000; no “2025-10” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_copilot_1.txt","shot":"/data/research/bench/gazou-2026-11-03/shots/copilot_06_t6.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":46,"why":"Figures are wrong: 2025-07 × (correct answer 32,900; no “2025-07” row) / 2025-08 × (correct answer 35,600; no “2025-08” row) / 2025-09 × (correct answer 23,300; no “2025-09” row) / 2025-10 × (correct answer 11,000; no “2025-10” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":14,"chars":46,"why":"Figures are wrong: 2025-07 × (correct answer 32,900; no “2025-07” row) / 2025-08 × (correct answer 35,600; no “2025-08” row) / 2025-09 × (correct answer 23,300; no “2025-09” row) / 2025-10 × (correct answer 11,000; no “2025-10” row)","instruction_follow":true,"answer_url":"/data/research/bench/gazou-2026-11-03/answers/t6_copilot_3.txt","shot":null}]}]}],"limits":["This is a single run on a single day. Rankings can change with the day or the task","The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier","Measurements use only the default model and default mode (features such as Deep Research are not used)","The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company","Only tasks that can be scored by machine are covered. Writing quality and readability are not measured","Measurements are taken from the same screen a person would use in a browser. These are not results from the API","For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is"],"license":"CC BY 4.0","attribution":"Please cite AInformation as the source, along with the URL of the page you used and the measurement date.","page_url":"https://ainformation.jp/research/bench/gazou-2026-11-03"}