{"bench_id":"b008-suushiki","slug":"suushiki-2026-10-08","title":"Writing working spreadsheet formulas (4 AIs, 6 tasks, 3 runs each)","field":"Excel and spreadsheet formulas","field_key":"suushiki","round":1,"run_date":"2026-10-08","timezone":"Asia/Tokyo","version":"v1","conductor":"AInformation","funder":"AInformation (self-funded)","sponsor":null,"method":{"tasks_n":6,"repeats":3,"ais_n":4,"trials_total":72,"trials_valid":72,"excluded":[],"new_chat_each_time":true,"order_rotated":true,"order_by_repeat":{"1":["ChatGPT","Gemini","Claude","Copilot"],"2":["Gemini","Claude","Copilot","ChatGPT"],"3":["Claude","Copilot","ChatGPT","Gemini"]},"timeout_seconds":300,"scoring":"Machine scoring only (not graded by AI)","materials_generated_at":"2026-10-07","materials_note":"A spreadsheet of sales data for the fictional Sample Trading Co. (column A: month, columns B–F: products, column G: total, rows 2–13: data). We actually calculated each formula the AI produced to check the answer","counted_outcomes":["scored","refused","timeout"],"excluded_outcomes":["limit","our_error","human_check"],"answer_cleanup":"When reading the answer off the screen, some AIs put a line meant to be read aloud, or a mid-thought summary, in the same box as the answer. Before scoring, we remove the part that appears at the start of the answer and then appears again, identical, later on. All four AIs are treated the same way. The uncleaned answers are kept as-is in answers/","csv_url":"/data/research/bench/suushiki-2026-10-08/latest.csv"},"ais":[{"name":"ChatGPT","plan":"ChatGPT Pro","model_shown":"Medium","checked_at":"2026-10-07 19:36"},{"name":"Gemini","plan":"Google AI Pro","model_shown":"Flash","checked_at":"2026-10-07 19:36"},{"name":"Claude","plan":"Claude Max","model_shown":"Opus 4.8 high","checked_at":"2026-10-07 19:36"},{"name":"Copilot","plan":"Microsoft 365 Personal","model_shown":null,"checked_at":"2026-10-07 19:36"}],"summary":{"headline":"We gave 6 tasks 3 times each, and the highest accuracy was a tie among ChatGPT, Gemini, and Claude at 100.0%","winner":null,"note":"Results from a single round of testing. Rankings may change on a different day or with different tasks"},"scores":[{"ai":"ChatGPT","correct":18,"trials":18,"accuracy":1,"accuracy_display":100,"seconds_median":14.199999999999999289457264239899814128875732421875,"seconds_min":14,"seconds_max":20.39999999999999857891452847979962825775146484375,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Gemini","correct":18,"trials":18,"accuracy":1,"accuracy_display":100,"seconds_median":19.10000000000000142108547152020037174224853515625,"seconds_min":19,"seconds_max":19.60000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Claude","correct":18,"trials":18,"accuracy":1,"accuracy_display":100,"seconds_median":13.9000000000000003552713678800500929355621337890625,"seconds_min":13.800000000000000710542735760100185871124267578125,"seconds_max":16.5,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0,"instruction_follow_display":0,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Copilot","correct":17,"trials":18,"accuracy":0.94440000000000001723066134218242950737476348876953125,"accuracy_display":94.400000000000005684341886080801486968994140625,"seconds_median":14.0999999999999996447286321199499070644378662109375,"seconds_min":13.9000000000000003552713678800500929355621337890625,"seconds_max":26.199999999999999289457264239899814128875732421875,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":0.94440000000000001723066134218242950737476348876953125,"instruction_follow_display":94.400000000000005684341886080801486968994140625,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}}],"tasks":[{"task_id":"t1","level":"易","prompt":"The following table has been pasted into a spreadsheet starting at A1 (row 1 is the header, rows 2–13 are data).\nWrite a single formula that calculates the total for Product A (column B) for the full year.\n\nRules\n- Your answer must be exactly one line. Write only a formula starting with =\n- Do not include any explanation or the calculated result\n\n---- Table start ----\nMonth,Product A (Tumbler),Product B (Eco Bag),Product C (Cooling Towel),Product D (Lunch Box),Product E (Insulated Bottle),Total\n2025-01,9400,13600,10200,14100,5200,52500\n2025-02,9400,12800,11100,14400,5200,52900\n2025-03,10300,10200,12600,14600,7900,55600\n2025-04,9400,7600,10700,14800,11800,54300\n2025-05,9400,6000,10500,15100,17000,58000\n2025-06,9400,4200,10000,15300,23600,62500\n2025-07,9400,3400,9800,15600,31400,69600\n2025-08,9400,3400,9100,15800,34100,71800\n2025-09,9400,5100,10200,16000,22300,63000\n2025-10,9400,8500,9000,16300,10500,53700\n2025-11,9400,11900,8200,16500,6600,52600\n2025-12,10300,14400,9700,16700,5200,56300\n---- Table end ----","scorer":"formula_run","answer_key":"(Machine-graded: formula_run) We actually calculated the formula the AI produced and checked whether it matches Product A's annual total = 114,600","materials":["/data/research/bench/suushiki-2026-10-08/materials/uriage.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_chatgpt_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/chatgpt_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.300000000000000710542735760100185871124267578125,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_gemini_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/gemini_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.60000000000000142108547152020037174224853515625,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":54,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_claude_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/claude_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":54,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14,"chars":54,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":12,"why":"Ran the formula =SUM(B2:B13): Product A's annual total — Pass (formula result 114600.0, correct answer 114600.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t1_copilot_3.txt","shot":null}]}]},{"task_id":"t2","level":"易","prompt":"The following table has been pasted into a spreadsheet starting at A1 (row 1 is the header, rows 2–13 are data, column G is the monthly total).\nWrite a single formula that calculates the grand total of sales for the full year.\n\nRules\n- Your answer must be exactly one line. Write only a formula starting with =\n- Do not include any explanation or the calculated result\n\n---- Table start ----\nMonth,Product A (Tumbler),Product B (Eco Bag),Product C (Cooling Towel),Product D (Lunch Box),Product E (Insulated Bottle),Total\n2025-01,9400,13600,10200,14100,5200,52500\n2025-02,9400,12800,11100,14400,5200,52900\n2025-03,10300,10200,12600,14600,7900,55600\n2025-04,9400,7600,10700,14800,11800,54300\n2025-05,9400,6000,10500,15100,17000,58000\n2025-06,9400,4200,10000,15300,23600,62500\n2025-07,9400,3400,9800,15600,31400,69600\n2025-08,9400,3400,9100,15800,34100,71800\n2025-09,9400,5100,10200,16000,22300,63000\n2025-10,9400,8500,9000,16300,10500,53700\n2025-11,9400,11900,8200,16500,6600,52600\n2025-12,10300,14400,9700,16700,5200,56300\n---- Table end ----","scorer":"formula_run","answer_key":"(Machine-graded: formula_run) We actually calculated the formula the AI produced and checked whether it matches the annual total = 702,800","materials":["/data/research/bench/suushiki-2026-10-08/materials/uriage.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.699999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_chatgpt_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/chatgpt_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_gemini_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/gemini_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14,"chars":54,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_claude_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/claude_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":54,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":54,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_copilot_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/copilot_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":26.199999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =SUM(G2:G13): annual total — Pass (formula result 702800.0, correct answer 702800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t2_copilot_3.txt","shot":null}]}]},{"task_id":"t3","level":"中","prompt":"The following table has been pasted into a spreadsheet starting at A1 (column G is the monthly total).\nWrite a single formula that calculates the average of the monthly totals (across 12 months).\n\nRules\n- Your answer must be exactly one line. Write only a formula starting with =\n- Do not include any explanation or the calculated result\n\n---- Table start ----\nMonth,Product A (Tumbler),Product B (Eco Bag),Product C (Cooling Towel),Product D (Lunch Box),Product E (Insulated Bottle),Total\n2025-01,9400,13600,10200,14100,5200,52500\n2025-02,9400,12800,11100,14400,5200,52900\n2025-03,10300,10200,12600,14600,7900,55600\n2025-04,9400,7600,10700,14800,11800,54300\n2025-05,9400,6000,10500,15100,17000,58000\n2025-06,9400,4200,10000,15300,23600,62500\n2025-07,9400,3400,9800,15600,31400,69600\n2025-08,9400,3400,9100,15800,34100,71800\n2025-09,9400,5100,10200,16000,22300,63000\n2025-10,9400,8500,9000,16300,10500,53700\n2025-11,9400,11900,8200,16500,6600,52600\n2025-12,10300,14400,9700,16700,5200,56300\n---- Table end ----","scorer":"formula_run","answer_key":"(Machine-graded: formula_run) We actually calculated the formula the AI produced and checked whether it matches the average monthly total = 58,566.7","materials":["/data/research/bench/suushiki-2026-10-08/materials/uriage.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.39999999999999857891452847979962825775146484375,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_chatgpt_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/chatgpt_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.4000000000000003552713678800500929355621337890625,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_gemini_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/gemini_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":62,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_claude_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/claude_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":62,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":62,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":14,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":16,"why":"Ran the formula =AVERAGE(G2:G13): average monthly total — Pass (formula result 58566.666666666664, correct answer 58566.7)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t3_copilot_3.txt","shot":null}]}]},{"task_id":"t4","level":"中","prompt":"The following table has been pasted into a spreadsheet starting at A1 (column G is the monthly total).\nWrite a single formula that finds the largest value among the monthly totals.\n\nRules\n- Your answer must be exactly one line. Write only a formula starting with =\n- Do not include any explanation or the calculated result\n\n---- Table start ----\nMonth,Product A (Tumbler),Product B (Eco Bag),Product C (Cooling Towel),Product D (Lunch Box),Product E (Insulated Bottle),Total\n2025-01,9400,13600,10200,14100,5200,52500\n2025-02,9400,12800,11100,14400,5200,52900\n2025-03,10300,10200,12600,14600,7900,55600\n2025-04,9400,7600,10700,14800,11800,54300\n2025-05,9400,6000,10500,15100,17000,58000\n2025-06,9400,4200,10000,15300,23600,62500\n2025-07,9400,3400,9800,15600,31400,69600\n2025-08,9400,3400,9100,15800,34100,71800\n2025-09,9400,5100,10200,16000,22300,63000\n2025-10,9400,8500,9000,16300,10500,53700\n2025-11,9400,11900,8200,16500,6600,52600\n2025-12,10300,14400,9700,16700,5200,56300\n---- Table end ----","scorer":"formula_run","answer_key":"(Machine-graded: formula_run) We actually calculated the formula the AI produced and checked whether it matches the largest monthly total = 71,800","materials":["/data/research/bench/suushiki-2026-10-08/materials/uriage.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.199999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_chatgpt_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/chatgpt_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_gemini_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/gemini_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":54,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_claude_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/claude_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":54,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":54,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":12,"why":"Ran the formula =MAX(G2:G13): largest monthly total — Pass (formula result 71800.0, correct answer 71800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t4_copilot_3.txt","shot":null}]}]},{"task_id":"t5","level":"難","prompt":"The following table has been pasted into a spreadsheet starting at A1 (row 2 is January, row 13 is December, column G is the monthly total).\nWrite a single formula that calculates the grand total for the first half of the year (January through June).\n\nRules\n- Your answer must be exactly one line. Write only a formula starting with =\n- Do not include any explanation or the calculated result\n\n---- Table start ----\nMonth,Product A (Tumbler),Product B (Eco Bag),Product C (Cooling Towel),Product D (Lunch Box),Product E (Insulated Bottle),Total\n2025-01,9400,13600,10200,14100,5200,52500\n2025-02,9400,12800,11100,14400,5200,52900\n2025-03,10300,10200,12600,14600,7900,55600\n2025-04,9400,7600,10700,14800,11800,54300\n2025-05,9400,6000,10500,15100,17000,58000\n2025-06,9400,4200,10000,15300,23600,62500\n2025-07,9400,3400,9800,15600,31400,69600\n2025-08,9400,3400,9100,15800,34100,71800\n2025-09,9400,5100,10200,16000,22300,63000\n2025-10,9400,8500,9000,16300,10500,53700\n2025-11,9400,11900,8200,16500,6600,52600\n2025-12,10300,14400,9700,16700,5200,56300\n---- Table end ----","scorer":"formula_run","answer_key":"(Machine-graded: formula_run) We actually calculated the formula the AI produced and checked whether it matches the first-half total = 335,800","materials":["/data/research/bench/suushiki-2026-10-08/materials/uriage.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_chatgpt_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/chatgpt_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.39999999999999857891452847979962825775146484375,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.39999999999999857891452847979962825775146484375,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_gemini_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/gemini_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":52,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_claude_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/claude_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14,"chars":84,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":52,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":241,"why":"No formula (an expression starting with =) in the answer","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":11,"why":"Ran the formula =SUM(G2:G7): first-half total — Pass (formula result 335,800.0, correct answer 335,800.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t5_copilot_3.txt","shot":null}]}]},{"task_id":"t6","level":"難","prompt":"The following table is pasted into the spreadsheet starting at A1 (row 2 is January, row 13 is December, column D is Product C).\nWrite a single formula that calculates the first-half (January through June) total for Product C.\n\nRules\n- The answer must be exactly one line. Write only a formula starting with =\n- Do not write any explanation or the calculation result\n\n---- Table start ----\nMonth,Product A (Tumbler),Product B (Eco bag),Product C (Cooling towel),Product D (Bento box),Product E (Insulated bottle),Total\n2025-01,9400,13600,10200,14100,5200,52500\n2025-02,9400,12800,11100,14400,5200,52900\n2025-03,10300,10200,12600,14600,7900,55600\n2025-04,9400,7600,10700,14800,11800,54300\n2025-05,9400,6000,10500,15100,17000,58000\n2025-06,9400,4200,10000,15300,23600,62500\n2025-07,9400,3400,9800,15600,31400,69600\n2025-08,9400,3400,9100,15800,34100,71800\n2025-09,9400,5100,10200,16000,22300,63000\n2025-10,9400,8500,9000,16300,10500,53700\n2025-11,9400,11900,8200,16500,6600,52600\n2025-12,10300,14400,9700,16700,5200,56300\n---- Table end ----","scorer":"formula_run","answer_key":"(Automated scoring: formula_run) Run the submitted formula and check whether it matches the Product C first-half total of 65,100","materials":["/data/research/bench/suushiki-2026-10-08/materials/uriage.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":16.39999999999999857891452847979962825775146484375,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_chatgpt_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/chatgpt_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.39999999999999857891452847979962825775146484375,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_gemini_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/gemini_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":16.5,"chars":52,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_claude_1.txt","shot":"/data/research/bench/suushiki-2026-10-08/shots/claude_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":13.800000000000000710542735760100185871124267578125,"chars":52,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":52,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":false,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.60000000000000142108547152020037174224853515625,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":14,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":11,"why":"Ran the formula =SUM(D2:D7): Product C first-half total — Pass (formula result 65,100.0, correct answer 65,100.0)","instruction_follow":true,"answer_url":"/data/research/bench/suushiki-2026-10-08/answers/t6_copilot_3.txt","shot":null}]}]}],"limits":["This is a single run on a single day. Rankings can change with the day or the task","The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier","Measurements use only the default model and default mode (features such as Deep Research are not used)","The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company","Only tasks that can be scored by machine are covered. Writing quality and readability are not measured","Measurements are taken from the same screen a person would use in a browser. These are not results from the API","For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is"],"license":"CC BY 4.0","attribution":"Please cite AInformation as the source, along with the URL of the page you used and the measurement date.","page_url":"https://ainformation.jp/research/bench/suushiki-2026-10-08"}