{"bench_id":"b007-kossi","slug":"kossi-2026-10-03","title":"Building a document outline with a set slide count and character count (4 AIs, 6 tasks, 3 runs each)","field":"Document outlining","field_key":"kossi","round":1,"run_date":"2026-10-03","timezone":"Asia/Tokyo","version":"v1","conductor":"AInformation","funder":"AInformation (self-funded)","sponsor":null,"method":{"tasks_n":6,"repeats":3,"ais_n":4,"trials_total":72,"trials_valid":72,"excluded":[],"new_chat_each_time":true,"order_rotated":true,"order_by_repeat":{"1":["ChatGPT","Gemini","Claude","Copilot"],"2":["Gemini","Claude","Copilot","ChatGPT"],"3":["Claude","Copilot","ChatGPT","Gemini"]},"timeout_seconds":300,"scoring":"Machine scoring only (not graded by AI)","materials_generated_at":"2026-10-02","materials_note":"Conditions for an internal document at Sample Trading Co., a fictional company (number of pages, number of bullet points, character counts). The conditions change every time.","counted_outcomes":["scored","refused","timeout"],"excluded_outcomes":["limit","our_error","human_check"],"answer_cleanup":"When reading the answer off the screen, some AIs put a line meant to be read aloud, or a mid-thought summary, in the same box as the answer. Before scoring, we remove the part that appears at the start of the answer and then appears again, identical, later on. All four AIs are treated the same way. The uncleaned answers are kept as-is in answers/","csv_url":"/data/research/bench/kossi-2026-10-03/latest.csv"},"ais":[{"name":"ChatGPT","plan":"ChatGPT Pro","model_shown":"Medium","checked_at":"2026-10-02 19:57"},{"name":"Gemini","plan":"Google AI Pro","model_shown":"Flash-Lite","checked_at":"2026-10-02 19:57"},{"name":"Claude","plan":"Claude Max","model_shown":"Opus 4.8 high","checked_at":"2026-10-02 19:57"},{"name":"Copilot","plan":"Microsoft 365 Personal","model_shown":null,"checked_at":"2026-10-02 19:57"}],"summary":{"headline":"Six tasks were given three times each for a total of 18 trials. ChatGPT had the highest accuracy rate, passing 15 of 18 trials (83.3%); Gemini was second at 77.8%","winner":"ChatGPT","note":"Results from a single round of testing. Rankings may change on a different day or with different tasks"},"scores":[{"ai":"ChatGPT","correct":15,"trials":18,"accuracy":0.83330000000000004067857162226573564112186431884765625,"accuracy_display":83.2999999999999971578290569595992565155029296875,"seconds_median":20.39999999999999857891452847979962825775146484375,"seconds_min":14.300000000000000710542735760100185871124267578125,"seconds_max":23.60000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Gemini","correct":14,"trials":18,"accuracy":0.77780000000000004689582056016661226749420166015625,"accuracy_display":77.7999999999999971578290569595992565155029296875,"seconds_median":19.300000000000000710542735760100185871124267578125,"seconds_min":19.10000000000000142108547152020037174224853515625,"seconds_max":22.5,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":1,"limit":0,"our_error":0,"human_check":0}},{"ai":"Claude","correct":11,"trials":18,"accuracy":0.611099999999999976552089719916693866252899169921875,"accuracy_display":61.10000000000000142108547152020037174224853515625,"seconds_median":35.5,"seconds_min":13.9000000000000003552713678800500929355621337890625,"seconds_max":51,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":1,"instruction_follow_display":100,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Copilot","correct":5,"trials":18,"accuracy":0.277799999999999991384669328908785246312618255615234375,"accuracy_display":27.800000000000000710542735760100185871124267578125,"seconds_median":17.10000000000000142108547152020037174224853515625,"seconds_min":14.199999999999999289457264239899814128875732421875,"seconds_max":20.199999999999999289457264239899814128875732421875,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":0.94440000000000001723066134218242950737476348876953125,"instruction_follow_display":94.400000000000005684341886080801486968994140625,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}}],"tasks":[{"task_id":"t1","level":"易","prompt":"Read the following conditions and write only the title of each page in the document.\n\n■ Conditions for building the outline of a document (fictional: Sample Trading Co.)\n\nTopic: Explanation of year-end EC (e-commerce) measures\nAudience: 6 department heads within the company\nTime: 10 minutes\n\nRules\n- The document must have exactly 5 pages\n- Each page must have exactly 1 title line and exactly 3 bullet lines\n- The title must be 18 characters or fewer\n- Each bullet line must be 40 characters or fewer\n- Use numbers wherever numbers can be used\n\nHow to write it\n- Write 1 line per page, containing only the title text\n- The number of lines must equal exactly the number of pages given in the conditions\n- No line may exceed the title character limit given in the conditions\n- Do not add numbering, symbols, or preamble","scorer":"format_check","answer_key":"(Automated scoring: format_check) {'type': 'line_count', 'min': 5, 'max': 5} / {'type': 'max_line_chars', 'value': 18}","materials":["/data/research/bench/kossi-2026-10-03/materials/shiryo_brief.txt"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.300000000000000710542735760100185871124267578125,"chars":50,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_chatgpt_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/chatgpt_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.60000000000000142108547152020037174224853515625,"chars":52,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17.199999999999999289457264239899814128875732421875,"chars":49,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":64,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_gemini_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/gemini_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":66,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":76,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.10000000000000142108547152020037174224853515625,"chars":211,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_claude_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/claude_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":20.10000000000000142108547152020037174224853515625,"chars":178,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":198,"why":"All format checks passed: line_count OK (5 lines, within 5–5) / max_line_chars OK (0 lines over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":39,"why":"Format did not match: line_count FAIL (1 line; required exactly 5) / max_line_chars FAIL (1 line over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_copilot_1.txt","shot":null},{"n":2,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":46,"why":"Format did not match: line_count FAIL (1 line; required exactly 5) / max_line_chars FAIL (1 line over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":40,"why":"Format did not match: line_count FAIL (1 line; required exactly 5) / max_line_chars FAIL (1 line over 18 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t1_copilot_3.txt","shot":null}]}]},{"task_id":"t2","level":"易","prompt":"What is the document with the following conditions about?\n\n■ Conditions for building the outline of a document (fictional: Sample Trading Co.)\n\nTopic: Explanation of year-end EC (e-commerce) measures\nAudience: 6 department heads within the company\nTime: 10 minutes\n\nRules\n- The document must have exactly 5 pages\n- Each page must have exactly 1 title line and exactly 3 bullet lines\n- The title must be 18 characters or fewer\n- Each bullet line must be 40 characters or fewer\n- Use numbers wherever numbers can be used\n\nHow to write it\n- Write exactly 1 line for the answer. Write the topic exactly as given in the conditions\n- Do not add an explanation","scorer":"contains_all","answer_key":"(Automated scoring: contains_all) Topic = Explanation of year-end EC (e-commerce) measures","materials":["/data/research/bench/kossi-2026-10-03/materials/shiryo_brief.txt"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.300000000000000710542735760100185871124267578125,"chars":10,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_chatgpt_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/chatgpt_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.39999999999999857891452847979962825775146484375,"chars":10,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17.199999999999999289457264239899814128875732421875,"chars":10,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"timeout","seconds":300.5,"chars":0,"why":"Timed out (the answer did not finish before the time limit)","instruction_follow":null,"answer_url":null,"shot":"/data/research/bench/kossi-2026-10-03/shots/gemini_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":10,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":21,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":14.0999999999999996447286321199499070644378662109375,"chars":83,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_claude_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/claude_02_t2.png"},{"n":2,"ok":false,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":61,"why":"0 of 1 item present, missing: Topic","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":37,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17,"chars":77,"why":"0 of 1 item present, missing: Topic","instruction_follow":false,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_copilot_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/copilot_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.300000000000000710542735760100185871124267578125,"chars":10,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":14.199999999999999289457264239899814128875732421875,"chars":10,"why":"The one item was present","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t2_copilot_3.txt","shot":null}]}]},{"task_id":"t3","level":"中","prompt":"Build the outline of a document using the following conditions.\n\n■ Conditions for building the outline of a document (fictional: Sample Trading Co.)\n\nTopic: Explanation of year-end EC (e-commerce) measures\nAudience: 6 department heads within the company\nTime: 10 minutes\n\nRules\n- The document must have exactly 5 pages\n- Each page must have exactly 1 title line and exactly 3 bullet lines\n- The title must be 18 characters or fewer\n- Each bullet line must be 40 characters or fewer\n- Use numbers wherever numbers can be used\n\nHow to write it\n- For each page, write one line as Page N: Title, followed by bullet lines starting with - for the number of bullets given in the conditions\n- The number of pages must match the conditions exactly, no more and no fewer\n- Do not add preamble or comments. Do not add blank lines","scorer":"format_check","answer_key":"(Automated scoring: format_check) {'type': 'line_prefix', 'pattern': '^(Page ?[0-9]+[:]|[\\\\-])'} / {'type': 'line_count', 'min': 20, 'max': 20}","materials":["/data/research/bench/kossi-2026-10-03/materials/shiryo_brief.txt"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.699999999999999289457264239899814128875732421875,"chars":351,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_chatgpt_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/chatgpt_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":20.300000000000000710542735760100185871124267578125,"chars":320,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":20.300000000000000710542735760100185871124267578125,"chars":339,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.39999999999999857891452847979962825775146484375,"chars":423,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_gemini_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/gemini_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":395,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":425,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":47.89999999999999857891452847979962825775146484375,"chars":500,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_claude_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/claude_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":41.7999999999999971578290569595992565155029296875,"chars":488,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":47.7000000000000028421709430404007434844970703125,"chars":474,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":283,"why":"Format did not match: line_prefix OK (0 of 1 line with inconsistent formatting) / line_count FAIL (1 line; required exactly 20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_copilot_1.txt","shot":null},{"n":2,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":309,"why":"Format did not match: line_prefix OK (0 of 1 line with inconsistent formatting) / line_count FAIL (1 line; required exactly 20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":17,"chars":280,"why":"Format did not match: line_prefix OK (0 of 1 line with inconsistent formatting) / line_count FAIL (1 line; required exactly 20)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t3_copilot_3.txt","shot":null}]}]},{"task_id":"t4","level":"中","prompt":"Turn the structure of the document under the following conditions into a markdown table.\n\n■ Conditions for building the outline of a document (fictional: Sample Trading Co.)\n\nTopic: Explanation of year-end EC (e-commerce) measures\nAudience: 6 department heads within the company\nTime: 10 minutes\n\nRules\n- The document must have exactly 5 pages\n- Each page must have exactly 1 title line and exactly 3 bullet lines\n- The title must be 18 characters or fewer\n- Each bullet line must be 40 characters or fewer\n- Use numbers wherever numbers can be used\n\nHow to write it\n- Use exactly 2 columns, in the order Page / Title\n- The number of rows must equal exactly the number of pages given in the conditions (excluding the header row and the separator row)\n- Output only the table. Do not add preamble or explanation","scorer":"format_check","answer_key":"(Automated scoring: format_check) {'type': 'table_cols', 'value': 2} / {'type': 'line_count', 'min': 7, 'max': 7}","materials":["/data/research/bench/kossi-2026-10-03/materials/shiryo_brief.txt"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":20.60000000000000142108547152020037174224853515625,"chars":72,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (6 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_chatgpt_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/chatgpt_04_t4.png"},{"n":2,"ok":false,"outcome":"scored","seconds":20.39999999999999857891452847979962825775146484375,"chars":77,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (6 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":17.300000000000000710542735760100185871124267578125,"chars":77,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (6 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":88,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (6 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_gemini_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/gemini_04_t4.png"},{"n":2,"ok":false,"outcome":"scored","seconds":22.300000000000000710542735760100185871124267578125,"chars":84,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (6 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":499,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (26 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":32.39999999999999857891452847979962825775146484375,"chars":157,"why":"Format did not match: table_cols FAIL (no table) / line_count OK (7 lines, within 7–7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_claude_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/claude_04_t4.png"},{"n":2,"ok":false,"outcome":"scored","seconds":32.39999999999999857891452847979962825775146484375,"chars":155,"why":"Format did not match: table_cols FAIL (no table) / line_count OK (7 lines, within 7–7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":29.39999999999999857891452847979962825775146484375,"chars":154,"why":"Format did not match: table_cols FAIL (no table) / line_count OK (7 lines, within 7–7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":20.199999999999999289457264239899814128875732421875,"chars":322,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (21 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_copilot_1.txt","shot":null},{"n":2,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":283,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (21 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":17.300000000000000710542735760100185871124267578125,"chars":290,"why":"Format did not match: table_cols FAIL (no table) / line_count FAIL (21 lines; required exactly 7)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t4_copilot_3.txt","shot":null}]}]},{"task_id":"t5","level":"難","prompt":"Build the outline of a document using the following conditions.\n\n■ Conditions for building the outline of a document (fictional: Sample Trading Co.)\n\nTopic: Explanation of year-end EC (e-commerce) measures\nAudience: 6 department heads within the company\nTime: 10 minutes\n\nRules\n- The document must have exactly 5 pages\n- Each page must have exactly 1 title line and exactly 3 bullet lines\n- The title must be 18 characters or fewer\n- Each bullet line must be 40 characters or fewer\n- Use numbers wherever numbers can be used\n\nHow to write it\n- For each page, write one line as Page N: Title, followed by bullet lines starting with - for the number of bullets given in the conditions\n- The number of pages and the number of bullets must match the conditions exactly\n- No line may exceed 46 characters\n- Use numbers wherever numbers can be used. Do not add preamble or comments. Do not add blank lines","scorer":"format_check","answer_key":"(Automated scoring: format_check) {'type': 'line_prefix', 'pattern': '^(Page ?[0-9]+[:]|[\\\\-])'} / {'type': 'line_count', 'min': 20, 'max': 20} / {'type': 'max_line_chars', 'value': 46}","materials":["/data/research/bench/kossi-2026-10-03/materials/shiryo_brief.txt"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":23.60000000000000142108547152020037174224853515625,"chars":408,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_chatgpt_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/chatgpt_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":23.5,"chars":343,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":20.5,"chars":370,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":651,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_gemini_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/gemini_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":506,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.39999999999999857891452847979962825775146484375,"chars":517,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":47.89999999999999857891452847979962825775146484375,"chars":438,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_claude_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/claude_05_t5.png"},{"n":2,"ok":true,"outcome":"scored","seconds":47.7999999999999971578290569595992565155029296875,"chars":452,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":51,"chars":464,"why":"All format checks passed: line_prefix OK (0 of 20 lines with inconsistent formatting) / line_count OK (20 lines, within 20–20) / max_line_chars OK (0 lines over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":312,"why":"Format did not match: line_prefix OK (0 of 1 line with inconsistent formatting) / line_count FAIL (1 line; required exactly 20) / max_line_chars FAIL (1 line over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_copilot_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/copilot_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":277,"why":"Format did not match: line_prefix OK (0 of 1 line with inconsistent formatting) / line_count FAIL (1 line; required exactly 20) / max_line_chars FAIL (1 line over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":17.300000000000000710542735760100185871124267578125,"chars":290,"why":"Format did not match: line_prefix OK (0 of 1 line with inconsistent formatting) / line_count FAIL (1 line; required exactly 20) / max_line_chars FAIL (1 line over 46 characters)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t5_copilot_3.txt","shot":null}]}]},{"task_id":"t6","level":"難","prompt":"Build the outline of a document using the following conditions, and output only JSON.\n\n■ Conditions for building the outline of a document (fictional: Sample Trading Co.)\n\nTopic: Explanation of year-end EC (e-commerce) measures\nAudience: 6 department heads within the company\nTime: 10 minutes\n\nRules\n- The document must have exactly 5 pages\n- Each page must have exactly 1 title line and exactly 3 bullet lines\n- The title must be 18 characters or fewer\n- Each bullet line must be 40 characters or fewer\n- Use numbers wherever numbers can be used\n\nHow to write it\n- Output only JSON. Do not add explanatory text before or after it\n- Use the shape {\"pages\": [{\"no\": 1, \"title\": \"title\", \"bullets\": [\"...\"]}]}\n- The number of items in pages must equal exactly the number of pages given in the conditions\n- The number of items in bullets must equal exactly the number of bullets given in the conditions","scorer":"format_check","answer_key":"(Automated scoring: format_check) {'type': 'json_valid'} / {'type': 'json_keys', 'keys': ['pages']} / {'type': 'json_list_len', 'key': 'pages', 'value': 5}","materials":["/data/research/bench/kossi-2026-10-03/materials/shiryo_brief.txt"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.5,"chars":482,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_chatgpt_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/chatgpt_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":23.60000000000000142108547152020037174224853515625,"chars":520,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":20.699999999999999289457264239899814128875732421875,"chars":519,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":22.5,"chars":730,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_gemini_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/gemini_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":19.199999999999999289457264239899814128875732421875,"chars":979,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":646,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":38.60000000000000142108547152020037174224853515625,"chars":723,"why":"Format did not match: json_valid FAIL (could not be parsed as JSON) / json_keys FAIL (JSON shape does not match) / json_list_len FAIL (no pages list found)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_claude_1.txt","shot":"/data/research/bench/kossi-2026-10-03/shots/claude_06_t6.png"},{"n":2,"ok":false,"outcome":"scored","seconds":44.7999999999999971578290569595992565155029296875,"chars":765,"why":"Format did not match: json_valid FAIL (could not be parsed as JSON) / json_keys FAIL (JSON shape does not match) / json_list_len FAIL (no pages list found)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":41.7000000000000028421709430404007434844970703125,"chars":792,"why":"Format did not match: json_valid FAIL (could not be parsed as JSON) / json_keys FAIL (JSON shape does not match) / json_list_len FAIL (no pages list found)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":468,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_copilot_1.txt","shot":null},{"n":2,"ok":true,"outcome":"scored","seconds":20.199999999999999289457264239899814128875732421875,"chars":472,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":469,"why":"All format checks passed: json_valid OK (parsed as valid JSON) / json_keys OK (keys match) / json_list_len OK (pages has 5 of 5 items)","instruction_follow":true,"answer_url":"/data/research/bench/kossi-2026-10-03/answers/t6_copilot_3.txt","shot":null}]}]}],"limits":["This is a single run on a single day. Rankings can change with the day or the task","The plan used is each AI's paid tier (ChatGPT Pro / Google AI Pro / Claude Max / Microsoft 365 Personal). These are not results from the free tier","Measurements use only the default model and default mode (features such as Deep Research are not used)","The tasks use original data for the fictional company Sample Shoji Co., Ltd. These are not results using data from a real company","Only tasks that can be scored by machine are covered. Writing quality and readability are not measured","Measurements are taken from the same screen a person would use in a browser. These are not results from the API","For any AI whose on-screen model name could not be read in a given run, that field is left blank (null). When the name could be read, it is shown as is"],"license":"CC BY 4.0","attribution":"Please cite AInformation as the source, along with the URL of the page you used and the measurement date.","page_url":"https://ainformation.jp/research/bench/kossi-2026-10-03"}