{"bench_id":"b010-meibo","slug":"meibo-2026-10-19","title":"Unificar las variantes de escritura de los nombres de socios comerciales (4 IA, 6 tareas, 3 rondas cada una)","field":"Limpieza de datos y unificación de nombres","field_key":"meibo","round":1,"run_date":"2026-10-19","timezone":"Asia/Tokyo","version":"v1","conductor":"AInformation","funder":"AInformation (financiado por la propia empresa)","sponsor":null,"method":{"tasks_n":6,"repeats":3,"ais_n":4,"trials_total":72,"trials_valid":72,"excluded":[],"new_chat_each_time":true,"order_rotated":true,"order_by_repeat":{"1":["ChatGPT","Gemini","Claude","Copilot"],"2":["Gemini","Claude","Copilot","ChatGPT"],"3":["Claude","Copilot","ChatGPT","Gemini"]},"timeout_seconds":300,"scoring":"Solo calificación mecánica (la IA no califica)","materials_generated_at":"2026-10-18","materials_note":"Lista de transacciones de 4 empresas clientes ficticias, con nombres escritos de forma dispersa usando 'Co., Ltd.' en distintas variantes, katakana de ancho medio, espacios, etc. La respuesta correcta queda fijada al crearla","counted_outcomes":["scored","refused","timeout"],"excluded_outcomes":["limit","our_error","human_check"],"answer_cleanup":"Al leer la respuesta desde la pantalla, en algunas IA una línea pensada para lectura en voz alta o un resumen del razonamiento en curso aparece en el mismo recuadro que la respuesta. Antes de calificar, se elimina la parte que está al principio de la respuesta y que luego se repite de forma idéntica más adelante. Se aplica el mismo proceso a las 4 IA. Las respuestas sin este recorte se conservan tal cual en answers/","csv_url":"/data/research/bench/meibo-2026-10-19/latest.csv"},"ais":[{"name":"ChatGPT","plan":"ChatGPT Pro","model_shown":"GPT-6.1 Sol ligero","checked_at":"2026-10-18 20:21"},{"name":"Gemini","plan":"Google AI Pro","model_shown":"Flash","checked_at":"2026-10-18 20:21"},{"name":"Claude","plan":"Claude Max","model_shown":"Opus 4.8 alto","checked_at":"2026-10-18 20:21"},{"name":"Copilot","plan":"Microsoft 365 Personal","model_shown":null,"checked_at":"2026-10-18 20:21"}],"summary":{"headline":"Al dar 6 tareas 3 veces cada una, el mayor porcentaje de aciertos fue el de ChatGPT, con 15/18 intentos (83.3 %) (en segundo lugar quedó Gemini, con 77.8 %)","winner":"ChatGPT","note":"Resultado de una sola verificación. El orden puede cambiar si cambian el día o las tareas"},"scores":[{"ai":"ChatGPT","correct":15,"trials":18,"accuracy":0.83330000000000004067857162226573564112186431884765625,"accuracy_display":83.2999999999999971578290569595992565155029296875,"seconds_median":38.7000000000000028421709430404007434844970703125,"seconds_min":33,"seconds_max":54.60000000000000142108547152020037174224853515625,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.83330000000000004067857162226573564112186431884765625,"instruction_follow_display":83.2999999999999971578290569595992565155029296875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Gemini","correct":14,"trials":18,"accuracy":0.77780000000000004689582056016661226749420166015625,"accuracy_display":77.7999999999999971578290569595992565155029296875,"seconds_median":63.5,"seconds_min":19.10000000000000142108547152020037174224853515625,"seconds_max":134.900000000000005684341886080801486968994140625,"consistency":0.888900000000000023447910280083306133747100830078125,"consistency_display":88.900000000000005684341886080801486968994140625,"source_alive":null,"source_alive_display":null,"instruction_follow":0.83330000000000004067857162226573564112186431884765625,"instruction_follow_display":83.2999999999999971578290569595992565155029296875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Claude","correct":12,"trials":18,"accuracy":0.66669999999999995932142837773426435887813568115234375,"accuracy_display":66.7000000000000028421709430404007434844970703125,"seconds_median":20.199999999999999289457264239899814128875732421875,"seconds_min":16.89999999999999857891452847979962825775146484375,"seconds_max":32.2000000000000028421709430404007434844970703125,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.333299999999999985167420391007908619940280914306640625,"instruction_follow_display":33.2999999999999971578290569595992565155029296875,"failures":{"refused":0,"timeout":0,"limit":0,"our_error":0,"human_check":0}},{"ai":"Copilot","correct":12,"trials":18,"accuracy":0.66669999999999995932142837773426435887813568115234375,"accuracy_display":66.7000000000000028421709430404007434844970703125,"seconds_median":17.10000000000000142108547152020037174224853515625,"seconds_min":13.9000000000000003552713678800500929355621337890625,"seconds_max":17.39999999999999857891452847979962825775146484375,"consistency":1,"consistency_display":100,"source_alive":null,"source_alive_display":null,"instruction_follow":0.85709999999999997299937604111619293689727783203125,"instruction_follow_display":85.7000000000000028421709430404007434844970703125,"failures":{"refused":0,"timeout":4,"limit":0,"our_error":0,"human_check":0}}],"tasks":[{"task_id":"t1","level":"易","prompt":"En la siguiente lista de transacciones, el nombre de la empresa cliente aparece escrito de formas distintas. ¿En cuántas empresas quedan agrupadas si se unifican las que son iguales?\n\nReglas\n- La respuesta debe ser exactamente una línea, con el formato N empresas\n- No escribas los nombres de las empresas ni una explicación\n\n---- Lista de transacciones, inicio ----\nID,Nombre del cliente,Encargado,Monto\nA001,Sample Unyu Co., Ltd.,Takahashi,730000\nA011,(Co.) Sample Shokuhin,Takahashi,563000\nA007,Sample Kogyo (Co.),Ito,761000\nA010,Sample Shokuhin Co., Ltd.,Tanaka,231000\nA009,Sample Shokuhin (Co.),Tanaka,799000\nA003,Sample Bussan (Co.),Takahashi,250000\nA002,(Co.) Sample Unyu,Ito,483000\nA004,Sample Bussan (Co.),Kato,502000\nA008,Sample Kogyo (Co.),Tanaka,474000\nA006,(Co.) Sample Bussan,Kobayashi,641000\nA005,Sample Bussan Co., Ltd. ,Nakamura,314000\n---- Lista de transacciones, fin ----","scorer":"contains_all","answer_key":"(Corrección automática, contains_all) El número de empresas tras agrupar = 4 o 4 empresas o 4 casos","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":33,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":36.2999999999999971578290569595992565155029296875,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.60000000000000142108547152020037174224853515625,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":127.5,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":61.60000000000000142108547152020037174224853515625,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":52.39999999999999857891452847979962825775146484375,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.5,"chars":124,"why":"El único elemento está presente","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":64,"why":"El único elemento está presente","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":62,"why":"El único elemento está presente","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.39999999999999857891452847979962825775146484375,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_01_t1.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":13.9000000000000003552713678800500929355621337890625,"chars":2,"why":"El único elemento está presente","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t1_copilot_3.txt","shot":null}]}]},{"task_id":"t2","level":"易","prompt":"Agrupa los nombres de las empresas clientes de la siguiente lista de transacciones y enumera todas las empresas.\n\nReglas\n- Una línea por empresa. Cada línea debe empezar con -\n- Las empresas iguales se agrupan en una sola línea. No escribas montos ni cantidades\n\n---- Lista de transacciones, inicio ----\nID,Nombre del cliente,Encargado,Monto\nA001,Sample Unyu Co., Ltd.,Takahashi,730000\nA011,(Co.) Sample Shokuhin,Takahashi,563000\nA007,Sample Kogyo (Co.),Ito,761000\nA010,Sample Shokuhin Co., Ltd.,Tanaka,231000\nA009,Sample Shokuhin (Co.),Tanaka,799000\nA003,Sample Bussan (Co.),Takahashi,250000\nA002,(Co.) Sample Unyu,Ito,483000\nA004,Sample Bussan (Co.),Kato,502000\nA008,Sample Kogyo (Co.),Tanaka,474000\nA006,(Co.) Sample Bussan,Kobayashi,641000\nA005,Sample Bussan Co., Ltd. ,Nakamura,314000\n---- Lista de transacciones, fin ----","scorer":"set_match","answer_key":"(Corrección automática, set_match) Sample Bussan [Sample Bussan] / Sample Shokuhin [Sample Shokuhin] / Sample Kogyo [Sample Kogyo] / Sample Unyu [Sample Unyu] Se permiten 0 adiciones inventadas","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":36.2999999999999971578290569595992565155029296875,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":36.39999999999999857891452847979962825775146484375,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":50.10000000000000142108547152020037174224853515625,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":26.10000000000000142108547152020037174224853515625,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":89.099999999999994315658113919198513031005859375,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.199999999999999289457264239899814128875732421875,"chars":263,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":23.89999999999999857891452847979962825775146484375,"chars":298,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":32.2000000000000028421709430404007434844970703125,"chars":229,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.199999999999999289457264239899814128875732421875,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_02_t2.png"},{"n":2,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":47,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t2_copilot_3.txt","shot":null}]}]},{"task_id":"t3","level":"中","prompt":"En la siguiente lista de transacciones, calcula la cantidad de casos por empresa después de agrupar las que son iguales.\n\nReglas\n- Una línea por empresa, con el formato Nombre de la empresa: N casos\n- Ordena de mayor a menor cantidad. No escribas una explicación\n\n---- Lista de transacciones, inicio ----\nID,Nombre del cliente,Encargado,Monto\nA001,Sample Unyu Co., Ltd.,Takahashi,730000\nA011,(Co.) Sample Shokuhin,Takahashi,563000\nA007,Sample Kogyo (Co.),Ito,761000\nA010,Sample Shokuhin Co., Ltd.,Tanaka,231000\nA009,Sample Shokuhin (Co.),Tanaka,799000\nA003,Sample Bussan (Co.),Takahashi,250000\nA002,(Co.) Sample Unyu,Ito,483000\nA004,Sample Bussan (Co.),Kato,502000\nA008,Sample Kogyo (Co.),Tanaka,474000\nA006,(Co.) Sample Bussan,Kobayashi,641000\nA005,Sample Bussan Co., Ltd. ,Nakamura,314000\n---- Lista de transacciones, fin ----","scorer":"exact_number","answer_key":"(Corrección automática, exact_number) Sample Bussan = 4 (margen ±0) / Sample Shokuhin = 3 (margen ±0) / Sample Kogyo = 2 (margen ±0) / Sample Unyu = 2 (margen ±0)","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":39.7000000000000028421709430404007434844970703125,"chars":59,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.7999999999999971578290569595992565155029296875,"chars":59,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":35.7999999999999971578290569595992565155029296875,"chars":59,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":19.300000000000000710542735760100185871124267578125,"chars":43,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":113.400000000000005684341886080801486968994140625,"chars":43,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":27,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 4; falta la línea de Sample Bussan) / Sample Shokuhin incorrecto (respuesta correcta 3; falta la línea de Sample Shokuhin) / Sample Kogyo incorrecto (respuesta correcta 2; falta la línea de Sample Kogyo) / Sample Unyu incorrecto (respuesta correcta 2; falta la línea de Sample Unyu)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":20.39999999999999857891452847979962825775146484375,"chars":121,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_03_t3.png"},{"n":2,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":119,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.89999999999999857891452847979962825775146484375,"chars":197,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.39999999999999857891452847979962825775146484375,"chars":43,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 4) / Sample Shokuhin correcto (respuesta correcta 3) / Sample Kogyo correcto (respuesta correcta 2) / Sample Unyu correcto (respuesta correcta 2)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t3_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_03_t3.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":301.6000000000000227373675443232059478759765625,"chars":0,"why":"Se agotó el tiempo (la respuesta no se detuvo antes del límite de tiempo)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":301,"chars":0,"why":"Se agotó el tiempo (la respuesta no se detuvo antes del límite de tiempo)","instruction_follow":null,"answer_url":null,"shot":null}]}]},{"task_id":"t4","level":"中","prompt":"En la siguiente lista de transacciones, calcula la suma del monto por empresa después de agrupar las que son iguales.\n\nReglas\n- Una línea por empresa, con el formato Nombre de la empresa: N yenes\n- Ordena de mayor a menor monto. Escribe los números en caracteres de ancho medio. No escribas una explicación\n\n---- Lista de transacciones, inicio ----\nID,Nombre del cliente,Encargado,Monto\nA001,Sample Unyu Co., Ltd.,Takahashi,730000\nA011,(Co.) Sample Shokuhin,Takahashi,563000\nA007,Sample Kogyo (Co.),Ito,761000\nA010,Sample Shokuhin Co., Ltd.,Tanaka,231000\nA009,Sample Shokuhin (Co.),Tanaka,799000\nA003,Sample Bussan (Co.),Takahashi,250000\nA002,(Co.) Sample Unyu,Ito,483000\nA004,Sample Bussan (Co.),Kato,502000\nA008,Sample Kogyo (Co.),Tanaka,474000\nA006,(Co.) Sample Bussan,Kobayashi,641000\nA005,Sample Bussan Co., Ltd. ,Nakamura,314000\n---- Lista de transacciones, fin ----","scorer":"exact_number","answer_key":"(Corrección automática, exact_number) Sample Bussan = 1,707,000 (margen ±0) / Sample Shokuhin = 1,593,000 (margen ±0) / Sample Kogyo = 1,235,000 (margen ±0) / Sample Unyu = 1,213,000 (margen ±0)","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":39.39999999999999857891452847979962825775146484375,"chars":91,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":83,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":83,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":65.400000000000005684341886080801486968994140625,"chars":67,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":31.199999999999999289457264239899814128875732421875,"chars":67,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":76.900000000000005684341886080801486968994140625,"chars":67,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":26.10000000000000142108547152020037174224853515625,"chars":165,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_04_t4.png"},{"n":2,"ok":true,"outcome":"scored","seconds":26.199999999999999289457264239899814128875732421875,"chars":161,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_claude_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":23.199999999999999289457264239899814128875732421875,"chars":153,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":67,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_04_t4.png"},{"n":2,"ok":false,"outcome":"timeout","seconds":301.1000000000000227373675443232059478759765625,"chars":0,"why":"Se agotó el tiempo (la respuesta no se detuvo antes del límite de tiempo)","instruction_follow":null,"answer_url":null,"shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":16.89999999999999857891452847979962825775146484375,"chars":67,"why":"Todos los valores coinciden: Sample Bussan correcto (respuesta correcta 1,707,000) / Sample Shokuhin correcto (respuesta correcta 1,593,000) / Sample Kogyo correcto (respuesta correcta 1,235,000) / Sample Unyu correcto (respuesta correcta 1,213,000)","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t4_copilot_3.txt","shot":null}]}]},{"task_id":"t5","level":"難","prompt":"Agrupa los nombres de las empresas clientes de la siguiente lista de transacciones y preséntalos en una tabla en formato markdown.\n\nReglas\n- Exactamente 3 columnas, en el orden Nombre de la empresa / Cantidad de casos / Suma de montos\n- Las empresas iguales se agrupan en una sola línea. Ordena de mayor a menor monto\n- Muestra solo la tabla. No escribas introducción ni explicación\n\n---- Lista de transacciones, inicio ----\nID,Nombre del cliente,Encargado,Monto\nA001,Sample Unyu Co., Ltd.,Takahashi,730000\nA011,(Co.) Sample Shokuhin,Takahashi,563000\nA007,Sample Kogyo (Co.),Ito,761000\nA010,Sample Shokuhin Co., Ltd.,Tanaka,231000\nA009,Sample Shokuhin (Co.),Tanaka,799000\nA003,Sample Bussan (Co.),Takahashi,250000\nA002,(Co.) Sample Unyu,Ito,483000\nA004,Sample Bussan (Co.),Kato,502000\nA008,Sample Kogyo (Co.),Tanaka,474000\nA006,(Co.) Sample Bussan,Kobayashi,641000\nA005,Sample Bussan Co., Ltd. ,Nakamura,314000\n---- Lista de transacciones, fin ----","scorer":"exact_number","answer_key":"(Corrección automática, exact_number) Sample Bussan = 1,707,000 (margen ±0) / Sample Shokuhin = 1,593,000 (margen ±0) / Sample Kogyo = 1,235,000 (margen ±0) / Sample Unyu = 1,213,000 (margen ±0)","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":107,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":107,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_chatgpt_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":38.89999999999999857891452847979962825775146484375,"chars":107,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":119.5,"chars":103,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":80,"chars":87,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_gemini_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":134.900000000000005684341886080801486968994140625,"chars":103,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17,"chars":172,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17,"chars":170,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":20.10000000000000142108547152020037174224853515625,"chars":188,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":95,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_05_t5.png"},{"n":2,"ok":false,"outcome":"scored","seconds":17,"chars":95,"why":"Los valores no coinciden: Sample Bussan incorrecto (respuesta correcta 1,707,000; respuesta dada 41707000.0) / Sample Shokuhin incorrecto (respuesta correcta 1,593,000; respuesta dada 31593000.0) / Sample Kogyo incorrecto (respuesta correcta 1,235,000; respuesta dada 21235000.0) / Sample Unyu incorrecto (respuesta correcta 1,213,000; respuesta dada 21213000.0)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t5_copilot_2.txt","shot":null},{"n":3,"ok":false,"outcome":"timeout","seconds":301.80000000000001136868377216160297393798828125,"chars":0,"why":"Se agotó el tiempo (la respuesta no se detuvo antes del límite de tiempo)","instruction_follow":null,"answer_url":null,"shot":null}]}]},{"task_id":"t6","level":"難","prompt":"Agrupa los nombres de las empresas clientes de la siguiente lista de transacciones y genera solo un JSON.\n\nReglas\n- La salida debe ser solo JSON. No escribas texto explicativo antes ni después\n- El formato debe ser {\"companies\": [{\"name\": \"Nombre de la empresa\", \"count\": cantidad, \"total\": suma de montos}]}\n- Las empresas iguales se agrupan en una sola. Los números deben ir como valores numéricos (sin comas)\n\n---- Lista de transacciones, inicio ----\nID,Nombre del cliente,Encargado,Monto\nA001,Sample Unyu Co., Ltd.,Takahashi,730000\nA011,(Co.) Sample Shokuhin,Takahashi,563000\nA007,Sample Kogyo (Co.),Ito,761000\nA010,Sample Shokuhin Co., Ltd.,Tanaka,231000\nA009,Sample Shokuhin (Co.),Tanaka,799000\nA003,Sample Bussan (Co.),Takahashi,250000\nA002,(Co.) Sample Unyu,Ito,483000\nA004,Sample Bussan (Co.),Kato,502000\nA008,Sample Kogyo (Co.),Tanaka,474000\nA006,(Co.) Sample Bussan,Kobayashi,641000\nA005,Sample Bussan Co., Ltd. ,Nakamura,314000\n---- Lista de transacciones, fin ----","scorer":"set_match","answer_key":"(Corrección automática, set_match) Sample Bussan [Sample Bussan] / Sample Shokuhin [Sample Shokuhin] / Sample Kogyo [Sample Kogyo] / Sample Unyu [Sample Unyu] Se permiten 0 adiciones inventadas","materials":["/data/research/bench/meibo-2026-10-19/materials/meibo_dirty.csv"],"per_ai":[{"ai":"ChatGPT","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":54.60000000000000142108547152020037174224853515625,"chars":207,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_chatgpt_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/chatgpt_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":38.7000000000000028421709430404007434844970703125,"chars":207,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_chatgpt_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":42.2000000000000028421709430404007434844970703125,"chars":207,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_chatgpt_3.txt","shot":null}]},{"ai":"Gemini","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":43.39999999999999857891452847979962825775146484375,"chars":357,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_gemini_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/gemini_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":116.5,"chars":357,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_gemini_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":19.10000000000000142108547152020037174224853515625,"chars":357,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_gemini_3.txt","shot":null}]},{"ai":"Claude","trials":[{"n":1,"ok":false,"outcome":"scored","seconds":26.10000000000000142108547152020037174224853515625,"chars":449,"why":"No se pudo leer como JSON (no siguió la indicación de formato)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_claude_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/claude_06_t6.png"},{"n":2,"ok":false,"outcome":"scored","seconds":20.199999999999999289457264239899814128875732421875,"chars":441,"why":"No se pudo leer como JSON (no siguió la indicación de formato)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_claude_2.txt","shot":null},{"n":3,"ok":false,"outcome":"scored","seconds":23.10000000000000142108547152020037174224853515625,"chars":439,"why":"No se pudo leer como JSON (no siguió la indicación de formato)","instruction_follow":false,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_claude_3.txt","shot":null}]},{"ai":"Copilot","trials":[{"n":1,"ok":true,"outcome":"scored","seconds":17.10000000000000142108547152020037174224853515625,"chars":207,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_copilot_1.txt","shot":"/data/research/bench/meibo-2026-10-19/shots/copilot_06_t6.png"},{"n":2,"ok":true,"outcome":"scored","seconds":14.4000000000000003552713678800500929355621337890625,"chars":207,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_copilot_2.txt","shot":null},{"n":3,"ok":true,"outcome":"scored","seconds":17,"chars":191,"why":"Coinciden los 4, 0 elementos inventados","instruction_follow":true,"answer_url":"/data/research/bench/meibo-2026-10-19/answers/t6_copilot_3.txt","shot":null}]}]}],"limits":["Se trata de una medición realizada una vez al día. El orden puede cambiar según el día o la tarea","El plan corresponde a la versión de pago de cada IA (ChatGPT Pro, Google AI Pro, Claude Max, Microsoft 365 Personal). No son resultados de la versión gratuita","La medición se realizó solo con el modelo y el modo predeterminados (no se usaron funciones como Deep Research)","Las tareas usan datos creados por nosotros para la empresa ficticia Sample Shoji Co., Ltd. No son resultados con datos de una empresa real","Solo se incluyen tareas que se pueden calificar de forma automática. No se mide la calidad ni la legibilidad del texto","La medición se realizó desde la misma pantalla del navegador que usan las personas. No son resultados de la API","En las rondas en las que una IA no permite leer el nombre del modelo que aparece en la pantalla, ese campo se deja vacío (null). En las rondas en las que sí se pudo leer, se muestra tal cual"],"license":"CC BY 4.0","attribution":"Como fuente, escribe AInformation junto con la URL de la página utilizada y la fecha de medición.","page_url":"https://ainformation.jp/research/bench/meibo-2026-10-19"}