RECEIPT: six flagship AIs on our first 300 MMLU-Pro questions (paid run)

Written 2026-09-17T06:41:24Z. Run: 2026-09-16 (one main run and two gap-fill runs), scored 2026-09-16T18:22:53Z. Every file below is fingerprinted with sha256. Scoring re-read from scoreboard_300.json, not re-run.

Total billed by the six vendors: $65.6364 (every HTTP 200 call, retries included).

AI companyright / scored (of 300)accuracywith a claimed sourcewrong with no sourcebilledseconds per question
Anthropic280 / 30093.3%87.7%0.7%$44.334635.71
OpenAI269 / 29890.3%82.2%1%$7.483812.71
Google271 / 30090.3%52%4.3%$3.577411.06
xAI271 / 30090.3%50%5.7%$5.672155.5
DeepSeek259 / 29886.9%81.5%0.3%$4.253892.63
Mistral190 / 30063.3%39%22.3%$0.31470.93

One shared denominator (questions all six scored): 296. Holes (empty reply, answer outside options) are excluded from both numerator and denominator. Provenance here is CLAIMED (what the model cited); sources were not verified. OpenAI output tokens include reasoning; other vendors report reasoning separately.

Comparison (separate receipt, the first 300 by code): our stored cards, run by plain code, 300/300 on the same set with zero model calls on two machines; cards built for these questions, 16 repaired using the key.

Evidence files (sha256)

evidence (run records available to reviewers on request)bytessha256what
run record scoreboard_300.json134031c2f9dc191ea55159f43446b7ecbd316a48bc141514802633458c6431dc89c78scoreboard: per-vendor scored results (computed 2026-09-16T18:22:53Z)
run record score_six_300.cjs105917c027d9316090e66b8b775a4ba7898d99fa02ffb6f65a65d0c7105a6f9144050scorer (read-only, no vendor call)
run report89298c15f4b6c905f523fc5b9b8f9894af059c37eb59d3e25e56a83452e370ffbadascoring report
run record our_first_300_questions.json281323d0d258da766affd18ae52c67f8992ecca7434cc1bd464b79980b2af939445de2the 300 question set
public MMLU-Pro test split, mmlu-pro-test.jsonl983014651d7b60cb50f027a564d57df2e8db671af7f084c34481cfb37285b6d3eb0c1b1MMLU-Pro test corpus (keys)
run record first300_phase1.jsonl.partial62184180e86bedd8c81e1394c0d125c3e65a7699597c26163e248a5cd521d224234b360raw vendor call rows (1215 rows used)
run record first300_phase2_mistral.jsonl158699921c1f4029d6f8a2b6b87dc5e47ee0b97874d4c54bda24018933aca1f92c1ddf9raw vendor call rows (300 rows used)
run record run_deepseek.jsonl333364798e0bd206bed10a38947ef8a34b4cf51b5112bc5254153dc37060bce8a5cec3raw vendor call rows (65 rows used)
run record run_google.jsonl293677596f69f81b4bb6503ef21ad33362f791ed21cefdcab65b2243768192c08f9e1fraw vendor call rows (57 rows used)
run record run_openai.jsonl293433d826003b2c0443e0b030132522f996f1152d7350b6275033f2c4c67c713f43faraw vendor call rows (60 rows used)
run record run_xai.jsonl2969502cf928a7875692554a5bc28c27543ac3bfcab6d95ed27352e08af9298a8f8551raw vendor call rows (59 rows used)
run record anthropic_gap.jsonl1470321cf3e9d7a2f064b36f917eb3e89d10b0dadc51f9ea2c24ed94978142feb3aa501raw vendor call rows (265 rows used)
run record anthropic_gap2.jsonl6720618064124fb78a9e411c9e1a8ac8fbbeeb1251f2045f18b8d8dbcc36a164f7fb96raw vendor call rows (128 rows used)
run record run3_anthropic.jsonl17100f50ed591044b1607ec441c34de305bb9e061b6ad343cc150d0ca68a05c7dd982raw vendor call rows (4 rows used)
run record run3_openai.jsonl9270a7e8b2d0fb74d26d6427d98ac87d9e26315b601941017560387acfc81bbeacabraw vendor call rows (2 rows used)
run record run3_xai.jsonl137139a55d6ed0995a978a4f42b2301f9f2c2bf434d60dd58c018960d3969dba38bddraw vendor call rows (3 rows used)
run record run4_anthropic.jsonl11856821a958e4aaffcdb9bc697c5f417076714a420d14253e0ff7bf42e23d1b08abbraw vendor call rows (2 rows used)
run record run4_xai.jsonl908553210fb507c3718cd572b7ccc9911ea62a25b16aed072d1a49f3d342ac5ab0dfraw vendor call rows (2 rows used)
run record run5_xai.jsonl10400302026b356e1756477fdeddf651e0a821d54572e97bda79e1155aa55a66f45e6raw vendor call rows (2 rows used)
run record run6_deepseek.jsonl9771bb8dc536441c1bbacd19419175e4d52afafa976681443df5a8f8530e3ab6803draw vendor call rows (2 rows used)
run record run_anthropic.jsonl112222cc10eba30e975daae81be5aa0e17b06044d3b35081a283bddceda9c50f667134raw vendor call rows (21 rows used)
run record run_deepseek.jsonl2150202020191ec47f29a66fe58ebe9b18850d2e498dceaaa3417096e352a94acdf00raw vendor call rows (4 rows used)
run record run_openai.jsonl6076875e9c9daeab00a38a98586284305a4878356e18311e2a0bf3e501768c95789eraw vendor call rows (2 rows used)
run record run_xai.jsonl10853bb27e248b29cb11bc74db2a8fde3c13dbc289ab23505eca87f526533c198525eraw vendor call rows (3 rows used)
run record first300_phase1.spend.json22575d831cbdc36a1d9f65d44846b55b393b78554e7d3ff5baad29bedbda86a93e8fspend meter / denominator audit
run record first300_phase2_mistral.spend.json795fdeb96c94949ba0375f988da34f36bbd857610a498df45d2638cf7ab2ef19275spend meter / denominator audit
run record stubtest.spend.json6492f76b90f54050d321e7223ceb4cdcf8e8f053cc35ce9ab1496a68a349572a66cspend meter / denominator audit
run record validate_2q.spend.json21475769aaae19be8ee7b46522487698b711f74b2f1811919d4cb6d374f5c88e8517spend meter / denominator audit
run record waittest.spend.json750f2b32fa14c586177e391a8f38b92c5a3bb9890f876fdeb917bd361dcb5027a2bspend meter / denominator audit
run record run_deepseek.spend.json7754e0bf06f8383e814b7a15504c887d6f1ac4d4ea37d7bc6e231ca42059c4a0deaspend meter / denominator audit
run record run_google.spend.json74910c21ed9e58a83a499159f5366e0d46532bbcc38f77373aa9a3645d35f66b4eespend meter / denominator audit
run record run_openai.spend.json7085893fdda1215bb574ce9dbfcfd536c79cd2d696877a99f5ec22bca3c1b10774espend meter / denominator audit
run record run_xai.spend.json7816e0b9680b67642d329713e5a7b6cb82d749633087346af6ce6b5a978e168c869spend meter / denominator audit
run record anthropic_gap.spend.json830b9359b2b8bccc40cdbf6c93f4344829fc94a9137f0326363f1817f5582861506spend meter / denominator audit
run record anthropic_gap2.spend.json828ca8d3fdcad0796747f51a8a1ff026ded761ebce72ade7cbbb0e0e80122381b7aspend meter / denominator audit
run record validate_1q.spend.json728b4055df11c602be5f076bc3fa15ac303773018517fe21b6abfc5bab8acec60c1spend meter / denominator audit
run record validate_1q_b.spend.json6776ab6cd508e481382c7d4456a6267ec0cc57a162b87da30de3aec2390fee483c2spend meter / denominator audit
run record run3_anthropic.spend.json7537f72c48bbdd5a15eab405b481907dc010477adad2767240d11967fb7ac6c9d0cspend meter / denominator audit
run record run3_openai.spend.json709f601ada9b7d37ba3e7a6bafbfbd10760088b1a5346a9c9faa8d61e290a70254aspend meter / denominator audit
run record run3_xai.spend.json69816963698f8c9e7b02456a1b6ceaccc7e71d65e3239b9876a7e7409b04e3e2bb1spend meter / denominator audit
run record run4_anthropic.spend.json67909eeca5621281670311cc20cc688aacb0e356d3ec0aeca5c6136b77475aea91aspend meter / denominator audit
run record run4_xai.spend.json667aaa903185039c27ecc42216bcdc7366686ca73aa0e9ebc2386dc3fc743d4fa96spend meter / denominator audit
run record run5_xai.spend.json6711d9a2324a26a09521fe4c7a34ec70b5718c83356b6f9ce1ff65fc4f093166aadspend meter / denominator audit
run record run6_deepseek.spend.json689ce60fd17144788fa1f4ab88bf219d7f44d15f5af9ac9702428140be23d2e5609spend meter / denominator audit
run record run7_deepseek.spend.json698db2234e1ab613e9b003a256314dd0510b0107255df3065cfad4987d9c19170daspend meter / denominator audit
run record run_anthropic.spend.json809b002bfc62f1bde4297b06f9e65aa878dcbc6d6edd394d3fb111794bdaffde22aspend meter / denominator audit
run record run_deepseek.spend.json7250a833bd239342492472b4e6bed46ba1ab2a58a691cf934ddf57fd4ec6248d187spend meter / denominator audit
run record run_openai.spend.json7265b60d1a0767fa5688aea926eb4ce79783d70762094322eea86947d50b1476ba1spend meter / denominator audit
run record run_xai.spend.json741697dbdc76fa76d9aad9fd722dabf6f4dc46fd9955a354317c7a07c45e346f9efspend meter / denominator audit
run record run3_anthropic.DENOMINATOR_AUDIT.json3474fcfb431eb507ee3e91921fe7e6207c2b9745bff6a5fc86ba740cfe1fa346d40spend meter / denominator audit
run record run3_openai.DENOMINATOR_AUDIT.json34478d4597526a626c1a1a6b81c154d972bec4e2078d4de0c39f9672347348094c4spend meter / denominator audit
run record run3_xai.DENOMINATOR_AUDIT.json34168ea87c983623467bd21a8a9c3583c612de7ec54de1a50e537cdf7120f8f5019spend meter / denominator audit
run record run4_anthropic.DENOMINATOR_AUDIT.json34718e8e54f754b3e4ebb15d5d3d49b8cfbd9e944f494ef5583fa5090d1aa34caeespend meter / denominator audit
run record run4_xai.DENOMINATOR_AUDIT.json341fe43d35e5a47d747cb92ae9a5306937f3e3f03a527454b60bdb891a2a783e459spend meter / denominator audit
run record run5_xai.DENOMINATOR_AUDIT.json341ab443ea7b1c2cf0bd1ac442f77ad81114eed1a7a5e0d0c1649e8b7e841238138spend meter / denominator audit
run record run6_deepseek.DENOMINATOR_AUDIT.json346a7f541a4ecbd8e520ccef907f8a06469dc2ba448d204c271042e4a70432eba74spend meter / denominator audit
run record run7_deepseek.DENOMINATOR_AUDIT.json346d16a4e12a217ae89b144c083de6ce7f80fe03db06090d13e22a75b88e397192aspend meter / denominator audit
run record run_anthropic.DENOMINATOR_AUDIT.json35040fae6082b65dcf43f77272ac8a3a3e95ccb1fa84b45879ee2ed33166054f2b8spend meter / denominator audit
run record run_deepseek.DENOMINATOR_AUDIT.json346cbe9fe07cfb4a98cbc2419c35ce6169d4f0da1edf83881852c102932963393a3spend meter / denominator audit
run record run_openai.DENOMINATOR_AUDIT.json344e5a37ea4142d5a28021d2f5fb8d8f8ce49a70817f34a34c0b8fb49134fb3bbdcspend meter / denominator audit
run record run_xai.DENOMINATOR_AUDIT.json3412987ad9e52096586b5a541602e12bc0c8d0405b3f175b8807c4478adb139a4a6spend meter / denominator audit
run note959faefb62ac82b41c7a3971e9ad4f14fef2b7a41838e7e27eb07d4b76fbfdaad5brun status note