Agentic Determinism Index (ADI)

Reference leaderboard by serving tuple. Top three earn medals for the latest snapshot.

Last run: 2026-09-02 15:44 UTC Reference runs: 4 Run: r/f7zyafnnc4sa/

providers

models

cases

scored tuples

Rank	Serving tuple	Model	Score	Mode share	Byte-exact replay	Deterministic runs	Mean distinct	Cases
1	1st Groq via OpenRouter 7c6m…k43x	meta-llama/llama-3.1-8b-instruct	100.0	100%	Yes · byte-exact	2 / 2 streak 2	1.0000	1
2	2nd CoreWeave via OpenRouter 96qg…ddwp	meta-llama/llama-3.1-8b-instruct	65.5	90%	No	1 / 2 streak 0	2.0000	1
3	3rd NIM hosted (default) nscq…sj2x	meta/llama-3.2-11b-vision-instruct	58.5	80%	No	0 / 2 streak 0	2.0000	1
4	NIM hosted + NIM_FORCE_DETERMINISTIC header khqy…38tq	meta/llama-3.2-11b-vision-instruct	55.0	75%	No	0 / 2 streak 0	2.0000	1
5	DeepInfra via OpenRouter cazx…9sfb	meta-llama/llama-3.1-8b-instruct	44.5	60%	No	0 / 2 streak 0	2.0000	1
6	SiliconFlow via OpenRouter 5kxg…ahgd	deepseek/deepseek-chat-v3.1	21.71	30%	No	0 / 2 streak 0	7.0000	1
7	Azure via OpenRouter 7bw4…xexc	openai/gpt-4o-mini	21.71	30%	No	0 / 2 streak 0	7.0000	1
8	Amazon Bedrock via OpenRouter sw58…sng6	amazon/nova-lite-v1	14.62	20%	No	0 / 2 streak 0	8.0000	1

Byte-exact replay is the property that matters for audit trails: green rows returned identical bytes on every successful repeat of the same request. Deterministic runs counts how many scored reference runs that serving tuple stayed fully byte-exact (N of M), plus the current consecutive streak. Mode share is the fraction matching the most common completion (can be high without bit-identity). All-error probes are omitted. Scores recompute from raw transcripts; community replications are not merged.

Stack-drift timeline

When a provider changes system_fingerprint / model version across reference runs.

Provider	Model	Drift events	Timeline
nvidia_nim	meta/llama-3.2-11b-vision-instruct	0	2026-09-01T110738Z: [n/a] / [meta/llama-3.2-11b-vision-instruct] → 2026-09-02T154024Z: [n/a] / [meta/llama-3.2-11b-vision-instruct]
openrouter	amazon/nova-lite-v1	0	2026-09-01T110738Z: [Amazon Bedrock] / [amazon/nova-lite-v1] → 2026-09-02T154024Z: [Amazon Bedrock] / [amazon/nova-lite-v1]
openrouter	deepseek/deepseek-chat-v3.1	0	2026-09-01T110738Z: [SiliconFlow] / [deepseek/deepseek-chat-v3.1] → 2026-09-02T154024Z: [SiliconFlow] / [deepseek/deepseek-chat-v3.1]
openrouter	meta-llama/llama-3.1-8b-instruct	2	2026-09-01T100929Z: [DeepInfra, Novita, fp_03e8423237, fp_6a1eabf260, fp_8639719ff2] / [meta-llama/llama-3.1-8b-instruct] → 2026-09-01T110738Z: [DeepInfra, Novita, fp_03e8423237, fp_6a1eabf260, fp_8639719ff2, vllm-0.22.1.dev0+g0b3ba88f1.d20260529-25c7fbc0] / [meta-llama/llama-3.1-8b-instruct] → 2026-09-02T154024Z: [DeepInfra, fp_03e8423237, fp_4387d3edbb, fp_8639719ff2, vllm-0.22.1.dev0+g0b3ba88f1.d20260529-25c7fbc0] / [meta-llama/llama-3.1-8b-instruct]
openrouter	openai/gpt-4o-mini	0	2026-09-01T100929Z: [fp_369e662417] / [openai/gpt-4o-mini] → 2026-09-01T105457Z: [fp_369e662417] / [openai/gpt-4o-mini] → 2026-09-01T110738Z: [fp_369e662417] / [openai/gpt-4o-mini] → 2026-09-02T154024Z: [fp_369e662417] / [openai/gpt-4o-mini]

Maintained by Lemma Ventures AG. Source: lemma-ventures/agentic-determinism-index (MIT).