Agentic Determinism Index (ADI)

Reference leaderboard by serving tuple. Each row uses that tuple’s latest successful score (carried across due-only runs). Top three earn medals for this snapshot.

Byte-exact replay is one score. Every repeat of the same request in that run returned the same bytes (for tool call cases, the same normalized call sequence: same tool names and arguments in the same order, ignoring transport ids and indexes). The pill states that fact. It does not color the row. A green row means the latest scored run held. That run was byte-exact, and it returned the same response as the previous run. A broken streak is not a green row. The count under the status (held N of M runs) is history. It is not a second verdict. A medal needs 3 consecutive held runs over at least 72 hours. The streak line marks that bar with the word medal. Mode share is the fraction matching the most common completion (can be high without bit-identity). All-error probes are omitted. Scores recompute from raw transcripts; community replications are not merged. When a due-only run scores a subset, other tuples keep their previous score (shown under the stack id).

Parked tuples

Out of the score and watch rotation. Their history stays in the repository and earlier scores stay on the board, marked parked. A run where every request failed is never published as a score.

• EigenAI (EigenCloud, bit-exact claim) (gpt-oss-120b-f16): No API key is configured on the scoring host, so the target was always skipped. Never scored. Re-activate when a key is available. last score never scored, parked 2026-10-26
• Cohere via OpenRouter (cohere/command-r-08-2024): Cohere's endpoint for this model on OpenRouter is not zero-data-retention (ZDR), and the index account only routes to ZDR endpoints. Every request since the tuple was added on 2 Sep 2026 has failed with HTTP 404; never scored. last score never scored, parked 2026-10-26
• AtlasCloud via OpenRouter (SGLang partner) (deepseek/deepseek-chat-v3.1): AtlasCloud no longer serves this model on OpenRouter, and none of its OpenRouter endpoints is zero-data-retention. Never scored. last score never scored, parked 2026-10-26
• Crusoe via OpenRouter (meta-llama/llama-3.3-70b-instruct): Crusoe no longer serves this model on OpenRouter (last served 13 Sep 2026; pinned requests fail with HTTP 404 since 16 Sep 2026). Earlier scores are history. last score 2026-09-07T123353Z, parked 2026-10-26
• SambaNova via OpenRouter (meta-llama/llama-3.3-70b-instruct): OpenRouter no longer routes the 'sambanova' slug for this model; SambaNova serves it only as the 'sambanova-turbo' endpoint, measured as a separate tuple (SambaNova Turbo via OpenRouter). Never scored under this pin. last score never scored, parked 2026-10-26
• Together via OpenRouter (meta-llama/llama-3.3-70b-instruct): Together no longer serves this model on OpenRouter (last served 17 Oct 2026; pinned requests fail with HTTP 404 since 25 Oct 2026). Earlier scores are history. last score 2026-10-05T201706Z, parked 2026-10-26
• OpenAI via OpenRouter (openai/gpt-4o-mini): OpenAI's endpoint for this model on OpenRouter is not zero-data-retention (ZDR), and the index account only routes to ZDR endpoints. Every request since the tuple was added on 2 Sep 2026 has failed with HTTP 404; never scored. last score never scored, parked 2026-10-26
• Thinking Machines direct (batch-invariant claim) (thinkingmachines/inkling:free): OpenRouter serves this free model only to listed agentic apps (HTTP 403 for any other caller), and its endpoint is not zero-data-retention. Never scored. last score never scored, parked 2026-10-26

Stack-drift timeline

When a serving tuple changes system_fingerprint / model version across reference runs. First non-exact is the earliest reference stamp where byte-exact replay failed.

Watch-tick stack IDs

Hourly cheap probes on the private host. Each change of fingerprint / model version is a drift event.

Maintained by Lemma Ventures AG. Source: lemma-ventures/agentic-determinism-index (MIT).