Spark-HermesSN74What every round is measured against. A change to any row changes what miners are scored on, so it is a deliberate, dated act — and a new era when it changes the reference arms' comparability.
Current (era e1, from the first round after r0055)
| Component | Pin | Where |
|---|---|---|
| hermes-agent | commit d84ece48b8552501660be229797e2d2aa4cee8db (main, 2026-09-15T16:29Z, v0.21.3) | sh/validator/images/hermes-ubuntu/Dockerfile |
| model | gittensor-model-hub/SparkHermes-27B-v1-NVFP4 at revision 70a4b710b7a4149719e4aa570fe989bc939c2b78 (2026-10-05; never main): ukisai/Swift-Qwen3.8-27b fine-tuned on Spark-Hermes agent trajectories, NVFP4 with the Swift NVFP4 recipe (model card on the Hub). Pre-staged on the worker at /models/sparkhermes-v1-nvfp4 and served with MODEL_DIR set to it, so the entrypoint never downloads. Before it serves a round the staged revision (.cache/huggingface/download/*.metadata, line 1) must be this one and each safetensors file's sha256 its LFS hash: model-00001-of-00002.safetensors f90c15988a6c146227ccc567c640f27804a48a38bdacb027158482d2dd37a21f, model-00002-of-00002.safetensors 5b988abaf4731285524781134c4ca047f84b975343bc1099e3c72dd6d46bbc7d. 91.224.44.227, 2026-10-05: both hold | HF |
| inference engine | ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:0.6.5 = sha256:040ffc5695ff1092d6ba62dd27b437eda7b169c9ba9c1e7b6f3133059abb82d9 (from r0051, re-evaluated from scratch on it on 2026-10-03; until then 0.5.7 = sha256:c919d647e8ae873837ad2cbcb83935cb9949765f1480637f9495c1ae7643d85b). 0.6.x speculates by default: a drafter proposes tokens and the pinned model verifies them, so the output distribution is the pinned model's (a request speculating alone gives the same tokens it would without the drafter); it is faster, not different, so it opened no era. Started with CTX=65536 and the volume below; the image's own entrypoint, unchanged | ghcr |
| speculative drafter | z-lab/Qwen3.8-27B-DFlash2 (Apache-2.0) at revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4, staged into the sh-models volume at /models/qwen38-dflash2 before the engine starts, so the entrypoint never downloads it: model.safetensors sha256 67fc76d68dc5a9415511a4f394ef744d67510cd20e93b37cc2cc7d28e4bab65c (its LFS hash), config.json sha256 873e3556509b0da06e29654ba00d4944888d4b5e8a33afde25f7eb27d321e980. Without the files the 0.6.5 entrypoint downloads main | HF, sh-models |
| context | 65536 — Hermes requires ≥ 64K advertised; 262144 OOMed at concurrency 4 | engine |
| sampling | temperature 0.7, top_p 0.95, max_tokens 8192, proxy-enforced (a bundle cannot change them), and the proxy's own default: a proxy started without --sampling on the new worker ran at the retired 0.2 from the worker move until r0004's evaluation ended (r0003's re-evaluation, the SWE-smith pilot, r0004 and every screen until then — each round internally consistent). The proxy logs its sampling at start. 0.2 drives the model into repetition loops that burn 6× the decode and kill episodes on the timeout (docs/spikes.md; whole-round A/B: timeouts 5/12 → 2/12, canon 0/3 → 3/3) | sh/validator/proxy.py |
| agent image | hermes-ubuntu:pin — ubuntu:22.04 + uv (PyPI) + uv-managed CPython 3.12 in /opt/uvpython + hermes-agent at the commit above + pytest 8.3.5 on the system interpreter + the runner. The venv is last on PATH so a task's own python3 wins in the shell; the runner is started by absolute path. Built on each worker, so the image id differs per build (91.224.44.223: sha256:a763247f56e0…; 162.156.217.154, 2026-09-23: sha256:3fe812681c9e…); the agent's dependencies come from the commit's lockfile (uv sync --frozen) | sh/validator/images/build.sh |
| episode network | sh-ep 172.30.0.0/24, gateway 172.30.0.1, proxy on :8090; --dns 172.30.0.1 with 53 REJECTed (Docker 27 has no --dns none); a counted DROP rule per container → network_egress_attempt | sh/validator/net/ |
| sandbox | uid 1000, no capabilities, no new privileges, pids 512, 4 GB, tmpfs home (the grader: pids 256, 2 GB); the grader takes the protected paths' baseline from the task image itself, never from the runner (which shares the agent's uid), and the runner's output directory is recreated before anything is written to it; the workspace is the image's own tree for image-defined tasks (writable), /ep/ws otherwise (read-only rootfs); the grader is a fresh read-only container with a tmpfs at the workdir | sh/validator/episode.py, sh/validator/grade.py |
| scoring | sh-scoring-v3: an episode counts by its credit — the share of the task's withheld checks that hold (0–1; 0 when disqualified) — not all-or-nothing. On r0003 nobody passed every check while agents passed 19–41 % of them. Overfit = published share − withheld share ≥ 0.5 (tasks with a published half only). Paired Δ of credit vs the NULL arm on the same instances, se with the reference term, Δc = max(0, mean − 1.28·se); the efficiency term is off (w_e = 0, see payment); window pooled over the last 8 rounds for references and miners alike; min 8 window episodes; overfit / DQ / near-duplicate zeroing | sh/scoring/v2.py |
| commitment | salt = HMAC(secret, "sh-salt|task_id), commitment = "hmac-sha256:" + HMAC(salt, canonical_json(withheld)); revealed at close | private supply/seal.py, sh/validator/round.py |
Round clock (2026-09-16)
| Parameter | Pin | Where |
|---|---|---|
| submission window | 120 min from open; the seal is taken at close, by PR head SHA. A window that closes with no valid submission (an incumbent alone does not count) reopens: same round, same tasks, a fresh 120 min, recorded in window.json (reopened, reason) — nothing is sealed, evaluated or revealed. submissions/ carries exactly the current king: a dethroned incumbent is removed at announce | sh/validator/orchestrate.py --window-minutes |
| tasks per round | 8 from the first round minted after 2026-10-07 (6 before: more instances per round make the crown less of a coin flip, and the pooled 8-round window carries ~64 scored episodes per miner; one RTX 5090 evaluates ~18-25 episodes an hour; a round with fewer than 6 is re-minted), minted ahead by supply.queue (--ahead 3); future rounds published only as digests in rounds/queue.json, over the tasks and their previews. For a previewing family (swe_fix) miners get the previews at open and the evaluated tasks at close | private repo |
| attestation | sh-attestation-v2: sr25519(hotkey, "spark-hermes:<repo>:<round>:<bundle_sha256>:<signed_at>") in attestation.json. One submission per hotkey and per GitHub account per round (from r0052; an account's PRs for several hotkeys: the one it signed last; the king's account competes only through its crown): the latest signed counts (not the newest PR — signed bundles are public and could be reopened by anyone); a signing time more than 10 min in the future is refused. | sh/cli/attest.py, orchestrate.one_per_hotkey |
| PR scope | a strategy PR changes exactly one submissions/<hotkey>/ and nothing else (never sh/, config, another hotkey); its directory name must be a valid ss58. While a submission server is advertised (since r0005, 2026-09-23) it adds only attestation.json (and receipt.json): the prose is uploaded to the server, and a PR that adds prose is refused by CI and at the seal, with no fallback to the PR tree. The crown is merged only at the sealed head (--match-head-commit), so a push after the seal cannot slip unevaluated code into main | orchestrate.candidates, announce |
| S1 — copied answers | a challenger sharing more than 8 distinct solution lines (≥ 24 chars) or more than 40 distinct 12-token runs with the round's private reference/alternate solutions and verifiers (for swe_fix: the code its bugs replaced) — text that also appears in a task's prompt, or in a preview's, excluded — is refused at seal. | sh/validator/similarity.py |
| crown | best Δ credit vs baseline on this round's instances, > 0, ≥ 4 paired, not disqualified this round, not a near-duplicate of another author's earlier bundle; among the eligible a tie does not dethrone the incumbent; other ties go to the strategy that first appeared earliest (the round its author first sealed essentially this bundle), then a round-keyed hash (not the hotkey name). Until the first round closed after 2026-10-07 other ties went by pooled Δc. A round where nobody beats the baseline crowns nobody | sh/scoring/crown.py |
| dethrone | the incumbent leaves submissions/ at announce only when (a) a challenger is crowned over it, (b) its pooled 8-round window fails the correctness gate on enough evidence (mean_d + z·se < 0, the evidence payment uses), or (c) it has gone 3 rounds in a row without the crown. A single round that crowns nobody does not dethrone (on 6 instances a better strategy ties the baseline by noise about a third of the time) | orchestrate.dethroned |
| engine budget | one RTX 5090 serves two long agent contexts. A call the engine refuses for capacity is held by the proxy and retried (up to 240 s) instead of failing — a refusal would otherwise void the whole episode. The reference strategy (canon) runs only in calibration rounds, every 8th (--canon-every); it labels families and is never paid, and the pooled window carries its measurements between calibrations. Evaluation goes first: when a round is ready, a baseline screen holding the engine for a future round is ended at once and screened again later (from 2026-10-07; before, the round waited up to 90 min) | sh/validator/proxy.py, orchestrate.evaluate |
| reward | the crowned pull request, merged at the sealed head, is what the reward follows (Gittensor workflow). An incumbent that wins is paid only through a defense PR (2026-09-24): an attestation-only PR re-signing the bundle it already defends with for this round (same digest); the seal keeps it as the incumbent and merges it if the crown holds. Without one, the incumbent keeps the crown and nothing is merged. The crown is decided on this round's instances alone: best mean Δ vs baseline > 0, ≥ 4 paired, not disqualified. score/Δc/weight are published diagnostics over the pooled window, not the reward, and read 0.000 at current sample sizes | sh/scoring/crown.py, orchestrate.announce |
| pooled statistic | pooled over the last 8 rounds. score = the paired mean Δ of credit vs the baseline over the window, after the overfit and copy penalties (signed). Copy penalty (from the first round closed after 2026-10-07): each 12-token run of a bundle belongs to whoever sealed it first; the share of a bundle that other authors (another hotkey and another GitHub account) sealed in an earlier round than its author first used it is measured at close (close/copies.json). From 80 % the bundle is a near-duplicate: score 0 and never crowned; from 30 % its score is cut by half that share; below 30 % it is ordinary shared phrasing (originals share 1–16 %); weight = a score's share among the strategies above the baseline, summing to 1; Δc = the one-sided 90 % lower bound of that mean, published beside them as how sure the gain is. The efficiency term is off (w_e = 0): it penalised honest correctness gains and paid worse-than-baseline strategies for fewer calls. The reference variance is floored (max-entropy over n) so a degenerate all-identical null arm cannot make the baseline look certain | sh/scoring/v2.py, --window-from |
| training data | SFT: the king's episodes where every check holds; DPO: the king's episode with credit ≥ 0.8 vs another surface's with at least 0.5 less, same instance; no king → nothing uploaded | sh/exports/build.py |
Family #4 — swe_fix (2026-09-17)
| Component | Pin | Where |
|---|---|---|
| seed material | SWE-smith (SWE-bench/SWE-smith, MIT): bugs injected into real Python repositories, each with an issue-style problem statement, the bug's diff, the tests it breaks (FAIL_TO_PASS) and the tests it leaves passing (PASS_TO_PASS), on a per-repository environment image. Indexed: 23,129 usable of 59,136 rows (the dataset as of 2026-09-17; docs/seed-datasets.md cites the paper's 50,137) (a problem statement, 1–10 broken tests, ≥ 5 passing, a diff that touches no test infrastructure). Rounds draw from 12 pure-Python repositories. Each bug is used at most once | private supply/swesmith.py, supply/families/swe_fix/repos.json |
| pilot | 20 bugs over 5 repositories, pinned model with no strategy, alone on the GPU: 17 valid, mean credit 0.34, 4 fully resolved, 4 partly, 9 not at all; no kept test broken; 9 of 17 used all 30 turns | 2026-09-17 |
| task images | swe-task:<opaque key> = the environment image at the bug's branch with its git history cut to one commit and no bytecode compiled before the bug (a pilot agent read the injected diff out of git log), plus the Hermes layers of hermes-ubuntu:pin. Built once, on the worker; images/<key>/SOURCE.json names the dataset, license, repository and environment — never the bug | private supply/swesmith.py |
| previews | miners are shown different bugs from the ones scored: each evaluated bug has a preview — another bug from the same repository and environment, with its SWE-smith instance id so it can be reproduced locally — and only previews are in rounds/<id>/tasks/ while the window is open. A strategy is paid for debugging these codebases, not for knowing eight one-line answers. The evaluated tasks are published at close in rounds/<id>/evaluated/, and reveal.json names each one's SWE-smith instance id — the committed test ids are that bug's own tests, readable in the dataset; rounds/queue.json commits to both before the round opens | supply/families/swe_fix/family.py, orchestrate.shown |
| universe | one custom swe_test <node id> per broken test, all withheld (split withheld: nothing published, so no overfit rule applies and a partial fix is partial credit). The grader receives the broken tests' files (as they were before SWE-smith deleted those tests) and the kept tests in withheld.json.assets, never the agent | supply/families/swe_fix/checks.py |
| grading | in a fresh read-only container over the agent's final tree: bytecode deleted; test infrastructure the task did not ship (a new conftest.py, sitecustomize.py, .pth, pytest.ini, anything under tests/) removed; the broken tests written back; one pytest --verbose run parsed the way SWE-smith parses it. A broken test counts only if it passes and no kept test fails. Shipped test infrastructure is a protected path: modifying it disqualifies | checks.py, sh/validator/grade.py |
| kept tests | up to 60 passing tests from the broken tests' files and 40 from elsewhere, sampled per bug; at mint, only those that pass both on the bug and after the reference are kept | swesmith.sample_p2p, family.kept_tests |
| derivation | every broken test fails on the untouched tree and passes after the reference (the bug's diff reversed), twice; the gate re-proves it | private supply/derive.py, supply/gate.py |
| admission | one baseline episode per sealed task; admitted when the pinned model with no strategy does not fully fix it (credit < 1). A baseline 0 is admitted: unlike FACET, a real bug with its failing tests is fixable, and 0 is where strategies have the most room | ops/start-queue.sh --screen-band 0.0:0.99 |
| S1 | compared with the code the bug replaced (the diff's removed lines), not with the diff's context or the repository's tests, which quote the repository miners can read | sh/validator/similarity.py |
| limits | a token budget of 600k (prompt + completion, as the engine counts them; enforced by the proxy, which answers 402 and marks the episode — token_budget_spent, graded normally, never void) is the binding limit from r0005: what an episode costs does not depend on what else runs, and a lean context buys more turns. Backstops: max_turns 100, timeout_s 1800 (r0004 ran 30 turns / 600 s: 3 of 16 capped episodes were cut off by the clock while a third of that clock went to queueing). Tools terminal + file, network none; suite timeout 480 s | family.py, sh/validator/proxy.py |
Eras
e1(from the first round after r0055): modelgittensor-model-hub/SparkHermes-27B-v1-NVFP4@70a4b710. Engine, drafter, context, sampling, limits and scoring unchanged. Scores are pooled from the firste1round only: the reference arms of different models are not comparable. r0056 (its window opened by mistake for about 15 minutes after r0055) and the queued r0057 (screened against thee0model) were discarded, never evaluated.e0(r0001-r0055):gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090at revision5b7a687fc8211a5d631c8ca6a593dd37eb26ce33(2026-09-10; resolved 2026-09-18 — nevermain). The engine image does not carry the weights: its entrypoint downloadsmaininto thesh-modelsvolume on first start, and skips that whileconfig.jsonis there. So a new worker serves this revision only ifmainstill points at it then: before it serves a round, the downloaded revision (.cache/huggingface/download/*.metadata, line 1) must be this one and each safetensors file's sha256 must equal its LFS hash (line 2). 162.156.217.154, 2026-09-23: both hold
Retired
terminal_task(family #3, FACET-Terminal-6k; retired before launch): real terminal tasks with pytest verifiers, transformed at mint, cheats = partial deliverables. Retired 2026-09-17: the pinned model earned ~0 credit on most tasks (r0003: 6 of 8 at 0 % for every surface) and screening at 10–70 % admitted none of its first screens. Its code stays in the private repo so rounds already minted can be evaluated.posix_report(family #1, 2026-09-14): the pinned model solved it without a strategy; no headroom.process_lifecycle(family #2; retired before launch): authored template; the baseline passed 7/8, one instance of headroom. Removed with its code on 2026-09-16 (family-process-lifecycle:pin=sha256:cb21736d…,hermes-base:pin= the python:3.12-slim agent image these ran on).temperature 0.2(era A0): repetition loops; see sampling above.
Generated from docs/pins.md