Skip to content

Norn V3 lane statecharts ​

These bounded TLA+ checks are offline models. A passing model checks the specified state transitions and invariants; it does not prove live credentials, provider state, host behavior, availability, or the implementation itself.

On 2026-11-09, all 21 checked-in TLC configurations were rerun against Norn master 19297662 using the statechart skill. Every configured rule passed. Configurations with NoDeadlock enabled passed deadlock checking; the concurrent-admission config deliberately disables deadlock checking because terminal result states stutter. This includes the four missing-evidence admission cases (each 4 states / 6 steps) and the two-client concurrent store case (14 states / 18 steps), as well as the primary lane models listed below. Outputs were written under /tmp/norn-*-check; no generated outputs or provider effects were added to the repository. The statechart reports record the bounds and omissions for each model.

On 2026-11-12 at 17:35 UTC, eleven configurations covering ten release models were rerun from the current tree with the statechart skill. Mini direct checked 90 states / 179 steps; the Mini host-agent gate 56 / 138; DO all-stop release 299 / 1,036; existing-Fleet Genesis 69 / 107; online Norn upgrade 97 / 251; workload mixed versions 103 / 275 at capacity four and 24 / 52 at capacity three; Fresh-Fleet create 308 / 813; exact nine-node Fresh-Fleet recovery 102 / 210; General + Stateful pools 45 / 98; and pre-registration Norn authority 19 / 21. Every configured invariant and deadlock check passed. Capacity three still has no target-only route or terminal release path. Outputs are in /tmp/norn-*-20261112; no repository model/chart source changed and no provider or host action ran.

On 2026-11-10, ten focused TLC configurations across eight protocols were rerun against the current tree: Mini recovery (90 states / 179 steps), workload capacity four (103 / 275) and three (24 / 52), future General + Stateful pools (45 / 98), nine-node staging recovery (102 / 210), existing DO release (299 / 1,036), proposed online Norn upgrade (97 / 251), existing-Fleet Genesis (69 / 107), and the Genesis source-readback gate in normal (18 / 52) and disabled/stale-binding (2 / 2) configurations. All configured invariants and deadlock checks passed. Outputs stayed under /tmp; no provider or host action occurred. The complete 30-configuration sweep was subsequently rerun on 2026-11-10: 28 configs passed all rules, while the deliberately incorrect prior-runtime and missing-upgrade-mode configs each reproduced the expected later-upgrade hold.

At 03:04 UTC, a focused recovered-forward finalization model was checked in four configurations. The valid path reached 52 states / 96 transitions; bad adoption ancestry and bad transition-to-candidate ancestry each held in 3 / 5; normal API health unavailable held host-agent restart in 25 / 45. All six configured invariants and deadlock checks passed. The model starts from preservation-pending and abstracts signed Git verification and live implementation behavior; see its README for limits and reproduction commands.

At 09:50 UTC, the post-recovery forward-upgrade gate was modeled in two configurations. The signed-descendant path passed five invariants and deadlock checking at 6 states / 7 steps. The legacy exact-recovery-runtime assumption reproduced a later-upgrade hold after the first signed descendant in a 3-step counterexample. The actual shell helper now has a Git-backed integration test for candidate, signed-descendant, and unrelated-release cases.

At 12:01 UTC, the Mini queued-upgrade host-agent stability gate was rerun with a missing-agent observation. TLC checked 56 states / 138 steps; three stable launchd/agent samples admit the gate, while an absent agent resets the window and exhaustion leaves the source and schema unchanged. Five invariants and deadlock passed. The model abstracts the implementation's 30-sample bound, binary/signature checks, and real Launchd behavior; see the focused model README.

At 04:29 UTC on 2026-11-12, a held workload-release abandonment model checked 38 states / 77 steps, four safety invariants, and deadlock. It proposes readback and explicit authority before abandonment, retains encrypted input until a durable terminal disposition, and requires a no-references inventory before retiring a shared key-ring version. Three unsafe mutants reproduce the expected violations. This policy is not yet implemented.

LaneResultModel statusMain limit
Existing DO platform release299 states, 1,036 steps; 10 invariants pass (rerun 2026-11-12)Existing all-stop, forward-only release modelDoes not cover Genesis installation or fresh provider creation
Mini direct V3 upgrade and recovered-forward repair90 states, 179 steps; 8 invariants and deadlock check pass (rerun 2026-11-12)Includes the observed host-agent-fenced halt; ordinary same-intent retry is blocked until a distinct signed repair-controller action verifies lineage and resumes the exact generationThe model abstracts adoption-source ancestry, GUI LaunchAgent environment, repair authorization refresh, Launchd timing, SQL/filesystem durability, and key custody. The signed-ancestor finalization gate has a separate controller integration test.
Mini recovered-forward finalizationValid: 52 states / 96 transitions; invalid adoption and candidate ancestry: 3 / 5 each; unavailable normal API health: 25 / 45; six invariants and deadlock pass in all (2026-11-10)Exact preservation receipt and signed-lineage gate, database-fence release, normal API activation/readback, host-agent restart and terminal receipt; failed lineage holds before fence release, failed API health keeps the agent fencedRanks and health/evidence flags abstract signature/Git verification, generation authorization, SQL/Launchd durability, and workload behavior; a controller integration regression covers signed-ancestor acceptance.
Mini recovered-forward later upgradesValid signed-descendant rule: 6 states / 7 steps; legacy exact-release and missing-upgrade-mode configurations each reproduce a later-upgrade hold after 3 transitions; five invariants and deadlock pass in the valid model (2026-11-10)Retained recovery evidence gates future upgrades while allowing a verified signed descendant and admitted operation mode; unrelated signed release is heldModel abstracts signature checks as a boolean, Git ancestry as a finite relation, and mode-list membership as a boolean; Git-backed shell regressions exercise release ancestry and upgrade-mode admission.
Mini queued-upgrade host-agent stability gate56 states / 138 steps; five invariants and deadlock pass (rerun 2026-11-12)Requires three stable launchd/agent samples; a missing agent resets the window and bounded instability holds without changing the active source or schemaAbstracts the code's 30-sample bound to five, candidate byte/signature checks as successful, and real Launchd/process behavior; see focused model README.
Proposed online Norn platform upgrade97 states, 251 steps; 12 invariants and deadlock check pass (rerun 2026-11-12)Old release serves through additive schema expansion and passive candidate verification; writer/traffic handoff is fenced, crash recovery keeps the same intent, and schema contraction waits until old-writer retirementProposed protocol only; assumes binary/schema compatibility and post-handoff health; abstracts real migrations, writer discovery, fencing, supervision, workload/data integrity, and availability measurements
Existing Fleet Genesis69 states, 107 steps; 9 invariants and deadlock check pass (rerun 2026-11-12)Dormant install, signed intent, staged controls, passive-first promotion, recovery; fresh-Fleet request is explicitly blockedInputs are modeled as one valid/invalid gate; provider create workflow is absent from this model
DO Genesis readback source and input gateNormal recovery: 18 states / 52 steps; closed/stale gate: 2 / 2; 6 invariants and deadlock check pass in both (rerun 2026-11-10)Models closed protected workflow or stale exact state binding holding before intent/runner dispatch; normal recovery requires a fresh one-use intent after source updateDigests, signatures, protected installer, journal, and provider/runtime are abstract; no live dispatch or Genesis state change
Workload mixed-version resizeCapacity 3: 24 states / 52 steps; capacity 4: 103 / 275; 11 invariants and deadlock check pass in both (rerun 2026-11-12)Proposed protocol, not current controller behavior; capacity 3 has no target-only route or terminal path under the target-only-before-drain rule, while capacity 4 reaches terminal receiptAbstracts Nomad placement, app processes, database semantics, route implementation, and timing; live preflight must prove full overlap capacity
Fresh DO Fleet create and recovery308 states, 813 steps; 9 invariants and deadlock check pass (rerun 2026-11-12)Bounded protected plan/apply, partial create, readback, same-intent recovery, PKI/configuration, exact cleanupThree abstract node groups; detailed provider resources and secret lifecycle are omitted
Future fresh Fleet with two app pools45 states, 98 steps; 14 invariants and deadlock check pass (rerun 2026-11-12)General + stateful pool create/recovery, role-correct workload placement/readiness, stateful recovery proof, then shared inventory/configuration/receiptAbstract counts/identities, provider API, secret lifecycle, workload-specific data integrity, and implementation behavior
Nine-node staging-topology create and recovery102 states, 210 steps; 10 invariants and deadlock check pass (rerun 2026-11-12)3+2+4 partial-create, fresh exact readback, same-intent forward recovery, then inventory/PKI/configuration; pool counts match protected norn-fleet mainNot wired to the protected provider workflow; node identities abstracted, inputs/credentials aggregate, no provider, GitHub Actions, SSH, Ansible, or teardown behavior
Fresh-Fleet Norn pre-registration authority19 states, 21 steps; 9 invariants and deadlock check pass (rerun 2026-11-12)Exact intent and protected dispatch, attempt-bound verifier admission, one-use create grant, exact-remainder recovery, terminal receipt, then target registration/fence transferModels the Norn authority boundary only; provider/state evidence is abstract and no live grant or create is exercised
Workload old-generation stop recovery10 states, 15 steps; 7 invariants and deadlock check passBounded target-health and mixed-route gate, stop accepted/lost response, marker recovery, old allocation drainDoes not model effect-store crashes, signed admission, API dispatch, durable receipt loading, route publication, or live Nomad
Workload route publication and recovery55 states, 111 steps; 8 invariants and deadlock check passPer-node mixed-route publication; atomic active-route/receipt CAS; same-generation recovery and fresh pre-stop route/inventory proofDoes not prove live workload authority, inventory source, route CAS store, ingress transport, or controller implementation
Concurrent workload admission replay14 states, 18 transitions; 3 invariants passTwo independent etcd clients contend for a stable request; the fence loser rechecks durable identity and returns the winnerStore-level only; does not exercise HTTP handling or process restart
Workload admission replay after response loss and restart115 states, 365 steps; 10 invariants and deadlock check passDurable acceptance with a lost response, controller stop/restart without local acceptance memory, and exact same-key replayAbstract protocol model; does not exercise the HTTP handler, etcd adapter, or runtime worker/controller
Simultaneous workload release HTTP admission19 states, 28 steps; 6 invariants and deadlock check passTwo handlers miss initial replay lookup, read current evidence, then converge at atomic acceptanceThe simultaneous-acceptance test uses a deterministic store fake. A separate real-etcd HTTP integration now proves refusal and no durable acceptance when an owned serving route is absent; it does not prove successful fresh acceptance.

Model files ​

Findings to carry into implementation ​

  • The Mini model's writer gate follows protected backup and signed shadow preparation, and precedes journal binding and live host effects.
  • The Mini recovery model now represents the actual halt after host-agent fencing: the API remains on the old source, before write fencing or schema migration. Normal retry cannot resume that phase; the modeled repair action requires the exact journal target and preserved observed state. Its signature and source-lineage checks remain an implementation-refinement task.
  • The completed recovery receipt remains the authority for later upgrades, but the current API and host agent must be checked against the active signed descendant. Requiring them to remain at the original recovery SHA blocks the second upgrade; accepting any signed descendant without a Git ancestry check would admit unrelated releases. The new model and integration regression cover both boundaries.
  • The workload proposal stops effects while held, waits for healthy target capacity before draining old allocations, and requires fresh proof of the exact current route after recovery. Route publications move only from old to mixed to target; switching back requires a separately admitted intent. Loss of one canary allocation invalidates route proof and holds old-version drain until fresh readback confirms the route and same-intent recovery reconciles.
  • In the mixed-version workload model, three simultaneous allocation slots preserve safety but cannot complete the two-old/two-target rollout without weakening the route/drain fence. Four slots provide the modeled full-overlap path. Treat this as a preflighted scheduler-capacity requirement, not a reason to drain old allocations before target-only route proof.
  • The workload route-publication model keeps partial node writes under the same intent and generation after a lost response. Inventory drift, target-health loss, or authority loss holds progress. An old-stop authorization needs fresh inventory and active-route checks even after a historical route receipt was committed.
  • Genesis covers an existing Fleet; the fresh-Fleet model is separate. The provider workflow's integrated input/credential tests, DO token permission scope, and exact live cleanup remain open despite the passing bounded model.
  • The nine-node model's 3+2+4 pool counts match protected norn-fleet main and the provider inventory, but its aggregate inputs do not prove the exact source parser, protected workflow, or persistent staging recovery. It assumes provider-scope evidence at admission even though current preflight does not validate token scope; it has no teardown action because persistent staging has no sanctioned whole-Fleet destroy.
  • The online Norn upgrade now has a bounded proposed model, but no controller implementation or live online-upgrade qualification is established.

Each lane directory contains the TLA+ spec, TLC config, chart view, Mermaid chart, report, run output, and reproduction notes. Regenerate with the command in that lane's README.md.