Norn v3 launch history
Online Fleet launch checkpoint (2026-11-14 UTC)
The launch priority changed to controller-mediated online updates of the existing DO staging platform. Workload admission, fresh creation, resizing, and Mini follow-ups no longer gate that launch milestone.
Signed Norn cdf2081e passed all twelve qualification jobs and immutable publication. Fleet PR #1036 merged as f482f21f after full CI. The signed fa301fa5 baseline passed the joined three-member etcd guest rehearsal, and exact-pair producer 43553952979 passed bidirectional plan storage and control-fencing checks. No live online promotion followed. The full disposable controller/restart rehearsal and installed controller update remain pending.
Read-only work-account inspection found all eleven staging Droplets active. Public platform /version and canary /route-proof returned HTTP 200. These checks did not refresh private API/source lineage or prove continuity. The running management controller's installed closure predates the new online modules. No DO or Mini change ran for this checkpoint.
The prior long status pages are preserved in the launch-state archive and program archive. Current status is in launch state and the milestone map.
Fresh HTTP admission refuses an unowned serving route (2026-11-12 18:07 UTC)
The authenticated workload-release HTTP handler was exercised against disposable real etcd for Fleet placement, approved policy, operator identity, and current serving-route reads. With no control-owned serving route present, the handler returned HTTP 409 workload_release_route_unavailable; durable replay lookup confirmed that no acceptance was recorded. The qualification lookup used a narrow fixture because the test stops at the absent-route gate. The focused admission integration suite passed, then all eight TestEtcdWorkloadReleaseAdmission cases were rerun against disposable etcd at 18:11 UTC and passed, including this refusal case and durable replay after handler/store reconstruction. This proves fail-closed refusal at the route gate, not successful fresh admission, private-input acceptance through HTTP, or live DO workload admission. The temporary etcd container was stopped after the run. PR #457 merged to master at 18:18 UTC as ec91740c, recording this test and the related milestone corrections. No provider or host action ran.
DO staging runtime rechecked (2026-11-12 17:53 UTC)
The explicit read-only doctl --context work account was active for TurnKey Marketing. All eleven Norn staging Droplets were active: two management, three control, two ingress, and four app nodes. Platform /version returned HTTP 200 with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. Canary /route-proof returned HTTP 200 (norn-fleet-canary); /version returned HTTP 200 with source be5ca5d969ccf3509e62ce04a833516bb5c204db, node de960b60d29f, and allocation 079ea126-ed26-9943-02cc-afa6f7d3f255. The platform allocation was unchanged from 17:16; the canary node and allocation changed. These point-in-time public responses do not prove allocation-to-host lineage, signed workload admission, or continuous health. No provider or host action ran.
Release-lane statecharts reproduced offline (2026-11-12 17:35 UTC)
The statechart skill reran eleven configurations covering ten Mini, DO, and Fresh-Fleet protocols from the current tree. Mini direct: 90 states / 179 steps; Mini host-agent gate: 56 / 138; DO all-stop platform release: 299 / 1,036; existing-Fleet Genesis: 69 / 107; proposed online upgrade: 97 / 251; mixed-version resize: capacity four 103 / 275 and capacity three 24 / 52; Fresh-Fleet create: 308 / 813; nine-node recovery: 102 / 210; General + Stateful pools: 45 / 98; pre-registration Norn authority: 19 / 21. Every configured invariant and deadlock check passed. Capacity three still has no target-only route or terminal release path. Outputs are in /tmp/norn-*-20261112; checked-in model and chart sources were not changed. These bounded models do not qualify provider credentials, workflow wiring, live controller recovery, Nomad placement, or online availability. No provider or host action ran.
Protected workload-readback gate remains closed (2026-11-12 17:25 UTC)
The live Fleet main remains 11e4611cd9a28472a1c6b2a237ad18a796b54faf. A names-only audit compared the current protected workflow references with repository and work-staging variable/secret names: six of seven variables and all three secrets are present; NORN_WORKLOAD_READBACK_ENABLED remains absent. No values, protected files, modes, or credential contents were read. A query of the latest 30 Fleet workflow runs found no workload-readback dispatch. This is a bounded recent-history check, not a complete archive audit. No workflow dispatch or provider action ran.
DO runtime rechecked; local recovery qualification rerun (2026-11-12 17:16 UTC)
The explicit read-only doctl --context work account remained active (UUID prefix 17f40050). The staging-environment tag returned two management Droplets; the Fleet tag returned nine nodes (three control, two ingress, four app). All eleven were active. Platform /version returned HTTP 200 in 0.266 seconds with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. Canary /route-proof returned HTTP 200 in 0.256 seconds (norn-fleet-canary); /version returned HTTP 200 in 0.228 seconds with source be5ca5d969ccf3509e62ce04a833516bb5c204db, node 515ddcea44b8, and allocation 6b4b9efa-ca9a-2dc6-3e0b-d7cc3e27bcf8. These point-in-time reads do not establish host lineage, signed workload admission, or continuous health. The platform allocation changed since the earlier 17:03 sample; the canary also reported a different node/allocation.
The repeatable bash v2/scripts/test-workload-release-stage-nomad.sh qualification passed all three tests again at 17:15 UTC against disposable local etcd, Consul, and Nomad. It verifies exact terminal receipt and succeeded operation replay after etcd client/store reconstruction, plus recovery of a lost successful Nomad create response without duplicate registration. The route/drain effects use a test driver, so this is not controller-process or live-ingress qualification. No DO or provider mutation ran.
Workload-release terminal receipt replay qualified locally (2026-11-12 17:11 UTC)
The repeatable bash v2/scripts/test-workload-release-stage-nomad.sh qualification passed all three restart tests against disposable local etcd, Consul, and Nomad. The composed etcd-backed stage/route/drain test now applies the worker's app-lock-fenced terminal operation write, reconstructs the client and operation store, then verifies the same terminal receipt revision/digest and succeeded operation replay. The separate Nomad test drops a successful target create response, hides first readback, and proves successor adoption of the original reservation and exact Nomad job revision without a duplicate registration. Route/drain effects in the composed test use a test driver; this does not prove a live controller restart, ingress recovery, or DO admission. No DO or provider action ran.
DO staging runtime and readback-gate refresh (2026-11-12 17:03 UTC)
The explicit read-only doctl --context work account check remained active (UUID prefix 17f40050). All eleven staging Droplets were active: two management, three control, two ingress, and four app nodes. Platform /version returned HTTP 200 in 0.187 seconds, reporting allocation c776c80f-6ba8-0992-a552-bfc21c441f8a and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. The canary /route-proof returned HTTP 200 in 0.182 seconds (norn-fleet-canary); /version returned HTTP 200 in 0.169 seconds with source be5ca5d969ccf3509e62ce04a833516bb5c204db, node de960b60d29f, and allocation 079ea126-ed26-9943-02cc-afa6f7d3f255. The platform allocation and canary node/allocation differ from the 16:26 sample; these public responses do not establish host lineage, signed admission, or continuous health.
A names-only audit compared the workload-readback workflow at current norn-fleet main 11e4611c with repo and work-staging environment configuration. Six of seven referenced variable names and all three secret names are configured; NORN_WORKLOAD_READBACK_ENABLED is absent. Main's workflow file matches the audited copy. No variable values, secret values, or protected files were read, and no workflow dispatch or provider action ran. The readback gate remains closed; issuer provenance, file contents and modes, runner access, and a durable V3 admission receipt remain unverified.
Workload-release restart recovery exercised locally (2026-11-12 16:48 UTC)
Three existing workload-release restart integration tests passed against disposable local services. Two used a test driver with local etcd: one recovered a lost route-effect response after worker lease revocation and successor claim; the other composed target-stage and route/drain recovery across etcd client restarts and reached a terminal receipt without duplicate effect writes. The third used local etcd and a Docker-capable disposable Nomad agent: a proxy dropped Nomad's successful target-create response and hid the first readback; after etcd client restart, the successor adopted the existing effect reservation, bound Nomad's exact job version and modify index, and did not register a duplicate job. This qualifies target-stage recovery through real disposable Nomad, not the full private-input, health/ingress, old-cohort drain, or deployed-controller path. No DO or provider action ran.
DO staging refresh and readback configuration audit (2026-11-12 16:26 UTC)
Using the explicit work DigitalOcean context, the read-only account check matched UUID prefix 17f40050; all eleven Norn-tagged staging Droplets were active (two management, three control, two ingress, four app). Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.192 seconds with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and platform version af0f054358d9b9c7b68ea25132b52ee5eb099e12. Canary https://norn-fleet-canary.tkmkt-dev.com/route-proof returned HTTP 200 in 0.187 seconds with body norn-fleet-canary; /version returned HTTP 200 in 0.201 seconds with source be5ca5d969ccf3509e62ce04a833516bb5c204db, node 515ddcea44b8, and allocation 6b4b9efa-ca9a-2dc6-3e0b-d7cc3e27bcf8. The platform allocation changed since 16:15; the canary node and allocation match that sample. These are point-in-time endpoint proofs, not V3 admission, allocation-to-host lineage, or continuous health evidence.
Against norn-fleet main 27d7b33f, a names-only audit compared all seven variable references and three secret references in work-staging-workload-readback.yml with repository and work-staging environment configuration. Six variable names are configured; only NORN_WORKLOAD_READBACK_ENABLED is absent. All three state-backend secret names are configured. The NORN_FLEET_SECRET_DIR value was not displayed; a metadata-only check confirmed it is absolute and has no trailing slash. The audit did not verify protected file existence, modes, contents, digests, credential provenance, or runner access. Readback remains disabled. Workflow history still contains only failed candidate push 43117232561 with zero jobs and no protected readback. No workflow dispatch or provider mutation ran.
DO staging read-only refresh (2026-11-12 13:33 UTC)
Using explicit DigitalOcean context work, a read-only inventory found all eleven Norn staging Droplets active: two management, three control, two ingress, and four app nodes. Account UUID 17f40050233d1cd4962b44050e82a0dd01f8dff7 was active. Platform https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.267 seconds with allocation c776c80f-6ba8-0992-a552-bfc21c441f8a and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. The public canary https://norn-fleet-canary.tkmkt-dev.com/route-proof returned HTTP 200 with body norn-fleet-canary; /version returned HTTP 200 with source be5ca5d969ccf3509e62ce04a833516bb5c204db, node de960b60d29f, and allocation 079ea126-ed26-9943-02cc-afa6f7d3f255. These probes establish point-in-time endpoint responses, not V3 signed admission or allocation-to-host binding. The names-only work-staging audit still finds 18 required bindings missing. Workflow history contains one failed candidate-branch push with zero jobs and no protected readback run. No provider mutation, workflow dispatch, or deployment ran.
DO staging read-only refresh (2026-11-12 12:58 UTC)
Using explicit DigitalOcean context work, the read-only inventory found all eleven Norn staging Droplets active: two management, three control, two ingress, and four app nodes. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.256 seconds with version af0f054358d9b9c7b68ea25132b52ee5eb099e12 and allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05. This confirms provider power state and a responding platform API at that time. The canary was not re-probed; its latest recorded /route-proof and /version responses remain from 12:26 UTC. No provider mutation, workflow dispatch, or deployment ran.
DO workload-readback bindings rechecked (2026-11-12 12:52 UTC)
Reran the repository's name-only audit from protected norn-fleet main against the work-staging GitHub environment and current work-staging-workload-readback.yml. Eighteen required environment binding names are missing, including the enable flag and dedicated input-file paths. The five shared workflow variables and three state-backend secret names are present. The audit read no values and did not validate paths, file contents, digests, provenance, or runner access. No workflow was dispatched, and no provider or fleet mutation occurred. The protected readback remains unconfigured; these name bindings alone do not make it ready to run.
Fresh-Fleet Norn authority merged (2026-11-12 12:41 UTC)
Norn PR #437 merged to master as f41f70cc from exact head ff055988. Protected run 43219500086 passed API Go, CLI Go, UI, OpenAPI, workflow lint, platform gate, app catalog, and Fleet pilot workload checks. Companion run 43219500056 passed three-client durable-replica qualification. The API feature remains default-off. No signed release, DO deployment, Fresh-Fleet provider executor, or create workflow was published or run. This is protected source qualification, not live Fresh-Fleet qualification.
Fresh-Fleet workflow graph and fail-closed war-game rerun (2026-11-12 12:30 UTC)
In the existing local norn-fleet worktree, the apply-graph and Fresh-Fleet war-game suites passed 11 tests in 18.770 seconds. The graph requires the persistent-staging effect guard before apply; synthetic eight-of-nine create recovery remains held, and the workflow test uses a fake Norn attempt authority. No provider call or workflow dispatch occurred. The Norn typed grant-route candidate has since merged, but remains unpublished and undeployed; the protected Norn release deployed to DO cannot yet serve it. Provider-executor and protected-workflow integration remain open.
DO staging read-only refresh (2026-11-12 12:26 UTC)
The doctl --context work inventory returned all eleven named Norn staging Droplets active: two management, three control, two ingress, and four app nodes. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.254 seconds with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and version af0f054358d9b9c7b68ea25132b52ee5eb099e12.
The public canary /route-proof and /version endpoints both returned HTTP 200. The version response reported source be5ca5d969ccf3509e62ce04a833516bb5c204db, node de960b60d29f, and allocation 079ea126-ed26-9943-02cc-afa6f7d3f255. This is a fresh point-in-time workload response. It does not prove V3 signed admission/readback, current control-plane or etcd health, continuous serving, or resolve the earlier failed deployment receipt. No provider or deployment mutation ran.
Fresh-Fleet effect-grant routes and ambiguous-response recovery checked locally (2026-11-12 12:24 UTC)
Default-off routes now expose verifier grant issue/readback and exact runner-bound consume/status. A consumed-grant status response returns only the grant digest and consumed flag, not a second successful grant record. This gives the runner a status readback after a lost consume response without replaying the one-use authorization. Issue response loss recovers only the exact matching, unconsumed grant. A lost consume response returns an error; the runner must read status and reconcile before any provider effect or successor admission.
Focused route tests passed, including rejection of a wrong runner binding and checking that status JSON contains no grant field. Three focused real-etcd tests passed for issue/consume one-use behavior, committed issue recovery, ambiguous consume without grant-record replay, and terminal receipt/target-fence handoff recovery. The full v2/api module suite passed. These are local code and storage checks only. No protected CI, provider executor/workflow, DO readback, or provider effect ran; those integration and launch gates remain open.
Fresh-Fleet terminal receipt and target-fence handoff checked offline (2026-11-12 12:10–12:12 UTC)
Norn now records immutable complete-inventory terminal evidence only for the current create attempt after its exact admission and one-use grant are consumed. The verifier-scoped target-registration route then atomically writes the ordinary Fleet target, cluster alias, and free target fence while retiring that intent's pre-registration cluster fence. Ordinary registration still refuses all unrelated active create fences. The new routes remain behind the existing default-off feature and executor-lane gates; this work performed no provider effect or DO change.
A focused real-etcd test passed against disposable etcd 3.7.3 on macOS. It rejects an eight-of-nine terminal receipt, recovers simulated lost responses after terminal write and target handoff, verifies exact replay and target alias/fence readback, and confirms the create fence is retired. The full v2/api module suite passed locally at 12:12 UTC. The updated Norn-specific authority statechart passed nine invariants and deadlock across 19 states/21 steps; its explicit terminal receipt transition now precedes target registration. These are code and deterministic protocol checks only. Norn treats provider/state inventory digests as verifier assertions: no protected provider workflow, fresh-Fleet apply, protected CI run, or provider-backed inventory check occurred. Verifier provenance, a new protected rerun after runner loss, provider-executor and workflow integration, and the live DO workload-admission gate remain open.
DO staging live read-only refresh (2026-11-12 11:17 UTC)
The explicit doctl --context work account readback matched the documented TurnKey Marketing work-staging account (UUID prefix 17f40050), not the personal norn-pilot account. All eleven staging Droplets were active: two management, three control, two ingress, and four app nodes. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.260 seconds, source version af0f054358d9b9c7b68ea25132b52ee5eb099e12, and allocation c776c80f-6ba8-0992-a552-bfc21c441f8a.
The separate public canary https://norn-fleet-canary.tkmkt-dev.com returned HTTP 200 from /route-proof with body norn-fleet-canary and from /version with source be5ca5d969ccf3509e62ce04a833516bb5c204db. Five /version samples reached two allocation IDs on two node IDs. This confirms that the staging canary workload is serving at this time. It does not prove that the new V3 signed workload-admission/readback path authorized it, provide the earlier failed deployment's terminal receipt, or establish DO API, etcd, Nomad/Consul, or continuous health. No provider or deployment mutation ran.
Workload-readback binding names (2026-11-12 11:22 UTC)
Re-ran the repository's name-only audit against the current protected work-staging environment and current work-staging-workload-readback.yml in antiartificial/norn-fleet. Five shared variables and the three state backend secret names referenced by that workflow are present. Eighteen required workflow variables are missing, including NORN_WORKLOAD_READBACK_ENABLED and the contract, inventory, ingress host-key, Nomad/Consul ACL and CA/cert/key, observer, provider-token, provenance, and receipt-directory paths. No variable or secret values were read, and this audit does not validate paths, contents, digests, provenance, or whether the runner can access the files. The protected readback remains disabled and cannot run as configured. No workflow was dispatched.
At 11:28 UTC, the documented 15-module workload-readback, ACL, ingress, controller, and gate-installer suite passed all 146 tests against norn-fleet protected main at a22629597558c0098a920ae69e0f739ad68aef22. These are deterministic tests with fake APIs and local controller fixtures; they do not qualify live credentials, the protected DO runner, successful artifact retrieval, or controller restart against a real readback.
At 11:34 UTC, two statecharts were rerun from the same exact checkout. The workload-readback model passed nine safety invariants and deadlock across 70 states/120 steps. The input-provenance model passed 15 invariants and deadlock across 16 states/16 steps; all eight intentional mutants produced their named counterexample. Outputs were written under /tmp/norn-workload-readback-tlc-20261112 and /tmp/norn-readback-provenance-tlc-20261112. These models verify protocol properties, not runner files, credential provenance, or live staging behavior.
DO staging read-only refresh (2026-11-12 10:45–10:46 UTC)
A read-only doctl --context work compute droplet list inventory found all eleven Norn staging Droplets active: two management, three control, two ingress, and four app nodes. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.250 seconds with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and version af0f054358d9b9c7b68ea25132b52ee5eb099e12, unchanged from 04:55 UTC. This confirms provider-reported power state and one public response at that time; it does not bind the allocation to current app hosts or prove controller admission, continuous health, workload quality, or recovery. No protected workload readback or controller-admitted canary was run.
DO staging read-only refresh (2026-11-12 04:55 UTC)
A read-only doctl --context work inventory found all eleven Norn staging Droplets active: two management, three control, two ingress, and four app nodes. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.176 seconds with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. The current name-only audit found five shared Fleet variables and three state backend secrets present, while all 18 readback-specific bindings are missing from protected work-staging; it read no values and did not validate paths or contents. The public-pilot verifier's 12/12 reads at 02:40:57 UTC remain the latest recorded MySQL pilot check. The /version response does not establish controller admission, allocation-to-host binding, continuous health, or recovery. No protected workload readback or controller-admitted canary has run; the launch date remains uncommitted.
DO staging read-only refresh (2026-11-12 04:02 UTC)
A read-only doctl --context work inventory found all nine tagged norn:tk-staging-nyc3 Fleet Droplets active. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.231 seconds with allocation c776c80f-6ba8-0992-a552-bfc21c441f8a and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. The binding audit found the five shared Fleet variables and three state backend secrets present, with all 18 readback-specific environment binding names missing from GitHub Actions repository and work-staging configuration; no values were read. The only workload-readback run remains a failed candidate-branch push with zero jobs. This confirms provider power state and a public response at that time, not controller admission, allocation-to-host binding, continuous health, or recovery. No protected readback or controller-admitted canary has run; the launch date remains uncommitted.
DO staging read-only refresh (2026-11-12 03:41 UTC)
A read-only doctl --context work inventory found all nine tagged norn:tk-staging-nyc3 Fleet Droplets active. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.176 seconds with allocation de4645bf-25c0-93bd-124b-5e7f9c8cda05 and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. This confirms provider power state and a public workload response at that time. It does not bind the allocation to current app-node identities or prove controller admission, continuous health, or recovery. The public-pilot verifier passed 12/12 reads at 02:40:57 UTC and observed two allocation IDs. A 03:41 name-only audit found all 18 workload-readback-specific environment binding names missing from GitHub Actions repository-level and work-staging configuration; it read no values. The only workload-readback run remains a failed candidate-branch push with zero jobs. No protected readback or controller-admitted canary has run. The launch date remains uncommitted.
DO staging read-only refresh (2026-11-12 03:03 UTC)
A read-only doctl --context work inventory found all nine tagged norn:tk-staging-nyc3 Fleet Droplets active. Public https://staging.tkmkt-dev.com/version returned HTTP 200 in 0.307 seconds with allocation c776c80f-6ba8-0992-a552-bfc21c441f8a and version af0f054358d9b9c7b68ea25132b52ee5eb099e12. This confirms provider power state and a public workload response at that time. It does not bind the allocation to current app-node identities or prove controller admission, continuous health, or recovery. The earlier 12-read verifier passed at 02:40:57 UTC and observed two allocation IDs. The first controller-admitted workload launch date remains uncommitted.
DO V3 platform updated through the release controller (2026-11-11 05:21 UTC)
Norn source fa301fa5f881b847a26f61a501092f9a9dd090fd passed full protected qualification run 42963040596 with all 12 jobs successful. Publisher run 42963531936 published an immutable signed release with the pinned UI and four platform bundles. Protected state readback 42964119843 passed at lineage 9201526d-4ce8-c70b-6a70-7e94a68dc580, serial 23, and checked the three control identities.
The signed one-use controller intent dispatched Fleet run 42964798605 from Fleet main 9db8195ec49def208550a8c98f17537403c3182d. It upgraded the existing Fleet from 34178b8b90d52a2fa3f138ffae2dfc63744197e1 to fa301fa5f881b847a26f61a501092f9a9dd090fd through the all-stop lane. The postflight verified exactly one healthy private API with etcd backend and durable API/UI pins on all three controls. Controller receipt 42964798605.json records terminal-and-runner-retired. Service was interrupted during promotion.
This completes the existing-Fleet V3 platform deployment through a controller. It does not establish a responding application workload, controller-mediated application admission/resize, same-target recovery, or an online platform upgrade. The last provider inventory found all eleven staging Droplets active at 04:49 UTC; no newer provider inventory was taken. No overall V3 launch date is set. The deployment date of this platform release is 2026-11-11.
Mini direct success confirmed against the owner-set workload bar (2026-11-10 15:26 UTC)
Read-only checks were run over the Mini's verified SSH host key. The active ~/norn/current symlink selects signed release 34178b8b90d52a2fa3f138ffae2dfc63744197e1. Local /api/health returned HTTP 200, the bundled V3 CLI reported api ok and required services up, and host-runtime status reported the API, host agent, Nomad, Consul, supervisor, and Cloudflare agent up.
Nomad reported the Norn-managed vigil-gateway job and allocation running. A read-only HTTP GET to that allocation's declared /health endpoint returned HTTP 200. This satisfies the owner's Mini success criterion: V3 deployed and at least one workload responds. It does not claim all applications are healthy or qualify Git-driven admission. The three queued-upgrade failures remain a separate controller-recovery follow-up. No workload, platform, or provider mutation was performed during these checks.
DO V3 platform release and controller recovery installed (2026-11-11 03:16 UTC)
Protected state readback 42943628535 passed at lineage 9201526d-4ce8-c70b-6a70-7e94a68dc580, serial 23. The controller-mediated platform upgrade 42945021070 completed against Fleet source 2d168140c842d8614fe093c9ece28b458926eeaa. It promoted signed Norn release 34178b8b90d52a2fa3f138ffae2dfc63744197e1 through the all-stop lane. The postflight verified the target version, exactly one healthy private API backed by etcd, and durable API/UI pins on all three controls. The all-stop promotion interrupted service. Online upgrade remains unqualified.
Fleet PRs #979, #980, and #981 fixed terminal intent discovery, kept receipt validation in the pinned discovery module, and allowed a protected module refresh without changing the Linux gate. Gate install run 42951258208 installed the reviewed controller and discovery modules. Runner 1036 retired with a terminal receipt, both controller journals cleared, and the installed controller scanned the intent queue with zero eligible entries. The LaunchAgent restarted with the reviewed modules. The Linux runner service remained inactive and disabled. No Fleet node, workload, or provider infrastructure changed during controller installation.
The platform postflight does not prove that an application workload responded. Controller-driven workload admission, mixed-version rollout and resize, online upgrade, and fresh-Fleet creation remain separate open gates. The Mini is an independent direct lane.
DO provider inventory refreshed; dormant bootstrap did not start (2026-11-10 14:58 UTC)
A read-only doctl --context work inventory at 14:53 UTC found all eleven staging Droplets active in nyc3: two management nodes, three controls, two ingress nodes, and four app nodes. This confirms provider instance state only. It does not confirm Norn, etcd, Nomad, Consul, route, or workload health. The norn-pilot context belongs to a different DO account.
Signed DO bootstrap admission merged (2026-11-10 16:16 UTC)
Fleet PR #962 merged as 5400fd8a973266f6e211ce85ee6b3e0d7798cb16. It admits the dormant bootstrap workflow through owner-signed, state-bound intent, the one-use Linux runner controller, and remote job gate. Exact run-scoped runner labels are part of the admission binding. The exact-head contract passed; focused local verification covered 54 tests and the statechart checked 37 states / 53 transitions with nine safety invariants.
The bootstrap feature flag remains false. The merged remote job gate has not been installed on the DO management runner, and no bootstrap dispatch, Fleet install, or provider mutation occurred. The next gate is a reviewed install with the persistent effect runner stopped, followed by fresh runner readiness and a new reviewed intent. Do not replay canceled run 42838769029.
DO runner inventory identified install hold (2026-11-10 16:20 UTC)
A read-only GitHub runner inventory for antiartificial/norn-fleet returned one online self-hosted runner: norn-external-mac-finalizer-xenon, labeled norn-external-mac-finalizer. No registered runner had the required norn-work-management-bootstrap label or the protected Linux norn-fleet-protected effect-runner label. The external-Mac finalizer is a separate purpose and must not be substituted.
The merged work-staging-fleet-gate-install workflow requires the protected macOS management-bootstrap runner, so the remote job-gate install cannot yet be dispatched. No runner registration, install workflow, or provider mutation occurred in this read.
DO staging provider refresh (2026-11-10 16:02 UTC)
A fresh read-only doctl --context work compute droplet list returned all eleven Norn staging Droplets active in nyc3: two management nodes, three controls, two ingress nodes, and four app nodes. The work account is Turnkey Marketing (webdev@tkmkt.com, team Webdev, account UUID 17f40050233d1cd4962b44050e82a0dd01f8dff7). The separate default norn-pilot account is theartificial@hotmail.com (theartificial, team My Team, UUID 05fe610a42b6d50c65344dd87bd842ce35ab036d); it is not the staging account.
This refresh confirms provider-reported VM state only. It does not check the Norn API, etcd, Nomad, Consul, routes, or application response. Genesis readback, platform release/postflight, and workload qualification remain open.
Fleet PR #959 merged as 7f6433b06030d87726aa4055b7ff7b5bfa963f2b, adding exact route-continuity proof and deterministic source-archive and private-vars generation. Protected bootstrap run 42838769029 was canceled before its first step because no protected Linux effect runner was registered. The signed effect controller does not yet admit the bootstrap workflow. The bootstrap gate is off and no host changed.
The two latest Genesis readbacks, 42827741033 and 42831291445, proved backend-byte identity, serial-22-to-23 route continuity, and effective Ansible transport. Both failed installed-source verification before reading Genesis data. The management-host controller remains pinned to 2de3cbd832688a06fc549ec68172145dca4404fc; Genesis units remain disabled. Do not repeat the readback until the approved controller source is installed and verified and a fresh state- and host-key-bound intent is available.
Successor release published; direct Mini upgrade completed, queued path unresolved (2026-11-10 12:01 UTC)
PR #401 merged as 34178b8b90d52a2fa3f138ffae2dfc63744197e1. Exact-source qualification 42809544343 passed all 12 jobs, and protected release run 42810147654 published the immutable signed release with 16 assets. On Mini, passive preflight passed. A fresh protected backup and signed shadow passed with 68 tables, 404,527 rows, ledger 1–52, 66 fingerprinted application tables, and no source mutations.
Fresh queued operation e41badb6-b178-451c-a450-86f367864dc7 failed before promotion at the completed-recovery host-agent check. Its receipt identified LaunchAgent wrapper has no child process; the Mini remained on 2a2d712a, schema 52, with API health ok and mutation fence inactive. An idle process readback subsequently found the managed agent under its wrapper. The operation-window disappearance remains unexplained. The offline statechart now models an absent-agent sample resetting the stability window, but this does not identify or repair the runtime cause.
The same exact release was then promoted through the documented direct controller lane. It preserved 7 queued Fleet dispatch receipts, kept schema 52 unchanged, and reported successful API activation. Authenticated platform smoke passed health, operation drain, releases, events, and platform checks; host status showed API, host-agent, Nomad, Consul, and supervisor up; host doctor passed. This demonstrates the Mini direct lane only. The queued upgrade failure remains open.
Successor release qualified; queued Mini upgrade held before promotion (2026-11-10 11:26 UTC)
PR #400 merged at 1b2dc22973ad22a6ea92ceb866273cab6300821b. Protected exact- source qualification 42804580712 passed all 12 jobs, and protected release run 42805168466 published the immutable signed release. On Mini, its passive preflight passed. The fresh protected backup proof and signed shadow bound to this SHA passed: 68 tables, 404,484 rows, 66 application tables fingerprinted, ledger 1–52, two migrate-only passes, passive health, and no source mutations.
Queued operation 487811ae-03ea-428c-b108-ade3da070721 failed. After a new backup and shadow rehearsal, operation c8e6b37b-6d07-438b-93ab-06a9534548a8 failed with the same generic host-agent stability diagnostic. Source inspection locates this check in recovered_forward_completed_candidate_validate, before the ordinary upgrade promotion path. Authenticated post-attempt readback confirmed /Users/0xadb/norn/current still selects 2a2d712a, API health is ok, schema remains 52, the mutation fence is inactive, and host services are up. These failures did not promote the candidate, run a migration, or change workloads.
An idle live sample later observed matching managed/active host-agent bytes and stable LaunchAgent parent/child PIDs for the sampled interval. That did not reproduce the failing controller-time condition. A follow-up implementation now waits for three matching one-second process observations within 30 seconds and records the specific failed predicate. Its dedicated offline stability regression and complete 24-test platform-upgrade integration suite passed; shell syntax and diff checks passed. The fix remains unmerged and unreleased, so it has not been installed on Mini. The two terminal operations must not be replayed; next live attempt requires the qualified signed follow-up plus fresh candidate-bound backup and shadow evidence.
Mini same-generation finalization completed (2026-11-10 06:30 UTC)
The user authorized the exact Mini immutable publication and all testing/staging work. Protected release run 42765955784 published signed immutable source 5bb691de07c3eb865c5b7f636d07e668d824510c. The exact Mini bundle was imported and signature-verified. The signed repair controller resumed the existing generation gen-1a769550b1051bc9358ce8e8758265e5, transition mini-v3-20261110T0021Z-a4b9c3b3, with source d4dc88e0c37003a1013caed444983b2eeb50fbcb and candidate a4b9c3b313d65c8a054fb869b18455fde33d3a70. It reused the durable preservation receipt and made no new generation or migration.
The first run reached normal API activation and stopped when the host-agent process was not yet stable. The journal preserved the exact retry point. A second run through the same signed controller succeeded: the candidate host-agent remained running, the original API LaunchAgent was restored, and the journal reached candidate-promoted. The API serves the exact A4 source; authenticated platform smoke passed health, operation drain, releases, events, and overall platform checks. Host status returned ok.
The 06:29 inventory returned 30 apps and 46 services but still shows stopped or unhealthy app records, unknown cron/worker services, 25 active incidents, and an inconsistency between active-operation readings (0 in smoke, 7 in inventory). Treat the Mini as having a live V3 platform, not fully healthy workloads. The Mini control plane's staging configuration and historical read-only capacity plans do not prove DO provider or workload health; the last protected DO private state readback remains failed/held. No DO provider/Fleet mutation was made during this Mini finalization.
Prior-controller finalizer repair merged and qualified; publish awaiting approval (2026-11-10 06:07 UTC)
PR #391 merged to master as 5bb691de07c3eb865c5b7f636d07e668d824510c. Full protected V3 qualification run 42765513323 passed. Protected release run 42765955784 passed source authorization, pinned UI, and four platform bundle builds, then entered the protected platform-release approval wait. This exact SHA remains unsigned and unpublished. It has not been imported to Mini or activated. The last Mini runtime read remains the 05:31 authenticated check; the protected release build did not contact or mutate Mini.
Mini maintenance healthy; finalizer controller handoff blocked (2026-11-10 05:31 UTC)
Authenticated read-only checks returned /api/health=ok, A4 source a4b9c3b313d65c8a054fb869b18455fde33d3a70, and the exact pending maintenance status for transition mini-v3-20261110T0021Z-a4b9c3b3. The active generation remains gen-1a769550b1051bc9358ce8e8758265e5 at candidate-preservation-pending; the journal and pointer hashes match the 05:17 read. The preservation receipt exists and the finalization receipt does not. API LaunchAgent and HMAC sidecar remain bound to signed A8 controller a8aa2860c0ecc9ac76b2d852b46cd1ccfb634d63; host-agent LaunchAgent is unloaded.
The imported and signature-verified 2e8 helper's read-only launcher-verify rejects the prior A8 controller binding. The A8 launcher itself verifies against the existing plist and serves authenticated health/version/status. This exposes a cross-release handoff gap: the finalizer needs to preserve the valid A8 binding through the state comparison, then rebind at the receipt-bound normal activation boundary. No live finalizer action or service restart ran. The 2e8 release remains imported and immutable; a follow-on signed repair is required before activation.
Finalizer release imported and reverified; no activation (2026-11-10 05:17 UTC)
The owner-approved immutable release for source 2e8bd545547f4bfa375c68fac9be9667954276de was fetched by exact tag and signature-verified on Mini. The ARM64 bundle was imported into the retained release directory and passed the installed artifact verifier plus the configured external signature verifier. No service was restarted and no recovered-forward action was run.
A new read-only state check confirmed that the current symlink still selects A4 source a4b9c3b313d65c8a054fb869b18455fde33d3a70 and the active generation remains gen-1a769550b1051bc9358ce8e8758265e5, transition mini-v3-20261110T0021Z-a4b9c3b3, at candidate-preservation-pending. The journal and pointer digests remain 3eba3c4e83f025e96959604bf23215ab2434a2cf2b1434b58c44af7ba59e8749 and 84cbb71c138afd08b60b6e9a1c9ea14bf3d2ebccd0f7c6611a9c06d61e4d8e8a. Preservation receipt exists, finalization receipt does not, and the host-agent LaunchAgent remains unloaded. The repair helper must revalidate these exact inputs and create authorization for this digest before any finalizer effect.
Mini finalizer repair release published after exact-SHA approval (2026-11-10 05:10 UTC)
The owner approved the protected platform-release environment for exact source 2e8bd545547f4bfa375c68fac9be9667954276de. Run 42744929675 completed successfully. GitHub reports immutable tag platform-2e8bd545547f4bfa375c68fac9be9667954276de published at 05:08 UTC with platform bundles, manifests, SBOMs, and detached signatures. The approval covered publication only; the repair release was not imported or run on Mini, and no DO Fleet change occurred.
A fresh authenticated Mini inventory at 05:10 UTC returned /api/health=ok and A4 source a4b9c3b313d65c8a054fb869b18455fde33d3a70. It reported 19 of 30 app records running and healthy, two pending, one dead, and eight without Nomad status; 19 of 46 service records passed and 27 were unknown. Several control, host, incident, release, and readiness endpoints returned 404 and remain unknown.
The same 05:10 read resolved the active-generation pointer to gen-1a769550b1051bc9358ce8e8758265e5, transition mini-v3-20261110T0021Z-a4b9c3b3, source d4dc88e0c37003a1013caed444983b2eeb50fbcb, and candidate a4b9c3b313d65c8a054fb869b18455fde33d3a70. The current symlink still selects A4, phase is candidate-preservation-pending, preservation receipt exists, and finalization receipt does not. The journal digest is 3eba3c4e83f025e96959604bf23215ab2434a2cf2b1434b58c44af7ba59e8749; the pointer digest is 84cbb71c138afd08b60b6e9a1c9ea14bf3d2ebccd0f7c6611a9c06d61e4d8e8a. The signed helper must revalidate this state before authorizing same-generation repair; these observations are not a reusable authorization.
Finalizer ancestry correction merged; exact release awaits owner approval (2026-11-10 02:56 UTC)
Norn PR #385 merged as 2e8bd545547f4bfa375c68fac9be9667954276de. It applies signed-descendant verification to recovered-forward-finalize and adds a regression for an adoption receipt whose source is an ancestor of the transition source and candidate. All PR checks passed. Full protected-master qualification 42744444370 then passed all 12 jobs. Exact-source release workflow 42744929675 verified the merged SHA and built the pinned UI and four platform bundles; its protected signing job is waiting for owner approval. Nothing in this workflow deployed to Mini or changed DO staging.
Added a focused recovered-forward finalization statechart. It models the preservation receipt and signed-lineage gate, write-fence release, normal API restart and readback, host-agent restart intent, and terminal receipt. TLC checked the valid path at 52 states / 96 transitions; invalid adoption and candidate ancestry at 3 / 5 each; and unavailable normal API health at 25 / 45. All six configured invariants and deadlock checks passed. Numeric ranks abstract Git ancestry, while booleans abstract candidate evidence and health; the model does not prove signatures, runtime effects, or implementation refinement. Charts and reproduction commands are in the statechart directory.
Read-only host inspection at 02:54 UTC confirmed the Mini current symlink is A4, transition mini-v3-20261110T0021Z-a4b9c3b3 remains at candidate-preservation-pending, and its preservation receipt matches. No finalization receipt exists. The API LaunchAgent is loaded and has a loopback listener, but an unauthenticated health request returned 401, so authenticated API health is unverified by this check. The com.norn.host-agent LaunchAgent label is absent; host supervisor is not running. No write-fence release, normal activation, host-agent restoration, or workload readback occurred.
A 03:12 UTC followup read showed that the top-level recovered-forward/transition.json is an older October record. The current authoritative pointer is active-generation.json, which selects gen-1a769550b1051bc9358ce8e8758265e5; that generation's nested journal is at candidate-preservation-pending. The root-level legacy record was not changed. Future status and recovery checks should resolve the active generation before reading a journal.
Mini active-generation readback confirms preservation-pending (2026-11-10 03:23 UTC)
A 03:23 UTC read-only Mini inspection confirmed that active-generation.json still selects gen-1a769550b1051bc9358ce8e8758265e5 and its nested journal remains at candidate-preservation-pending, with preservationMatched=true and no finalization receipt. The current symlink still selects A4. The API LaunchAgent has a process, the host-agent LaunchAgent label is absent, and the host supervisor has no running PID. This does not establish authenticated API health or workload recovery. The old root-level recovered-forward/transition.json remains unrelated October data; resolve the active-generation pointer before reading the current journal.
Authenticated Mini API responds; generation still awaits finalization (2026-11-10 03:48 UTC)
The read-only inventory authenticated successfully. /api/health returned ok, and /api/version reported source a4b9c3b313d65c8a054fb869b18455fde33d3a70. The inventory included 30 app records and a 46-service manifest with unhealthy or unknown entries; it does not establish complete workload recovery. Unsupported endpoint probes from the inventory client returned 404 and were not used as health evidence.
The active-generation pointer still selects gen-1a769550b1051bc9358ce8e8758265e5, whose journal remains at candidate-preservation-pending with preservationMatched=true and no finalization receipt. The A8 API LaunchAgent is running as PID 93463. The com.norn.host-agent LaunchAgent label remains absent and the host supervisor is not running. No write-fence release or workload mutation occurred.
A8 controller resumed Mini into preservation-pending; finalizer gate found (2026-11-10 02:25 UTC)
After owner approval, protected release workflow 42730463793 signed and published immutable controller release platform-a8aa2860c0ecc9ac76b2d852b46cd1ccfb634d63. The release was imported to Mini and both the external signature verifier and release manifest verifier returned signed.
The controller resumed the existing A4 generation and preserved all seven queued DO-only fleet.github.apply-dispatch receipts. The first API fence attempt observed launchd's asynchronous unload before it completed; the durable journal recorded api-fencing, the listener was later absent, and the same signed controller resumed from that exact phase. Migration 52 then completed, and A4 passed passive preflight. Controller repair authorizations bind the journal digest, so each later phase required a fresh authorization.
The A4 maintenance API initially failed launcher validation because its GUI LaunchAgent did not receive NORN_PLATFORM_REPO from the SSH shell. The canonical Mini checkout path was added to the encrypted API environment, with the original encrypted file retained owner-only. The signed API then passed postflight in recovered-forward preservation mode, and the preservation receipt recorded an unchanged baseline. The API remains fenced in maintenance mode while the host agent remains stopped.
Finalization exposed a separate guard defect: the adoption receipt names an older signed ancestor than the current transition source, while the journal names the current source and A4 candidate. Forward recovery accepts this exact signed ancestry, but recovered-forward-finalize rejected it. No write-fence release or normal API/host-agent activation occurred. The correction and regression test are being prepared before finalization resumes. No DO Fleet change occurred.
Mini read-only inventory refreshed; assurance is still stale (2026-11-10 01:52 UTC)
Authenticated inventory of the independent Mini reports API/host status ok on source d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0), with Consul, Nomad, PostgreSQL, S3, SOPS, and the Nomad/Consul connector up; runtime mutation fence inactive; and 19 of 30 app records healthy. Seven active operations remain queued fleet.github.apply-dispatch reservations. There are 25 active incident groups, and production readiness remains blocked (18 failed, seven passed, one warning). The endpoint's latestAssurance receipt is still dated Sep 6, 2026, despite its current observedAt; therefore host status ok is not fresh host-assurance evidence. This readback does not corroborate the 01:16 entry's claim of a fresh assurance pass, so treat that claim as superseded unless a newer durable receipt is produced. No assurance command, runtime mutation, or DO readback was run.
Genesis readback pre-dispatch hold modeled offline (2026-11-10 01:35 UTC)
Added a second deterministic config for the currently observed pre-dispatch condition: the protected Genesis readback-enable variable is absent and the prior approved state binding no longer matches the configured digest. The workflow must hold before staging an intent or starting a runner. TLC checked the normal recovery path (18 states / 52 steps) and closed/stale gate (2 / 2); all six invariants and deadlock checks passed in both. Charts, reports, and reproduction commands are checked in. This is bounded offline evidence only; no fresh intent, runner dispatch, host action, or provider mutation occurred.
DO Genesis readback binding must be refreshed (2026-11-10 01:29 UTC)
Read-only inspection of the protected work-staging GitHub Environment found that the configured release-state snapshot digest is now 8cf77ae4f5ea5729a8aecde21fe05f82ca36066f66663f2bb5fe3ef14b19060c; the configured management member known-hosts digest remains dac5276e2e4970d19fa270433c405e4522bdb104a1507761f641c2fdae4edccd. The earlier owner-approved readback was bound to state digest 4c43bae56dc9e6c9d624f729dc4398d810043c7f4344df0975637a268c1a00cd, so it cannot authorize a dispatch against the changed configured binding. The matching saved intents are expired. The readback workflow also requires NORN_WORK_PLATFORM_GENESIS_READBACK_ENABLED=true, but that environment variable is currently absent, so the protected job fails closed. No fresh intent or runner dispatch was created. These environment hashes describe configured input bindings, not a new DigitalOcean state/runtime observation; no provider mutation occurred. Obtain current owner-reviewed state/host-key evidence and enable the exact readback gate before a new one-use dispatch.
Mini read-only inventory refreshed (2026-11-10 01:16 UTC)
The authenticated inventory confirms the independent Mini still serves d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0), with API health and host status ok and runtime mutation fence inactive. Nineteen of 30 app records are healthy. At the time of that inventory, the host-status endpoint's latest host.assure receipt was dated Sep 6; a later 01:52 UTC inventory did not corroborate a fresh 01:16 assurance pass. See the latest entry above. This is not a claim that all app records are healthy or that the V3 transition completed. Seven queued operations remain, all known DO fleet.github.apply-dispatch reservations. There are 25 active incident groups, four snapshot-retention warnings, and production readiness remains blocked. The current release has not changed, and the halted recovered-forward generation remains unresolved. No mutation was requested or performed by this inventory. DO staging was not read, so its current runtime health remains unverified.
Focused offline release models rerun (2026-11-10 01:05 UTC)
Nine TLC configurations across eight protocols reproduced on the current tree: Mini recovery (90 states / 179 transitions), workload capacity four (103 / 275) and three (24 / 52), General + Stateful fresh pools (45 / 98), nine-node staging recovery (102 / 210), existing DO release (299 / 1,036), proposed online Norn upgrade (97 / 251), existing-Fleet Genesis (69 / 107), and Genesis source-readback gate (18 / 52). All configured invariants and deadlock checks passed. Outputs stayed under /tmp; no provider or host action occurred. These bounded checks do not prove implementation refinement, secret lifecycle, provider effects, runtime health, or live Nomad placement.
DO staging plan refreshed without provider mutation (2026-11-10 00:55 UTC)
Protected norn-fleet main 35e1c893198efbdc9e407d708e5096a32b485f7d completed staging-only read-only plan 42731843397. The private plan summary reported one create for digitalocean_firewall.management_ingress_runtime[0], 28 no-op resources, and zero newly created nodes. The firewall attaches to two ingress Droplets and allows TCP 18082 for the observer and 18083 for the publisher from the management runner's private /32. No provider apply ran. This repeats the ingress firewall proposal recorded in PR #949; it does not prove live Norn API, etcd, Nomad/Consul, or workload health. Staging remains the DO testbed and is separate from the independent Mini. No production environment was selected.
Mini recovery statechart aligned to observed fence and schema (2026-11-10 00:46 UTC)
The Mini direct model previously advanced from schema 49 to 51 and allowed a generic same-intent retry after any held phase. The live Mini was at schema 51 and stopped at host-agent-fenced, before API fencing or migration; ordinary retry is rejected there. Updated the model to cover schema 51 to 52 and require a distinct signed repair-controller action after this exact halt. TLC explored 90 states and 179 transitions; eight invariants and deadlock checking passed. The chart keeps the observed recovery path visible with 18 arrows. The modeled signature, controller identity, source-lineage verification, Launchd, SQL and filesystem durability still need implementation-refinement evidence. No live Mini or DO action was performed.
Mini post-stop runtime inventory (2026-11-10 00:33 UTC)
The authenticated post-stop inventory confirms the Mini API remains on source d4dc88e0c37003a1013caed444983b2eeb50fbcb, schema 51, and health ok; Consul, Nomad, PostgreSQL, S3, SOPS, and the Nomad/Consul workload connector are up. The runtime mutation fence remains inactive and the exact signed a4b9c3b3 release is installed but not current. Nineteen of 30 app records remain running and healthy. The host-agent LaunchAgent is loaded but repeatedly exits at the durable fence, so host operations are unavailable even though current API and workload processes remain online. Seven queued DO Fleet dispatch receipts and 25 incident groups remain. The latest host-assurance receipt is still dated Sep 6; this readback is not a fresh host assurance. Production readiness is blocked. No Mini database migration, candidate promotion, DO readback, Fleet dispatch, or provider apply occurred.
Mini recovered-forward stopped before API fence and migration (2026-11-10 00:27 UTC)
The exact published release a4b9c3b313d65c8a054fb869b18455fde33d3a70 was imported into Mini and passed local manifest plus external signature verification. Passive preflight stopped before database mutation because schema 52 is required. A fresh protected backup was created at /Users/0xadb/.config/norn/m5-v3-rc1/backup/control-20261110T001956Z-39515a89.dump with proof and SHA-256 2ff131df39ac532cc59112aaa890f0d7480adac0d2c7c8f14bbd937681ed1911. The candidate-specific signed shadow passed private restore, two migrate-only runs, complete ledger through 52, passive health, and exact backup verification.
Transition mini-v3-20261110T0021Z-a4b9c3b3 created generation gen-1a769550b1051bc9358ce8e8758265e5 and acquired the promotion lock. The managed controller fenced and stopped the legacy norn-host-agent, then stopped at durable phase host-agent-fenced. It made no API fence, schema migration, or candidate promotion. The source API remains healthy on d4dc88e0, schema last observed at 51, with runtime mutation fence inactive.
The wrapper then received 404 from /api/v1/platform/recovered-forward/status because the source API is not serving the candidate's recovery-maintenance status surface yet. The exact same transition retry was rejected because the normal generation resolver does not permit the host-agent-fenced phase. A read-only launchctl bootstrap attempt did not restart the host agent; its managed wrapper correctly refused while the durable fence was active. The source release's signed controller-repair helper then rejected the new source-lineage evidence before creating a repair authorization. No journal, active-generation pointer, backup, or shadow evidence was edited or removed.
Do not use a new transition ID, bypass the fence, or edit the journal. Resume with a distinct later signed controller that contains the lineage verifier fix, using the generation's signed controller-repair path. After promotion, verify schema 52, exact API source, workload preservation, host-agent restoration, smoke, host assurance, and terminal receipt. No DO readback, Fleet dispatch, or provider apply occurred.
Mini authenticated read-only inventory refreshed (2026-11-10 00:15 UTC)
The machine-specific norn_inventory.sh --remote mini read completed authenticated API reads for health, version, host status, platform rollup, active operations, events, releases, and workload inventory. The Mini is still on source d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0), API health and host status are ok, and the runtime mutation fence is inactive. The exact replacement release a4b9c3b313d65c8a054fb869b18455fde33d3a70 is not in the installed release list. Nineteen of 30 app records are running and healthy. Seven active operations remain queued fleet.github.apply-dispatch receipts; all are the known DO Fleet reservations. The inventory reports 25 open incident groups, including old allocation-loss events, and production readiness is blocked. The most recent host-assurance receipt shown by the host-status endpoint is from Sep 6, so it is not current assurance evidence. This read did not obtain fresh schema or recovered-forward generation state.
No release import, backup, shadow, preflight, transition, Mini mutation, DO readback, GitHub dispatch, or provider apply occurred. The next safe step is exact signed release import and verification, followed by fresh candidate-bound backup, shadow, and preflight receipts. Keep the migration gate closed until the recovered-forward controller proves its current prerequisites.
Recovered-forward replacement release published (2026-11-10 00:10 UTC)
After exact-SHA owner approval, protected release run 42721807811 completed signing, verification, and publication for source a4b9c3b313d65c8a054fb869b18455fde33d3a70. GitHub exposes the immutable signed prerelease as platform-a4b9c3b313d65c8a054fb869b18455fde33d3a70. The run had passed its 12-job protected-master qualification and built the pinned UI plus Linux and macOS bundles for amd64 and arm64.
This was artifact publication only. No Mini import, Mini transition, DO readback, Fleet dispatch, or provider apply occurred. The last recorded Mini runtime snapshot is still the 22:47 UTC Nov 9 inventory; refresh Mini state before importing the release. The seven stale DO dispatch receipts remain queued because the current-lane binding guard stops reconciliation before GitHub lookup. PR #375's later actions:read permission correction is not in this immutable release.
Dispatch recovery now requests read-only GitHub Actions access (2026-11-10 00:01 UTC)
Norn PR #375 merged at 66690f93. RecoverBoundPlan now requests only the GitHub App actions:read permission. The recovery tests require that scope and verify that recovery never sends another workflow dispatch. Full PR CI passed, including API tests, Fleet pilot workload, and the three-client durability check.
This does not resolve the seven stale dispatch receipts. The reconcile handler still rejects their saved bindings against the current lane before it calls GitHub. The records remain queued, no GitHub lookup ran, and no receipt, Mini, or DO state changed. PR #375 is code only and is not in the a4b9c3b3 release candidate. Keep the current lane guard on ordinary dispatch and replay paths. Any historical lookup path must validate the durable saved binding and the exact external run before it writes a terminal receipt.
Verifier fix fully qualified; replacement release awaits owner approval (historical, 2026-11-09 23:42 UTC)
Norn PR #373 merged at exact source a4b9c3b313d65c8a054fb869b18455fde33d3a70. Its full protected-master qualification 42721291745 passed all 12 jobs. Exact-source release run 42721807811 passed authorization, pinned UI build, and all four platform bundles. Its protected publish job is waiting for owner approval; the release has not been signed or published. The Mini and DO were not changed. The already published 9d9d6208 release predates this verifier correction, so the exact replacement SHA must be published before repeating the Mini recovery sequence. The replacement workflow does not authorize or perform that Mini transition.
Recovered-forward verifier source-lineage fix merged (2026-11-09 23:29 UTC)
The signed Mini transition for candidate 9d9d6208654b995cddc8b76e13d089c665c52fd6 stopped before fencing or database mutation. Its evidence verifier incorrectly required the adopted source SHA to equal the currently serving source, even though the signed adoption source e117d5645f7764ac6ba6bc316aa19fa241521655 is an ancestor of current d4dc88e0c37003a1013caed444983b2eeb50fbcb. PR #373 fixes this by requiring both releases to verify as externally signed and checking exact Git ancestry, while keeping the historical incident candidate exactly bound. Its positive and negative regression tests passed, as did all required CI, including API/CLI tests, the Fleet pilot workload, Web UI build, and the three-client durable-replica qualification. It was squash-merged at a4b9c3b313d65c8a054fb869b18455fde33d3a70.
The approved 9d9d6208 candidate is already immutable and signed, but it does not contain this verifier fix. The Mini remains on d4dc88e0; health is ok, the runtime mutation fence is inactive, and no transition generation pointer, host fence, migration, or release promotion was made by the failed attempt. The attempt-specific r3 evidence was preserved and must not be reused. The fix needs protected-master qualification and a new exact-source release; the prior publish approval was specific to 9d9d6208 and does not authorize that new release. After publication, repeat candidate-specific import, backup, shadow, preflight, and recovered-forward admission with a fresh transition ID.
Seven stale Fleet dispatch receipts remain unreconcilable on current lane (2026-11-09 23:08 UTC)
The user approved another protected, lookup-only reconciliation attempt for the seven stale fleet.github.apply-dispatch receipts. A fresh authenticated Mini inventory confirmed all seven are still queued. Review of the deployed V3 handler shows it checks that each saved dispatch binding matches the currently configured Fleet environment before calling the lookup-only GitHub recovery client. These seven bindings fail that guard, so the endpoint returns fleet_github_dispatch_binding_mismatch without performing the GitHub lookup or changing the operation. This was already observed for each receipt at 21:31 UTC; repeating the POST against unchanged configuration would only produce another audited 409.
No receipt was canceled, rebound, or otherwise changed. No GitHub workflow dispatch or provider apply occurred. Do not edit these operation rows or temporarily change the Fleet lane to force a match. A future code path would need to prove each immutable saved binding against its exact GitHub run using read-only scope, persist only a positive recovered or verified-no-write/canceled result, and leave ambiguous cases queued. Until then the seven receipts remain preserved for the signed recovered-forward Mini transition; they are not individually reconciled and must not be counted as a cleared Fleet drain gate.
Fresh-Fleet General + Stateful role path verified (2026-11-09 23:02 UTC)
Read-only inspection of protected norn-fleet main 35e1c893198efbdc9e407d708e5096a32b485f7d found that the selected General + Stateful application-pool pair is now supported by its source: sanity.py accepts exactly the legacy single app pool or the two distinct general/stateful workload labels; generated inventory preserves each pool ID and role under the app group; bootstrap-topology.yml enforces at least two clients per app pool; and Nomad bootstrap registers each distinct pool before clients join. The Nomad client template keeps the real node-pool ID and derives its class from the workload role.
Three focused tests against that exact source passed for topology validation, inventory rendering, and Nomad bootstrap wiring. The checked-in staging configuration remains one workload: app pool. Stable future IDs are now recorded as general and stateful; no fresh-Fleet profile or provider plan was created. SKU, capacity, budget, placement/workload policy, stateful data recovery, and joined persistent apply/recovery with exact provider identities and terminal receipt remain open. This was a source review and synthetic test run only; no DigitalOcean read, apply, or host change occurred. The original norn-fleet checkout had unrelated uncommitted changes and was left untouched; verification used a separate clean worktree.
Recovered-forward repair merged; exact-source release prepared (2026-11-09 22:56 UTC)
Norn PR #370 merged at exact source 9d9d6208654b995cddc8b76e13d089c665c52fd6. It repairs the recovered-forward source-evidence path binding and gives legacy-baseline and recovered-forward separate queued Fleet receipt policies. The full protected-master CI dispatch 42713422353 completed successfully with all 12 jobs. Exact-source release workflow 42714046058 passed protected-master authorization, pinned UI build, and all four Linux/ Darwin amd64/arm64 bundle builds. At 22:56 UTC, its protected signing and publish job remains waiting at the owner-approval environment. No signature, immutable release, or Mini deployment has been created by that workflow.
All 21 checked-in TLC configurations were rerun from current master19297662 with the statechart skill into scratch outputs under /tmp (no repository artifacts or provider actions): Mini direct (82 states / 164 steps), existing DO platform release (299 / 1,036), fresh-Fleet create (308 / 813), current nine-node staging recovery (102 / 210), future two-app-pool Fleet (45 / 98), existing Fleet Genesis (69 / 107), Genesis readback source gate (18 / 52), proposed online Norn upgrade (97 / 251), mixed-version workload resize at capacity 4 (103 / 275) and capacity 3 (24 / 52), workload route publication (55 / 111), workload stop recovery (10 / 15), release admission (458 / 1,663), admission replay after restart (115 / 365), simultaneous HTTP admission (19 / 28), and private-input retention (12 / 25). The four missing-evidence admission configurations each explored 4 states / 6 steps and rejected incomplete admission; concurrent two-client store admission explored 14 states / 18 steps. Every configured rule passed. Deadlock checks passed where enabled; the concurrent-admission configuration intentionally disables deadlock checks for terminal stutter states. The capacity-3 resize model still has no target-only route or terminal receipt; capacity 4 does. These models validate bounded protocol contracts only; they do not establish code-to-model refinement, fresh-Fleet workflow integration, live DO readiness, workload availability, or Mini runtime state.
The latest authenticated Mini inventory was refreshed at 22:47 UTC: API/host ok, runtime mutation fence inactive, source d4dc88e0, schema 51, floors 5/33, and the latest host-assurance record succeeded. Nineteen of 30 app records were running and healthy; two were pending, one dead, and eight had no Nomad status. Seven active operations remain queued Fleet dispatch receipts; all 25 active incident groups and four snapshot-retention warnings remain. Production readiness is separately blocked (7 passed, 1 warning, 18 failed). The Mini can host current workloads; this is not evidence that all app records are healthy. The DO provider/runtime snapshot was not refreshed in this turn; its latest evidence remains 18:58–19:01 UTC and needs a new protected readback before claiming current Fleet health. No direct local provider query or provider mutation occurred. The exact-source replacement must be approved and published before importing it or preparing new source-bound backup and shadow evidence.
Mini recovered-forward gate refresh (2026-11-09 22:20 UTC)
Fresh authenticated Mini inventory at 22:08 UTC reported API and host ok, source d4dc88e0c37003a1013caed444983b2eeb50fbcb, schema 51, and reader/writer floors 5/33. The exact-source protected backup and signed e9a21d20 shadow receipt were checked: the artifact digest/size match the proof, and the shadow receipt records the same backup and candidate plus passing private-restore and runtime claims. Passive preflight stopped before DDL because migration 52 is required, as expected.
The standard upgrade invocation was refused before migration or promotion: the signed active recovery generation completed for candidate bba290db, while Mini currently serves its signed descendant d4dc88e0. The generation resolver and completed receipt verify; the current symlink still names d4dc. The successor preparation code verifies signed ancestry to bind that current source, so the old journal must remain unchanged.
Source review found two release blockers in the recovered-forward path. The candidate wrapper stages successor evidence in a private .generation-inputs directory that the candidate controller's evidence-path allowlist rejected. The lane drain helper was also shared with legacy-baseline, so its queued Fleet-receipt refusal would block the recovered-forward lane that must preserve those receipts. This worktree now validates the exact hash-derived, mode-0700 generation-input path and separates recovered-forward preservation from legacy-baseline refusal. Focused Go tests, all 21 platform-upgrade integration tests (Node 24.19.0 via NVM), and the full startup package passed (62.5 seconds). An attempted direct internal controller invocation without wrapper-generated evidence was rejected at argument validation; no generation pointer, host fence, migration, or release changed. The runbook now specifies the signed candidate wrapper as the entrypoint. These local fixes are not merged or signed; publish a new exact-source candidate and repeat the fresh backup/shadow gates before running the wrapper. The Mini database and release remain unchanged, and the seven stale DO receipts remain queued after their lookup-only reconciliations returned 409.
Mini release candidate and queued Fleet receipt review (2026-11-09 21:31 UTC)
The protected platform-release approval published immutable signed prerelease platform-e9a21d2095d60c8a96c1440e35e045dd0b6a7637 from exact source e9a21d2095d60c8a96c1440e35e045dd0b6a7637. The Darwin/arm64 bundle passed signature, manifest, archive-inventory, SBOM, and source-SHA verification. Mini remained healthy on d4dc88e0c37003a1013caed444983b2eeb50fbcb, schema 51. Candidate passive preflight required schema 52 and stopped before DDL or promotion; this is a safe hold, not an upgrade failure after mutation.
The seven stale DO dispatch receipts were submitted individually to the lookup-only reconciliation endpoint. Each returned 409 fleet_github_dispatch_binding_mismatch because the reservation's Fleet lane no longer matches current configuration. None was reconciled or modified, and no dispatch/apply was triggered. Preserve these receipts through Mini schema migration; do not rebind them to the present lane.
The live Mini /api/schema read reported active norn.startup/v2, schema 51, and reader/writer floors 5/33. The signed e9a21d20 candidate requires schema 52 with unchanged floors 5/33. The ordinary direct upgrade path is therefore contract-compatible after exact-candidate backup and private-shadow proof; its preflight migration-required result is an expected stop before DDL.
An initial source fix attempted to broaden queued-dispatch preservation to legacy-baseline. Exact-SHA qualification run 42706816151 failed the test_recovered_forward_alone_preserves_queued_fleet_apply_dispatches contract, which deliberately binds that exception to signed recovered-forward evidence. Corrected behavior refuses legacy-baseline before fencing when any queued Fleet dispatch exists. The ordinary Mini upgrade does not use that quiescent migrator. The corrected guard passes the focused Go tests and the 20 controller integration tests with Node 24.19.0 selected from the existing NVM install; it still needs merge and exact-SHA protected qualification. The Mini has not migrated or promoted the candidate. A fresh protected backup and private restore proof remain required before ordinary upgrade.
DO token-scope admission (2026-11-09 20:27 UTC)
norn-fleet PR #948 merged as 92b615d after its exact-head contract and security checks passed. The change requires an owner-attested scope record and redacted grant evidence bound to the run's account UUID and exact DigitalOcean token fingerprint. It rejects broad global aliases, missing plan-derived mutation scopes, stale evidence, and unsupported provider actions. The bounded admission statechart and offline tests passed. This is an offline contract, not DigitalOcean introspection or proof that the protected runner's live token has those grants.
Merged-main read-only plan run 42693143436 completed successfully. The staging plan proposed only management_ingress_runtime[0], which allows the exact private management runner to reach ingress ports 18082 and 18083. No provider apply ran. The production target was unconfigured and skipped. The separate norn-fleet runbook update in PR #949 records this repeated plan result.
DO staging provider inventory (2026-11-09 18:58 UTC)
Read-only doctl --context work queries found all nine tk-staging-nyc3 Droplets active (3 control, 2 ingress, 4 app), both staging-management Droplets active, and the tk-staging-nyc3-ingress load balancer active. The management-authority PostgreSQL and application PostgreSQL/MySQL databases report online. A read-only public HTTPS probe at 19:01 UTC returned norn-fleet-canary from /route-proof; five /version samples reported source be5ca5d969ccf3509e62ce04a833516bb5c204db across two allocation IDs on two nodes. This confirms the canary route is serving that source, but does not verify the DO Norn API, etcd quorum, Nomad/Consul health, application catalog, or terminal receipt for the earlier failed deployment. No provider, host, deployment, or route mutation was run.
Read-only Mini and DO refresh (2026-11-09 15:48–15:49 UTC)
The authenticated Mini inventory at 15:48 UTC reported API and host status ok, an inactive runtime mutation fence, and Consul, Nomad, Postgres, S3, SOPS, and the Nomad/Consul workload connector up on signed V3 repair release d4dc88e0c37003a1013caed444983b2eeb50fbcb. It contained 30 app records and 46 service records (19 passing, 27 unknown), with 21 of 21 allocations running. Seven fleet.github.apply-dispatch receipts remain queued from Oct 8–10; 25 incident groups remain open, and snapshot retention is over limit for four apps. Production readiness generated at 15:48:50 UTC remains blocked (7 passed, 1 warning, 18 failed). The latest assurance operation succeeded but its operation timestamp is still Sep 6; this refresh did not run route assurance. The Mini's Nov 7 direct adoption remains complete; no new upgrade was attempted.
Read-only doctl --context work queries at 15:49 UTC found all nine tk-staging-nyc3 Droplets active, the ingress load balancer active, and the management-authority PostgreSQL plus application PostgreSQL/MySQL databases online. This provider inventory does not verify current Norn API, etcd, Nomad/Consul, application catalog, or route health. The canary proof remains the separately dated 13:09 UTC probe recorded below. No provider, host, deployment, or route mutation was run.
Read-only Mini and DO refresh (2026-11-09 14:59–15:02 UTC)
The authenticated Mini inventory at 14:59 UTC reports API health and host status ok on signed V3 release d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0). It reports 30 app records, 46 services (19 passing, 27 unknown), seven queued fleet.github.apply-dispatch records last updated Oct 8–10, 25 active incident groups, and four snapshot-retention warnings. Production readiness is blocked (7 passed, 1 warning, 18 failed). The latest host assurance receipt is still the successful operation from Sep 6; this read did not run route assurance. These queued records and incidents remain unresolved; this read made no mutation.
Read-only doctl --context work reads at 15:02 UTC found all nine tk-staging-nyc3 Droplets active (3 control, 2 ingress, 4 app), the ingress load balancer active, and the management-authority PostgreSQL plus application PostgreSQL/MySQL databases online. The latest public canary probe remains the 13:09 UTC read: /route-proof returned norn-fleet-canary, and five /version responses reported source be5ca5d969ccf3509e62ce04a833516bb5c204db across two allocation IDs and two node IDs. Provider activity and that earlier route proof do not establish current DO Norn API, etcd, Nomad/Consul, or full catalog health. No provider, deployment, route, or host mutation was run.
Admission replay recovery model merged (2026-11-09)
Norn PR #333 merged as 443ca8404b8188c004d6229871b34792f0912c37. Its TLA+ model checks durable admission acceptance followed by a lost response, controller crash/restart without process-local acceptance memory, and same-key replay from the durable receipt. TLC explored 115 states and 365 steps; ten invariants and deadlock checking passed. This strengthens offline protocol evidence only. PRs #335 and #337 later exercised real-etcd HTTP replay across response loss and handler/ adapter reconstruction; PR #339 exercised shared-reconciler recovery across an etcd-backed successor claim with controlled external-effect fakes. Fresh HTTP admission with real qualification/private-input persistence and full production executor/worker restart remain unproven; no deployment or workload mutation was performed.
Read-only runtime and provider refresh (2026-11-09 12:29 UTC)
The authenticated Mini inventory again returned API and host status ok on source d4dc88e0c37003a1013caed444983b2eeb50fbcb. It reported 30 app records, 46 services (19 passing, 27 unknown), seven queued GitHub apply-dispatch receipts last updated Oct 8–10, 25 incident groups, and four snapshot-retention warnings. Production readiness generated at 12:29 UTC remains blocked (7 passed, 1 warning, 18 failed). This read did not run route assurance; the latest recorded assurance is still dated Sep 6.
Read-only doctl --context work queries found all nine tk-staging-nyc3 Droplets, both management Droplets, and the staging ingress load balancer active. These provider reads do not establish database, Norn/API, etcd, Nomad/Consul, workload, or public-route health. DO runtime health and Genesis readback remain unverified. No provider mutation, deployment, or host change was made.
DO staging provider and canary refresh (2026-11-09 13:09 UTC)
Read-only doctl --context work queries found all nine staging Droplets active (3 control, 2 ingress, 4 app), both management Droplets active, the ingress load balancer active, and the management-authority PostgreSQL plus application PostgreSQL/MySQL databases online. Verified public HTTPS probes returned 200 from /route-proof with body norn-fleet-canary; five /version reads all reported source be5ca5d969ccf3509e62ce04a833516bb5c204db across two allocation IDs and two node IDs. /api/health on the canary host returned 404. This confirms provider resources and the canary application route are serving, not Norn API health, etcd quorum, Nomad/Consul control-plane health, catalog state, or a terminal receipt for the earlier failed deployment. No provider mutation or deployment ran.
Fresh release-readiness reconciliation (2026-11-09 12:25 UTC)
Norn PR #324 merged at 3fcdc154e247d5b32a8ad025e4fb9690390b5022. It fixes a race where two independent etcd clients submitted the same workload release request, both missed the initial replay read, and the fence loser returned an error instead of the winner's durable acceptance. The two-client integration test passed ten consecutive runs against local etcd, a focused race-detector run, and the full API module suite. The pre-fix five-step counterexample and fixed 14-state / 18-transition model are checked in. This proves store-level same-key convergence only; concurrent HTTP handlers, worker/controller restart and response loss, and production drivers remain open. No live workload release was run.
An authenticated Mini inventory at 11:59 UTC returned API and host status ok, source d4dc88e0c37003a1013caed444983b2eeb50fbcb, 30 app records, 46 services (19 passing, 27 unknown), seven queued Fleet GitHub dispatch receipts, 25 incident groups, and four snapshot-retention warnings. Production readiness is blocked (7 passed, 1 warning, 18 failed). The latestAssurance receipt is still succeeded but dated 2026-09-06; this inventory did not run assurance, so public-route assurance is not fresh.
At about 12:00 UTC, read-only doctl --context work queries found all nine staging Droplets, both management Droplets, and the staging ingress load balancer active. The queries did not inspect the managed databases or any Norn API, etcd, Nomad/Consul, workload, or route state. DO runtime health remains unverified. Fleet PR #837 at head eef00505d23e37c66cdaed75c5f4c5504a1cd22b merged at 12:22 UTC as 6335c667693d0333d4239f91edf2efb9772ab63a. Its exact-head contract rerun 42578247221 passed after the earlier transient DigitalOcean provider v2.100.0 download failure; both Socket security checks also passed. This merged source adds a default-off protected ingress-enrollment prerequisite. No host was enrolled, provider mutation ran, or Genesis readback occurred. Fresh owner- reviewed inputs and separately authorized one-use dispatch are still required.
Release merge and gate update (2026-11-09 05:35 UTC)
Norn PR #300 (release lanes, milestones, and V3 API/client cleanup), #301 (Mini route assurance), and #302 (offline workload release controller) are merged to master; #302 is at c792564c9440805a467a832a4fcb6b433a1f68c3. The Mini remains an independent node, and #301's code merge is not a fresh host readback or a claim that production route assurance is complete. #302 is source and offline evidence only; the operator API/CLI admission path and live workload qualification remain open.
Fleet PRs #940 (release intent controller), #941 (fresh-Fleet offline war-game), and #945 (provider scope contract) are merged. Stacked PR #943 closed when #941's base merged; its credential-lifecycle changes were rebased as draft PR #947. PR #942 moves tests to module-owned collection. Both refreshed PR contracts failed at OpenTofu provider installation after GitHub returned 504 responses fetching the DigitalOcean provider checksum or signature. Earlier contract phases passed. GitHub branch policy blocks merge until the required contract passes; it was not bypassed. No provider apply, deployment, or Genesis readback ran as part of these merges.
This file preserves the dated observations and decision trail formerly collected in LAUNCH-STATE.md. Many entries are historical snapshots and must not be read as current status. For current gates and the next actions, see Norn v3 launch state. Use the relevant lane runbook for operator procedures.
DigitalOcean staging provider readback (2026-11-08 09:36 UTC)
Read-only DigitalOcean API queries using the configured work auth context confirm that the independent DO staging testbed exists and its provider resources report active/online. The inventory matches staging document antiartificial/norn-fleet@4451767624b12cbc41cc32e160a10d4b46006e0f:
- 3
control, 2ingress, and 4appDroplets are active innyc3, with the expected pool tags and sizes. tk-staging-nyc3-ingressis an activenyc3load balancer.tk-staging-nyc3-application-postgres(2 nodes) andtk-staging-nyc3-application-mysql(3 nodes) report online.- Project
norn-stagingand VPCtk-staging-nyc3(10.43.0.0/20) are present.
The default norn-pilot doctl context is a different DigitalOcean account and does not show this testbed; repeat future provider checks with the explicit work context. A second read-only inventory at 2026-11-08 12:41 UTC again found all nine tagged Norn staging Droplets active and the three Norn-related managed databases online. At 13:52 UTC, a fresh provider-only inventory again found all nine active (3 control, 2 ingress, 4 app), the ingress load balancer active, and management-authority PostgreSQL plus application PostgreSQL/MySQL online. These queries do not check node APIs, cluster membership, workloads, or routes; provider status is not runtime health.
The latest protected Genesis readback remains failed at the source capability check (run 42418784411). It passed runner, installed-input, and private-transport checks, then the installed controller refused because its protected source repository did not contain the pinned capability commit or signed target commit. The run used Oct 21 state and route pins; do not reuse them as current evidence. Diagnose the missing source objects, refresh state/route/host-key evidence, and bind any new read-only intent to those refreshed inputs before dispatch.
Fresh DO staging provider inventory (2026-11-09 01:51 UTC)
An explicit doctl --context work readback matched the pinned staging account and returned nine tk-staging-nyc3-* Droplets, all active in nyc3: 3 control, 2 ingress, and 4 app. This rechecks provider VMs only; it does not recheck load balancers or managed databases and proves nothing about Norn API, Nomad/Consul, workloads, routes, or Genesis journal health. The protected Genesis readback remains unpassed.
DO staging provider recheck (2026-11-09 02:17 UTC)
The explicit work account again matched the pinned staging account and was active. All nine NYC3 Droplets were active (3 control, 2 ingress, 4 app); the tk-staging-nyc3-ingress load balancer was active; both managed application databases were online (MySQL 3 nodes, PostgreSQL 2); and the staging VPC and norn-staging project were present. No provider mutation ran. These reads do not prove Norn API, Nomad/Consul, workloads, routes, or Genesis journal health; the protected Genesis readback remains unpassed.
DO staging provider recheck (2026-11-09 02:48 UTC)
Read-only queries with explicit doctl --context work confirmed the pinned account remains active and again returned nine active NYC3 staging Droplets (3 control, 2 ingress, 4 app), an active ingress load balancer, online application MySQL (3 nodes) and PostgreSQL (2 nodes), and VPC 10.43.0.0/20. No provider mutation ran. This is provider availability only; it does not establish Norn API, Nomad/Consul, workloads, or routes as healthy. The latest protected Genesis readback remains failed at run 42418784411 because the management source repository lacked pinned Git objects. The successful run 42419030676 is a separate release-state handoff, not a Genesis or runtime readback. Do not describe the DO Norn runtime as online based on either provider status or that handoff.
Latest Mini direct release (2026-11-08 09:20 UTC)
The independent Mini advanced from signed V3 RC bba290db041e9f497c7161543c679d9d348b742d (v3.0.0-rc.1-platform-251-gbba290db) to signed V3 RC d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0) through the direct platform-upgrade controller. The exact target was already installed, signature-verified, and passed passive candidate preflight before promotion. The upgrade retained schema version 51, reader floor 5, writer floor 33, and the database identity; it used the existing restart lane and did not claim zero interruption or online schema migration.
Postflight evidence: norn platform smoke passed health, operation drain, release marker, and event checks; norn host status showed Consul, Nomad, API, host agent, supervisor, assurance timer, and cloudflared up/armed; norn host doctor passed. The smoke reported activeOps=0 for Mini-affecting work while preserving seven queued Fleet GitHub dispatch receipts. Those records remain queued and are not Mini work or running provider applies. A read-only inventory at 2026-11-08 12:41 UTC again observed /api/health=ok, host status ok, and the same exact V3 RC source SHA. It also retained the seven queued Fleet dispatch receipts and 25 grouped open incidents. The latest host assurance receipt predates the upgrade, so this does not claim fresh route assurance or clear those incidents. Production readiness remains blocked and is not the Mini direct upgrade gate. No mutation was performed during this inventory.
Node v24.19.0 and pnpm 10.32.1 are now installed in the Mini user's NVM version. Non-interactive maintenance still needs the NVM bin directory on PATH and explicit NORN_NODE_BIN; the managed platform helper does not automatically load NVM. No UI rebuild was needed because the exact signed target bundle was present. The installed release is V3 RC, not a final stable release. PR #302 remains open and was not deployed.
Fresh Mini live check (2026-11-09 01:50 UTC)
The read-only authenticated inventory still reports /api/health=ok and V3 source d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0). norn host doctor exited 0 and passed current managed-file, LaunchAgent, runtime, and Cloudflare checks. The latestAssurance receipt is still the successful operation from 2026-09-06 17:23:57 -05:00, not fresh assurance; 25 active incident groups remain (22 critical, 3 warning). Host assurance and route reconciliation therefore remain open. No route or host mutation was run.
The Mini API's configured Fleet document names norn-disposable-nyc3 and declares only 3 control + 2 ingress nodes. This is configuration held by the independent Mini, not proof that Mini belongs to a Fleet or a readback of the separate DO staging topology.
Fresh Mini read-only inventory (2026-11-09 02:32 UTC)
The authenticated norn_inventory.sh read returned /api/health=ok, host status ok, and V3 source d4dc88e0c37003a1013caed444983b2eeb50fbcb (v3.0.0-rc.1-platform-252-gd4dc88e0). It found 30 app records, 46 service records, seven queued fleet.github.apply-dispatch receipts, and 25 active incident groups (22 latest-critical, 3 latest-warning). The latest successful host-assurance receipt remains from 2026-09-06 17:23:57 -05:00. This read did not run host assurance or change routes; route assurance remains unverified. The API's fleet_configured=true describes the separate target configuration with control and ingress pools, not Mini Fleet membership. The raw owner-only inventory is temporary and is not a release artifact.
Mini event-reconcile dry-runs and route fix qualification (2026-11-09)
- At 2026-11-08 23:55 UTC, the read-only event-reconcile dry-run scanned 200 events. It would acknowledge 54 task-restart events under the deterministic rule; 146 required review, including host-assurance failures, failed/lost allocations, and capacity warnings without later recovery evidence. No event was acknowledged.
- At 00:55 UTC, a second dry-run scanned 200 events: 53 matched the deterministic acknowledgement rule and 147 remained for review. No events were acknowledged. Host-assurance failures and lost allocations had no automatic acknowledgement rule. The seven queued Fleet dispatch receipts remained separate from Mini-affecting active operations.
- Norn PR #301 adds an allowlisted route-forge child through the managed SOPS environment while retaining the runtime-mutation fence for other children. At exact head
fef0d6b5e2fa364c9327f733f513e5b0ec01912a, focused shell checks passed 2 route-fence cases and 10 recovered-baseline cases. This is offline evidence; the change remains undeployed and live host assurance can publish or prune routes. Require a fresh receipt and route inventory after the protected release and controlled assurance action.
Mini direct v3 upgrade completed (2026-11-07)
The Mini has completed its controller-mediated direct upgrade as an independent node. It is not a Fleet member. The DigitalOcean deployment remains the only Fleet in this work and is a lower-resource staging testbed intended to resemble production.
- The exact signed target is
bba290db041e9f497c7161543c679d9d348b742d, published from protectedmasterin release run 42408397039 after full qualification run 42408083511. - Current source
e117d5645f7764ac6ba6bc316aa19fa241521655was verified as a signed descendant of the retained adoption source. A fresh protected backup and exact-target signed shadow passed before transition. - The controller created generation
gen-ac83feb81071c9f0e096d9df96d53983, transitionmini-v3-20261107-bba290db-r1. Its journal iscandidate-promoted, and its signed preservation receipt reportspreservationMatched=true. - The migration completed at schema 51 with reader floor 5 and writer floor 33. API health is
ok, the runtime mutation fence is inactive, and the API serves the exact target SHA. Platform smoke passed with zero active Mini operations; host doctor/status report Nomad, Consul, API, host agent and cloudflared up. - Seven queued
fleet.github.apply-dispatchreceipts were preserved through the Mini-only drain. They are not Fleet membership or evidence of running provider mutations. - A PID-reuse race interrupted host-agent fencing and required a distinct signed repair controller from
d4dc88e0c37003a1013caed444983b2eeb50fbcb(published in release run 42410387388). The target candidate and its evidence remained bound tobba290dthroughout repair. The regression fix is PR #298.
This proves the Mini direct adoption and a controller-recovered schema-compatible forward transition. It does not prove uninterrupted API availability, a general online migration protocol, rollback across future incompatible schema changes, production admission, or the separate DigitalOcean Fleet gates. The older dated Mini sections below are historical snapshots and do not describe the current runtime.
Fresh Mini live check (2026-11-08 07:29 UTC)
Authenticated read-only inventory confirms the API still serves the exact signed target bba290db041e9f497c7161543c679d9d348b742d (v3.0.0-rc.1-platform-251-gbba290db). /api/health and host status report ok; Consul, Nomad, PostgreSQL, and the workload connector report up. This confirms that the Mini V3 control plane is running on the intended release.
This is not a clean whole-host assurance result. The latest host-assurance incident is critical and reports public-route reconciliation failures for six apps; 25 incident groups remain open (22 latest-critical, 3 latest-warning). The newest assurance receipt returned by inventory is dated Sep 6, so the current host/status=ok must not be presented as a fresh successful assurance. The event details and route inventory are retained only in the local mode-0700 inventory directory; no credentials were copied into this document.
Seven fleet.github.apply-dispatch operations remain queued from Oct 8–10. They have not started as provider applies, but they remain live queue entries that could be claimed; classify or safely recover them before dispatching more Fleet work. The 25 incident groups include older Nomad lost-allocation and capacity events, and the service manifest still reports some services as unknown. Production readiness is blocked (7 pass, 1 warning, 18 fail), including the compatibility development profile, single-node quorum, ACL/TLS, local database, and recovery-readiness gates. That production admission result is not the direct V3 upgrade gate; route assurance and queued-operation classification remain separate operator-readiness work.
A read-only norn smoke platform passed after inventory: health, app-operation drain, release marker, and event-surface checks passed. Its activeOps=0 uses the Mini release drain semantics, which intentionally excludes queued fleet.github.apply-dispatch operations; it is not a claim that the shared control database has no queued operations. The smoke's critical/warning counts are recent-event query counts, not a zero-incident assertion. A read-only norn host doctor also passed its local dependency, managed-file, LaunchAgent, and runtime checks. Neither check ran host assurance or repaired routes.
The assurance log's route failures are HTTP 409 responses stating that the authenticated credential does not establish a stable operation actor. The assurance LaunchAgent carries only PATH, while the then-installed host-runtime invoked the managed CLI directly. Current source now routes only forge <app> assurance through the SOPS-backed platform env-exec path and adds a recovery-fence exception for the canonical host-runtime executable and its single allowlisted _assurance-forge child mode. Focused tests prove the app allowlist and fence dispatch; this source change is not yet on the Mini. A read-only platform smoke was rerun successfully using the managed Mini binary and SOPS. The encrypted env contains NORN_API_TOKEN and no NORN_TOKEN override (values withheld). Smoke proves the credential works for those read-only checks, not that the new route mutation is accepted with stable actor context. Do not claim route assurance is repaired until a protected release is published and a controlled assurance run produces fresh successful evidence.
Inventory artifacts were collected with the read-only norn_inventory.sh procedure at 2026-11-08 07:29 UTC. The Mini remains independent of the DO staging Fleet; its fleet_configured flag describes the separate target config loaded by its control plane, not Mini membership.
Fresh Mini inventory (2026-11-07 21:38 UTC)
Authenticated read-only inventory reconfirmed source bba290db041e9f497c7161543c679d9d348b742d (v3.0.0-rc.1-platform-251-gbba290db), API/host status ok, 30 app records and 46 service records. Mini is not a DO Fleet member. Its API does report fleet_configured=true: loadFleetInventory defines that flag as a Fleet config path being set, and the loaded document is the separate DigitalOcean staging target norn-disposable-nyc3 in nyc3, with control and ingress pools. This is target configuration on the Mini control plane, not Mini membership or proof of live provider state.
The API still reports seven queued fleet.github.apply-dispatch operations dated 2026-10-08 through 2026-10-10. They are durable dispatch receipts, not running provider mutations or Mini workloads; classify them before any new dispatch. The Mini has 25 open incident groups (22 latest-critical and 3 latest-warning), and its production-readiness endpoint is blocked by the compatibility development profile and single-node substrate. These findings do not invalidate the successful v3 API/host postflight, but host ok alone does not mean every application is healthy or that Mini is production-ready. A read-only norn events reconcile --dry-run at 21:01 UTC scanned 200 recent events: 80 matched deterministic reconciliation rules and 120 still need review. It acknowledged nothing; the incident groups remain open until a separate evidence-backed reconciliation.
Post-M8 protected evidence (2026-11-07)
The protected work-staging release-state readback (run 42345172234) passed at 06:03 UTC and verified the retained signed state on all three control nodes. A read-only application catalog readback (run 42341905690) passed at 05:27 UTC: all three controls selected the same immutable catalog and Norn remained active. This proves readback only; no promotion or rollback evidence is established by that run.
Grafana observability credentials passed protected lifecycle checks, but the two latest signed rollout attempts failed on control-01 while confirming etcd health after service changes. The newest rollout run 42348418095 stopped after 20 retries; its secret-safe diagnostic returned unclassified-etcdctl-error (exit 1). The exact etcd failure is therefore unknown; do not retry or continue the serial rollout until a protected, secret-safe diagnosis identifies the cause and fresh cluster-health evidence is available. The catalog readback above does not prove observability ingest.
The later protected DO staging Genesis readback (run 42398114376) stopped in its local runner preflight before contacting the management host: ansible-inventory was absent from PATH. It did not change Fleet state. Replacement Fleet PR #938 added the pinned Ansible virtualenv to GITHUB_PATH and was squash-merged at 21:10 UTC as 6f9b46886950f3109abe4c26eecead91f823965d after its protected checks passed.
A fresh owner-signed read-only intent and one-use runner authorization then dispatched protected run #42415121479. Its runner preflight, checkout, input verification, and private management transport passed; the runner was retired afterward. The final verifier failed because it ran Git as root against the protected bare source repository owned by norn-exec. Fleet PR #939 added an invocation-scoped safe.directory exception for that exact resolved repository path. The change is present in the current norn-fleet checkout as merge commit 44517676. The post-fix readback #42418784411 passed protected input validation and private management/control transport, then failed in the read-only source capability check. The installed controller reported that the management host's protected Norn source repository lacks the pinned capability commit or signed target commit. The log does not identify which object is absent. The intent and run are terminal and must not be reused. Do not retry until source-object availability is diagnosed and proven without changing the repository, then use a fresh one-use intent. Its workflow inputs were the original state and route files captured on Oct 21, not a fresh provider state readback. Its private SSH probes prove connectivity to those pinned addresses at run time; they do not prove current provider identity. The run did not change Fleet state.
Mini in-place upgrade checkpoint (historical observation, 2026-11-07)
PR #289 and PR #290 are merged; the latter is e8c8ddd37f6d0db5d411ec0e3ddd7ab98114abee, published as a signed prerelease in run 42358665477 and imported on the Mini. The active pointer still resolves to 3602e7e3d5fe5c0ef333b4041580d48ec15b24c5; API health is ok, schema is 49, and writer floor is 32. The candidate requires schema 51 and writer floor 33.
The prior recovered-forward transition from 3da50fe… to 3602e7e… has now been finalized and verified with its exact signed controller. The original adoption receipt was retained by digest; a successor receipt is now bound to the running 3602e7e… source. A fresh protected backup and exact-candidate signed shadow for e8c8ddd… passed: 65 tables and 398,805 rows matched, both migration passes reached schema 51, passive health passed, and the source was read-only. Seven queued fleet.github.apply-dispatch receipts remain preserved.
Ordinary upgrade correctly refused the writer-floor change because it cannot prove all writers are stopped. The recovery suite exposed two generation continuity cases: a prepared predecessor must remain supersedable without a receipt, and an already promoted generation may need a signed-descendant catch-up when the running source advanced later. The local fix retains old signed journals and receipts, binds catch-up to both externally verified release signatures and Git ancestry, and rejects unrelated or unverified sources. Focused tests now pass: 15 transition, 22 generation preparation, and 13 resolver tests. At the time of this observation, the change was not merged or released, and no migration or candidate promotion had occurred. See the dated Mini observation and online-upgrade capability proposal.
Fleet running-upgrade checkpoint (2026-11-02 00:10 UTC)
The first protected staging canary release remains live on public HTTPS. A second protected release from source be5ca5d969ccf3509e62ce04a833516bb5c204db built and attested image ghcr.io/antiartificial/norn-fleet-canary@sha256:33265daecc36215776c2029b7e0cd16463570e559c28678d3a94e249bd43ba52, but run 41364735627 failed during deployment completion. Its new Nomad job has two healthy allocations on app-03 and app-04. The previous job also has two healthy allocations, and the public /version route still serves the first release source 182b719bb8f5b84f8c7272e715f83553a00acb4d. The deployment operation 6d96db37-a54e-4231-b1b8-e3e616beb3ab is failed with manual recovery required; its durable Nomad effect says completed. The one-use runner was disabled and unregistered.
The proposed held Fleet deployment recovery procedure describes the checks and operator endpoint; it requires the reviewed recovery code to be deployed first.
The immediate release blocker is the route handoff for a second generation: the current initial route path rejects an already active route after creating the healthy replacement job. A fenced N+1 handoff and recovery of this failed operation are being implemented and tested. Keep both jobs for evidence and availability until a verified route switch and separate old-job retirement. Do not count the second release as deployed. The earlier replica scale and app-node drain proofs remain valid; running upgrade, rollback, loaded node-count contraction, HA fault/restore, soak, and production promotion remain open. No reliable production release date is established.
Fleet staging canary checkpoint (2026-11-01 23:49 UTC)
The first protected Fleet canary release completed in run 41359827250 from source 182b719bb8f5b84f8c7272e715f83553a00acb4d. Norn accepted the signed, attested, content-addressed image ghcr.io/antiartificial/norn-fleet-canary@sha256:1321f05a19f2e3af0bd3bd2df4b6c2c0878d0d38d0866a3d98f7f9f15b321f63. Deployment 1584a4b1-03c8-43be-86ef-b00b943431a4 and operation ff94b712-d5d6-4933-9b19-2a228db98c9f finished successfully. Nomad reported two healthy, running web allocations on distinct ready app-pool nodes app-01 and app-02. Public HTTPS returned 200 for /route-proof with the signed body digest 75790fc8d9898d5a12787ac491b6f471619cde0efc851be9f26db608b6fbb37c and for /version with the released source SHA; TLS validation passed. The one-use private runner stopped, consumed its authorization, and unregistered.
Norn then accepted an explicit web scale from two to three (operation 2ae52865-3834-49bb-9f56-6c7bc9b3f03b) and back to two (operation 4e9b6a39-e8a9-4f5e-a0e2-5825f8e34cc3). Both operations finished succeeded. At three, the scale-status API reported three desired, running and healthy allocations on distinct ready app nodes app-01, app-02 and app-04; 20 public /version requests reached all three allocation IDs with the released source SHA. After downscale, scale-status reported two desired, running and healthy allocations, and 12 public requests reached both remaining allocation IDs. The public /route-proof body still matched the signed digest during the three-replica phase.
A bounded app-node drain exercise then moved the live web allocation off app-01 to app-03, and a second drain moved the app-02 allocation to app-04. Each drain was disabled afterward; all four app nodes returned to ready with no drain strategy. During the second transition, 180 consecutive public HTTPS /version requests succeeded through curl certificate validation and returned the released source SHA. Requests reached the old app-02 allocation and both replacement allocations. Final Norn scale-status reported desired, placed, running and healthy counts of two. The public /route-proof body still matched the signed digest. This proves scheduler replacement and sampled public continuity during sequential drains; it does not prove node retirement, fault recovery, a sustained load budget or Norn's Fleet generation executor.
This is a staging release and live two-host workload proof. It advances replica-scale evidence only; it does not prove fresh-Fleet provisioning, platform upgrade, or mixed application-version rollout. The live 2→3→2 replica scale cycle and bounded app-node drain are proven, but loaded node-count 2→3→2 replacement, worker drain, leader failover during route mutation, running upgrade, rollback, whole-node restore, soak, and production promotion remain unproved. The ingress publishers currently name the elected control03 address 10.43.0.2 for route authority. An API leader change requires authority reconciliation before another route publication; the current successful route remains served. The Mini transition described below is independent of this Fleet checkpoint.
Mini adoption evidence checkpoint (historical, 2026-11-01 05:33 UTC)
Pinned-address SSH and authenticated loopback inventory verify that the Mini has completed its guarded PostgreSQL v2-to-v3 adoption as an independent development Norn. The active API reports exact source 3602e7e3d5fe5c0ef333b4041580d48ec15b24c5 and version v3.0.0-rc.1-platform-34-g3602e7e3; ~/norn/current resolves to that immutable release. The original com.norn.api LaunchAgent is running through the managed launcher in active/check mode. Health is ok, the runtime mutation fence is inactive, and Consul, Nomad, PostgreSQL, S3 and SOPS report up. The selected recovered-forward generation remains recorded as candidate-promoted, with its preservation receipt retained. Current inventory found 30 app records, 46 services and host status ok.
This confirms the Mini live-adoption portion of M9 and provides transition evidence for M5; M5's copied-environment rehearsal gate remains a separate assessment. It does not qualify a repeat without operator intervention, a current-master v3-to-v3 refresh, production admission, Fleet etcd or HA, or full M9 adoption. The completed transition required bounded operator intervention to correct the owner-only release-key path, fill one exact preservation digest under the promotion lock, and temporarily accommodate host-agent startup timing; those facts remain part of the evidence rather than an autonomous-repeat claim. The Mini checkout and current master may advance independently of the immutable runtime source, so neither is runtime provenance.
The DigitalOcean staging Fleet is the only Fleet in this work: a separate testbed intended to resemble production with lower resource use. Mini is not a member. The Mini API's configured DO target and queued Fleet dispatch receipts are control-plane records, not evidence that Mini is a Fleet node or that provider work is running. The DO testbed retains its own gates for directly deployed representative workloads and data, placement and loaded node-count 2-to-3-to-2 scaling with safe drain, running-version upgrade under traffic, whole-node fault/restore and HA, external admission, soak, and operational handoff. Classify the seven queued dispatch receipts before any new Fleet dispatch. Production readiness is still blocked. No Mini transition should be replayed or made dependent on the separate Fleet work.
Operating lanes and API retirement boundary (historical pre-upgrade snapshot, 2026-11-07)
The release has distinct execution lanes across independent environments: (1) Direct Mini for its independent, individually controlled node, (2) Fleet bootstrap for the DigitalOcean staging testbed with the default two application node pools, (3) Fleet upgrade for an existing Fleet with workloads, and (4) capacity changes as a separate, workload-aware operation. The direct Mini upgrade did not require Fleet signing authority or Fleet connectivity, and still used a verified signed release, protected backup and shadow evidence, controller-mediated fencing, health checks, and a recoverable journal. Mini's control API may hold configuration for the separate DO staging target; that does not make Mini a Fleet member. Fleet bootstrap is not the Fleet running-upgrade gate; online rollout should preserve availability where proven, with a controlled offline window as an explicit fallback.
The Mini's adoption and post-upgrade verification are now complete. A source review found no /api/v2 route namespace: the production router exposes the current /api/v1 contract and older unversioned /api handlers. The v2/api Go import path is the repository's package layout, not evidence of an HTTP API version. A specific Fleet offline-effect subprotocol is still explicitly versioned in the route /api/v1/fleet/offline/attempts/{planID}/{attemptID}/effect-grants/v2; its tests protect one-use evidence and recovery behavior. It needs a caller and protocol migration review before removal as part of V3-only support. Tests mentioning “legacy”, “V2”, or “Postgres” therefore cannot be removed by name alone; many protect durable state recovery, signed Fleet protocol revisions, or Postgres conformance. Before retiring any old unversioned handler, trace supported callers and replace its behavioral coverage at the current contract boundary. Keep migration/recovery and rollback-read compatibility coverage until the corresponding stored state is no longer supported. No test deletion is justified by the route/name search alone.
This is a correction to the pre-upgrade terminology in this historical snapshot. PR #295 merged the Mini generation catch-up changes as 3f51e35ad246c68a835977d1f1e3b8fdf0a8511c. The signed prerelease for 0701add87b34d0de7ead5031579b3d74cd829bd7 was published in run 42406296055. The generation catch-up logic preserves older signed journals/receipts and binds a completed generation to the exact current source through external signature verification and Git ancestry; it does not authorize migration or candidate promotion by itself.
Fresh authenticated Mini inventory on 2026-11-07 reports source e117d5645f7764ac6ba6bc316aa19fa241521655, version v3.0.0-rc.1-platform-248-ge117d564, API health ok, and runtimeMutationFence=inactive. ~/norn/current resolves to that source. The active recovered-forward generation is gen-1d647aa65bac5b2b5cdda675fdb8cb85, transition mini-v3-20261107-38fcec83, candidate 38fcec83380c7ba989d46e11db6cc449510fad86; its journal says candidate-promoted. Git confirms that candidate is an ancestor of the running source. This confirms a healthy V3 runtime and an earlier promoted candidate; it does not prove the pending e117-based schema migration. The signed 0701add8… release was published, but Mini preflight stopped before candidate preparation because its legacy adoption guard accepts only the original adoption source or exact journaled candidate. A follow-up fix allows evidence preparation and recovered-forward control only after both source releases verify externally and Git proves ancestry; ordinary upgrade remains gated on the completed recovered-forward journal. That fix is still unreleased. No candidate import, database migration, or candidate promotion was verified in this checkpoint.
This checkpoint supersedes older present-tense Mini source, schema, and generation statements below. Preserve those sections as historical evidence.
Track boundaries (2026-10-11)
The v3 launch has two independent environments. Mini is an independent, single-node Norn installation and is not part of a Fleet. It retains its unmanaged PostgreSQL, and its startup and upgrade do not require Fleet connectivity, Fleet etcd, or Fleet management. Separately, the only Fleet in this work is the DigitalOcean staging testbed, intended to resemble production with lower resource use. Its bootstrap and upgrade qualification are distinct Fleet gates, not Mini prerequisites.
The Mini track is: publish and verify the exact signed platform release, take and verify a fresh Mini-local recovery copy, run the protected Mini transition/upgrade, then verify API version, health, database access, preserved applications, and host assurance. Fleet deployment, etcd quorum, ingress, workload/scale testing, and teardown remain separate Fleet acceptance milestones; none is a prerequisite for upgrading the Mini.
As of 2026-10-11, the Mini drain corrections in PRs #137 and #138 are merged. The signed rehearsal correction in PR #140 is also merged as a059f2e5b42e8df30c3fe4460cbd6455a0ea4722. Its immutable signed prerelease is platform-a059f2e5b42e8df30c3fe4460cbd6455a0ea4722. The Mini imported and signature-verified that exact release. A fresh Mini-local backup and a signed private-database/LaunchAgent shadow rehearsal passed. The live upgrade then stopped before migration or promotion: the candidate raises the PostgreSQL writer floor from 31 to 32, and the ordinary restart lane cannot prove that every old writer is fenced. A prior transition is recorded as candidate-postflight-failed; its signed adoption receipt supports forward work but does not authorize replay of the old one-way procedure or prove live writer quiescence. Mini remains on its prior signed release and schema 47. The next Mini-specific engineering gate is a separately journaled, interruption-tested recovered-forward transition that proves all writers are quiesced before migration and only moves forward. Fleet deployment, Fleet management nodes, and Fleet etcd are not prerequisites for this Mini update.
Historical architecture note: Mini is the independent single-node development platform and uses its existing unmanaged PostgreSQL service. It does not host Fleet nodes or require Fleet management nodes or an etcd quorum. Earlier proposals considered making Mini a Fleet member; that is not the current topology. The DigitalOcean Fleet is a separate staging testbed. The Mini's current API target configuration for DO is recorded separately above.
At 2026-10-11 19:42 UTC, protected signed release 798c82847c04156d99a9876c0bb0d6d4b76f901b was published and independently verified; it is a v3.0.0-rc.1 prerelease, not a stable v3.0.0 release. Direct SSH and loopback API checks reconfirmed Mini still serves 26147c39a554b73b6761a371edee5a9591c81d3f (v2.20.0-platform-72-g26147c39), schema 47/writer floor 31, health ok, and single-node PostgreSQL/Consul/Nomad. No candidate was promoted.
The failed transition evidence binds historical candidate e7301f779041cd22f1872090323d3b8ed5c4611b, while the current signed runtime is 26147c39a554b73b6761a371edee5a9591c81d3f. Its reconciliation records matchesBefore=false; that is an explicit loss of proof for the old baseline, not evidence that current data should be discarded. The local correction now models these as separate identities (sourceSHA and incidentCandidateSHA), retains the v1 receipt and all historical files, and captures a fresh current Mini workload digest for the new forward transition. It still requires a fresh verified database backup and exact-candidate signed shadow restore before any live migration. This correction is not yet merged or released, and no live Mini state has been changed.
Observed 2026-10-11. This is the single current handoff for the pragmatic deployment objective. The milestone contract defines the exit criteria; percentages below are engineering estimates, not owner sign-offs. Recheck external state before any new mutation.
Latest Mini transition checkpoint (2026-10-11 21:00 UTC)
The Mini is being upgraded as its own single-node Norn instance. It uses its existing unmanaged PostgreSQL and does not need Fleet nodes, Fleet management nodes, or Fleet etcd. Fleet availability is not an upgrade prerequisite. Mini is not a Fleet member; its control API's configured DO staging target is separate from the Mini host and its workload topology.
Norn PR #144 merged as 85fb2830ac18a711b2e3e20de45c1adcbbd4935d; the protected signed prerelease platform-85fb2830ac18a711b2e3e20de45c1adcbbd4935d was published for that exact SHA. The Mini imported and verified it. A fresh local backup and exact-candidate shadow rehearsal passed. The live forward transition mini-v3-20261011-85fb2830 is paused in legacy-fenced: both com.norn.api and com.norn.host-agent are fenced with exit 78, current still points to 26147c39a554b73b6761a371edee5a9591c81d3f, and PostgreSQL remains at schema 47. No v3 migration or candidate promotion has occurred.
The immediate gate is seven queued fleet.github.apply-dispatch receipts for pilot pilot261006c. Their bound GitHub apply runs are terminal (five failed, two cancelled); the Apply verified plan step is skipped for each. The migration quiescence contract currently blocks on every nonterminal operation, while Mini's own preflight explicitly preserves these queued Fleet dispatch receipts because they do not mutate Mini. The fix is to align those contracts narrowly: allow only queued fleet.github.apply-dispatch receipts to remain unchanged through migration, while continuing to block every other active operation and any additional database client. Do not edit operation rows directly, bypass the guard, or alter the journal's candidate binding. Then resume the same signed transition through a supported guarded candidate retarget path, if its journal and schema checks still validate. Mini's independent upgrade must not be held for provisioning or starting a Fleet cluster.
Current decision
Norn v3 is not deployed on the DigitalOcean staging Fleet (status at this historical checkpoint). Mini's single-node upgrade is independent of Fleet deployment. Mini was fenced during its single-node upgrade at that time; it is not serving its normal API until the recovered-forward transition completes. Its development profile and preserved applications do not qualify Fleet or complete the release. A Fleet attempt remains a separate acceptance run and requires fresh credentials, a repaired partial-apply/recovery lane, and a complete workload and failure proof. Do not reuse pilot261004b state, runner, keys or approvals.
At 21:29 UTC, Norn PR #123 merged to protected master as 1ca28934034c48f28aed4b97b6d4971d69b642e3. Norn CI 37376169196, Repository CI 37376169212, and docs deployment 37376169180 all succeeded on that exact SHA. The Norn observer/status controller is now in protected master; this is not a published platform release or live Fleet deployment.
At 21:34 UTC, Fleet PR #258 merged as e4756190670aa3a8afa6d3cfdc6344a531dd4376. The staging doctor now accepts the verified GitHub squash-commit identity. It still reports three missing input groups. staging-plan is missing ADMIN_CIDRS_JSON, DIGITALOCEAN_TOKEN, and STATE_ACCESS_KEY_ID, STATE_BUCKET, STATE_ENDPOINT, STATE_REGION, and STATE_SECRET_ACCESS_KEY. staging is missing DIGITALOCEAN_TOKEN, NORN_API_URL, and the same five state-store values. The protected runner configuration is missing NORN_FLEET_BOOTSTRAP_VARS_FILE, NORN_FLEET_KNOWN_HOSTS_FILE, NORN_FLEET_PRIVATE_INTERFACE, NORN_FLEET_SSH_PRIVATE_KEY_FILE, and NORN_FLEET_SSH_USER. No provider operation has been run from staging.
At 00:10 UTC, the protected platform-release environment secret NORN_RELEASE_IMMUTABILITY_CHECK_TOKEN was updated after the release-check PAT was regenerated. At 00:12 UTC, workflow run 37392793546 was dispatched for exact merged SHA 1ca28934034c48f28aed4b97b6d4971d69b642e3. Authorization, pinned UI build, and all four Linux/macOS amd64/arm64 bundle jobs passed. The publish job is waiting for review in the protected platform-release environment; no immutable release has been published yet.
During browser inspection, the newly generated release-check PAT value appeared in an accessibility snapshot. Treat it as exposed and revoke/replace it before approving publication. The environment secret must then be refreshed with the replacement. The running publish job may fail if it reaches the immutability check after revocation; rerun the same exact SHA after rotation if needed. No token value is recorded here.
The current staging doctor still reports missing staging-plan secrets ADMIN_CIDRS_JSON, DIGITALOCEAN_TOKEN, STATE_ACCESS_KEY_ID, STATE_BUCKET, STATE_ENDPOINT, STATE_REGION, and STATE_SECRET_ACCESS_KEY; staging is missing DIGITALOCEAN_TOKEN, NORN_API_URL, and the five STATE_* values. Protected runner variables remain missing: NORN_FLEET_BOOTSTRAP_VARS_FILE, NORN_FLEET_KNOWN_HOSTS_FILE, NORN_FLEET_PRIVATE_INTERFACE, NORN_FLEET_SSH_PRIVATE_KEY_FILE, and NORN_FLEET_SSH_USER. No provider operation has been run from staging.
At 00:22 UTC, the newest authenticated Mini inventory reports source 26147c39a554b73b6761a371edee5a9591c81d3f (v2.20.0-platform-72-g26147c39), host ok, Fleet unconfigured with zero pools, and production readiness blocked (6 pass, 19 fail, 1 warning). It found 28 apps, 45 services, zero active operations, and 25 active incidents. Evidence is at /var/folders/c8/d4bdjwzn2qgfmg3hqz7qj7ch0000gn/T/norn-inventory-20261006T002155Z. The merged Norn SHA has not been released or promoted to Mini.
At 19:34 UTC, a fresh authenticated Mini inventory again reported host status ok, but fleet_configured=false, zero node pools, and production readiness blocked. It found 28 apps, 45 services, zero active operations, and 25 active incidents. The detailed safe inventory is at /tmp/norn-inventory-20261005T1934Z; this confirms the Mini is online on its v3 API source but does not establish Fleet deployment readiness.
At 20:01 UTC, a new authenticated read-only Mini inventory again reported source commit 26147c39a554b73b6761a371edee5a9591c81d3f (v2.20.0-platform-72-g26147c39), host status ok, Fleet configured=false with zero node pools, and production readiness blocked. It observed 28 app entries, 45 service entries, zero active operations, and 25 incidents. Local observation files are at /var/folders/c8/d4bdjwzn2qgfmg3hqz7qj7ch0000gn/T/norn-inventory-20261005T200109Z. This confirms the Mini v3 API is online; it does not show a Fleet deployment.
At 20:30 UTC, a fresh authenticated read-only inventory remained unchanged: source 26147c39a554b73b6761a371edee5a9591c81d3f, host ok, Fleet unconfigured with zero pools, and production readiness blocked (6 pass, 19 fail, 1 warn). It reported 28 app entries, 45 service entries, zero active operations, and 25 incidents. Evidence is at /var/folders/c8/d4bdjwzn2qgfmg3hqz7qj7ch0000gn/T/norn-inventory-20261005T203015Z.
At 20:11 UTC, the next authenticated read-only Mini inventory was unchanged: source 26147c39a554b73b6761a371edee5a9591c81d3f, host ok, Fleet unconfigured with zero pools, and production readiness blocked (6 pass, 19 fail, 1 warn). It again reported 28 app entries, 45 services, zero active operations, and 25 active incidents. Evidence is in /var/folders/c8/d4bdjwzn2qgfmg3hqz7qj7ch0000gn/T/norn-inventory-20261005T201102Z.
At 21:01 UTC, focused local tests for the separate disposable-retirement controller worktree passed (78 tests across the coordinator, delegation, deadline proposal, root retirement, backend cleanup, and manifest builder). These use injected test adapters and fixtures. The worktree still contains uncommitted changes based on Fleet main 5210307; the controller has no real executor/Spaces adapters, fresh deadline-time fence approval producer or renewal, or restart supervisor. Its runbook still classifies deadline retirement as not implemented and unrehearsed. This work was independent of the Norn observer-controller integration and remains a prerequisite to safely running another billable disposable Fleet pilot.
At 17:42 UTC, Fleet PR #256 merged to main as 5210307. Merged-main plan 37350374522 and contract validation 37350374566 both succeeded. The change binds resumable paired backend cleanup to the exact locked run and adds a validation-only deadline descriptor checker. It does not implement or qualify an unattended deadline controller. Fleet PR #255 remains merged at abb0b00; its merged-main validation (37336272528) and planning (37336272298) passed. Preceding commit 1fcaeb7 passed contract validation run 37343349906, including the full Python unittest suite, local retirement qualification, workflow validation and OpenTofu checks. Commit abaabc4 clarifies retirement CLI help text; validation run 37344577776 passed the same contract suite at 17:01 UTC. Commit d4eee3e adds a regression test proving an already-started backend cleanup can resume after its short approval expires, while a new cleanup cannot start after expiry. Full validation run 37346380888 passed at 17:15:58 UTC, including the full Python unittest suite, local retirement qualification, workflow checks, OpenTofu validation and disposable pilot validation. Review then found a race where a replaced cleanup scope could select another run after the shared lock was acquired. Commit 3f41503 binds the reopened scope and canonical ledger path to the locked run; its focused regression test and full validation run 37347927438 passed. PR #256 also corrects the disposable README's stale pilot261004b label to the actual pilot261005a budget-gate ID, adds a non-executable descriptor checker, and makes an already-authorized backend cleanup retryable after its approval expires. It records deadline-controller design requirements but does not provide a scheduler or make a pilot safe to start. The Fleet account has no matching pilot261005a or pilot261005c droplets, databases, load balancers, projects, VPCs, volumes, firewalls, reserved IPs, snapshots, or runner labels at 16:20 UTC; this is still not Spaces bucket-zero proof. Norn launch-state PR #120 was a documentation-only handoff and merged to protected master at 17:31 UTC as d67b8bf01346ef8e54f25e1119c7ffe532e05e1d. Its Repository CI run 37348527855 and Norn CI run 37348527835 both passed, including the required API/CLI, Fleet pilot workload, OpenAPI, workflow lint, Web UI, M4 durable replica, and release-bundle checks. These workflows had been manually disabled; both are now active. This verifies the current Norn branch gates, not a Fleet deployment. Fleet issuer PR #210 is conflicting with main; lifecycle PR #175 has a failed contract check. Neither is a current deployment gate result.
The approved pilot261004b window was $20 maximum, four hours active plus one hour for teardown, ending 2026-10-05 08:30 UTC. The protected apply run 37273763546 at Fleet main ca52148d69eecd70a843938b72c2bc882a066c5d stopped during provider creation at 06:50:30 UTC with exit 143. The runner was online and idle after termination; the signal sender is unproven. Norn attempt eb349686-c1ac-439d-b7b6-cf1fc2443d9f was later marked abandoned after its heartbeat expired. The remote Fleet state persisted at serial 3, lineage ffe289b7-a711-4ca6-b84d-7887c67f4d5c, with five droplets, load balancer, VPC, firewalls, project and tags. It did not contain either managed database, although DigitalOcean had created both. No bootstrap, public ingress, etcd quorum, application workload or scale proof followed. Automatic recovery run 37274493490 failed at source-run name validation; the repository recovery inventory would also hold on the two provider-only databases. Plain apply replay was unsafe.
At 13:47 UTC, well after the teardown deadline, a fresh DigitalOcean inventory still showed seven pilot droplets, three managed databases, the Fleet load balancer, four firewalls, two VPCs and two projects. This was an overrun incident. Exact-ID emergency deletion began after preserving private state and provider evidence. By 13:58 UTC, fresh DigitalOcean list calls showed zero pilot261004b droplets, databases, load balancers, firewalls, VPCs, projects, tags, volumes, snapshots, reserved IPs and SSH keys. Both pilot Spaces state buckets, every retained object version and both scoped Spaces keys were removed; a new full-access Spaces inventory key found zero pilot buckets and was revoked. The exact GitHub pilot runner and the staging secrets/variables created for this pilot were removed. DNS A lookup for pilot-pilot261004b.betabung.com returned no address. These are technical absence observations after emergency unprotected cleanup, not a signed protected final-zero receipt. The final pilot-specific DigitalOcean bill is not yet available. The account's month-to-date balance cannot be attributed to this pilot, so compliance with the $20 cap remains unproven.
Controller and executor state
There are three distinct controller/executor workstreams; none alone proves an autonomous bounded pilot or a live deployment:
- The Norn Fleet observer/controller implementation began on
feature/v3-fleet-controllerover the weekend and was integrated by PR #123 into protected master on Oct 5. It is merged, tested in post-merge CI, and present in source; it derives and stores Fleet target, observation and status. It does not provision provider infrastructure. Release publication, Mini promotion and Fleet deployment remain separate gates. - Fleet executor PR #257 remains a draft on
feature/fleet-controller-executorat64cb7b0, based on Fleet main5210307; its contract check passed. This is separate newer work, not the weekend Norn observer/controller integration. The PR explicitly supports only the legacy PostgreSQL no-drain lifecycle, refuses v3/etcd before provider calls, and does not ship the Norn executor API route. It is not a v3 deployment path and must not be installed on a v3/etcd staging root. No provider-backed E2E was run. - The checkout
norn-fleet-deadline-retirementis still on merged Fleetmaincommit5210307with uncommitted edits, not on an independent commit branch. Those edits add delegated owner authority/journaling at existing root/backend retirement boundaries and a post-create root approval producer. Newscripts/disposable_retirement_controller.pyprovides a local-only persisted coordinator with injected stage adapters, separate controller locking, HMAC/hash-chained intents, proof checks, restart recovery hooks, and T0+5 breach-and-continue behavior. The full Fleet Python unittest suite passed at 20:06 UTC after the latest coordinator proof-validation changes: 1,643 tests, four skipped, in 258.317 seconds. The earlier coordinator run had 1,641 tests, four skipped, in 278.771 seconds. This suite includes existing executor and fixture tests; it does not qualify a live provider adapter, unattended restartable orchestration, or deployment. Subsequent local changes now enforce the full root-zero and backend-cleanup receipt field sets, digest shapes, delegated run/root scope, independent provider-observation hashes, bucket bindings, and prior-stage receipt digests; 11 focused tests,py_compileandgit diff --checkpass. This is structural binding only: the coordinator still does not reopen and validate retained state exports, executor ledgers, or external provider evidence. The existing per-stage CLI executors do not yet have an adapter or launchd supervisor. The complete 1,641 tests, four skipped, in 278.771 seconds on 2026-10-05. Both Fleet and launch-stategit diff --checkpass. It has no real root/backend command, approval, fence-renewal, or Spaces adapter and no CLI or launchd supervisor, and has not passed sleep/reboot/partial-apply rehearsal.
The three workstreams cover separate boundaries (Norn desired-state observation, executor-host execution, and disposable-pilot retirement). They need an explicit integration/release path; do not infer they compose safely because their local tests pass.
After merged Fleet PR #256, a separate local Fleet checkout norn-fleet-deadline-retirement on branch codex/executable-deadline-controller contains new, unmerged work. The scripts/disposable_retirement_delegation.py module validates an owner-HMAC run delegation and maintains an owner-only hash-chained retirement journal. The journal records and fsyncs a T0+5 breach marker before later cleanup can continue. Both the root executor and paired-backend apply now have opt-in delegation checks at their existing shared-lock mutation boundaries, with account, budget, activation, source, evidence, lock, state/root, provider-scope, and ordering bindings. New disposable_deadline_approval.py and manifest builder support derive a short-lived HMAC approval from the owner delegation and fresh post-create root output, state, provider scope, retention, fence approval digest, and MySQL receipt. The root executor verifies that approval against the exact signed delegation. Focused tests after these edits pass: 8 manifest-builder, 26 root-retirement, 19 backend-cleanup, 11 delegation/journal, and two deadline-proposal tests. The earlier Python suite passed 1,632 tests with four skipped in 300.372 seconds at 19:14 UTC, before the coordinator module was added. The subsequent full run including the coordinator also passed: 1,641 tests with four skipped in 278.771 seconds. A refreshed Mini inventory at 19:34 UTC again reports Fleet unconfigured and production readiness blocked; the controller suite pass does not alter the deployment gate. The deadline proposal validator accepts the complete delegation CLI bundle but remains validation-only. There is still no fully wired end-to-end controller, fresh fence-approval producer/renewal, restart supervisor, or sleep/reboot/partial-apply rehearsal. Do not use this branch to authorize or start a billable pilot.
The two offline pilot261004b Tailscale management records (100.120.209.89 and 100.122.147.29) were removed after action-time user confirmation; the console then showed 14 machines and no pilot261004b machines. The five one-use Fleet auth keys are invalidated. The broad OAuth client that issued them was previously revoked; a replacement scoped client was requested but not generated. The pilot261004b DigitalOcean API token was revoked through DigitalOcean’s token revocation endpoint, and a follow-up account request returned HTTP 401. Its local token file was removed after a hash-only receipt was saved. Technical external cleanup is now observed, but the final pilot bill and a signed protected final-zero receipt remain unavailable.
Reusable DigitalOcean capability: The authenticated doctl context and DigitalOcean API can eliminate repeated dashboard sign-ins for supported account, resource, and token-management operations. An already-issued pilot personal access token can revoke itself through DigitalOcean’s API, so cleanup of that token does not require a second dashboard sign-in when its local token file is still available. For pilot261004b, POST to DigitalOcean’s OAuth revocation endpoint with that token as both Bearer authorization and the token form parameter returned HTTP 200; a subsequent /v2/account call with the same token returned 401. This can eliminate a dashboard sign-in specifically for revoking an existing token; it does not create a token, restore expired access, or prove provider resources are gone. Record only a hash and response statuses, then remove the local file. DigitalOcean documents the endpoint for OAuth tokens; support for a dashboard-issued personal access token is a live observation to recheck before relying on it in another pilot.
At 17:12 UTC, the signed-in local doctl default context authenticated successfully (a fresh doctl account get) to the intended theartificial@hotmail.com account (UUID 05fe610a42b6d50c65344dd87bd842ce35ab036d). Supported DigitalOcean API inspection and operations can use this existing CLI session without a new dashboard sign-in, including later token revocation when the active CLI context has the required account authority. It does not supply fresh run-scoped credentials or approval, and doctl cannot enumerate Spaces buckets in this workflow; retain the separate Spaces API/S3 inventory for that proof. Read-only CLI checks found no pilot261005a name matches among droplets, databases, load balancers, projects, or VPCs. At 17:44 UTC, corrected read-only doctl name checks also found no pilot261004b, pilot261005a, or pilot261005c matches among droplets, databases, load balancers, firewalls, VPCs, or projects. These name checks are not complete provider-zero evidence and did not inspect Spaces buckets. A fresh read-only check at 18:14 UTC again authenticated to the intended account and found no pilot261005a or pilot261005c matches in droplets, databases, load balancers, firewalls, VPCs, or projects. This is still only a name-collision check; volumes, snapshots, reserved IPs, SSH keys, project resources, DNS, and Spaces buckets were not checked in this observation. No provider mutation was made. A plain Mini norn fleet pools CLI request at 17:45 returned HTTP 401 authenticated_principal_required; the documented owner-host SOPS inventory path then authenticated successfully at 17:50. It confirmed Fleet configured=false, zero node pools and zero plans. No Fleet plan or provider mutation was made in this turn.
Operational follow-up: Use the authenticated doctl context first for supported DigitalOcean account, resource, and token operations. This avoids re-entering the dashboard for routine API-capable tasks and may allow a known active token to be revoked through the API during cleanup. Use the dashboard only for actions that require interactive UI authentication or are not exposed by the API. Before a future pilot, verify the selected context/account and required scopes with a read-only request; this convenience does not replace a fresh run-scoped credential, bounded approval, or a separate Spaces inventory.
Mini and source
On 2026-10-05 at 17:50 UTC, a fresh authenticated owner-host inventory showed the Mini loopback API healthy on source 26147c39a554b73b6761a371edee5a9591c81d3f, displayed as v2.20.0-platform-72-g26147c39. The platform version label's source includes v3 work. Authenticated production-readiness returned blocked: 6 of 26 checks passed, 19 failed, and 1 warned. Current failures include the development profile and compatibility auth, single-member Nomad/Consul, disabled ACLs and incomplete TLS, local control database without PITR/replica, application deployment provenance, snapshot/restore posture, and missing recovery drills. The host status is ok; this does not override production readiness.
At 19:15 UTC, authenticated owner-host inventory was repeated using the Norn-platform inventory script. It again reported healthy Mini API source 26147c39a554b73b6761a371edee5a9591c81d3f (v2.20.0-platform-72-g26147c39), host status ok, Fleet configured=false, zero node pools, zero plans, and production readiness blocked with 6 pass, 19 fail, and 1 warn. It observed 28 app entries, 45 service entries (18 passing, 27 unknown), zero active operations, and 25 active incidents. Snapshot-retention warnings remain for field-harbor and turnkey-offer-intake; mail-mcp and signal-sideband still need secret attention. This read-only inventory made no provider or control-plane mutation. The local observation files are in /var/folders/c8/d4bdjwzn2qgfmg3hqz7qj7ch0000gn/T/norn-inventory-20261005T191507Z.
At 20:38 UTC, a new authenticated owner-host inventory again found source 26147c39a554b73b6761a371edee5a9591c81d3f, host status ok, Fleet configured=false with zero pools, zero active operations, 25 active incidents, and production readiness blocked at 6 pass, 19 fail, and 1 warn. It listed 28 app entries and 45 services. The snapshot is in /var/folders/c8/d4bdjwzn2qgfmg3hqz7qj7ch0000gn/T/norn-inventory-20261005T203815Z.
The authenticated doctl context still resolves to the intended DigitalOcean account. Fresh read-only name-filtered lists found no pilot261004b, pilot261005a, pilot261005c, norn-staging, or norn-management matches in Droplets, managed databases, load balancers, firewalls, VPCs, volumes, snapshots, reserved IPs, or projects. doctl does not inventory Spaces buckets; this is not complete account-zero evidence. Staging Fleet's offline scripts/sanity.py validation passed. Its setup doctor still reports missing staging provider/state secrets and runner configuration. The protected squash metadata policy is now merged and passes staging doctor. The three remaining groups are staging-plan secrets, staging secrets and protected runner configuration. No cloud resource was created or changed.
The exact pilot261004b protected apply log confirms its pre-issued public ingress TLS check passed before provider apply. The apply then remained in managed database creation until exit 143 at 06:50:30 UTC. The signal sender is unproven; do not attribute this interruption to the TLS gate.
These app and service counts include discovered and inactive entries; they do not prove identity preservation through the earlier one-way transition or health for every app.
Norn PR #115 merged the bridge title correction as c56fa0c. The signed platform-c56fa0c release contains four verified bundles. Publication is not Mini promotion or Fleet fitness. The local codex/v3-release-integration branch remains unmerged and must not be treated as protected release authority. The main Norn checkout has unrelated local changes that must be preserved.
Pilot lineage
| Pilot | Most recent useful evidence | Disposition |
|---|---|---|
pilot261003c / pilot261003d | c and d exposed earlier gaps in provider evidence, TLS handoff and emergency cleanup. The d Fleet apply stopped before provider mutation; management was retired by emergency unprotected cleanup. | Historical failure evidence, not launch proof. Old c/d access items were retired in the later owner observation. |
pilot261004a | Exact-SAN TLS and replacement GitHub App key supported complete management handoff and protected Fleet planning. Its apply stopped at the pre-provider retained-bucket inventory gate. Management and both backends were retired before its approved deadline. | Clean cloud/backend abort; two Tailscale device records and replacement OAuth client were still pending at the last observation. Recheck them separately. |
pilot261004b | Protected plan 37273380169, Norn plan a25b25fa-c0b0-47e0-b9ec-836ce6cde3ed, and apply 37273763546 reached live provider creation. State missed both databases. | Fleet never bootstrapped; emergency technical teardown occurred after the 08:30 UTC deadline. Technical access cleanup is observed; billing closure remains open. |
Pilot a was the first complete management-to-Fleet handoff, but b was the most recent and exposed the material partial-apply failure. A fresh pilot should be based on the repaired contracts, not resumed from a or b.
pilot261005a is a proposed fresh run ID, not an approved or active pilot. At 16:20 UTC on 2026-10-05, read-only checks against the intended theartificial@hotmail.com account found no matching droplets, databases, projects, load balancers or VPCs; no registered GitHub runner carries its Fleet label. The broader name checks also found no matching c-suffixed droplet, database, project, load balancer, VPC, volume, firewall, reserved-IP or snapshot names. These are name-collision checks, not a full provider-zero or Spaces-bucket proof. The canonical owner-only pilot261005a directory, its budget contract/activation files, and its DigitalOcean token file do not exist; staging aggregate-budget variables still point to the retired b paths and digest. Fleet PR #255 merged the signed c budget, plan, apply, recovery and exact topology gates as abb0b00; it rejects reuse of retired pilot261004b. The PR contract check and merge-main plan and validation passed (37335410031, 37336272298, 37336272528). This is code validation, not live proof. Read-only checks at 16:20 found no c-named DigitalOcean resources in the categories above and no registered c runner. The independent deadline retirement path remains unimplemented. Existing approvals require post-create state and expire within an hour, so signed budget expiry alone cannot guarantee teardown. The existing owner-Mac retire-by watchdog performs only a local finalizer; it has no provider, GitHub, Spaces or token-exchange capability. Fleet PR #256 contains a validation-only descriptor checker, not a controller. Its runbook records that root destroy plans must be built after resources exist. Fleet PR #256 commit 1fcaeb7 adds a same-lock, ledgered backend cleanup start: a retry can continue after the short approval expires only if the first start event bound the complete exact authority while that approval was live. Full contract validation passed on both 1fcaeb7 and abaabc4. This does not authorize a new start after expiry or provide Fleet/management deadline-time approvals. A read-only workflow audit confirmed no existing workflow can mint post-create retirement approvals or execute the ordered retirement chain. There is no installed or wired deadline-retirement executable. The separate branch's coordinator and delegation/journal primitives alone do not close this gate. Do not start a new billable run until a restartable, run-scoped owner-Mac controller can fence the runner, perform Fleet → management → backend cleanup automatically from T0+4, retain each stage's receipts, and continue retrying through and beyond T0+5 until final zero. A timer alone is not a hard spend cap when the owner host or provider access is unavailable.
At 19:20 UTC, the authenticated doctl context again identified the intended DigitalOcean account. Name-filtered lists contained no pilot261004b, pilot261005a, or pilot261005c matches: 2 droplets, 0 managed databases, 0 load balancers, 0 firewalls, 6 VPCs, 1 volume, 2 snapshots, 0 reserved IPs, and 3 projects were listed. This is a scoped resource-name collision check; Spaces buckets/object versions, DNS, GitHub runners, Tailscale devices, and provider resources without these names were not proven absent. No mutation was made.
Historical release-gate snapshot (reconciled 2026-11-07)
This table preserves the program status as recorded on 2026-11-07; it is not the current release status. The latest Mini recovery and DO staging evidence is at the top of launch state, and the current work queue is in release program milestones. The detailed exit criteria remain in the release fitness plan.
| Lane/gate | State at 2026-11-07 | Next proof as known then |
|---|---|---|
| Mini direct | V3 transition and recovery completed; independent node, not Fleet. | Keep this as the direct-lane reference. Measure interruption on future changes; do not claim uninterrupted or generally online migration. |
| DO staging Fleet / soak | 72h35m soak completed with 4,254 valid hash-chained samples and no recorded alarms/gaps. Fresh provider readback at 09:36 UTC confirms all nine expected Droplets active, the staging load balancer active, and the two staging managed databases online. | Fresh control-plane/node health, etcd diagnosis, service-route proof, host identity, and exact control roster. Provider status does not prove runtime or application health. |
| DO Genesis readback | Latest protected readback 42418784411 reached the management/control hosts, then failed read-only because a required capability or signed target commit is absent from the protected source repository. Inputs used Oct 21 state/route pins. | Diagnose object presence without mutation; refresh state/route/host-key evidence; only then issue a fresh one-use readback intent. |
| Existing DO Norn release | All-stop, forward-only controller is the supported release design. | Wait for fresh M3 readback and health gates; then execute exact-candidate release, postflight, and same-target recovery evidence. |
| Fresh DO Fleet | Future fresh-Fleet contract remains separate from staging. Current staging main source at 4451767624b12cbc41cc32e160a10d4b46006e0f declares control 3, ingress 2, and one app pool at 4; the provider readback matches those nine nodes. It does not implement the future two-app-pool General + Stateful default. Genesis is a separate existing-Fleet controller-install path. | Complete the future General + Stateful pool contract, provider source, IDs/sizes/capacity bounds, placement semantics, and integrated workflow. Continue token-scope admission and per-file secret provenance/lifecycle. Persistent staging has no sanctioned whole-Fleet destroy path. |
| Workload resize | Fixed-image replica scale, app-pool resize, and mixed-version application release remain separate lanes. M5 has an offline stateless release contract, durable PostgreSQL/etcd receipt CAS, signed-acceptance verification, claim-bound route intents with independent proof and atomic promotion, target-health gating, and old-job stop/readback recovery. PR #302 (c55efd56) fixes the stop-ordering defect: mixed traffic cannot authorize stopping the old job; stop requires target-only route proof. Focused Go tests, all PR CI checks, and the bounded statechart pass. | PR #302 remains open and unmerged; it is not live-qualified. The operator API/CLI admission path is absent, so the internal worker is not yet usable as a release action. A production-store worker process/lease crash matrix, target-specific public traffic evidence, live canary qualification, and stateful data-compatibility proof remain open. Prior replica 2→3→2 and sequential app-node drains do not establish mixed-version release safety. See the M5 protocol. |
| Online Norn upgrade | Unsupported and not modeled. | Keep all-stop until compatibility, writer fencing, quorum, interruption and recovery contracts pass model and staging fault exercise. |
| V2 cleanup | No general /api/v2 namespace; current V3 CLI/UI still call unversioned /api, and the protected norn-fleet runner still calls /effect-grants/v2. | Migrate callers route-by-route and qualify a replacement grant protocol; retain V2 read/recovery support for saved grants until its recovery window closes. See the surface-retirement inventory. |
No single percentage represents readiness across these independent lanes. The DO Fleet is not currently confirmed healthy from fresh evidence, despite its completed soak. No new protected release or provider mutation is authorized by this status update.
The doctl work context returned the work-staging account UUID prefix 17f40050 enforced by its preflight and was used for the provider readback above. The current norn-pilot context is a different account (prefix 05fe610a) and must not be used for staging inventory. Provider inventory is now confirmed, but a fresh protected Norn readback is still required for control-plane, node, route, and application health. The failed Genesis readback is not that proof and must not be retried with its expired intent.
Evidence locations
- Owner-only
pilot261004bhandoff, raw Fleet and management state snapshots, two complete pre-cleanup inventories, run logs and exact-ID deletion records:/Users/arti/.config/norn/fleet-next-full-e2e/successor-pilot261004b/fleet-protected-handoff/. - Protected Fleet apply and automatic recovery.
- Exact dry-run incident planner and tests in isolated Fleet checkout
/Users/arti/.config/norn/fleet-next-full-e2e/worktrees/incident-fleet-orphan-cleanupateb7db92e5cb7f136472c5f43c2c4a8e5c9d4bed0. It was not the executor used for the later emergency teardown and does not provide protected proof. pilot261004aabort and backend receipts under/Users/arti/.config/norn/fleet-next-full-e2e/successor-pilot261004b/and older pilot evidence under/Users/arti/.config/norn/fleet-next-full-e2e/.
The final invoice remains independent of the current technical zero observation. The emergency cleanup did not produce a protected final-zero receipt.