Norn v3 execution milestones
For the current runtime state, milestone dispositions, findings and next actions, see Norn v3 launch state. This document defines the release contract and exit criteria.
Release contract
The current launch milestone is the online upgrade of the existing DO staging Fleet. Its working target is the week of 2026-11-16, subject to controller, restart, and measured continuity proof. The broader program below also covers fresh creation, workload admission, resizing, and database lifecycle work. Those follow-on features are not prerequisites for this platform launch.
The broader program tracks two independent deployment paths:
- Existing single-machine Mini: v2/PostgreSQL → v3/PostgreSQL, preserving running workloads and application data.
- DigitalOcean HA Fleet: online v3 control upgrades in existing staging. Fresh creation with three etcd control members and independent application databases remains a separate qualification.
The existing Mini runs as an independent single-node development control plane on its existing PostgreSQL database. It is not a member of the DigitalOcean Fleet and its upgrade does not depend on that testbed. The Mini control API may hold a configuration document for the separate DO staging target; that does not make the Mini part of its node topology. The DO Fleet's validation is independent of upgrading or running Norn on Mini.
The DigitalOcean staging Fleet is a separate testbed intended to resemble production with lower resource use. It is a separate Norn deployment for fresh bootstrap, quorum, workload communication, scaling and recovery. Its etcd and management-node requirements apply to that Fleet deployment, not to the Mini. These paths share release qualification but are not prerequisites for one another. No general PG↔etcd control conversion is required. A targeted v2 prerequisite release is needed only if the inspected Mini version cannot safely upgrade directly.
The Mini is a development machine. Its v3 upgrade may use a fresh control backup in a directory on that Mini, with a verified private restore and a recorded rollback decision. Loss of the Mini or its storage may also lose that backup; this release makes no Mini disaster-recovery RPO or RTO claim. Off-host, bounded-loss recovery is a separate production qualification, not an M0/M5 Mini prerequisite. The Mini upgrade can advance independently of Fleet provisioning. Fleet's etcd snapshots, restore, quorum and fault gates remain required before claiming that staging exercise complete.
Mini is an independent single-node v3 development installation, not a Fleet member. Its direct upgrade does not depend on the DO staging Fleet. The Mini API may separately hold a configuration document for that target; this does not make Mini a node in its topology. The DigitalOcean Fleet is the only Fleet in this work and is a separately deployed staging testbed intended to resemble production with lower resource use; a future production Fleet is another distinct environment. Deploy applications directly to the staging Fleet. Mini-to-Fleet application promotion or data migration is not an initial release gate. If cross-environment mobility is offered later, qualify explicit identity, data and provenance mappings without cloning Mini's control database or sharing its authority.
Milestone order
| ID | Outcome | Depends on | Accountable repositories | Exit evidence |
|---|---|---|---|---|
| M0 | Freeze initial contracts and establish Mini baseline | Planning review | Norn, Fleet, NornUI | Reviewed decision register, source/runtime inventory, fixture and budget plan |
| M1 | Shared control semantics and working PG adapter | M0 | Norn | Store invariant tests, ownership fixes, old-data compatibility |
| M2 | Shared profiles, database bindings and bounded history | M1 | Norn, Fleet, NornUI | Local profile integration, binding resolution and archive/replay tests |
| M3 | Etcd-backed control plane and fresh HA bootstrap | M1; M2 archive contract | Norn, Fleet | Three-member bootstrap/restore, authority/fencing and quorum tests |
| M4 | Fleet app pools, persistent scaling and replacement | M1; M2 profiles | Norn, Fleet, NornUI | Verified placement, 2→3→2 nodes, replica growth and safe drain |
| M5 | Rehearsed Mini v2→v3 upgrade on PG | M1–M2 | Norn, NornUI | Isolated upgrade/rollback from representative Mini state; jobs and routes unchanged |
| M6 | Running upgrades and app database lifecycle | M2–M4 | Norn, Fleet, app owners | v3→v3 rolling rehearsal; app DB upgrade/cutover and interrupted recovery |
| M7 | Direct Fleet application deployment | M3–M4; M6 app DB lifecycle | Norn, Fleet, selected app repo | Fleet-owned web/worker workload, data binding, public traffic, recovery and rollback proof |
| M8 | v3 release qualification | M3–M7 | All release owners | Signed candidate, client parity, version matrix, soak/fault results and operator runbooks |
| M9 | Controlled adoption | M8; explicit rollout scope | Operators + release owners | Mini upgrade and separate staging Fleet deployment with directly deployed applications independently verified |
After M1, profile/archive, etcd and scheduling work can overlap at agreed interfaces. M5 proceeds without a live DO Fleet. Provider-backed tests follow local/Linux isolated qualification; their budget and exact targets are established before provisioning. M8 qualifies the package; M9 is actual deployment, not an automatic side effect of completion.
M0 — Decisions, baseline and fixtures
- Record deployed Mini binary/source/schema versions, supervisor configuration, app specs, enabled/disabled state, actual job IDs/counts, routes/ports, secrets references and volume/database ownership. Prior conversation snapshots are not sufficient.
- Measure PG table/index sizes, growth and connection use; classify all observed control tables and non-SQL stores. Record archive, encryption/signing keys and restore requirements without printing credentials.
- Build a sanitized representative fixture for routine CI. Separately plan a private restore rehearsal for fidelity, with denied production network egress and all workers/webhooks/cron/provider mutation disabled.
- Resolve scale precedence: proposed rule is a revisioned desired-scale object seeded by import, with explicit scale changes retained across code deploys. Applying a changed declared scale is an explicit configuration operation with conflict detection, not an implicit deploy reset.
- Decide API handoff contract, pool/ingress roles, resource schema ownership, first app engine support, diagnostic budget and log backend spike. Draft numerical targets in planning-contracts are not yet accepted SLAs.
- Assign named owners and PR boundaries. Re-estimate at M1/M3 based on actual work; no calendar commitment before the baseline and spikes.
Gate: review-ready contracts and measurable targets, a verified Mini upgrade-backup/restore strategy, no implicit ownership ambiguity. Scope decisions can be accepted independently of unmeasured targets. A production disaster-recovery target is tracked separately.
M1 — Shared control behavior
Suggested review units:
- Domain interfaces and a PG adapter preserving observable behavior; inventory and eliminate bypassing direct SQL at control boundaries.
- Atomic acceptance/idempotency/audit intent and explicit execution ownership/fencing contract.
- Owner-aware exec-session and operation recovery, including candidate startup with workers disabled.
- Versioned additive schema migrations with one migration owner, compatibility guard and rollback window.
- Canonical backup/export format for inspection and disaster recovery; stable identifiers and original receipt/signature bytes.
Gate: concurrent request/recovery tests, ambiguous response reconciliation, no global failure of healthy sessions on candidate startup. Any downstream action that cannot be fenced has a defined reconciliation or proven-stop rule.
Rollback: retain old readable schema; do not contract fields during the Mini upgrade window. Export format does not imply a supported cross-backend converter.
M2 — Profiles, database references and retention
Suggested review units:
- Local-PG and Fleet-etcd profiles with common logical resource names and independent control-plane identities.
- Database service/binding resolver shared by apps, workers, migrations, backups, restore and probes. Preserve legacy Mini connection resolution through an explicit compatibility adapter.
- Allocation/task-aware logs, node rotation and bounded collector spool; authenticated historical queries.
- Evidence archive/outbox and verified pruning; event replay expiry/resync; local filesystem and Fleet object adapters.
- Capability/status UI exposing current profile/backend, desired vs observed scale, binding generation, archive health and unsupported operations.
Gate: an unchanged imported Mini app resolves to its existing DB and route; a new Fleet app resolves independently; no secrets leak in exported topology; evidence survives archive/index recovery. Ordinary WordPress uses a MySQL adapter. CockroachDB is deferred.
Rollback: disable collection/pruning independently; retain compatibility reads for archived records. Binding changes never silently rewrite live application connections.
M3 — Etcd and fresh Fleet
Suggested review units:
- Etcd adapter with shared invariant suite, bounded keys/indexes, lease-independent operation records and revisioned event cursors.
- Host-supervised member bootstrap, TLS, membership changes, snapshots, compaction, monitoring and offline restore.
- Provider-independent readiness for control backend; remove mandatory SQL probes from etcd deployments without weakening admission.
- Fresh Fleet workflow creates three control members, independent discovery/scheduler groups and evidence/log destinations. Bootstrap identity/keys without depending on a functioning Norn queue.
Gate: fresh bootstrap and full restore without control PG; one member lost; quorum lost; stale executor; corrupted snapshot; disk/quota alarm; archive outage. Verify no accidental default/local PG dependency anywhere, including host agents and tools.
Rollback: disposable rehearsal topology can be recreated from supported backups. Production member changes require quorum-safe recovery; binary downgrade support is version-specific. Do not replace all control voters together for a generic blue/green VM policy.
M4 — App capacity and placement
Suggested review units:
- Role-based pool schema, inventory/bootstrap/LB wiring and actual Nomad pool enrollment, including arbitrary names.
- Durable replica intent and drift visibility; scale/config precedence; explicit per-process placement and identity mapping where jobs split.
- Required/preferred host spreading, resources, graceful drain, web/worker priorities and stateful-storage constraints.
- Generation-based surge/join/readiness/drain/retire executor with exact node identities and interruption recovery.
Gate: VM growth independently from replicas; redeploy preserves desired scale; placement matches policy; 2→3→2 under load; failed drain blocks retirement; N-1 and rollout reserve fit actual allocations. Worker drain preserves acknowledged work. For routed services, bind the deployment's signed InfraSpec and accepted region to a server-selected Fleet plan and durable prior route, persist a generation-fenced route intent, then prove every ingress node and the public endpoint before recording an active traffic weight. A desired weight in Nomad or Consul metadata is insufficient. Node addition alone is not a rebalance guarantee.
Rollback: retain prior node generation until replacement readiness; restore desired revisions through reviewed operations. Restore old job identity only through a declared mapping/traffic handoff.
M5 — Mini upgrade rehearsal
Use a copied control DB and private copies of specs/state. The rehearsal must not hold usable production mutation credentials or network access to production services. An isolated Nomad/Consul environment proves translation compatibility; read-only identity comparisons capture the live workload preservation requirement.
Rehearse backup → drain Mini-affecting Norn operations → install candidate → additive migration → health/auth/history validation → release switch → worker resumption → smoke. Preserve queued Fleet-only GitHub dispatch receipts in the shared Mini control database; they are not claimed by the Mini operation worker. A running dispatch or any Mini-affecting operation still blocks the transition. Upgrade API/agent consumers compatibly. Keep existing application PG, app data, Nomad allocations and routes in place for the real upgrade; no Fleet adoption or app redeploy is implied.
Gate: original IDs/credentials/receipts survive; no duplicated periodic work; new deployment and recovery operate correctly on fixtures; a fresh local Mini backup can be retrieved and restored privately before the protected transition. Qualify the installed legacy release's actual rollback boundary and use the reviewed legacy-baseline fence where old-binary return is unsupported. Record control interruption separately from application request availability. A single-host reboot still interrupts that host's workloads.
Output: exact supported source version, signed target candidate, commands/runbook, backup manifest and rollback conditions. Live Mini cutover is M9.
M6 — Running upgrades and app databases
Qualify Norn v3 version A→B under traffic, rather than only first installation. Prove candidate isolation, session/worker ownership and compatible old/new schemas. Upgrade Nomad, Consul, etcd and OS independently, one voter/node at a time with readiness checks. Do not label single-active handoff as continuous API service unless tests meet the agreed target.
Define app consistency groups and a migration journal outside the DB being changed. Provision target, validate engine/extensions/roles/networking, rehearse restore, fence all web/worker/cron writers, perform final synchronization, update consumer generation, reconnect, verify, then retain source for the acceptance window. DO managed-PG-to-managed-PG online migration requires a separately qualified path; dump/restore is the baseline.
Gate: PG and selected MySQL adapter backup/restore; planned engine upgrade or cluster replacement; ambiguous commit/retry handling; process crashes at every cutover checkpoint; no old writers after activation. After target writes, return requires reverse reconciliation, not merely old credentials.
Control-PG replacement on the Mini and general PG↔etcd conversion may follow later; app database cutovers are required here.
M7 — Direct Fleet application deployment
Select one representative low-risk database-backed application created for Fleet, with explicit data/file/queue ownership and synthetic traffic. Include web and worker behavior in the fixture; prove other storage patterns separately before offering support for them.
Define the application in Fleet, provision its database and other dependencies, deploy web and worker processes, and verify routes and externally visible traffic. Exercise deployment recovery, scheduled work, node replacement and rollback while the application is running. Prove the workload remains available within the agreed service budget during the M4 scale/drain and M6 running-upgrade exercises.
Gate: auth/URLs/files function; data and acknowledged work remain correct across restart, drain and rollback; no duplicate schedules or writers; Fleet database backup/restore and recovery are evidenced. Mini remains an independent development Norn and is not a deployment source for this gate. Record which application data/engine classes are actually supported.
M8–M9 — Release and adoption
Release evidence includes exact Norn/Fleet/NornUI/runner and upstream versions, workload fixtures, topology and resource reservations, request/error/latency results, operation/effect reconciliation, retention growth and restore records. Etcd soak and failure qualification are initial HA release gates. Any deferred capability is explicitly absent/experimental in negotiation.
Adoption has two independent paths: upgrade Mini on PG for development, and deploy a separate staging Fleet on etcd with applications deployed directly to it. A future production Fleet has its own reviewed rollout and qualification. Prioritize staging Fleet fitness while retaining M8 qualification and a reviewed protected plan before provider changes. A failed Fleet launch must not force a Mini rollback; a successful Mini upgrade must not be mistaken for HA qualification.
The immediate Fleet objective is to qualify the DigitalOcean staging testbed: prove provider-backed bootstrap, control-plane recovery, ingress, directly deployed workloads, scaling and running upgrades against M0–M9 exit criteria. The earlier bounded Fleet fitness pilot supplies historical test cases, not a spend cap or retirement requirement for work staging. Record each result only against the gate it actually proves.
General cross-backend conversion, local single-member etcd, Fleet-PG qualification, autonomous cloud scaling, multi-region HA and additional database engines remain later milestones unless scope is explicitly expanded.
Execution status
The current dated status, observations, findings and next actions are maintained in Norn v3 launch state. No full M0–M9 gate has a recorded owner sign-off as of 2026-11-07. The Mini direct v3 adoption subgate is complete; the broader M6 low-interruption, rollback and application-database criteria and the separate DO staging Fleet gates remain open. Keep this document as the milestone contract and update the launch-state page when new live or protected evidence changes a disposition.