Upgrading Norn
Use this runbook when Norn itself is installed as the local LaunchAgent com.norn.api.
For V3 release planning, use the current platform release lanes and workload resizing guide. Mini is an independent node; the DO staging Fleet is a separate environment. The existing DO Norn release protocol is all-stop and forward-only. Mixed-version Norn control upgrades are not qualified. Mini is an independent node, not a Fleet member. Its API uses the norn.startup/v2 contract with reader/writer floors 5/33 and catalog schema 52. The signed platform release 34178b8b90d52a2fa3f138ffae2dfc63744197e1 is now active on Mini after a successful direct controller upgrade. The existing recovered-forward generation gen-1a769550b1051bc9358ce8e8758265e5 remains candidate-promoted with its A4 recovery candidate; the forward upgrade preserved that evidence and did not advance the schema beyond 52. Authenticated platform smoke passes. Host status and host doctor also pass. Three durable queued upgrades stopped before promotion at the completed-recovery host-agent stability check. The offline model now covers an absent-agent sample, but the operation-window host-agent disappearance remains unexplained. Queued upgrades remain unqualified even though the direct lane now runs 34178b8b. The inventory still contains stopped/unhealthy apps and unknown background services, so platform health is not a claim that every workload is healthy. See the current Mini status.
The retained failed-transition sentinel means every later forward upgrade must verify the completed recovered-forward journal, its signed evidence, and the exact adoption receipt. After the first forward release, bind health, API version/schema, and the managed host agent to the current active release, then verify that exact release is signed and descends from the journaled recovery candidate. Do not require the API to keep reporting the original recovery SHA; that would block the next upgrade. The standard upgrade lane is available when those checks pass. Do not substitute legacy-baseline: that fenced lane requires zero queued Fleet dispatch receipts, while signed recovered-forward can preserve them.
The safe upgrade path restarts only the Norn API process. Do not use make down for a production-ish local upgrade because it also stops Nomad and Consul, which can disrupt hosted apps.
First-Class Platform Lane
cd /path/to/norn
git fetch origin
release_sha=$(git rev-parse origin/master)
# Build into a versioned release directory and boot a candidate API on :18800.
norn platform preflight "$release_sha"
# Promote the release, restart only com.norn.api, and rollback if postflight fails.
norn platform upgrade "$release_sha"
# On proxy-fronted hosts, switch the managed upstream instead of restarting launchd.
NORN_PROXY_RELOAD=true norn platform upgrade "$release_sha" --proxy
norn platform releases
norn platform rebuild "$release_sha" --verify
norn platform rollback <sha-prefix>
norn platform smoke
norn platform env -- norn smoke platform
norn platform proxy-plan
norn platform proxy-status
norn platform proxy-render
norn platform proxy-switch 18802
norn smoke platform
# Preferred remote/UI lane: durable and restart-safe.
norn platform queue-preflight "$release_sha"
norn platform queue-upgrade "$release_sha"
norn platform queue-rollback <sha-prefix>
norn platform queue-smokeThe Mini's 2026-11-10 later-upgrade attempts exposed successive recovered-state gate mismatches. The d414b3ab and 640b8b89 attempts stopped before promotion at evidence-verification checks. The 1b2dc229 follow-up passed exact-source qualification and fresh backup/shadow rehearsal, but two queued operations failed before promotion at the host-agent stability check. A later signed release 34178b8b passed those preflight/backup/shadow gates; its queued operation reported LaunchAgent wrapper has no child process. The Mini was then upgraded successfully to 34178b8b through the direct controller lane, with schema unchanged at 52. A new durable queued platform.smoke has since passed, and the current LaunchAgent wrapper plus exact managed host-agent child remained stable for three one-second observations. This does not resolve or qualify the queued upgrade path: the earlier failed receipt has no process-state trace, and no root cause is established. Before another queued attempt, use a fresh signed release and exact-candidate backup/shadow, then inspect the terminal receipt; never replay a failed operation's idempotency key. See the current incident and evidence record.
Use the exact pushed SHA intended for promotion. HEAD is convenient for a local development rehearsal but can move between review and upgrade.
Mini single-node v3 transition
Mini runs Norn as a single node with its existing PostgreSQL database. It does not require the Fleet management or etcd nodes; Fleet is an independent staging environment. Optional Fleet integration can be configured separately: the current Mini implementation reads a local NORN_FLEET_CONFIG document and uses a protected GitHub App/workflow bridge for Fleet operations. It does not need a direct connection to a Fleet-node API, and this integration is not an upgrade prerequisite.
For the Mini bootstrap lane, use the installed managed script through the authenticated runtime wrapper so the encrypted API environment is loaded only for the child process. The signed release must include the evidence helpers. After importing and verifying the exact candidate release, capture a fresh protected database backup and rehearse that candidate against a private shadow restore before running the standard upgrade:
norn platform env -- /Users/0xadb/.config/norn/host/bin/platform-upgrade mini-protected-backup
norn platform env -- /Users/0xadb/.config/norn/host/bin/platform-upgrade mini-signed-shadow <full-candidate-sha>
norn platform env -- /Users/0xadb/.config/norn/host/bin/platform-upgrade preflight <full-candidate-sha>
norn platform env -- /Users/0xadb/.config/norn/host/bin/platform-upgrade upgrade <full-candidate-sha>Do not run the backup or shadow helpers from an unsigned checkout on Mini. Keep the backup fresh, verify the shadow receipt names the exact candidate and passes both private-restore and runtime claims, and inspect the preflight result before the database-changing upgrade. A schema migration requirement can be a valid reason for ordinary upgrade after those gates; it is not evidence that the live database was migrated. Smoke and host-assurance checks are still required afterward.
If the retained failed-transition sentinel is present, ordinary upgrade is intentionally blocked until recovered-forward evidence and preservation receipts verify. Start that lane through the exact signed candidate's wrapper so it can bind a fresh backup, signed shadow, current workload baseline, and successor generation before calling the managed controller:
The Nov 10 Mini recovery is complete and recorded in the current recovery record. For any future interrupted transition, preserve its durable phase and resume only through the signed controller repair path that binds the exact active generation, journal digest, source evidence, and release. Refresh authorization after each journal phase change. Do not hand-edit the journal or restart a fenced host agent by hand. Publication alone does not prove that a node is ready.
The signed recovery API launcher also checks the adopted baseline's Git ancestry. Set NORN_PLATFORM_REPO in the encrypted API environment to the canonical Mini checkout; an SSH shell variable does not reach the GUI LaunchAgent. After a repair step changes the journal, create a fresh repair authorization for its new digest. If launchctl bootout returns while the job still appears loaded, inspect the durable phase, process, and listener before resuming the same signed transition. Do not reuse an authorization for an older journal digest.
The adopted baseline may be an older signed ancestor of the transition source. The finalizer must verify that same source-to-candidate lineage before it releases the write fence; it must not require the transition source SHA to equal the adoption receipt SHA.
Keep replaced adoption receipts as owner-only archived files named .norn-recovered-baseline-adoption-v2.json.<suffix> in ~/.config/norn. Immutable predecessor evidence binds the receipt's raw SHA-256, so changing the current receipt for a newly accepted baseline must not invalidate the earlier generation. The evidence verifier resolves only an archive with the exact signed digest (in deterministic name order) and verifies it with the historical audit key. If no exact archive is present, stop and recover the evidence; do not rewrite predecessor evidence or copy a different receipt into place.
norn platform env -- \
/Users/0xadb/norn/releases/<candidate-sha>/bin/mini-recovered-forward-transition \
<candidate-sha> \
--source-sha <current-mini-release-sha> \
--transition-id <fresh-transition-id> \
--backup-proof /absolute/private/backup-proof.json \
--backup-artifact /absolute/private/control.dump \
--shadow-receipt /absolute/private/signed-shadow-receipt.jsonThe candidate SHA must be the exact signed release already installed under ~/norn/releases. When the active recovery generation completed on an earlier signed release, the wrapper verifies signed Git ancestry from that promoted candidate to the current source before it creates a successor generation. Do not hand-edit the generation pointer or journal. The wrapper owns this roll-forward path; platform-upgrade recovered-forward is its internal controller handoff and requires the signed evidence that the wrapper creates.
macOS and Homebrew Maintenance
Homebrew owns executable installation; Norn owns the cloudflared service definition. Do not run brew services start/restart cloudflared, because it regenerates a bare-binary LaunchAgent without named-tunnel arguments.
brew update
brew upgrade
norn host install --repo /path/to/norn
norn host cloudflared-recover --cloudflared-probe https://service.example.com/health
norn host doctor
norn host statusUse the same doctor and status checks after a macOS upgrade or reboot. The managed plist points through Homebrew's stable binary path, validates the ingress config before activation, and retires only a matching or broken legacy homebrew.mxcl.cloudflared job.
Signed GitHub Release Checklist
For an operational signed release, merge the reviewed change to protected master before building. First dispatch full .github/workflows/v3-ci.yml qualification on protected master at that exact SHA and require its first attempt to succeed. Dispatch .github/workflows/platform-release.yml from master with the exact 40-character SHA and qualification run ID. Approve only the matching platform-release environment run after authorization, UI, and all four OS/architecture bundle jobs succeed.
Assign the release a version-first operator label in the established v<major>.<minor>.<patch>-platform[-<distance>-g<shortsha>] form, or for a candidate v<major>.<minor>.<patch>-rc.<n>-platform[-<distance>-g<shortsha>]. For example: v2.21.0-platform, v2.21.0-platform-3-g1a2b3c4, or v3.0.0-rc.1-platform. Candidate GitHub Releases must be marked prerelease consistently with the signed bundle version; stable releases remain stable. Release history in the API, CLI, web UI, and native clients should lead with that value and show a short SHA only as supporting provenance. Do not rename the immutable GitHub tag, full-SHA release directory, manifest binding, or signed asset names to match the display label.
Before host promotion, verify all of the following:
- the human-facing version uses the intended version-first platform label and is consistent across all platform bundles;
- tag
platform-<full-sha>targets the same commit and reportsimmutable=true; - every supported platform has one archive, manifest, signature, and SPDX SBOM;
- the host uses the protected workflow's Ed25519 public key and owner-only fetch/verify configuration;
- public repositories download anonymously; private mirrors use only an owned, non-symlink, exact-mode-
0600Contents-read token file; - the Mini-affecting operation drain is clear and host fetch/verify hooks are executable. Queued
fleet.github.apply-dispatchreceipts in the shared control database are preserved because the Mini worker does not claim them; running Fleet dispatches, active Mini operations, or malformed/truncated queue evidence still block the upgrade.
With require-signed, rollback, preflight, and upgrade invoke the configured fetch hook when the exact release directory is absent. They require an exact full-SHA import, then run both the local manifest verifier and external Ed25519 verifier before candidate startup or promotion. If the command begins a local build instead, stop: the selected script or configuration is stale.
rebuild --verify is different from promotion: it first verifies the retained release under the configured signature policy, then creates a fresh unsigned staging build, validates that staging manifest locally, and compares its canonical release manifest with the retained artifact. The temporary rebuild is never promoted or treated as a signed release. The rebuild also reuses the verified retained manifest's version string; it must not recompute embedded versions from the current tag graph because release publication can add tags after the original build.
The platform lane builds from an isolated git worktree into $HOME/norn/releases/<sha>, writes a $HOME/norn/current symlink, installs compatibility binaries into $HOME/go/bin, and health-checks a candidate API with recovery and operation workers disabled so preflight does not mark running work failed or claim queued jobs.
Upgrade, rollback, and manual proxy switching are host-serialized with an advisory lock in the release root. A concurrent direct promotion fails closed; a crashed process automatically releases the kernel lock, so rerunning the intended operation is the recovery action.
Promotion also synchronizes and byte-verifies the persistent host agent, its private CLI copy, platform-upgrade, platform-release-manifest, the signed GitHub fetch/verify helpers, and host-runtime. This keeps unattended recovery and prerequisite checks on the same revision as the API instead of leaving launchd to invoke an older managed copy after an otherwise successful upgrade. Pre-manifest releases are unverifiable and are refused. A first, development-only migration can opt into one legacy rollback target with NORN_ALLOW_LEGACY_RELEASES=true; production always refuses that escape hatch.
The queued lane is preferred when the operator is not already on the host. It returns a durable operation ID, is executed by com.norn.host-agent, and can be followed with norn operations <operation-id> even after the API restarts.
Platform subcommands add existing /opt/homebrew/bin and /usr/local/bin directories to the managed child process before invoking the upgrade script. This keeps direct platform maintenance reliable through non-interactive SSH shells with a minimal PATH; queued work continues to execute in the managed host-agent environment. The enrichment neither mutates the parent shell nor replaces prerequisite checks, and missing tools still fail visibly.
Script discovery is portable and deterministic: an explicit --script or --repo wins, followed by the current checkout, an adjacent installed script, the persistent managed host copy, and finally $HOME/projects/norn. It does not depend on a hard-coded operator account. Routine remote upgrades therefore use the same managed script that the previous promotion synchronized. A managed script likewise resolves its source checkout from an explicit NORN_PLATFORM_REPO, its own repository, the current directory, or $HOME/projects/norn, in that order.
A clean checkout can still be stale or historically divergent. In that case, passing only --repo selects its checkout-local script. For bootstrap or repair, pass the synchronized managed platform-upgrade with --script explicitly and use --repo only for source-object resolution. If an old script attempts to remove an immutable SHA directory, stop; do not relax directory permissions.
macOS release parents may carry com.apple.provenance. Importer f830b3c and later keeps the staging root owner-writable through the atomic no-replace rename and immediately seals the installed root, while descendants are sealed before publication. Do not remove provenance metadata to work around an older importer. A completed import must remain read-only and verify as signed.
For a first development migration from a pre-manifest current release, use a temporary owner-only configuration with allow-unsigned and NORN_ALLOW_LEGACY_RELEASES=true for that promotion only. Keep the persistent configuration at require-signed, remove the temporary file on every exit, and never use this escape under NORN_PROFILE=production. After two signed releases exist locally, rehearse rollback, authenticated smoke, and re-promotion without the escape.
Use these environment variables when the repo or host layout differs:
| Variable | Default | Description |
|---|---|---|
NORN_PLATFORM_REPO | script repo | Norn checkout to build |
NORN_RELEASES_DIR | $HOME/norn/releases | Versioned release directory |
NORN_NODE_BIN | detected node@24, then node | Node.js executable; its version must exactly match the repository pin |
NORN_EXPECTED_NODE_VERSION | repository pin, then v24.19.0 | Exact Node release-build version |
NORN_EXPECTED_PNPM_VERSION | repository pin, then 10.32.1 | Exact pnpm release-build version |
NORN_CURRENT_LINK | $HOME/norn/current | Current-release symlink |
NORN_BIN_DIR | $HOME/go/bin | Compatibility install directory |
NORN_HOST_CLI_BIN | $HOME/.config/norn/host/bin/norn | CLI copy used by persistent host recovery and assurance |
NORN_CANDIDATE_PORT | 18800 | Alternate-port candidate API |
NORN_TOKEN / NORN_API_TOKEN | — | Optional bearer token for active-operation drain checks |
NORN_DRAIN_MODE | fail | fail, wait, or force for active-operation drains |
NORN_DRAIN_EXCLUDE_OPERATION_ID | — | Host-agent operation omitted from its own drain query |
NORN_SKIP_CANDIDATE_API | false | Skip side-by-side candidate boot |
NORN_PLATFORM_UPGRADE_MODE | restart | restart or proxy; --proxy sets this for upgrades |
NORN_API_ENV_FILE | $HOME/.config/norn/api.env.enc.json | SOPS JSON env file for platform smoke and platform env |
NORN_SOPS_BIN | sops | SOPS executable used for encrypted env loading |
NORN_PROXY_DIR | $HOME/norn/proxy | Managed proxy state directory |
NORN_PROXY_CANDIDATE_PORT | 18802 | Private candidate API port used by proxy upgrade mode |
NORN_PROXY_PID_FILE | $NORN_PROXY_DIR/api.pid | Current proxy-managed API pid |
NORN_PROXY_RELOAD | false | Reload Caddy after proxy-switch |
NORN_RELEASE_SIGNATURE_POLICY | allow-unsigned | require-signed is forced by NORN_PROFILE=production |
NORN_RELEASE_FETCH_HOOK | — | Optional missing-release fetch adapter |
NORN_RELEASE_VERIFY_HOOK | — | External signature verifier; mandatory in production |
NORN_RELEASE_CONFIG_FILE | $HOME/.config/norn/release.env | Optional mode-0600 allowlisted host configuration loaded by direct and queued maintenance |
NORN_RELEASE_REPOSITORY | — | GitHub owner/repository used by the fetch adapter |
NORN_RELEASE_PUBLIC_KEY | — | Host path to the pinned Ed25519 public PEM |
NORN_RELEASE_GITHUB_TOKEN_FILE | — | Preferred owned, non-symlink, exact mode-0600 narrow Contents-read token file for private downloads |
NORN_RELEASE_GITHUB_TOKEN | — | In-process narrow Contents-read token fallback for private downloads; public repositories need neither token form |
NORN_ALLOW_LEGACY_RELEASES | false | Explicit one-time development migration escape; ignored in production |
For signed off-host artifacts, configure the protected workflow and host hooks described in Platform Upgrades. The workflow accepts only an exact commit already merged to protected master, keeps the signing key in the protected publish job, produces Linux and macOS bundles for amd64 and arm64, and resumes an interrupted draft without replacing an existing asset.
Manual Fallback
The old direct-binary path still works when the platform lane itself is broken:
export NORN_PLATFORM_REPO=/path/to/norn
export NORN_BIN_DIR="${NORN_BIN_DIR:-$HOME/go/bin}"
cd "$NORN_PLATFORM_REPO/v2"
cd ui && pnpm build
cd ..
make build
mkdir -p "$NORN_BIN_DIR"
install -m 0755 bin/norn-api "$NORN_BIN_DIR/norn-api"
install -m 0755 bin/norn "$NORN_BIN_DIR/norn"
launchctl kickstart -k gui/$(id -u)/com.norn.apiSmoke Checks
norn version
norn ops platform
norn services
norn status
curl -sf http://127.0.0.1:8800/api/healthOpen http://127.0.0.1:8800 and check the Platform tab. The Platform tab should show service counts, deployment provenance, snapshot retention state, access counts, and observability status.
The Platform tab also lists installed platform releases and can start a rollback through the same platform-upgrade script used by the CLI.
When an authenticated API token is available, run:
norn smoke platformThis checks health, operation drain, current release metadata, and recent warning/critical Beacon events.
If the API token is only available through the LaunchAgent/SOPS runtime env, use:
norn platform smokeFor arbitrary authenticated checks under the same env:
norn platform env -- norn events --severity critical --limit 5Rollback
norn platform upgrade rolls back automatically when postflight health fails and $HOME/norn/current pointed at a previous release.
Prefer the first-class rollback command:
norn platform releases
norn platform rollback <sha-prefix>The following direct copies are break-glass recovery only. They bypass the normal manifest/signature gate and do not update every managed helper:
releases_dir="${NORN_RELEASES_DIR:-$HOME/norn/releases}"
bin_dir="${NORN_BIN_DIR:-$HOME/go/bin}"
backup="$releases_dir/<previous-sha>"
mkdir -p "$bin_dir"
install -m 0755 "$backup/bin/norn-api" "$bin_dir/norn-api"
install -m 0755 "$backup/bin/norn" "$bin_dir/norn"
launchctl kickstart -k gui/$(id -u)/com.norn.apiThen rerun the smoke checks.
Notes
- The root
Makefilestill targets the older non-v2 tree. Usev2/Makefilefor v2 releases. NORN_UI_DIRcan point at an explicit dashboard build. If it is unset, the API serves$HOME/norn/current/uiwhen that release directory exists.- Keep Nomad, Consul, Postgres, and app allocations running during a Norn API upgrade unless you are intentionally rebuilding the whole dev environment.
- A normal candidate API is a preflight check, not the active control plane. On proxy-fronted hosts,
norn platform upgrade --proxyperforms a managed upstream cutover; see Platform Upgrades. - App deploys, preflights, and rollbacks are queued in control-plane Postgres. The drain gate checks those active rows before platform upgrades; read-only preflights can retry, while interrupted mutable deploy stages fail visibly rather than being replayed blindly.
norn platform proxy-planprints the no-blip proxy design.proxy-status,proxy-render, andproxy-switchmanage an optional local proxy config and upstream state.platform upgrade --proxyuses that state only when the host is already proxy-fronted andNORN_PROXY_RELOAD=true.