Skip to content

Protected M6 staging app rollback ​

This source adds a staging-rollback lane to the reusable Norn app release workflow. It restores only an earlier successful staging deployment with an exact deployment UUID, source SHA, and OCI digest. The Norn API rechecks its durable GitHub release candidate, current InfraSpec binding, immutable artifact attestation, and a different current deployment before queuing the rollback. No Nomad job write occurs in the workflow.

The staging release caller supports a protected workflow_dispatch for the first post-M8 release, so the already-reviewed current master source can be qualified without a no-op app commit. Dispatch it only after catalog promotion/readback and runtime preparation are complete. The dispatch must name the exact current master SHA, the terminal signed M8 completion receipt SHA-256, and release-hello-norn-mysql-staging-<source_sha>-<m8_receipt_sha256>. The receipt hash must match the protected NORN_M8_COMPLETION_RECEIPT_SHA256 repository variable. Keep NORN_MANAGED_MYSQL_STAGING_RELEASE_ENABLED false until those prerequisites are complete; the staging job also requires the acting and triggering user to be antiartificial and a protected master ref.

The lane is inactive until a caller on the protected default branch pins the merged reusable workflow by its full commit SHA, and the staging control API runs a signed release containing this handler. The staging GitHub OIDC policy must allow the pinned reusable workflow, workflow_dispatch, release:rollback, the exact app/repository binding, and the staging environment. Do not use a relative reusable workflow reference; the exchange requires a SHA-pinned job_workflow_ref.

The private one-use Linux runner must have labels self-hosted, Linux, X64, norn-fleet-protected, and norn-work-staging-fleet-effect. Register it with the exact name norn-work-staging-work<run_id>-runner after the dispatch exists; the job checks this name before reading the private route or obtaining a token. Provide the existing CA file at /etc/norn-fleet-executor-api-route/ca.pem and its reviewed SHA-256 in NORN_FLEET_RUNNER_API_CA_SHA256. The staging NORN_API_URL must be https://localhost:18443. Retire the runner after this one job.

The dispatch must supply lane: staging-rollback, rollback_environment: staging, runner_private: true, rollback_confirmation: ROLLBACK, the exact admitted target, an existing database-backed record ID, and the reviewed SHA-256 of /route-proof. Set NORN_MANAGED_MYSQL_PUBLIC_URL to the pathless HTTPS origin of the managed app. The job reads that record and route proof before and after, samples public /version every two seconds while Norn operates, records non-200 samples, and accepts only a successful durable Norn operation. Sample counts are evidence of measured interruption, not a zero-downtime guarantee.

The staging release caller is hello-norn-mysql-release.yml. It currently pins reusable workflow SHA 864cd7783027af6e8b1b69e894e29b3f0f019c12 and selects the private Fleet effect runner. The protected runtime oidc.env must allow this exact job_workflow_ref, with the exact hello-norn-mysql-managed=antiartificial/norn@<repository ID>@<owner ID> binding, staging environment, protected master ref, and workflow_dispatch event. Preserve the other approved bindings when adding these exact values. The rollback caller is hello-norn-mysql-staging-rollback.yml, pinned to reusable workflow SHA 6bd8304153686ab70bf9c7874b62b1a6a4f06bf6. Keep NORN_MANAGED_MYSQL_STAGING_ROLLBACK_ENABLED false until the Fleet one-use job gate and owner-controlled OIDC job_workflow_ref allowlist match the exact caller and workflow_dispatch. The runner authorization must bind the exact deployment UUID, source SHA, artifact digest, record ID, and route-proof SHA-256.

The managed MySQL activation has a real hostname, deploy: true, and a promoted catalog that was read back on all three controls. DNS resolves, but TLS still presents the Traefik default certificate. Keep the immutable image repository and restricted ingress policy. Confirm that the reviewed route-proof body hash remains c2f21436748914ff673d5bb71c7e5fb50bcc64eced18dae230eface10d65b1cc. Install and verify the hostname certificate on both ingress nodes before enabling release. Provision the four revision-job-scoped Nomad Variable keys MYSQL_DSN, MYSQL_CA_PEM, MYSQL_PINNED_IP, and PILOT_WRITE_TOKEN through a protected executor. Verify MySQL TLS and that the app identity has only its intended data access. Activate the zero-service staging-app-runtime database profile and signed Norn app runtime workers through their protected lanes if they are not already active.

For A/B proof, dispatch two protected staging releases from exact reviewed master source SHAs. Change only the pilot app source between A and B. Supply the current M8 completion receipt hash and required owner acknowledgement for each release. Each release must carry its own source SHA, OCI digest, GitHub candidate, and successful durable app.deploy operation. Both A and B must include the record response's immutable createdAt field. Before B, create an immutable test record through the app's authorized write path and retain its ID and creation timestamp. Keep the schema, DB connection, job name, namespace, replica bounds, and route unchanged between A and B. The rollback dispatch targets A's exact successful deployment; Norn must prove A's artifact remains admitted and B is the current successful deployment. Compare a stable hash of record ID and creation timestamp before and after. Sample it during rollback alongside public /version, and retain the public sample artifact. A repository source change that also changes the InfraSpec digest invalidates rollback admission and requires a new compatibility review.

The managed MySQL InfraSpec now declares the real hostname, and the catalog is selected and read back. DNS resolves, but TLS still presents the Traefik default certificate. Before activation, complete TLS, provision the revision-scoped Nomad Variables, verify the app and database identities, and confirm the signed runtime workers and protected release binding. Then deploy version A and version B through Norn and qualify rollback. The older direct Nomad pilot does not qualify as the rollback target. If schema or writer-floor changes are incompatible with version A, use a compatible pair or forward repair; do not force an old binary or image back into service.