Release-recovery runbook
Recovery procedures for a release that fails mid-flight under the additive release model, stage by stage, without rewriting published history.
This runbook restores a working state when a vX.Y.Z release of
ahmed-g-gad/apothem fails part-way. It follows the same release model as the
release-cycle runbook: an additive model.
Each release adds one commit on main and one signed, annotated tag; the tag
push runs the release and npm workflows. Nothing is force-pushed, and history
on main is never rewritten.
Published history is permanent. Every release tag, GitHub Release and npm
version the project has published stays published. npm versions are
immutable: a published name@version can never be replaced, and a version
number that was unpublished can never be used again. Recovery therefore moves
forward with a new patch release; it does not delete or re-point anything a
user may already have installed.
1. Stage taxonomy
| Stage | Action | Failure mode | Recovery path |
|---|---|---|---|
| 1 | Land the release commit on main (§2.1 of the release-cycle runbook) | CI fails on the release commit | Fix forward with another commit on main; do not tag until CI is green |
| 2 | Push the signed, annotated tag | The push is rejected, or the tag is unsigned or on the wrong commit | §2 below |
| 3 | release.yml builds, signs and publishes the GitHub Release | A job fails, or assets are missing | §3 below |
| 4 | publish-npm.yml publishes @ahmed-g-gad/apothem (runs on the same tag push) | The publish fails | §3 below |
| 5 | Pages and repository metadata | The site or metadata is stale | §3 below |
Every recovery action is idempotent. Running it on a clean state is a no-op; running it on a partial state restores the expected post-stage condition.
2. Tag recovery
Diagnose the tag:
git fetch origin --tags
git ls-remote origin refs/tags/vX.Y.Z
git cat-file -t vX.Y.Z # "tag" for an annotated tag
git tag -v vX.Y.Z # signature checkWhat to do depends on whether anything was published from the tag:
-
Nothing published yet (the release workflow failed before its publish job, and
publish-npm.ymldid not publish): delete the tag and re-tag the same commit with-a -s.git push --delete origin vX.Y.Z git tag -d vX.Y.Z git tag -a -s vX.Y.Z -m "vX.Y.Z" <release-commit-sha> git push origin vX.Y.Z -
Anything published (a GitHub Release with assets, or the npm version): never move, delete or reuse the tag. Fix forward with
vX.Y.(Z+1).
A tag name is never reused for a different commit.
3. Publisher recovery
Each publisher is retried on its own:
| Surface | Diagnostic | Recovery |
|---|---|---|
| Release workflow | gh run view <run-id> --log names the failing job | gh run rerun <run-id> --failed. A failure at Check the plugin version changed with the plugin means the version was not bumped: see the release-cycle runbook's failure table |
| GitHub Release | gh release view vX.Y.Z --json tagName,isDraft,isPrerelease,assets | Re-run the publish GitHub Release job; it edits the existing release and re-uploads assets with --clobber |
| npm | npm view @ahmed-g-gad/apothem version and npx @ahmed-g-gad/[email protected] --version | Re-run publish-npm.yml for vX.Y.Z (gh workflow run publish-npm.yml --field tag=vX.Y.Z) after confirming npm trusted publishing for the package. If the version is already on the registry the workflow skips publication, since npm versions are immutable |
| Pages | gh run list --workflow=publish-static-site.yml --limit=1 and curl -sSI https://apothem.ahmedgad.com/ | Re-run publish-static-site.yml; verify the CNAME still resolves to ahmed-g-gad.github.io |
| Repository metadata | gh repo view ahmed-g-gad/apothem --json description,homepageUrl,repositoryTopics,visibility | Patch description, homepage and topics in place; never recreate the repository |
4. Rollback
A bad release is rolled back by releasing its fix, not by removing it:
-
Revert the offending commits on
main(an ordinary revert commit). -
Cut
vX.Y.(Z+1)with the release-cycle runbook. -
Mark the bad version: add a
### Yankednote to itsCHANGELOG.mdsection, edit its GitHub Release notes to point at the fix, and deprecate it on npm:npm deprecate @ahmed-g-gad/[email protected] "Contains <defect>; use X.Y.(Z+1)"
The bad tag, GitHub Release and npm version stay published, so pinned installs and reproducible builds keep resolving. Log the rollback as a release incident.
5. The release-facade rule
The repository's release-facade rule (the Release Facade section of the contributor instructions) states the additive model above: every published tag, GitHub Release and npm version stays. Its current-version-only requirement covers what public surfaces say. This runbook never deletes or rewrites a published tag, GitHub Release or npm version.
6. Cross-references
- Release procedure: release-cycle runbook.
- Release policy: release-engineering policy.
- Tag objects and how to verify each one: the release-signing section of SECURITY.md.
- Pages configuration:
site/content/docs/runbooks/pages-enablement.mdx.