Release-cycle runbook

This runbook drives a vX.Y.Z release of apothem from a clean working tree to verified distribution surfaces. The procedure is written for the maintainer wielding gh, git, and the release-tier shell; it terminates with a verified Release page, a filed CHANGELOG, and an install-path smoke confirming each user-facing surface answers the canonical install command.

The release model is additive. Each release adds one commit on main and one signed, annotated vX.Y.Z tag; nothing is force-pushed and no earlier tag, GitHub Release or npm version is removed. Every release keeps its tag, its GitHub Release and its npm version, and all stay published: npm versions are immutable (a published name@version can never be replaced), and pinned installs depend on the tags.

The tag push starts three workflows. .github/workflows/release.yml builds, signs, attests, and publishes the GitHub Release with its full artifact set: sdist, wheel, CycloneDX SBOM, Sigstore cosign bundles, SLSA-3 provenance, and the platform runtime archives. publish-npm.yml publishes @ahmed-g-gad/apothem to npmjs.com through npm trusted publishing, and publish-vscode.yml packages the VS Code extension and publishes it to the Visual Studio Marketplace. The versioned CHANGELOG section lands in the release commit; the operator runs the install-path smoke after both workflows finish.

1. PrerequisitesCopy Anchor Link

The maintainer confirms the following before tagging:

• Working tree. Clean (git status shows nothing pending). All feature work for the release is on main.

• CI green. Latest main gh run list --branch=main --limit=1 --json conclusion --jq '.[0].conclusion' returns success.

• Tooling. gh (GitHub CLI) version 2.40 or later authenticated against ahmed-g-gad; git 2.40 or later with the GPG signing subkey loaded; the npm trusted-publisher binding for @ahmed-g-gad/apothem is live (verified per the publisher-account-setup runbook).

• Version anchors aligned. The version string declared at pyproject.toml matches the planned tag without the v prefix (§2.1 sets it, with every other anchor). Verify:

  Copy Text

  grep '^version' pyproject.toml

  Expected output (for a vX.Y.Z release; placeholders stand for the in-flight release tag):

  Copy Text

  version = "X.Y.Z"

• CHANGELOG entries staged. The release bullets sit under ## [Unreleased] in Keep-a-Changelog headings; §2.1 turns them into the dated version section in the commit that lands the tag.

2. Tag and triggerCopy Anchor Link

2.1 Author the release commitCopy Anchor Link

The version lives in more than a dozen files, so do not edit them by hand. One command moves all of them:

git checkout main
git pull --ff-only origin main
python scripts/release/bump_version.py X.Y.Z

bump_version.py updates every version anchor together:

• Every packaging, extension, plugin and marketplace manifest, plus the citation file (version and release date). The full list:

  Copy Text

  pyproject.toml
  package.json
  site/package.json
  site/package-lock.json            (the project entries only)
  vscode-extension/package.json
  gemini-extension.json
  qwen-extension.json
  .claude-plugin/marketplace.json   (top level and metadata)
  plugins/apothem/plugin.json
  plugins/apothem/.codex-plugin/plugin.json
  CITATION.cff

• CHANGELOG.md: the [Unreleased] entries become a dated ## [X.Y.Z] section, [Unreleased] compares from vX.Y.Z, and a [X.Y.Z] link is added.

• SECURITY.md: a new minor gets a supported X.Y.x row. Whether older minors stay listed is your decision; the script prints a reminder.

It then regenerates the derived trees: the Claude Code plugin package, the behavior-diff goldens, and the docs reference and changelog pages. Pass --dry-run to list the changes first, --date YYYY-MM-DD to set the release date (default: today in UTC), or --no-regen to skip the regeneration. The script refuses a malformed version or a downgrade, and re-running it with the current version only re-aligns stray anchors.

Review the diff, then commit:

git add -A
git commit -S -m "chore(release): cut vX.Y.Z"
git push origin main

The -S flag signs the commit with the GPG key. The push triggers the standard CI matrix (lint, test, build); wait for green before tagging.

The readiness check in §2.4 also needs CodeQL and Conformity green on the release commit. Both run only when their own paths change, and a release commit usually changes none of them, so start them once the release commit is the head of main, then confirm each run names that commit:

gh workflow run codeql.yml --ref main
gh workflow run conformity.yml --ref main
gh run list --workflow codeql.yml --commit "$(git rev-parse HEAD)" --limit 1
gh run list --workflow conformity.yml --commit "$(git rev-parse HEAD)" --limit 1

Wait for both to succeed before tagging, and push nothing else to main until the release is out: a later push moves main away from the release commit, and CI cancels a run still in progress on the same branch.

2.2 Sign and push the tagCopy Anchor Link

git tag -a -s vX.Y.Z -m "vX.Y.Z"
git push origin vX.Y.Z

The -a -s combination produces an annotated, GPG-signed tag. The push triggers .github/workflows/release.yml, which builds, signs, attests, and publishes.

2.3 Confirm the release workflow completedCopy Anchor Link

gh run watch $(gh run list --workflow=release.yml --limit=1 --json databaseId --jq '.[0].databaseId')

The workflow runs build (sdist + wheel, then a CycloneDX SBOM of those distributions written beside them), keyless cosign signing, SLSA-3 provenance generation and verification, platform archive builds (darwin / linux / windows), archive signing with an aggregated SHA256SUMS manifest, and the final GitHub Release publish. The maintainer scans the publish GitHub Release job log for the asset-upload confirmations.

The publish GitHub Release job runs in the release environment, which requires the maintainer's approval, so the run waits there once the earlier jobs pass. Approve it on the run's page (Review deployments), or from the shell:

run_id=$(gh run list --workflow=release.yml --limit=1 --json databaseId --jq '.[0].databaseId')
env_id=$(gh api repos/ahmed-g-gad/apothem/environments/release --jq .id)
gh api -X POST "repos/ahmed-g-gad/apothem/actions/runs/${run_id}/pending_deployments" \
  -F "environment_ids[]=${env_id}" -f state=approved -f comment="vX.Y.Z"

The tag push also starts the npm and VS Code extension workflows. Their publish jobs wait for the same approval until §2.4.

2.4 Confirm the npm publishCopy Anchor Link

The tag push also runs publish-npm.yml beside release.yml. The npm workflow does not wait for the release workflow; the release environment approval is the gate. Once the release workflow from §2.3 has finished, confirm readiness. The check reads releases, workflow runs and the published SBOM, so set up two inputs first:

• GH_TOKEN. The script refuses to run without it, even when gh is already logged in. Use a fine-grained personal access token limited to ahmed-g-gad/apothem, with Actions: Read (workflow runs) and Contents: Read (releases). The check writes nothing, so no write permission is needed. A token from gh auth token carries every scope of the login, so prefer the fine-grained token.
• The published SBOM. The check reads dist/release-assets/sbom.cdx.json, the CycloneDX file the release attaches. Download the copy the release published rather than a local build.

read -rs GH_TOKEN && export GH_TOKEN
mkdir -p dist/release-assets
gh release download vX.Y.Z --repo ahmed-g-gad/apothem \
  --pattern sbom.cdx.json --dir dist/release-assets --clobber
bash scripts/release/check-release-ready.sh vX.Y.Z

The check fails when the release is a draft or a prerelease, when a required workflow for the tag's commit is not green, when the SBOM is missing, when it describes another version, or when it omits a vendored package. The failure message names the cause.

When the check passes, approve the waiting Publish npm run the same way as in §2.3 (its run ID comes from gh run list --workflow=publish-npm.yml). The workflow publishes @ahmed-g-gad/[email protected] to npmjs.com through npm trusted publishing (OIDC); no registry token is stored in the repository. Approve the waiting Publish VS Code Extension run as well. It packages apothem.vsix and publishes it to the Visual Studio Marketplace as ahmed-g-gad.apothem-vscode. The publish step runs only while the repository variable VSCODE_MARKETPLACE_READY is true and one credential is configured: the variable VSCE_AUTH=oidc (Marketplace Trusted Publishing, no stored secret), the variables VSCE_AUTH=entra, AZURE_CLIENT_ID and AZURE_TENANT_ID (a short-lived Microsoft Entra ID token, no stored secret), or the VSCE_PAT secret. With none, the run packages the extension and skips publication. The credential is a switch, not a retry: a failed publish on one route does not fall back to another by itself. The setup for each route is in Marketplace publishing with Trusted Publishing (preferred) and Marketplace publishing with Microsoft Entra ID below.

To publish again after a failed run, dispatch the workflow for the same tag (gh workflow run publish-npm.yml --field tag=vX.Y.Z). It skips publication when the version is already on the registry, because npm versions are immutable.

Marketplace publishing with Trusted PublishingCopy Anchor Link

Trusted Publishing lets the Marketplace accept this repository's GitHub OIDC token directly: vsce publish --oidc exchanges the publish job's token for a short-lived Marketplace credential. There is no stored secret and no Azure resource, so this is the preferred route. The Entra route below and the VSCE_PAT secret stay as alternatives that the operator selects.

1. Use a vsce release that supports it. On this route the publish job checks the vsce release locked in tools/vsce before it publishes, and stops with an error that names the minimum release when the lock is older. Dependabot proposes vsce updates for that lockfile; merge the one that reaches the minimum before you switch.

2. Create the policy. Sign in as an Owner of the publisher, open https://marketplace.visualstudio.com/manage/publishers/ahmed-g-gad, and add a policy under Trusted Publishing:

   • Policy name: any name, such as apothem-release
   • Repository owner: Ahmed-G-Gad
   • Repository: apothem
   • Workflow: publish-vscode.yml
   • Environment: release

   The field names are the ones the VS Code publishing guide lists, in its section on publishing with GitHub Actions and Trusted Publishing (https://code.visualstudio.com/api/working-with-extensions/publishing-extension). Unconfirmed: whether every publisher sees the Trusted Publishing tab yet. If yours does not, use the Entra route.

3. Switch the route. Keep VSCODE_MARKETPLACE_READY set to true and set the selector. Set it when no tag is waiting, because the next tag push publishes through this route.

   Copy Text

   gh variable set VSCE_AUTH --body oidc

4. Publish. The next tag push, or a dispatch for an existing tag, publishes through Trusted Publishing. This route has no dry run: the first publish is the check, and a dispatch with dry_run only warns that nothing was verified.

5. Fall back if it fails. Delete the selector and dispatch again for the same tag; with the VSCE_PAT secret set, the token route publishes. A failed publish published nothing, so the same tag can be dispatched again.

   Copy Text

   gh variable delete VSCE_AUTH

Marketplace publishing with Microsoft Entra IDCopy Anchor Link

The Entra route replaces the stored personal access token with a short-lived token. publish-vscode.yml asks GitHub for an OIDC token, the Azure/login action exchanges it for an Entra sign-in through a federated credential, and vsce publish --azure-credential publishes with the resulting token. No client secret exists. The VSCE_PAT secret stays as a fallback that the operator selects; the workflow never switches to it on its own.

The vsce release pinned in tools/vsce supports this route. It needs an Entra tenant, an application registration and publisher membership, which Trusted Publishing above does not; use it when Trusted Publishing is unavailable to the publisher.

1. Check the preconditions. Find the account that owns the publisher ahmed-g-gad (the Members tab at https://marketplace.visualstudio.com/manage/publishers/ahmed-g-gad) and the Entra tenant where you can register applications. Unconfirmed: whether the publisher owner's tenant must match the application's tenant, and whether the Azure DevOps organization Ahmed-G-Gad must be connected to Entra (dev.azure.com/Ahmed-G-Gad, Organization settings, Microsoft Entra). No source states either. If step 5 or 6 fails, check these first.

2. Register the application. In https://entra.microsoft.com, open App registrations, then New registration. Name it apothem-vscode-publish, choose a single tenant, and leave the redirect URI empty. From Overview, copy the Application (client) ID and the Directory (tenant) ID. Create no client secret and grant no API permission. The step needs application-owner rights or an Entra role such as Application Administrator.

3. Add the federated credential. On that application, open Certificates & secrets, Federated credentials, Add credential, and choose the scenario "GitHub Actions deploying Azure resources". Enter organization Ahmed-G-Gad, repository apothem, entity type Environment, environment name release, and credential name apothem-release. The generated fields must read:

   Copy Text

   issuer:   https://token.actions.githubusercontent.com
   subject:  repo:Ahmed-G-Gad/apothem:environment:release
   audience: api://AzureADTokenExchange

   Matching is exact and case-sensitive. If a later sign-in fails with AADSTS700213 or AADSTS7002138, compare the credential with the subject claim line that Azure/login writes to the job log. AADSTS70021 right after creation means the credential has not propagated yet; wait a few minutes and retry.

4. Set the repository variables. While VSCODE_MARKETPLACE_READY is true, the Entra route is active as soon as all three are set, so set them when no release tag is waiting for approval. Without VSCODE_MARKETPLACE_READY=true nothing signs in and the run skips the publish.

   Copy Text

   gh variable set AZURE_CLIENT_ID --body "<application-client-id>"
   gh variable set AZURE_TENANT_ID --body "<directory-tenant-id>"
   gh variable set VSCE_AUTH --body entra

   Leave VSCODE_MARKETPLACE_READY and VSCE_PAT as they are. When VSCE_AUTH is entra but AZURE_CLIENT_ID or AZURE_TENANT_ID is empty, the workflow logs a warning, whether or not VSCE_PAT is set, and does not use the Entra route. It then uses the VSCE_PAT secret if that is set; with no secret it publishes nothing and the run still ends green, so look for the warning.

5. Run the dry run. Dispatch the workflow for an existing release tag with dry_run set. On the Entra route it signs in, prints the Azure DevOps profile, and verifies publisher access without publishing. The token route has no dry run: if the log shows Marketplace credential mode: pat, the run ends green with a warning and has verified nothing. Approve the waiting release environment run.

   Copy Text

   gh workflow run publish-vscode.yml --ref main -f tag=vX.Y.Z -f dry_run=true

   Expected: Marketplace credential mode: entra, a subject claim line equal to the subject in step 3, and the profile id and displayName printed by the first dry-run step. The verify step is expected to fail with a 403 until step 6. Unconfirmed: that a sign-in without an Azure subscription yields a token for the Marketplace through the Azure CLI. This run is the first test of it. If it fails here, register a user-assigned managed identity in a subscription instead, which is the route Microsoft documents.

6. Add the application to the publisher. In the Members tab from step 1, sign in as a publisher Owner and add a member using the profile id from step 5, with the role Contributor. Unconfirmed: the dialog's field label, and whether it accepts the id before the application is a member of an Azure DevOps organization. Azure DevOps adds application identities per organization as a general rule. If the dialog rejects the id, add the service principal to the organization Ahmed-G-Gad (Organization settings, Users, using the Enterprise application Object ID) and retry. Also unconfirmed: whether the profile exists before the identity's first Azure DevOps call.

7. Repeat the dry run. The verify step now succeeds with "verification succeeded for the publisher"; its messages still say Personal Access Token, because the Entra token travels the same path. The check proves publisher membership (Creator, Owner, Contributor or Reader), not publishing rights. Unconfirmed: that Contributor is enough to publish an update. Microsoft's role description says it is, and no source shows a publish with it, so the first real publish is the proof.

8. Publish a release. Tag as in §2.2 and approve the run. The step Publish to Visual Studio Marketplace (Entra ID) runs, its environment header lists only AZURE_TENANT_ID, the token fallback step is skipped, and vsce prints DONE Published ahmed-g-gad.apothem-vscode. The Marketplace gallery lists the new version after about four minutes.

9. Decide on the token. Keep VSCE_PAT until the first successful Entra publish, then delete the secret if no fallback is wanted. Once it is gone, a deleted or empty Entra variable leaves the run green with no publish: with a warning when VSCE_AUTH is still entra, and with only the skip notice when it is not. Read the run's annotations after each tag. To keep a fallback past the retirement of global personal access tokens, create an organization-scoped token for Ahmed-G-Gad with the custom scope Marketplace Manage and the shortest expiry you accept, and store it as VSCE_PAT. Unconfirmed: the portal labels in this step are from memory, not from a primary source; Microsoft calls organization-scoped tokens a temporary workaround and can stop honoring them; and Microsoft's pages disagree on the retirement date of global tokens.

10. Fall back to the token. Delete the selector and dispatch again, then approve the run. A failed Entra publish published nothing, so the same tag can be dispatched again; check the Marketplace page first, because a version that landed cannot be published twice.

    Copy Text

    gh variable delete VSCE_AUTH
    gh workflow run publish-vscode.yml --ref main -f tag=vX.Y.Z

When the Entra publish fails, read the step log:

• A message that begins "Can not acquire a Microsoft Entra ID access token" means the sign-in or the credential chain failed. Check the Sign in to Microsoft Entra ID step and the federated credential first. If the sign-in is green, remove AZURE_TENANT_ID from the vsce step's env in the workflow; GitHub's own extension release workflow does not export it.
• "InvalidAccessException" or "not allowed" means the token's identity is not a publisher member. Compare the profile id from the dry run with the Members tab.
• A 403 from the dry-run verify step means the application has no role yet.

Three limits apply to this route:

• id-token: write is granted to the whole publish job, so every step in it can request an OIDC token, in token mode too. The token's subject is bound to the release environment, which requires a reviewer's approval.
• The release environment has no deployment branch or tag policy, so any ref a maintainer approves can obtain the environment-subject token. Restricting it to v*.*.* tags needs a check of what release.yml and publish-npm.yml dispatch first.
• A repository rename or transfer changes the OIDC subject format and silently breaks the federated credential. Copy the subject from the Azure/login log into the credential again after any rename.

2.5 Refresh the template-digest pinCopy Anchor Link

After the tag exists, set _LATEST_RELEASE and _LATEST_RELEASE_DIGESTS in tests/unit/test_template_digests.py to the new release and the SHA-256 digest of each plain-text template as that release shipped it (git show vX.Y.Z:<template path> | sha256sum). Do it in the next change to main, and add the previous digest to the manifest entry's earlier_digests when the template changed. The release-preflight CI job checks out every tag, so it fails with the tag names until the pin is refreshed.

3. Distribution SurfacesCopy Anchor Link

Each surface has a verification command answering "did the release land here?":

The GitHub Release is the canonical artifact surface; the npm package and the script installers are the install paths layered on top of it. The npm publish runs beside the release workflow on the same tag push.

4. CHANGELOG disciplineCopy Anchor Link

The CHANGELOG.md carries one section per release under the Keep-a-Changelog convention. The release commit's CHANGELOG edit:

• Confirms the version section uses the ## [X.Y.Z] - YYYY-MM-DD heading (a hyphen; bump_version.py writes it, and extract_release_notes.py matches the ## [X.Y.Z] prefix).
• Confirms the standard headings — Added, Changed, Deprecated, Removed, Fixed, Security — are used where applicable. Empty headings are dropped from the rendered file.
• Captures capabilities, not implementation. Domain language only; no plan-internal references; no commit hashes; no plan or phase identifiers.

A CHANGELOG section that survived the maintainer's review reads as the release-notes anchor for the announcement narrative.

5. Install-path smokeCopy Anchor Link

After both stages complete, the maintainer runs the smoke. The smoke is the canonical verification that the user-facing install commands answer.

5.1 Claude Code pluginCopy Anchor Link

/plugin marketplace add ahmed-g-gad/apothem
/plugin install apothem@apothem

Expected: the plugin installs and apothem commands are available inside Claude Code.

5.2 Node.js (npx)Copy Anchor Link

npx @ahmed-g-gad/[email protected] --version

Expected output: Apothem, version X.Y.Z.

5.3 One-shot script installer (POSIX)Copy Anchor Link

The installer writes the apothem launcher to ~/.local/bin, or to the directory named by APOTHEM_BIN_DIR when that variable is set. The check below needs that directory on PATH. When it is not, the installer prints the export PATH line to add. Until that directory is on PATH, the shell cannot find apothem, and the check fails. If the installer cannot place the launcher, it prints a warning and no export PATH line. On a host that does not hold the maintainer signing key yet, the first line imports it; the installer stops without it.

curl -fsSL https://github.com/ahmed-g-gad.gpg | gpg --import
curl -fsSL https://apothem.ahmedgad.com/install.sh | sh
apothem --version

Expected output: Apothem, version X.Y.Z.

5.4 One-shot script installer (Windows)Copy Anchor Link

The first line imports the maintainer signing key through Git's own bash, whose gpg keyring is the one git verify-tag reads.

& "$env:ProgramFiles\Git\bin\bash.exe" -c 'curl -fsSL https://github.com/ahmed-g-gad.gpg | gpg --import'
irm https://apothem.ahmedgad.com/install.ps1 | iex
apothem --version

Expected output: Apothem, version X.Y.Z.

5.5 Artifact verification (supply-chain evidence)Copy Anchor Link

Download every asset into an empty directory, then verify. The certificate identity is the release workflow at this exact tag. Identity checks are case-sensitive, and the certificates carry the owner as GitHub reports it, Ahmed-G-Gad, so a lowercase owner fails.

mkdir release-vX.Y.Z && cd release-vX.Y.Z
gh release download vX.Y.Z --repo ahmed-g-gad/apothem
cosign verify-blob \
  --bundle apothem-X.Y.Z-py3-none-any.whl.cosign.bundle \
  --certificate-identity https://github.com/Ahmed-G-Gad/apothem/.github/workflows/release.yml@refs/tags/vX.Y.Z \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  apothem-X.Y.Z-py3-none-any.whl
slsa-verifier verify-artifact \
  --provenance-path provenance.intoto.jsonl \
  --source-uri github.com/Ahmed-G-Gad/apothem \
  --source-tag vX.Y.Z \
  apothem-X.Y.Z-py3-none-any.whl apothem-X.Y.Z.tar.gz
bash ../scripts/release/verify-provenance.sh .

Expected: Verified OK, PASSED: SLSA verification passed, and verify-provenance: chain OK (3 of 3 layers verified). The script checks SHA256SUMS, every signature bundle, and the provenance in one run; it derives the tag from the wheel name (--tag vX.Y.Z overrides it).

The smoke produces a one-line PASS / FAIL row per surface; record the rows in the maintainer's release log.

6. Failure recoveryCopy Anchor Link

The release workflow has documented failure modes for the release engine and the npm publisher:

If a recovery cycle fails to land a surface within one diagnostic pass, the surface is logged as a known-failure in the release notes with a link to the open issue; downstream users are pointed to the working surfaces while the broken one is fixed in a patch release.

7. Cross-referencesCopy Anchor Link

• Publisher setup. The publisher-account-setup runbook establishes the npm trusted-publisher binding and the GitHub-side prerequisites this runbook assumes.
• Recovery flow. The release-recovery runbook follows the same additive model: it recovers a failed tag or publisher stage by stage and rolls back by releasing the fix as a new patch version, never by deleting or re-pointing a published tag, release or npm version.
• Pages enablement. The publish-static-site.yml workflow validates and deploys the project website to the custom domain; the Pages-enablement runbook (site/content/docs/runbooks/pages-enablement.mdx) is the upstream procedure that established the custom domain in the first place.
• Versioning policy. site/content/docs/governance/release-engineering-policy.mdx carries the SemVer + signing + deprecation policy this runbook honors.