Skip to content

Docker Hub recovery runbook

Steps to recover from a release the pipeline left incomplete, or whose two registries disagree. Most often that is a half-published one where docker.io accepted a per-arch push but ghcr.io failed, leaving an orphan arch-suffixed tag that carries no attestation or signature — every attest and sign step runs after the push loop, so the failing arch's never ran — while the sibling arch's job may have completed fully before the failure. A naive retry of release-on-bump.yml would silently overwrite the orphan.

A post-publication tag rewrite lands here too — see the third path under When this applies.

This runbook is referenced from the auto-filed release-on-bump-failure issue body, and from verify-latest-release.yml's cross-registry-manifest-mismatch arm. The release-on-bump-failure body intentionally links here rather than embedding the snippets inline, so:

  • The snippets stay correct (JSON construction via jq -n is quote-safe, unlike shell-interpolated JSON).
  • The procedure is one file to update if Docker Hub's API contract changes, not N copies across the workflow.

When this applies

The push loop inside release-on-bump.yml per-arch jobs runs docker.io first, then ghcr.io. Its two failure modes are:

  • docker.io push failed. Nothing was pushed to either registry for that arch, so that arch needs no registry-side cleanup; on the retrigger-PR path only, the GitHub release needs deleting before the retry (the tag always stays — see step 1).
  • ghcr.io push failed after docker.io succeeded. docker.io has the arch-suffixed tag ({version}-{arch}) but ghcr.io does not. The manifest job is skipped, so neither registry has the unsuffixed {version} or :latest tags. Delete the orphan docker.io arch tag before retrying: the push itself would succeed and silently overwrite it. The orphan carries no attestation or signature (those steps run after the push and never ran), so deleting it loses nothing.

A third path arrives here from elsewhere: verify-latest-release.yml files its verify-latest-release-failure issue with Reason: cross-registry-manifest-mismatch and points it at this runbook. That reason covers any failure of the verify ghcr.io and docker.io serve the same manifest step, and its log says which. A log that prints ghcr.io and docker.io disagree on the tag, followed by a non-empty manifest for each registry, shows the two registries disagreeing, and the issue body says to treat the Docker Hub token as compromised until proven otherwise. A log that stops at a fetch error or a jq parse error, or that prints an empty manifest for a registry, compared nothing: dispatch the workflow again, and this path applies only if a run prints a mismatch. A cross-registry mismatch is a post-publication tag rewrite, not a half-finished push, and the steps below assume the opposite — they spend DOCKERHUB_TOKEN_DELETE on the working assumption that the credential is sound. Take these first instead:

  1. Revoke DOCKERHUB_TOKEN_RW and DOCKERHUB_TOKEN_DELETE per the rotation guidance in SECURITY.md. No republish below runs until they are reissued.
  2. Capture both registries' current manifests before changing anything — docker buildx imagetools inspect --raw against each, as verify-latest-release.yml does to compare the registries — so the rewritten bytes are recoverable evidence rather than something the republish overwrites. Both repositories are public, so this step needs no credential and does not wait on the reissue.
  3. Only then apply the republish path below, with reissued credentials. Take it at Step-by-step step 3 and use the force-republish dispatch: steps 1 and 2 of that section are scoped to the half-published case. The Prerequisites below assume a release-on-bump-failure issue; on this path VERSION comes from gh release view --json tagName instead, and there is no single ARCH.

Prerequisites

  • A Docker Hub access token with Delete scope. Use the DOCKERHUB_TOKEN_DELETE repository secret value (the _RW token returns 403 on the tag-delete endpoint). Fetch it from your local secret vault — do not print gh secret values inside the auto-filed issue body. If no copy survives, see No local copy of the token below.
  • DOCKERHUB_USERNAME matching the repo owner (rvenutolo).
  • A failing run identified in the release-on-bump-failure issue — record VERSION (the pin version, e.g. 20260516-deadbee) and ARCH (amd64 or arm64) before starting.

No local copy of the token

Docker Hub shows an access token once at creation and never re-displays it, so a lost copy cannot be recovered — it can only be replaced. Create a fresh Delete-scoped token at https://hub.docker.com/settings/security, resync the repo secret, confirm the recovery commands below authenticate, and only then delete the token you replaced — the old one is the rollback target until the new one is proven, and dockerhub-sync.yml consumes this secret on every release.

gh secret set DOCKERHUB_TOKEN_DELETE --repo rvenutolo/linPEAS-flake

Under suspected compromise the order inverts — revoke first, per SECURITY.md — but a lost local copy is not a compromise.

Step-by-step

1. Delete the GitHub release (retrigger-PR path only)

Skip this step if you will re-run with force-republish (step 3, first bullet). On the force-republish path the image-amd64, image-arm64, manifest, changelog, and verify jobs re-run against the current pin while the release job's release-creation step stays gated by the "tag exists" guard, so the release and its assets survive and nothing needs deleting. The release job itself still runs. Its sign pin file (cosign sign-blob, keyless) step is ungated and re-uploads linpeas-pin.json.sigstore with --clobber, which is how this path recovers a missing sidecar.

Delete only when recovering by landing a retrigger PR (step 3, second bullet). Release-creation is gated on the release not existing (a gh release view probe, surfaced as the tag-exists output), so an orphan release left in place makes the re-push skip release creation, and skips image-amd64, image-arm64, manifest, changelog and verify as well (the same five jobs force-republish re-runs), leaving the recovery incomplete.

VERSION="<pin-version>"            # e.g. 20260516-deadbee
gh release delete "${VERSION}" \
  --repo rvenutolo/linPEAS-flake \
  --yes

Do not add --cleanup-tag: the release-tag-protection ruleset blocks tag deletion with no bypass actors, so the tag delete errors and the command fails. The tag survives, still pointing at the original pin commit, and release creation reuses it — an existing tag's commit wins over the step's --target pin, so the re-created release attaches to that original commit while the retry's attestations record the retrigger SHA.

2. Delete the orphan docker.io arch tag

Only required if ghcr.io failed after docker.io succeeded. Skip to step 3 otherwise.

The simplest path is the Docker Hub web UI: navigate to rvenutolo/linpeas tags, find {VERSION}-{ARCH}, click "Delete".

For automation / scripted recovery, use the API:

DOCKERHUB_USERNAME=rvenutolo
DOCKERHUB_TOKEN_DELETE="<paste DOCKERHUB_TOKEN_DELETE value>"
VERSION="<pin-version>"
ARCH="<amd64 or arm64>"

# Build a quote-safe JSON body via jq -n --arg. Shell-interpolated
# JSON construction breaks when the password contains a single quote
# or backslash; `jq -n` guarantees correct escaping.
TOKEN=$(curl --fail --silent --show-error \
    --request POST 'https://hub.docker.com/v2/users/login' \
    --header 'Content-Type: application/json' \
    --data "$(jq --null-input \
        --arg u "${DOCKERHUB_USERNAME}" \
        --arg p "${DOCKERHUB_TOKEN_DELETE}" \
        '{username:$u, password:$p}')" \
  | jq --raw-output .token)

curl --fail --silent --show-error \
  --request DELETE \
  --header "Authorization: JWT ${TOKEN}" \
  "https://hub.docker.com/v2/repositories/${DOCKERHUB_USERNAME}/linpeas/tags/${VERSION}-${ARCH}/"

3. Re-trigger the release pipeline

Either:

  • Dispatch the workflow from the Actions UI — the Run workflow button, i.e. workflow_dispatch — if the pin commit is still HEAD on main, setting force-republish: true so the image, manifest, verify, and changelog jobs run for the current pin even though the GitHub release already exists. Re-running the original run is not a substitute: a re-run replays that run's inputs and cannot set a new one, so on a push-triggered run force-republish stays false. Under "Re-run all jobs" the release job then finds the release, reports tag-exists true, and the image, manifest, verify, and changelog jobs skip — the recovery looks green while doing nothing. The release job's release-creation step stays gated by the "tag exists" guard, so the existing release assets are preserved and step 1 does not apply on this route. This route is faster and leaves no retrigger commit behind, OR

  • Open a PR that touches linpeas-pin.json itself with a prettier-stable edit (one blank line between two properties survives treefmt; other whitespace changes are reverted and fail the flake-check formatting gate) and a chore: retrigger release subject. main accepts no direct push — the protect-main ruleset has no bypass actors — so the commit lands only after the required checks pass. The workflow's push trigger filters on that path, so an empty commit or an edit to any other file starts nothing.

4. Confirm green end-to-end

Watch the next release-on-bump run finish green. Specifically confirm:

  • release published a new release (or kept the existing one with force-republish).
  • preflight emitted image-mode=full. On the recovery paths above it performs no registry probe — the state classification only runs in backfill mode, where backfill-tag is non-empty.
  • image-amd64 and image-arm64 both pushed clean to docker.io and ghcr.io.
  • manifest built and pushed.
  • changelog re-ran, which it does on the force-republish path used above.
  • verify passed per-arch attestation re-verify, manifest re-resolution, and :latest matches :VERSION per-arch digests.

Those are the seven jobs the workflow's own failure issue enumerates (the two per-arch image jobs share a bullet above), so a green run means all seven, not just the ones Docker Hub touches.

Close the release-on-bump-failure issue with a one-line root-cause comment (e.g., transient: docker.io 502 on push, retry green). On the third path there is no such issue: close the verify-latest-release-failure one instead, and only after a fresh verify-latest-release run passes.

Common Docker Hub failure modes

Token expired or revoked. Rotate DOCKERHUB_TOKEN_RW (and/or DOCKERHUB_TOKEN_DELETE independently) at https://hub.docker.com/settings/security, then gh secret set DOCKERHUB_TOKEN_RW.

Username mismatch. DOCKERHUB_USERNAME must equal the owner segment of the rvenutolo/linpeas repo path.

Docker Hub partial outage. Re-trigger as step 3 describes, after any cleanup steps 1 and 2 call for. Once the GitHub release exists, a workflow_dispatch with default inputs skips the image jobs. If multiple retries fail with the same shape, check https://status.docker.com.

DOCKERHUB_TOKEN split (RW + DELETE)

  • DOCKERHUB_TOKEN_RW — Read, Write. Used by release-on-bump.yml.
  • DOCKERHUB_TOKEN_DELETE — Read, Write, Delete. Used by dockerhub-sync.yml — the only workflow consumer (peter-evans/dockerhub-description requires Delete scope to PATCH repo metadata; Read, Write-only PAT returns 403) — and by the manual recovery snippets in this runbook.

Binding (in workflows, each secret reference is matched with the ASCII letters of secrets and of the name in either case, as GitHub Actions reads a secret's name whatever its case, in the index form secrets['NAME'] as in the dot form secrets.NAME, and in the decoded text of a double-quoted scalar, tagged or not, where an escape can hide it from the file text):

  1. secrets.DOCKERHUB_TOKEN_RW must never be consumed in dockerhub-sync.yml or the anonymous/read-only verify-latest-release.yml (the write-scoped token is release-only).
  2. secrets.DOCKERHUB_TOKEN_DELETE must never be consumed in release-on-bump.yml or verify-latest-release.yml.
  3. Any Docker Hub tag delete (--request DELETE or -X DELETE) in a fence the lint reads, in a tracked Markdown file outside tests/ — including this runbook's own step-2 recovery snippet — must name DOCKERHUB_TOKEN_DELETE and must not name DOCKERHUB_TOKEN_RW (the _RW token returns 403). Which fences the lint reads is stated in the header of scripts/check-dockerhub-token-scope-split.sh. It counts a fence as a Docker Hub delete when it performs a DELETE and either addresses hub.docker.com or names a DOCKERHUB_TOKEN variant.
  4. No unsuffixed DOCKERHUB_TOKEN secret may exist; only _RW and _DELETE variants are authoritative.
  5. No workflow may use the secrets context without a literal name (toJSON(secrets), secrets[format(...)], secrets[env.NAME], secrets.*, a double-quoted index, the word secrets inside a string literal), leave an expression unclosed on its line, or give a job secrets: inherit: no name in the text says which token such a use reads or hands on.
  6. release-on-bump.yml must consume secrets.DOCKERHUB_TOKEN_RW and dockerhub-sync.yml must consume secrets.DOCKERHUB_TOKEN_DELETE — the lint fails closed if either producer is removed, so a deletion is a reviewed edit rather than a silent hole that lets the release-on-bump.yml and dockerhub-sync.yml arms of items 1 and 2 above pass trivially. The verify-latest-release.yml arms of those items have no producer guard: that file's absence reads as "does not consume" and passes.

Read the two failure codes apart before rotating anything. Docker Hub answers 401 when it got no usable credential at all, and 403 with {"message":"access denied: insufficient scope"} when the credential is valid but its scope does not cover the call. Every 403 this runbook cites is the second shape, and the scope check runs ahead of the lookup — a delete of a tag that does not exist still answers 403, never 404. So a 403 says the token is the wrong one, not that it expired; a 401 says the secret is empty, malformed, or revoked.

That is also where the step 2 snippet fails if the _RW value gets pasted into it. The /v2/users/login exchange succeeds — the token is valid, just not delete-scoped — so the first curl --fail returns a JWT and it is the DELETE that dies. A failure on the login call instead means the token itself is bad, not merely under-scoped.

Rotation: no calendar cadence — on suspected compromise, or when a token has expired or been revoked.

Notify-body parity invariant

Two other copies of this failure-cause list exist and carry the same three causes as this runbook's "Common Docker Hub failure modes" section — each names the token its own job consumes, and this page carries the longer remediation:

  • release-on-bump.yml's notify-failure issue body carries it as a ## Common Docker Hub causes subsection.
  • dockerhub-sync.yml's notify-failure issue body carries the same three bullets as a Common causes: list, phrased against DOCKERHUB_TOKEN_DELETE rather than _RW because that is the token its own job consumes. It adds a sync-specific note about a no-op parent release-on-bump run — not a failure cause, since a no-op parent still yields a green sync — which carries no parity obligation here.

Nothing enforces that three-cause agreement, so an edit here has to reach both workflows by hand.

The third path under When this applies carries a one-directional dependency of the same kind: it restates verify-latest-release.yml's cross-registry-manifest-mismatch guidance, so a reword there leaves this page stale with nothing to notice.