Publishing packages: defer pushing tags until lerna publish succeeds#78253
Conversation
|
Size Change: 0 B Total Size: 7.97 MB ℹ️ View Unchanged
|
14a9097 to
38461cb
Compare
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @demo@local, @Copilot. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
Since both creating the git tags and publishing the npm packages can independently fail, a natural question I have for the direction proposed here is: What happens if the npm publish succeeds and then the git tagging fails? That leaves us with a published npm version and without the git tag, right? Maybe this is still an improvement if it's more likely for one to fail than the other that it makes sense to flip the behavior. But to me it feels like if either one fails, we should roll back to undo the other. And since (if my reading is correct) we cannot unpublish new versions of the npm package but we can delete git tags, it seems like we'd want a procedure like: gitTag();
try {
npmPublish();
} catch {
deleteGitTag();
} |
Lerna's default `lerna version` pushes the per-package git tags to origin before `lerna publish` runs. If publish fails afterwards (npm 503, OTP/auth blip, runner interrupted), the tags remain on origin and the next retry crashes with `tag '@wordpress/<pkg>@<version>' already exists`. Pass `--no-push` to `lerna version` so the version commit and tags stay local until `lerna publish from-package` reports success, then push the branch and tags together with `--follow-tags`. A failure between version and publish now leaves nothing on origin, so retries can re-version cleanly. Only the `latest` flow is changed. `next` tags include a unique timestamp and `bugfix`/`wp` use `lerna publish <bump>` which doesn't push tags until publish succeeds.
Wrap the final `git push --follow-tags` in a 3-attempt retry with linear backoff. Shrinks the window where a successful npm publish could be followed by a failed push, leaving npm-published versions without their tags on origin. The push is idempotent: tags already present on origin at the same SHA are a no-op, so retrying is safe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
You're right @aduth, the asymmetric window exists, but I think the current seems more reliable than
To reduce the chance of that happening for git tags, I added a retry attempt, so it should increase the success rate of that command. I'm curious to hear your thoughts on this new proposal. |
mikachan
left a comment
There was a problem hiding this comment.
Thanks for working on this! Deferring the tags until the publish action succeeds makes sense to me.
the asymmetric window exists, but I think the current seems more reliable than
tag -> try publish -> catch deleteTag
It would be good to hear what @aduth thinks, but this reasoning sounds good to me.
|
Yeah, on second thought, it seems like it's much less likely for the git tagging to fail and much easier to recover from. The built-in retry as proposed could further help reduce the likelihood of this failing. This direction seems like it would be an improvement 👍 |
Co-authored-by: Sarah Norris <1645628+mikachan@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
…-publish Co-authored-by: paulopmt1 <1044309+paulopmt1@users.noreply.github.com>
…ry log Agent-Logs-Url: https://github.com/WordPress/gutenberg/sessions/c587d776-0727-4f03-a254-ddff459d64be Co-authored-by: paulopmt1 <1044309+paulopmt1@users.noreply.github.com>
Resolved in 3db81ff. The trunk had migrated |
|
Flaky tests detected in 63d3db4. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/26057776719
|
Break `err.message` onto its own line and inline `attempt * 5` to satisfy prettier after the err.message addition in 53fdde7. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
I was fixing a similar mess few months ago in #74413. Back then the failure was:
At that time I didn't yet know that the changes made in step 1 are pushed to The issue being fixed here is a bit different, here the step 2 failed. One downside of this PR (push to On the other hand, it's always a tradeoff, as has been discussed. There's no perfect solution to this distributed network operation 🙂 |
Slack thread
What
Add
--no-pushto thelerna versioncall in thelatestflow ofbin/plugin/commands/packages.js, and push the version commit + package tags only afterlerna publish from-packagereports success. The post-publish push is wrapped in a small retry loop.Why
Lerna's default
lerna versioncreates the package tags and pushes them to origin beforelerna publishruns. If publish fails afterwards (npm 503, OTP/auth blip, runner interrupted), the tags remain on origin and the next retry of the workflow crashes here:This is currently blocking the
23.2.0-rc.1release. Attempt 1 of the Build Gutenberg Plugin Zip workflow (run 25801224562) created and pushed the@wordpress/*@4.46.0tags but failed before publishing to npm; attempt 2 hit the duplicate-tag error; attempt 3 is queued and would fail the same way. On npm,@wordpress/a11y@latestis still4.45.0— so4.46.0is tagged in git but unpublished.With
--no-push, the version commit and tags stay local untillerna publish from-packagesucceeds. A failure between version and publish leaves nothing on origin, so the next retry can re-version cleanly.Failure modes
Splitting
versionandpushintroduces two narrow failure windows. Both are now handled or recoverable:lerna version(the original bug): local commit + tags exist on the runner, but nothing was pushed to origin. The runner is ephemeral, so the local state dies with it. A workflow retry re-runslerna versioncleanly because origin is untouched. ✅lerna publish from-packageis a no-op for versions already on npm, and the push re-attempts and succeeds (push is idempotent — tags already on origin at the same SHA are a no-op). No manual tag deletion required for this scenario, unlike the original bug.We considered an explicit
try { publish() } catch { deleteTags() }rollback. Rejected because:deleteTags()has the same network/auth failure modes as the push, so the rollback can itself fail — relocating the risk rather than removing it.lerna publish from-packageover ~100 packages).One-time cleanup of the currently stuck state
This PR does not auto-recover the existing dangling tags on
wp/latestfrom the pre-change failure. Before the next23.2.0-rc.1rerun, the release manager needs to delete those tags from origin:After that one-time deletion, re-run the Build Gutenberg Plugin Zip workflow. Going forward, no failure can leave stale tags on origin, so retries are clean by construction.
Scope
Only the
latestflow is changed. Thenext,bugfix, andwpflows are untouched:nexttags include a unique timestamp (next.v.<ts>), so retries never collide.bugfix/wpuselerna publish <bump>(version + publish in one call), which doesn't push tags until publish succeeds.Test plan
@wordpress/*@4.46.0tags on origin, then re-run the stuck Build Gutenberg Plugin Zip workflow and confirm it publishes@wordpress/*@4.46.0to npm with thelatestdist-tag and pushes the version commit + tags.--follow-tagsand that simulating a publish failure (e.g. with bad NPM_TOKEN locally) leaves no tags on origin.