build: use Actions to validate commit message #32417
Conversation
|
Annotations in action:
I tried to treat the commit as filename by creating files with the commit messages in the Action, but GitHub does not allow files that are not committed in the Pull Request. |
| "pattern": [ | ||
| { | ||
| "regexp": "^not ok \\d+ (.*)$", | ||
| "message": 1 |
addaleax
Mar 22, 2020
Member
I’m not entirely clear on whether it works this way or whether the JSON values all have to refer to regexp capture groups, but could we use something like "severity": "warning" (or "severity": "notice"? not sure if that would work either) so that this does not appear like a failed build, given that this is ultimately not essential for PR authors to address?
I’m not entirely clear on whether it works this way or whether the JSON values all have to refer to regexp capture groups, but could we use something like "severity": "warning" (or "severity": "notice"? not sure if that would work either) so that this does not appear like a failed build, given that this is ultimately not essential for PR authors to address?
mmarchini
Mar 22, 2020
Author
Member
The severity only works for annotations, the Check status can only be Success or Failure
The severity only works for annotations, the Check status can only be Success or Failure
addaleax
Mar 22, 2020
Member
That’s unfortunate :/ Any ideas for how else we could make clear that this is non-essential? I’d rather not have explicit failures because of this…
That’s unfortunate :/ Any ideas for how else we could make clear that this is non-essential? I’d rather not have explicit failures because of this…
mmarchini
Mar 22, 2020
Author
Member
We could prefix the workflow name with "[not a blocker]" or hijack the annotation to have this as a message. Unfortunately I don't think there's anything we can do using GitHub's features (the best thing would be for the Action leave a comment, but we can't do that with Actions for PRs coming from forks), and I would rather not have the check on CI than mark as "allowed to fail", because if we do so it'll be misleading for newcomers (the check will always be green).
We could prefix the workflow name with "[not a blocker]" or hijack the annotation to have this as a message. Unfortunately I don't think there's anything we can do using GitHub's features (the best thing would be for the Action leave a comment, but we can't do that with Actions for PRs coming from forks), and I would rather not have the check on CI than mark as "allowed to fail", because if we do so it'll be misleading for newcomers (the check will always be green).
mmarchini
Mar 22, 2020
Author
Member
Switching to warning instead of error on the annotation is probably still worth though. I'll update the PR
Switching to warning instead of error on the annotation is probably still worth though. I'll update the PR
mmarchini
Mar 22, 2020
Author
Member
Hum, reading the Checks API docs again it seems there a "neutral" status.
https://developer.github.com/v3/checks/runs/
I never seen it, and I'm not sure we can set it without directly calling the API (which is not possible in the Action), but it's worth a shot.
(There's also a "Action Required" status, but it's visually worse than "Failure" IMO)
Hum, reading the Checks API docs again it seems there a "neutral" status.
https://developer.github.com/v3/checks/runs/
I never seen it, and I'm not sure we can set it without directly calling the API (which is not possible in the Action), but it's worth a shot.
(There's also a "Action Required" status, but it's visually worse than "Failure" IMO)
mmarchini
Apr 13, 2020
Author
Member
GitHub Actions has no way of setting the "neutral" status on runs without calling the Checks API directly (which needs a write-permission token, which means it wouldn't work on Pull Requests from non-collaborators). So, we can either:
- Have the check fail
- Set it to
continue-on-error
- Don't check the commit message on CI
I'd rather not set it to continue-on-error since any failures will fly under everyone's radar. I'm fine not landing this change though, if folks think failing the CI because the commit messages don't follow our guidelines is unwelcoming. Personally I think the annotations are helpful for folks who want to make sure their commits are following the guidelines. Maybe we could force one annotation with a "not required" message, but I'm not sure how useful and clear that would be.
On the other hand we could also use this check to verify Signed-off-by metadata on commits once #31976 lands, or have another Action doing the same verification. Either way the CI would fail if the commit is not signed, so maybe it's a good thing to introduce this checks now instead of later.
One alternative I've seen in the past is keeping the check "pending" when things like Signed-off-by are missing, but then again, I don't think this is possible with the current Actions permission model.
GitHub Actions has no way of setting the "neutral" status on runs without calling the Checks API directly (which needs a write-permission token, which means it wouldn't work on Pull Requests from non-collaborators). So, we can either:
- Have the check fail
- Set it to
continue-on-error - Don't check the commit message on CI
I'd rather not set it to continue-on-error since any failures will fly under everyone's radar. I'm fine not landing this change though, if folks think failing the CI because the commit messages don't follow our guidelines is unwelcoming. Personally I think the annotations are helpful for folks who want to make sure their commits are following the guidelines. Maybe we could force one annotation with a "not required" message, but I'm not sure how useful and clear that would be.
On the other hand we could also use this check to verify Signed-off-by metadata on commits once #31976 lands, or have another Action doing the same verification. Either way the CI would fail if the commit is not signed, so maybe it's a good thing to introduce this checks now instead of later.
One alternative I've seen in the past is keeping the check "pending" when things like Signed-off-by are missing, but then again, I don't think this is possible with the current Actions permission model.
|
How can we move forward with this? |
|
If we want to move forward as-is (fail the build), I just need to rebase. Otherwise I think we should drop it, since GitHub Actions doesn't allow us to set an intermediate state, and using |
BridgeAR
force-pushed the
nodejs:master
branch
2 times, most recently
from
8ae28ff
to
2935f72
|
We should be able to write a better Action once nodejs/github-bot#272 lands, as it will allow us to set the Check state to |
|
I just tested if it was possible to use
We can also add a summary of guideline violations for when the user clicks the "details" link in the Action. This is probably the best we can do with Actions to lint the commit message without a big red X in the PR. These are the options we have for status:
In the example above I used cc @addaleax since you were interested in this before :) |
|
I'll go ahead and implement the above with "Action Required". With the commit queue I believe it's more relevant to show when the commit message is not following our guidelines (and therefore will prevent us from using the commit queue). This signal the contributor that their PR might take more time to land compared to a PR following the commit guidelines. If Action Requested turns out to be too disruptive or drives contributors away, we can revisit and move to Neutral or Pending. |
Actions interface has a better integration with GitHub, and with Annotations and Problem Matcher we can display all failed checks in a single place, so that users don't have to go through the logs to figure out what's wrong. Since the job on Travis was allowed to fail and is not as easy to read, remove it from our Matrix. The Action will check every commit in the Pull Request, skipping commits with "fixup" or "squash".
mmarchini
force-pushed the
mmarchini:gh-actions-commit-message
branch
from
a875a81
to
46d6b02
|
Revisiting this PR and I'm inclined to move forward with the current implementation. Using cc @nodejs/tsc since this affects the collaborators (especially new collaborators) experience. |
| - name: Use Node.js 12 | ||
| uses: actions/setup-node@v1 | ||
| with: | ||
| node-version: 12.x |
aduh95
Jun 10, 2021
Contributor
Suggested change
- name: Use Node.js 12
uses: actions/setup-node@v1
with:
node-version: 12.x
- name: Use Node.js ${{ env.NODE_VERSION }}
uses: actions/setup-node@v2
with:
node-version: ${{ env.NODE_VERSION }}
| - name: Use Node.js 12 | |
| uses: actions/setup-node@v1 | |
| with: | |
| node-version: 12.x | |
| - name: Use Node.js ${{ env.NODE_VERSION }} | |
| uses: actions/setup-node@v2 | |
| with: | |
| node-version: ${{ env.NODE_VERSION }} |
| # Last 100 commits should be enough for a PR | ||
| fetch-depth: 100 |
aduh95
Jun 10, 2021
Contributor
I'm unsure if this syntax is valid YAML, but you can (and should IMO) use the actual number of commits.
Suggested change
# Last 100 commits should be enough for a PR
fetch-depth: 100
fetch-depth: ${{ github.event.pull_request.commits }} + 1
I'm unsure if this syntax is valid YAML, but you can (and should IMO) use the actual number of commits.
| # Last 100 commits should be enough for a PR | |
| fetch-depth: 100 | |
| fetch-depth: ${{ github.event.pull_request.commits }} + 1 |
| name: "Commit messages adheres to guidelines at https://goo.gl/p2fr5Q" | ||
|
|
||
| on: [pull_request] | ||
|
|
aduh95
Jun 10, 2021
Contributor
Suggested change
env:
NODE_VERSION: 14.x
| env: | |
| NODE_VERSION: 14.x | |
|
|
|
|
|
LGTM |
Actions interface has a better integration with GitHub, and with Annotations and Problem Matcher we can display all failed checks in a single place, so that users don't have to go through the logs to figure out what's wrong. Since the job on Travis was allowed to fail and is not as easy to read, remove it from our Matrix. The Action will check every commit in the Pull Request, skipping commits with "fixup" or "squash". PR-URL: #32417 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>
|
Landed in 53a63de |
|
Looks like a rebase is needed for the PR's those actions ran on. |
|
Shouldn't we actually use the |
|
@targos shouldn't be needed since we don't use anything that needs extra privileges. @richardlau hmmm, so it seems like GitHub Actions doesn't rebase/merge PRs on |
Not for the privileges, but to ensure the matcher is available. |
|
Found the issue, I set |
Actions interface has a better integration with GitHub, and with Annotations and Problem Matcher we can display all failed checks in a single place, so that users don't have to go through the logs to figure out what's wrong. Since the job on Travis was allowed to fail and is not as easy to read, remove it from our Matrix. The Action will check every commit in the Pull Request, skipping commits with "fixup" or "squash". PR-URL: #32417 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>
Actions interface has a better integration with GitHub, and with
Annotations and Problem Matcher we can display all failed checks in a
single place, so that users don't have to go through the logs to figure
out what's wrong. Since the job on Travis was allowed to fail and is not
as easy to read, remove it from our Matrix.
The Action will check every commit in the Pull Request, skipping commits
with "fixup" or "squash".
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passes