I’m not entirely clear on whether it works this way or whether the JSON values all have to refer to regexp capture groups, but could we use something like "severity": "warning" (or "severity": "notice"? not sure if that would work either) so that this does not appear like a failed build, given that this is ultimately not essential for PR authors to address?

The severity only works for annotations, the Check status can only be Success or Failure

That’s unfortunate :/ Any ideas for how else we could make clear that this is non-essential? I’d rather not have explicit failures because of this…

We could prefix the workflow name with "[not a blocker]" or hijack the annotation to have this as a message. Unfortunately I don't think there's anything we can do using GitHub's features (the best thing would be for the Action leave a comment, but we can't do that with Actions for PRs coming from forks), and I would rather not have the check on CI than mark as "allowed to fail", because if we do so it'll be misleading for newcomers (the check will always be green).

Switching to warning instead of error on the annotation is probably still worth though. I'll update the PR

Hum, reading the Checks API docs again it seems there a "neutral" status.

https://developer.github.com/v3/checks/runs/

I never seen it, and I'm not sure we can set it without directly calling the API (which is not possible in the Action), but it's worth a shot.

(There's also a "Action Required" status, but it's visually worse than "Failure" IMO)

GitHub Actions has no way of setting the "neutral" status on runs without calling the Checks API directly (which needs a write-permission token, which means it wouldn't work on Pull Requests from non-collaborators). So, we can either:

1. Have the check fail
2. Set it to continue-on-error
3. Don't check the commit message on CI

I'd rather not set it to continue-on-error since any failures will fly under everyone's radar. I'm fine not landing this change though, if folks think failing the CI because the commit messages don't follow our guidelines is unwelcoming. Personally I think the annotations are helpful for folks who want to make sure their commits are following the guidelines. Maybe we could force one annotation with a "not required" message, but I'm not sure how useful and clear that would be.

On the other hand we could also use this check to verify Signed-off-by metadata on commits once #31976 lands, or have another Action doing the same verification. Either way the CI would fail if the commit is not signed, so maybe it's a good thing to introduce this checks now instead of later.

One alternative I've seen in the past is keeping the check "pending" when things like Signed-off-by are missing, but then again, I don't think this is possible with the current Actions permission model.