I recommend looking at the documentation on "categories" in code scanning: https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning#runautomationdetails-object.

A category is an additional metadata string that can be provided with an analysis, which helps distinguish analyses produced by the same tool on the same commit/ref but in different circumstances (e.g. different jobs in a matrix, different subsets of a repo or analysis, different target language). In SARIF 2.1.0, the category can be provided as part of the <run>.runAutomationDetails.id field. See the docs for the precise details.

In the scenario you describe, I suggest being careful to ensure your push runs include results for all the categories you define. Your pull_request runs can include results for whichever subset of categories are relevant to that PR. As long as there were baseline results for the same category from an earlier push on the base branch, Code Scanning will correctly compute the alert diff for the PR for that category.

(For reference, you may also wish to read these docs on using categories with the CodeQL Action, which may be useful information even though your use case is independent of the CodeQL tool.)

I hope this helps -- please get in touch if you have further questions!

Something that's not entirely clear is:

• Do I need multiple SARIF files?
• Can I use a single SARIF file containing multiple runs entries, i.e. one or each category?
• Can I pack multiple categories in one id in a single SARIF file? In the example .github/workflows/codeql-analysis.yml:analyze/language:javascript/os:linux: Is this a single category or is it interpreted as category .github/workflows/codeql-analysis.yml:analyze + category language:javascript + category os:linux? In which case I could pack multiple categories in one id.

Thanks for you help!

• Do I need multiple SARIF files?
• Can I use a single SARIF file containing multiple runs entries, i.e. one or each category?

I think either of these approaches will work. Either multiple SARIF files each containing one run, or a single SARIF file containing multiple runs each with a different category. The important detail is that the different runs (whether in the same SARIF file or multiple SARIF files) must have different categories, otherwise they will overwrite each other when uploaded. Experiment with both and let us know if the behaviour is not what you expect.

I find it easiest to think about one SARIF file per Actions job. If you're doing multiple analyses in a single job, one SARIF file with multiple runs may be convenient. If you're doing analyses in different jobs in an Actions workflow (e.g. a job matrix, which is nice for parallelism too), one SARIF file per job is convenient.

Can I pack multiple categories in one id in a single SARIF file? In the example .github/workflows/codeql-analysis.yml:analyze/language:javascript/os:linux: Is this a single category or is it interpreted as category .github/workflows/codeql-analysis.yml:analyze + category language:javascript + category os:linux? In which case I could pack multiple categories in one id.

One category per run object. In the above example, there is a single category .github/workflows/codeql-analysis.yml:analyze/language:javascript/os:linux.

Brilliant, that clarifies everything!

follow-up question: If the analysis contains no findings, is it better to return an empty list of runs, or populate runs each with empty results, or something else?

I want to be sure that if the reported vulnerability is fixed by developers, it's automatically removed from the scanning result dashboard. After several attempts, what I found promising (I need to test more) is to populate Tools, no to populate rules field and set results to []. Is this the intended way?

After several hours of debugging, I hit another problem: it seems there is a limit on the maximum number of runs GitHub allows. If I provide more than 15, the findings are not longer taken into account. Is that correct?

I've attached 2 files so you can verify. Thanks
15-runs.txt
16-runs.txt

Let me clarify how my tool works. I have 16 analysis, which we call checks. On a push, we run all the checks; on a pull request we only run 4 of them. What I did above is use a run for each check/analysis, using the check name as category.
An alternative would be to create 2 categories: one for push event and one for pull request events. Then use each check as a rule.

I'm not sure which is the right one. Can you advise?

follow-up: I think the right approach is to use one run per check, otherwise GitHub is unable to compare results of a push request vs main branch, since the analysis are different (category "pull-request" vs "push")

So I'm stuck with the 15 upper limit, it seems. Can anyone help?

follow-up question: If the analysis contains no findings, is it better to return an empty list of runs, or populate runs each with empty results, or something else?

If the analysis ran successfully and contains no findings, create a single run with an empty results list.
An empty runs list indicates that no analysis ran (which makes the SARIF not very useful).

After several hours of debugging, I hit another problem: it seems there is a limit on the maximum number of runs GitHub allows. If I provide more than 15, the findings are not longer taken into account. Is that correct?

Checked with my colleagues that maintain the code scanning backend. You are correct that there is a limit of 15 runs per analysis.

Let me clarify how my tool works. I have 16 analysis, which we call checks. On a push, we run all the checks; on a pull request we only run 4 of them. What I did above is use a run for each check/analysis, using the check name as category. An alternative would be to create 2 categories: one for push event and one for pull request events. Then use each check as a rule.

I'm not sure which is the right one. Can you advise?

Thank you for the context! With that in mind, I think my initial suggestion of one category per check is not going to work. Instead, your best bet is:

• Have two analysis categories: let us call them limited and full.
• A run with the limited category should contain the 4 checks you want to run on PRs. Each check can indeed be a rule in the SARIF.
• A run with the full category should contain all 16 checks. Each check can indeed be a rule in the SARIF.
• On push events, produce run objects for both limited and full. This ensures that PR runs will always have a baseline for limited. You don't need to actually run the 4 overlapping checks twice, just ensure there are two runs in the SARIF on push events.
• On pull_request events, produce run objects only for limited.

Can you give this a try and let us know if it serves your purpose?

Thanks. So I tried something similar this morning and it seems to work, but I'd like to know if it's equivalent to your suggestion. I've highlighted the main different below:

• Have two analysis categories: let us call them limited and full.

I did this.

• A run with the limited category should contain the 4 checks you want to run on PRs. Each check can indeed be a rule in the SARIF.

I did this

• A run with the full category should contain all 16 checks. Each check can indeed be a rule in the SARIF.

This is where I implemented something slightly different. full always contains 12 checks, and limited always contains 4 checks. When I run on a push event, I run both categories limited and full. When I run on pull request, I run only limited.

Is this what you are proposing in your comment below?

• On push events, produce run objects for both limited and full. This ensures that PR runs will always have a baseline for limited. You don't need to actually run the 4 overlapping checks twice, just ensure there are two runs in the SARIF on push events.
• On pull_request events, produce run objects only for limited.

Can you give this a try and let us know if it serves your purpose?

That sounds like it will work, yes. The important detail is that the 4 limited checks need to be present on both push and pull_request events, and have the same category for both, which your solution does.

Thanks for the confirmation. When you say 4 limited checks need to be present on both push and pull_request events, does it mean the rules need to be present even if there are no findings for a check? Today I only populate the rule for checks that have findings and it seems to work, but I'm not sure if this is best practice or not. Do you know?

My phrasing of "need to be present" was too strict. I was thinking only of the case where there were results produced by each of those 4 checks, which needed to be matched across pull_request and push runs.

If a particular check doesn't have any results, I think it's a good practice to include it in rules just to show that the check was actually run, but I don't believe the code scanning service cares about the presence of the rule object when there are no corresponding results.

The SARIF spec is not prescriptive about this: rules says "each of which provides information about an analysis rule supported by the tool component". For CodeQL we populate all the rules that were run, so that users can know from the output exactly which of our queries were run even if all the queries didn't find results.

Gotcha. I'll update my code to generate rules regardless of the results. Thanks for the advice.