Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruby: extractor performance optimisations #7219

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

@nickrolfe
Copy link
Contributor

@nickrolfe nickrolfe commented Nov 23, 2021

I suggest looking at the changes commit-by-commit.

The main changes are:

  • Refactoring trap-writing into its own module,
  • Pre-computing the set of valid types for each field, so we don't repeat that work for every node encountered during extraction,
  • Sorting TRAP output: all label definitions first, and then tuples, grouped by name. This makes the TRAP compress better, which then makes TRAP import faster.

Locally, I consistently see these changes make overall codeql database create time 5-10% faster. On DCA runs, however, it usually seems to be a wash.

nickrolfe added 6 commits Nov 23, 2021
We were previously doing this during extraction, i.e. for each field
node we encouter, which meant we were repeating a lot of work. The
`type_matches_set` function was a fairly significant hot-spot in
profiling results, so this should improve performance.
First, emit labels with fresh ids. Then other labels. Then tuples,
grouped by name. Hopefully this will help both with the compression
ratio but also with branch prediction in the TRAP importer.
This is the same idea as Java's LinkedHashMap: it gives the same O(1)
insertion and lookup as HashMap, but preserves insertion order for
iteration.
@nickrolfe nickrolfe added the Ruby label Nov 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant