Thanks @sauyon ! Yeah, this looks something like what I was after. Looks like the JavaScript extractor is also open-source. I've only dabbled in golang, but it looks to support a new language you need to:

1. create a relational db schema for the AST of the language you're working on; this will need to be written into a .dbscheme file in the language repo, then copied into databases by an extractor. It looks like there is no documentation on the format for dbscheme files, so I guess it'll take some reverse-engineering from examples
2. write an extractor that will build a source project including dependencies, and translate the AST of the built project into the relational schema defined above
3. the extractor will write all the symbols and relations found to trap files - I don't see any documentation on this file format either, so probably pulling apart some example databases or hooking into the golang / js extractors is the best bet here
4. write a qll interface for dealing with your new language's schema; written in CodeQL and starting from database types (the ones defined in dbscheme above). These should be in a ql/ folder which will be searched from the CodeQL home dir, for organizational purposes
5. update a bunch of miscellaneous files like queries.xml , qlpack.yml , .qlpath , etc so that codeql understands what your package name is and when to use it

Any steps I've left out, or additional documentation I should check? Cheers,

Thanks for your interest! Describing what's necessary to add support for a new language to CodeQL is something we'd very much want to do. The public-facing documentation for this is simply not there at the moment, but hearing that it would be useful definitely makes it easier to prioritise cleaning up our internal docs. 😄

• create a relational db schema for the AST of the language you're working on; this will need to be written into a .dbscheme file in the language repo, then copied into databases by an extractor. It looks like there is no documentation on the format for dbscheme files, so I guess it'll take some reverse-engineering from examples

That's right. Designing the right schema is the part that will likely involve most thought.

The .dbscheme language is pretty simple: You declare your tables, giving a representation type and a logical type for each column; in addition, you can introduce unions of column types. [Documenting it publicly is on our TODO list.]

• write an extractor that will build a source project including dependencies, and translate the AST of the built project into the relational schema defined above

For Ruby, I expect you may not need to do a build -- you may be able to follow a model that's more similar to how we handle JavaScript. As it's an interpreted dynamic language, we simply represent the syntax in the database, and handle dependencies through explicit modelling in the CodeQL libraries. This reflects the fact that you could just run any JS file through the interpreter, without dependencies.

• the extractor will write all the symbols and relations found to trap files - I don't see any documentation on this file format either, so probably pulling apart some example databases or hooking into the golang / js extractors is the best bet here

Yup, another internal format whose public documentation is TODO... sorry! TRAP is designed to be easy to generate, and if the codeql-go trap-writer class isn't sufficiently clear, we'd be happy to answer any questions.

• write a qll interface for dealing with your new language's schema; written in CodeQL and starting from database types (the ones defined in dbscheme above). These should be in a ql/ folder which will be searched from the CodeQL home dir, for organizational purposes

The directory structure and naming is convention, but the root of your QL code hierarchy should contain the special qlpack.yml file you note below.

• update a bunch of miscellaneous files like queries.xml , qlpack.yml , .qlpath , etc so that codeql understands what your package name is and when to use it

.qlpath and queries.xml are legacy files; you should be able to get by with just a qlpack.yml.

Any steps I've left out, or additional documentation I should check? Cheers,

Actually write some queries to use your shiny new database! It'll likely take a bit of work to put the basics of analysis in place; I'd suggest looking closely at how the JavaScript or Go libraries deal with control-flow, data-flow and taint tracking, as those tend to be building blocks you need for everything else.

1. create a relational db schema for the AST of the language you're working on; this will need to be written into a .dbscheme file in the language repo, then copied into databases by an extractor. It looks like there is no documentation on the format for dbscheme files, so I guess it'll take some reverse-engineering from examples

Take a look at our dbscheme generator and table definitions. I think that should help with understanding the format, and in theory you could even use the Go dbscheme generator to generate a dbscheme and work from there. They could do with some extra documentation, I think, but they should be relatively legible and vaguely contain the syntax you need to know.

3. the extractor will write all the symbols and relations found to trap files - I don't see any documentation on this file format either, so probably pulling apart some example databases or hooking into the golang / js extractors is the best bet here

The trap writer package Pavel mentioned.

Any steps I've left out, or additional documentation I should check? Cheers,

All of the files required to make a CodeQL language pack are generated or copied by the extractor target in our Makefile. That should be able to serve as a simple checklist for creating a pack. Alternatively, you could just run make and use the files in build/codeql-extractor-go as a reference.

Thanks so much @p0 & @sauyon ! Really appreciate all the help and reference links.

For Ruby, I expect you may not need to do a build -- you may be able to follow a model that's more similar to how we handle JavaScript. As it's an interpreted dynamic language, we simply represent the syntax in the database, and handle dependencies through explicit modelling in the CodeQL libraries.

Interesting! I will read more through the JS extractor, then. I was worried that since Ruby and Rails have deferred run-time auto-loading, they might parse the wrong version of a method if you haven't run through a build script or test suite, but maybe I'm over-thinking it.

Actually write some queries to use your shiny new database!

Yes, haha, for sure! I was planning to start with the simplest possible thing; a dependency-free command-line script that does eval $1 , and then write a query to find "eval called with user input" , and try to implement the minimum AST & dbscheme that would support that file & query. Then once I have that working, add a gem dependency, and work my way up to more complicated cases.

I'd suggest looking closely at how the JavaScript or Go libraries deal with control-flow, data-flow and taint tracking, as those tend to be building blocks you need for everything else.

Thanks for the tip! 👍

All of the files required to make a CodeQL language pack are generated or copied by the extractor target in our Makefile. That should be able to serve as a simple checklist for creating a pack. Alternatively, you could just run make and use the files in build/codeql-extractor-go as a reference.

Thank you! Didn't think to check through the Makefile.

Much appreciated! This gives me a lot to get started with. Hopefully I'll have some time this week & next weekend to work on a proof-of-concept for Ruby.