Add the AST Viewer by aeisenberg · Pull Request #492 · github/vscode-codeql

aeisenberg · 2020-07-09T20:09:11Z

This commit adds the AST Viewer for viewing the QL AST of a file in a
database.

The different components are as follows:

There is a new view codeQLAstViewer, which displays the AST
This view is backed by the AstViewerDataProvider and AstViewer classes in astView.ts
To generate an AST, we use contextual queries, similar to how Find references/declarations are implemented. In particular, in definitions.ts there is TemplatePrintAstProvider which provides an AST for a given source buffer.

Similar to the other queries, we first determine which database the buffer belongs to.
Based on that, we generate a synthetic qlpack and run the templatized printAst.ql query
We plug in the archive-relative path name of the source file.
After the query is run, we wrap the results in an AstBuilder instance.
When requested, the AstBuilder will generate the full AST of the file from the BQRS results.
The AST roots (all top-level elements, functions, variable declarations, etc, are roots) are passed to the AstViewer instance, which handles the display lifecycle and other VS Code-specific functions.

There are a few unrelated pieces here, which can be pulled out to another PR if required:

The codeQLQueryHistory view now has a welcome message to make it more obvious to users how to start.
definitions.ts is moved to the contextual subfolder.
fileRangeFromURI is extracted from definitions.ts to its own file so it can be reused.

Also, note that this relies on github/codeql#3931 for the C/C++ query to be available in the QL sources. Other languages will need similar queries.

Replace this with a description of the changes your pull request makes.

Checklist

CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
Issues have been created for any UI or other user-facing changes made by this pull request.
@github/product-docs-dsp has been cc'd in all issues for UI or other user-facing changes made by this pull request.

aeisenberg · 2020-07-10T14:47:06Z

Note: Now rebaased on #494. Let's get that one in first.

This commit adds the AST Viewer for viewing the QL AST of a file in a database. The different components are as follows: 1. There is a new view `codeQLAstViewer`, which displays the AST 2. This view is backed by the `AstViewerDataProvider` and `AstViewer` classes in astView.ts 3. To generate an AST, we use contextual queries, similar to how Find references/declarations are implemented. In particular, in `definitions.ts` there is `TemplatePrintAstProvider` which provides an AST for a given source buffer. - Similar to the other queries, we first determine which database the buffer belongs to. - Based on that, we generate a synthetic qlpack and run the templatized `printAst.ql` query - We plug in the archive-relative path name of the source file. - After the query is run, we wrap the results in an `AstBuilder` instance. - When requested, the `AstBuilder` will generate the full AST of the file from the BQRS results. - The AST roots (all top-level elements, functions, variable declarations, etc, are roots) are passed to the `AstViewer` instance, which handles the display lifecycle and other VS Code-specific functions. There are a few unrelated pieces here, which can be pulled out to another PR if required: - The `codeQLQueryHistory` view now has a _welcome_ message to make it more obvious to users how to start. - `definitions.ts` is moved to the `contextual` subfolder. - `fileRangeFromURI` is extracted from `definitions.ts` to its own file so it can be reused. Also, note that this relies on github/codeql#3931 for the C/C++ query to be available in the QL sources. Other languages will need similar queries.

* Fix command-linting tests. * Fix failing windows test and Use Uri.parse(_, true) * Use Uri.parse(_, true). That is the preferred API. * Reformat comments.

This commit adds the AST Viewer for viewing the QL AST of a file in a database. The different components are as follows: 1. There is a new view `codeQLAstViewer`, which displays the AST 2. This view is backed by the `AstViewerDataProvider` and `AstViewer` classes in astView.ts 3. To generate an AST, we use contextual queries, similar to how Find references/declarations are implemented. In particular, in `definitions.ts` there is `TemplatePrintAstProvider` which provides an AST for a given source buffer. - Similar to the other queries, we first determine which database the buffer belongs to. - Based on that, we generate a synthetic qlpack and run the templatized `printAst.ql` query - We plug in the archive-relative path name of the source file. - After the query is run, we wrap the results in an `AstBuilder` instance. - When requested, the `AstBuilder` will generate the full AST of the file from the BQRS results. - The AST roots (all top-level elements, functions, variable declarations, etc, are roots) are passed to the `AstViewer` instance, which handles the display lifecycle and other VS Code-specific functions. There are a few unrelated pieces here, which can be pulled out to another PR if required: - The `codeQLQueryHistory` view now has a _welcome_ message to make it more obvious to users how to start. - `definitions.ts` is moved to the `contextual` subfolder. - `fileRangeFromURI` is extracted from `definitions.ts` to its own file so it can be reused. Also, note that this relies on github/codeql#3931 for the C/C++ query to be available in the QL sources. Other languages will need similar queries.

* Fix command-linting tests. * Fix failing windows test and Use Uri.parse(_, true) * Use Uri.parse(_, true). That is the preferred API. * Reformat comments.

Set `codeQL.experimentalAstViewer` to true in settings in order for component to be enabled.

…codeql into aeisenberg/ast-viewer

extensions/ql-vscode/CHANGELOG.md

jcreedcmu · 2020-07-21T17:13:47Z

extensions/ql-vscode/src/cli.ts

  async bqrsDecode(bqrsPath: string, resultSet: string, pageSize?: number, offset?: number): Promise<DecodedBqrsChunk> {
    const subcommandArgs = [
-      '--entities=url,string',
+      '--entities=id,url,string',


I don't suppose there's any performance penalty for including more things than we need for any particular query here, is there? Just idle speculation as to whether we might want to parameterize over the set of entities, since the other bqrs decoding code paths didn't need id until now.

I originally implemented this as optionally including the id field. I could add it back.

Do whatever you think is reasonable; I can see the side of the argument for it being awkward to make it optional too, since that makes the type vary.

jcreedcmu · 2020-07-21T17:17:47Z

extensions/ql-vscode/src/contextual/definitions.ts

      const destFile = dest.url && fileRangeFromURI(dest.url, db);
-      if (srcFile && destFile && filter(srcFile.file.toString(), destFile.file.toString())) {
-        localLinks.push({ targetRange: destFile.range, targetUri: destFile.file, originSelectionRange: srcFile.range, originUri: srcFile.file });
+      if (srcFile && destFile && filter(srcFile.uri.toString(), destFile.uri.toString())) {


This file -> uri replacement is slightly scary to me since I don't have this fully paged into my brain. I would have thought those would have different behavior. Did this not work before? Does it definitely continue working after this change?

Ahhh...I had forgotten that I did this. We are comparing 2 strings to see if they are a match and if so, then we mark the tuple as a match. Earlier on, we call getDefinitions and getReferences with document.uri.toString(). So, I am not sure how this worked in the past. I'll go try this out now and see.

Right. I extracted fileRangeFromURI to its own file and I made it return a vscode.Location object. It's the same as before, except it is returning a vscode api type. file and uri are both vscode.Uri instances.

extensions/ql-vscode/src/vscode-tests/no-workspace/contextual/astBuilder.test.ts

extensions/ql-vscode/src/contextual/astBuilder.ts

jcreedcmu · 2020-07-22T13:55:41Z

extensions/ql-vscode/src/contextual/astBuilder.ts

+
+    // Build up the parent-child relationships
+    edgeTuples.tuples.forEach(tuple => {
+      const from = tuple[0] as EntityValue;


Not a huge deal, but I'd have a slight preference for establishing what type we expect tuple to have right away, like

// Build up the parent-child relationships edgeTuples.tuples.forEach(tuple => { const [src, tgt, tupleType, orderValue] = tuple as [EntityValue, EntityValue, string, number]; const toId = src.id!; const fromId = tgt.id!; switch (tupleType) { case 'semmle.order': astOrder.set(toId, Number(orderValue)); break; case 'semmle.label': { childToParent.set(toId, fromId); let children = parentToChildren.get(fromId); if (!children) { parentToChildren.set(fromId, children = []); } children.push(toId); break; } default: throw new Error(`unexpected tuple type ${tupleType}`); } });

I don't know if the names I provided are the most appropriate for the fields of the tuples but that's sort of my point --- I'm a little uncertain reading this code what the expected schema/interpretation is.

(and if the interpretation of some columns depends on the values of others, which looks like it might be the case here with respect to tuple[3] depending on tuple[2], it would be similarly nice to clarify that)

Fair point. I will clarify.

jcreedcmu · 2020-07-22T13:57:16Z

extensions/ql-vscode/src/contextual/astBuilder.ts

+
+    // populate parents and children
+    nodeTuples.tuples.forEach(tuple => {
+      const entity = tuple[0] as EntityValue;


Same comment as above about deconstructing tuple here

jcreedcmu · 2020-07-22T14:00:16Z

extensions/ql-vscode/src/contextual/astBuilder.ts

+    });
+
+    // find the roots and add the order
+    for(const [, item] of idToItem) {


I didn't know you didn't have to write an explicit undefined to avoid binding a tuple deconstructor field! Nice.

jcreedcmu · 2020-07-22T14:02:02Z

extensions/ql-vscode/src/contextual/definitions.ts

  switch (keyType) {
-    case KeyType.DefinitionQuery: return 'ide-contextual-queries/local-definitions';
-    case KeyType.ReferenceQuery: return 'ide-contextual-queries/local-references';
+    case KeyType.DefinitionQuery:


It seems like a bunch of these definitions should be hoisted out of definitions.ts into some more general file in contextual/ but that needn't happen in this PR.

Sure. I agree that it's getting a bit messy. I'll work on that next.

jcreedcmu · 2020-07-23T13:17:09Z

extensions/ql-vscode/src/contextual/astBuilder.ts

-        children.push(toId);
+
+        default:
+        // ignore other tupleTypes since they are not needed by the ast viewer


Ah, ok, great --- I didn't know that other values of tupleType existed, and it's great to clarify here that we're explicitly ignoring them.

Thanks for suggesting that change. I think it makes the code clearer.

aeisenberg requested review from dbartol and jcreedcmu July 9, 2020 20:10

aeisenberg force-pushed the aeisenberg/ast-viewer branch from 121f8ea to d0053e7 Compare July 10, 2020 14:46

aeisenberg force-pushed the aeisenberg/ast-viewer branch from d0053e7 to c7c27fe Compare July 10, 2020 15:05

aeisenberg added 3 commits July 10, 2020 08:16

Update changelog

4aa7521

Fix tests and reformatting

3639dcb

* Fix command-linting tests. * Fix failing windows test and Use Uri.parse(_, true) * Use Uri.parse(_, true). That is the preferred API. * Reformat comments.

aeisenberg force-pushed the aeisenberg/ast-viewer branch from c7c27fe to 3639dcb Compare July 10, 2020 15:18

This was referenced Jul 16, 2020

Tracking issue for PrintAST queries in supported languages #505

Closed

Implement PrintAST for JavaScript github/codeql#3954

Closed

Implement PrintAST for Java github/codeql#3955

Closed

Implement PrintAST for Go github/codeql-go#260

Closed

aeisenberg added 4 commits July 16, 2020 14:42

Update changelog

af965c9

Fix tests and reformatting

8459edb

* Fix command-linting tests. * Fix failing windows test and Use Uri.parse(_, true) * Use Uri.parse(_, true). That is the preferred API. * Reformat comments.

Add feature flag for ast viewer

36d612e

Set `codeQL.experimentalAstViewer` to true in settings in order for component to be enabled.

aeisenberg mentioned this pull request Jul 17, 2020

Implement PrintAST for Python github/codeql#3956

Closed

jcreedcmu and others added 2 commits July 17, 2020 10:53

Merge branch 'main' into aeisenberg/ast-viewer

aebab08

Merge branch 'aeisenberg/ast-viewer' of github.com:aeisenberg/vscode-…

f21296e

…codeql into aeisenberg/ast-viewer

Marcono1234 reviewed Jul 21, 2020

View reviewed changes

extensions/ql-vscode/CHANGELOG.md Outdated Show resolved Hide resolved

Remove duplicate changelog line

65b5b68

jcreedcmu suggested changes Jul 22, 2020

View reviewed changes

Clean up ast builder code

0045891

aeisenberg force-pushed the aeisenberg/ast-viewer branch from 572e38c to 0045891 Compare July 22, 2020 20:34

jcreedcmu reviewed Jul 23, 2020

View reviewed changes

jcreedcmu approved these changes Jul 23, 2020

View reviewed changes

aeisenberg merged commit 1017741 into github:main Jul 23, 2020

Comments

Conversation

aeisenberg commented Jul 9, 2020

Checklist

Uh oh!

aeisenberg commented Jul 10, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants