Python: Add `ModuleVariableNode` to keep track of global reads and writes by tausbn · Pull Request #4265 · github/codeql

tausbn · 2020-09-14T16:23:06Z

Still partly a WIP. A few remaining issues need to be addressed:

Picking a suitable location and enclosing callable for these nodes. Since the location should be unique, and global variables may be defined in multiple places, I feel like the only reasonable choice here is to pick the module itself as the location (but I welcome any other suggestions). PR now uses the module location as the location of the node. This does present a small problem with regard to inline test expectations (where do you put a module-level comment?) but for now I'm just excluding these nodes from the test output.
Currently, the changes in this PR affect the behaviour of local flow at the top level (e.g. between global SSA variables in the same basic block). I'm not sure what's causing this, but it seems to be removing flow that we would actually want to have. Local flow at the top level now seems to work as intended.
Flow across module boundaries is not currently handled (and will probably not be handled in this PR).

The missing `global g` annotation meant `g = x` was interpreted as a local assignment.

Some of the places where flow has disappeared look a bit suspect, so I don't consider this to be the final word on these tests.

RasmusWL

Overall looks really nice 👍 I think we need to restore the global flow, but I also think I figured out the problem (see inline comment below).

Picking a suitable location and enclosing callable for these nodes. Since the location should be unique, and global variables may be defined in multiple places, I feel like the only reasonable choice here is to pick the module itself as the location (but I welcome any other suggestions).

I agree with using the module for the location 👍

RasmusWL · 2020-09-15T09:37:44Z

 predicate jumpStep(Node nodeFrom, Node nodeTo) {
-  // As we have ESSA variables for global variables,
-  // we include ESSA flow steps involving global variables.
-  (
-    nodeFrom.(EssaNode).getVar() instanceof GlobalSsaVariable
-    or
-    nodeTo.(EssaNode).getVar() instanceof GlobalSsaVariable
-  ) and
+  EssaFlow::essaFlowStep(nodeFrom, nodeTo) and
  (
-    EssaFlow::essaFlowStep(nodeFrom, nodeTo)
+    nodeFrom instanceof ModuleVariableNode
    or
-    // As jump steps do not respect chronology,
-    // we add jump steps for each def-use pair.
-    nodeFrom.asVar() instanceof GlobalSsaVariable and
-    nodeTo.asCfgNode() = nodeFrom.asVar().getASourceUse()
+    nodeTo instanceof ModuleVariableNode
  )
 }


The fact that we only allow jumpStep with a global variable that escapes (ModuleVariableNode) seems to be a problem, and what is causing us to loose flow.

python/ql/test/experimental/dataflow/basic/test.py contains:

def obfuscated_id(x): y = x z = y return z a = 42 b = obfuscated_id(a)

And used to have flow from a -> x, a -> y, a -> z, a -> b. But since a doesn't escape, we won't include it anymore.

I agree with your assessment, but I don't think allowing non-escaping variables is the solution. Because jump steps ignore chronology, something like the following would have flow from source to sink if foo has a ModuleVariableNode:

foo = NON_SOURCE SINK(foo) foo = SOURCE

Now, I was hoping that the necessary flow in the above would fall out from just the regular SSA flow, but it seems it doesn't.

Note that if foo escapes, then we do get the bad flow above anyway, and this is in some sense a bit spurious again (since the second line will never execute a second time with foo containing SOURCE). One solution would be to not consider all top-level reads and writes to global variables, but only the ones that ultimately define the variable in question.

Here's another problematic test example:

foo = SOURCE foo = NON_SOURCE SINK(foo)

Currently we would have flow from source to sink if foo is considered to be escaping.

Either way, I think we need to have "local" Global SSA flow for modelling assignments during module initialisation, and flow to and from module variable nodes to model steps via reads and writes of globals outside of initialisation.

I've named this `DataFlowModuleScope` since it's not really a callable (and all of the relevant methods are empty anyway).

tausbn · 2020-09-15T12:31:19Z

Hooking up the getEnclosingCallable machinery made most of the missing global flow reappear. 🎉
This is now looking closer to being mergeable, but I still would like to add test cases for (and possibly prevent) some of the kinds of bad global flow I outlined above.

RasmusWL · 2020-09-15T12:38:49Z

I'm confused why the fix to getEnclosingCallable broguth back flow for python/ql/test/experimental/dataflow/basic/test.py 😕 at least my explanation of only considering jumpSteps for global variables that escapes doesn't seem to hold true any longer 😞 😕

tausbn · 2020-09-15T12:45:46Z

Yeah, I'm not entirely sure.
I think for that test file, the only missing flow was from the definition of a to its use. The rest is handled locally in obfuscated_id and through the parameter passing and return steps, none of which should have been affected by this PR. I'll debug it for a bit to see if I can figure out what's going on with the flow.

RasmusWL

Nice 👍 a few minor things, and we're good to go

Unrelated to this PR: @yoff do you know why all the results in python/ql/test/experimental/dataflow/basic/globalStep.expected are duplicated? 😕

Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>

RasmusWL

@tausbn seems like you forgot to add the new tests to a commit?

The test cases are removed in ee76d9b#diff-7d9bc8520abc0d5daba111c8d9d5b922L9-L26, but I can't see them anywhere else 😅

tausbn · 2020-09-17T10:07:32Z

This is what happens when I go fast and break things. Sometimes I get too used to just looking at the "unstaged changes", and I forget about "untracked files" entirely.

RasmusWL

Nice 👍

I did not look deeply into the .expected changes of the (non taint realted) dataflow tests, I simply didn't know what to look for. -- hopefully @yoff can sign off on those 😊

I'm using the QL code from this PR locally right now, so I'm already unblocked (and getting this PR merged RIGHT NOW, wouldn't change that).

yoff · 2020-09-17T15:49:19Z

Unrelated to this PR: @yoff do you know why all the results in python/ql/test/experimental/dataflow/basic/globalStep.expected are duplicated? 😕

This is possibly because they happen in different contexts. Note that globalStep.ql references PathNodes which are context sensitive.

yoff

I am a bit confused about the strategy here. The PR leaves GlobalSsaVariables in the dataflow graph and reenables flow between them during local flow nut removes them from jump steps (I think this accounts for many of the test output changes). It then adds dataflow nodes for module variables and add jump steps for those. Could you clarify, which mechanisms should be in play for modelling which flow? 🙏

I love the clean output in the consistency check, but I actually do not understand how it is achieved 🤨

yoff · 2020-09-17T08:54:42Z

 * excludes SSA flow through instance fields.
 */
 predicate simpleLocalFlowStep(Node nodeFrom, Node nodeTo) {
+  not nodeFrom instanceof ModuleVariableNode and


Why are these moved out of the quantifier? A post-update node cannot be a ModuleVariableNode.
(I concede that you could move out nodeTo.)

Also, now we allow GlobalSsaVariables is that intended?

Removes steps from `ModuleVariableNode`s from `essaFlowStep`, and instead puts them only in `jumpStep`. This cleans up the logic a bit. This slightly broke the type tracker implementation (as it relied on `essaFlowStep` being fairly liberal), so I have rewritten it to explicitly rely on just familiar predicates for local and jump steps. Additionally, we disallow Essa-to-Essa steps where exactly one of the two nodes corresponds to a global variable (i.e. only local-local and global-global steps).

yoff

This looks good for now. Might I just ask that you write down in a comment somewhere:
Global SSA variables and ModuleVariables both refer to variables in module scope, but flow to and from

Global SSA variables correspond to "module-local flow" that happens at module initialisation time.
ModuleVariables correspond to jumpSteps that happen at run-time.
thus only ModuleVariables are allowed to mix with local variables.

I guess this should not be restricted to just the variables but extended to their def-use chains? Do we do this correctly?

tausbn · 2020-09-21T09:36:17Z

I guess this should not be restricted to just the variables but extended to their def-use chains? Do we do this correctly?

I'm not quite sure what "this" is referring to in the first sentence above. 😕

yoff

Good documentation 👍

tausbn added 6 commits September 11, 2020 18:17

Python: Fix broken test of global typetracker flow

e0f5b20

The missing `global g` annotation meant `g = x` was interpreted as a local assignment.

Python: Add ModuleVariableNode to dataflow

5fb33c9

Python: Update type tracking and strange-essaflow tests

0b641c5

Merge branch 'main' into python-add-global-flow-steps

e197f52

Python: Fix up merge weirdness

0bb726f

Python: Record test changes

3727c48

Some of the places where flow has disappeared look a bit suspect, so I don't consider this to be the final word on these tests.

tausbn requested a review from a team as a code owner September 14, 2020 16:23

github-actions Bot added the Python label Sep 14, 2020

RasmusWL reviewed Sep 15, 2020

View reviewed changes

tausbn added 2 commits September 15, 2020 14:23

Python: Add "enclosing callable" for ModuleVariableNode

d5e9f36

I've named this `DataFlowModuleScope` since it's not really a callable (and all of the relevant methods are empty anyway).

Python: Add a few function-local import tests

2e737ed

tausbn added 2 commits September 15, 2020 18:25

Python: Disregard module-time reads.

7cdd290

Python: Cleanup, docs, and an extra test case

2d3e23e

tausbn requested review from RasmusWL and yoff September 16, 2020 12:47

RasmusWL requested changes Sep 16, 2020

View reviewed changes

Comment thread python/ql/test/experimental/dataflow/global-flow/test.py Outdated

Comment thread python/ql/test/experimental/dataflow/global-flow/test.py Outdated

Comment thread python/ql/test/experimental/dataflow/tainttracking/basic/test.py Outdated

tausbn and others added 2 commits September 16, 2020 18:21

Python: Apply suggestions from code review

e179df7

Co-authored-by: Rasmus Wriedt Larsen <rasmuswriedtlarsen@gmail.com>

Python: Clean up tests

ee76d9b

tausbn requested a review from RasmusWL September 16, 2020 17:05

RasmusWL requested changes Sep 17, 2020

View reviewed changes

Python: Add missing global flow test

9458861

RasmusWL requested changes Sep 17, 2020

View reviewed changes

Comment thread python/ql/test/experimental/dataflow/tainttracking/unwanted-global-flow/test.py Outdated

tausbn added 3 commits September 17, 2020 13:20

Python: Clean up global flow test

797ac23

Python: Fix misnamed variable.

1d462ae

Python: Fix typo

f93c44a

RasmusWL previously approved these changes Sep 17, 2020

View reviewed changes

yoff reviewed Sep 17, 2020

View reviewed changes

tausbn dismissed RasmusWL’s stale review via 11c85f0 September 18, 2020 16:20

yoff requested changes Sep 21, 2020

View reviewed changes

tausbn added 2 commits September 21, 2020 11:46

Python: Add a bit more documentation to ModuleVariableNode

1d6558b

Merge branch 'main' into python-add-global-flow-steps

9d7a2d2

yoff approved these changes Sep 21, 2020

View reviewed changes

yoff merged commit 557db33 into github:main Sep 21, 2020

tausbn deleted the python-add-global-flow-steps branch February 12, 2021 18:03

Conversation

tausbn commented Sep 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RasmusWL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RasmusWL Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

tausbn Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

tausbn commented Sep 15, 2020

Uh oh!

RasmusWL commented Sep 15, 2020

Uh oh!

tausbn commented Sep 15, 2020

Uh oh!

RasmusWL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RasmusWL left a comment

Choose a reason for hiding this comment

Uh oh!

tausbn commented Sep 17, 2020

Uh oh!

Uh oh!

RasmusWL left a comment

Choose a reason for hiding this comment

Uh oh!

yoff commented Sep 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yoff left a comment

Choose a reason for hiding this comment

Uh oh!

yoff Sep 17, 2020

Choose a reason for hiding this comment

Uh oh!

yoff Sep 17, 2020

Choose a reason for hiding this comment

Uh oh!

yoff left a comment

Choose a reason for hiding this comment

Uh oh!

tausbn commented Sep 21, 2020

Uh oh!

yoff left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tausbn commented Sep 14, 2020 •

edited

Loading

yoff commented Sep 17, 2020 •

edited

Loading