C++: Use IR alias analysis for field flow by MathiasVP · Pull Request #4142 · github/codeql

MathiasVP · 2020-08-26T14:00:06Z

Fixes https://github.com/github/codeql-c-analysis-team/issues/95.

Prior to this PR we wouldn't catch the following flow:

struct S {
  int x, y;
};

void test() {
  S s;
  s.x = source();

  S s_copy = s;
  int* px = &s_copy.x;
  sink(*px);
}

because the FieldAddressInstruction was "far away" from the actual LoadInstruction, so we wouldn't recognize that the load generated by *px loaded from the field address x. This meant that it wasn't picked up as a readStep in IR dataflow.

However, the IR already computed that this is a load that totally overlaps with the definition S s_copy = s, and so it is reasonably easy to hook into the IR internals and grab this information.

…loading it afterwards

…dataflow read and store steps.

MathiasVP · 2020-08-26T14:04:27Z

CPP-difference job started: ~~https://jenkins.internal.semmle.com/job/Changes/job/CPP-Differences/1365/~~ https://jenkins.internal.semmle.com/job/Changes/job/CPP-Differences/1367/

jbj

I was hoping for a much less disruptive PR where offsets were only used to fix up cases where the field can't be determined by SSA.

jbj · 2020-08-27T08:55:32Z

cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivate.qll

 private import semmle.code.cpp.ir.IR
 private import DataFlowDispatch
+private import semmle.code.cpp.ir.implementation.aliased_ssa.internal.AliasedSSA
+private import semmle.code.cpp.ir.internal.IntegerConstant


I don't think we can merge this before we find a way to avoid implementation and internal imports here.

I agree. I'll have a chat with @dbartol about this later today.

cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivate.qll

…Interval predicate to NonPhiMemoryOperands.

…dds a Class column to the FieldContent branch so FieldContent has a pretty toString implementation again.

MathiasVP · 2020-09-01T09:39:26Z

I've added a new iteration of this PR now. Here's an overview:

1e13a39 implements the new API on ChiInstruction and NonPhiMemoryOperand that @dbartol and I discussed on Zoom the other day.
9de570b uses the new API for field flow, and adds a Class column to the FieldContent branch for more a user-friendly toString.
3cca74e accepts the test changes. The new results aren't as impressive as the first iteration of this PR since we added the Class column as part of the FieldContent, and this means that only source-level fields can be considered FieldContent.
This means we don't catch lambda-captured variables as fields as we did before adding the Class column. In the future we could consider generalizing this column to include "synthesized classes", which could include lambda captures and global variables.

I've started a new CPP-difference here: https://jenkins.internal.semmle.com/job/Changes/job/CPP-Differences/1379/

jbj · 2020-09-01T10:14:51Z

This means we don't catch lambda-captured variables as fields as we did before adding the Class column. In the future we could consider generalizing this column to include "synthesized classes", which could include lambda captures and global variables.

Why do we no longer catch lambda-captured variables? I thought they were represented as Fields of a synthetic Class in our db.

MathiasVP · 2020-09-01T10:20:56Z

Why do we no longer catch lambda-captured variables? I thought they were represented as Fields of a synthetic Class in our db.

Oh, you're right. They are! I thought that was the reason we caught https://github.com/github/codeql/blob/main/cpp/ql/test/library-tests/dataflow/dataflow-tests/lambdas.cpp#L46 before I added the Class column, but not afterwards. I'll do a more thorough investigation of why we lose that new result after the changes in 1e13a39 and 9de570b.

…the shared dataflow library. This is the tuple counts on ChakraCore before the fix: (5539s) Tuple counts for DataFlowPrivate::readStep#fff: 3208924 ~0% {2} r1 = SCAN Operand::NonPhiMemoryOperand::getAnyDef_dispred#3#ff AS I OUTPUT I.<1>, I.<0> 3208924 ~2% {2} r2 = JOIN r1 WITH DataFlowUtil::TInstructionNode#ff AS R ON FIRST 1 OUTPUT r1.<1>, R.<1> 751306 ~8% {2} r3 = JOIN r2 WITH Instruction::CopyInstruction::getSourceValueOperand_dispred#3#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r2.<1> 751306 ~0% {3} r4 = JOIN r3 WITH DataFlowUtil::TInstructionNode#ff AS R ON FIRST 1 OUTPUT r3.<0>, r3.<1>, R.<1> 751306 ~0% {4} r5 = JOIN r4 WITH Instruction::CopyInstruction::getSourceValueOperand_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r4.<1>, r4.<0>, r4.<2> 751306 ~2% {4} r6 = JOIN r5 WITH Operand::NonPhiMemoryOperand::getAnyDef_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r5.<1>, r5.<2>, r5.<3> 209341 ~0% {4} r7 = JOIN r6 WITH Instruction::Instruction::getResultType_dispred#fb AS R ON FIRST 1 OUTPUT R.<1>, r6.<1>, r6.<2>, r6.<3> 7115323 ~1% {6} r8 = JOIN r7 WITH DataFlowPrivate::FieldContent#class#ffff_1023#join_rhs AS R ON FIRST 1 OUTPUT R.<2>, R.<3>, r7.<1>, r7.<2>, r7.<3>, R.<1> 81341188413 ~0% {5} r9 = JOIN r8 WITH SSAConstruction::Cached::getUsedInterval#fff_120#join_rhs AS R ON FIRST 2 OUTPUT r8.<3>, R.<2>, r8.<2>, r8.<4>, r8.<5> 137684 ~7% {3} r10 = JOIN r9 WITH Instruction::CopyInstruction::getSourceValueOperand_dispred#3#ff AS R ON FIRST 2 OUTPUT r9.<2>, r9.<4>, r9.<3> 751306 ~0% {3} r11 = JOIN r4 WITH Instruction::LoadInstruction::getSourceAddress_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r4.<1>, r4.<2> 94306 ~0% {3} r12 = JOIN r11 WITH Instruction::FieldInstruction::getField_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r11.<1>, r11.<2> 152363 ~6% {3} r13 = JOIN r12 WITH DataFlowPrivate::FieldContent::getField_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT r12.<1>, R.<1>, r12.<2> 290047 ~22% {3} r14 = r10 \/ r13 return r14 and after: (851s) Tuple counts for DataFlowPrivate::readStep#fff: 3208924 ~0% {2} r1 = SCAN Operand::NonPhiMemoryOperand::getAnyDef_dispred#3#ff AS I OUTPUT I.<1>, I.<0> 3208924 ~2% {2} r2 = JOIN r1 WITH DataFlowUtil::TInstructionNode#ff AS R ON FIRST 1 OUTPUT r1.<1>, R.<1> 751306 ~8% {2} r3 = JOIN r2 WITH Instruction::CopyInstruction::getSourceValueOperand_dispred#3#ff_10#join_rhs AS R ON FIRST 1 OUTPUT R.<1>, r2.<1> 751306 ~0% {3} r4 = JOIN r3 WITH DataFlowUtil::TInstructionNode#ff AS R ON FIRST 1 OUTPUT r3.<0>, r3.<1>, R.<1> 751306 ~0% {4} r5 = JOIN r4 WITH Instruction::CopyInstruction::getSourceValueOperand_dispred#3#ff AS R ON FIRST 1 OUTPUT r4.<0>, r4.<1>, r4.<2>, R.<1> 751306 ~0% {5} r6 = JOIN r5 WITH Instruction::CopyInstruction::getSourceValueOperand_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r5.<1>, r5.<0>, r5.<2>, r5.<3> 751306 ~0% {5} r7 = JOIN r6 WITH Operand::NonPhiMemoryOperand::getAnyDef_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r6.<1>, r6.<2>, r6.<3>, r6.<4> 209341 ~1% {5} r8 = JOIN r7 WITH Instruction::Instruction::getResultType_dispred#fb AS R ON FIRST 1 OUTPUT R.<1>, r7.<1>, r7.<2>, r7.<3>, r7.<4> 7115323 ~0% {7} r9 = JOIN r8 WITH DataFlowPrivate::FieldContent#class#ffff_1023#join_rhs AS R ON FIRST 1 OUTPUT r8.<4>, r8.<1>, r8.<2>, r8.<3>, R.<1>, R.<2>, R.<3> 7116087 ~0% {9} r10 = JOIN r9 WITH SSAConstruction::Cached::getUsedInterval#fff@staged_ext AS R ON FIRST 1 OUTPUT r9.<1>, r9.<2>, r9.<3>, r9.<0>, r9.<4>, r9.<5>, r9.<6>, R.<1>, R.<2> 449879 ~4% {9} r11 = SELECT r10 ON r10.<5> <= r10.<7> 193804 ~1% {9} r12 = SELECT r11 ON r11.<5> >= r11.<7> 154980 ~0% {9} r13 = SELECT r12 ON r12.<6> <= r12.<8> 137684 ~0% {9} r14 = SELECT r13 ON r13.<6> >= r13.<8> 137684 ~7% {3} r15 = SCAN r14 OUTPUT r14.<0>, r14.<4>, r14.<2> 751306 ~0% {3} r16 = JOIN r4 WITH Instruction::LoadInstruction::getSourceAddress_dispred#ff AS R ON FIRST 1 OUTPUT R.<1>, r4.<1>, r4.<2> 94306 ~0% {3} r17 = JOIN r16 WITH Instruction::FieldInstruction::getField_dispred#3#ff AS R ON FIRST 1 OUTPUT R.<1>, r16.<1>, r16.<2> 152363 ~6% {3} r18 = JOIN r17 WITH DataFlowPrivate::FieldContent::getField_dispred#ff_10#join_rhs AS R ON FIRST 1 OUTPUT r17.<1>, R.<1>, r17.<2> 290047 ~22% {3} r19 = r15 \/ r18 return r19

MathiasVP · 2020-09-02T14:20:03Z

Oh, you're right. They are! I thought that was the reason we caught https://github.com/github/codeql/blob/main/cpp/ql/test/library-tests/dataflow/dataflow-tests/lambdas.cpp#L46 before I added the Class column, but not afterwards. I'll do a more thorough investigation of why we lose that new result after the changes in 1e13a39 and 9de570b.

Oops! I understand why we're losing that result (along with the other new ones) now. The flow isn't through a lambda-captured variable, but rather through a reference parameter that gets assigned in the body of the lambda. It's the same reason we lose the result in 3cca74e#diff-17a3dcef69a5018db3bc4790a32f25b0:

The assignment was recognized as a storeStep because FieldContent only required the [begin, end) offset, so the assignment didn't have to be to a field. But once we introduced the Class column in 9de570b we lose that since it's not an assignment to a field.

MathiasVP · 2020-09-02T14:28:01Z

I've started a (hopefully final) CPP-difference that should show that performance is fine now. The internal PR should also be up-to-date.

It would be great if you could comment on 1e13a39 and 91a2309, @dbartol.

MathiasVP · 2020-09-04T15:28:55Z

Oops. I was too trigger happy with my rebase. Sorry! My previous join order fix didn't preserve the behavior, so I've rebased to the correct solution now. It turned out that the changes in the internal tests were caused by multiple toString results in FieldContent, which was fixed by c679516. So I've removed the internal dependency label.

Sorry for the annoyingly long git conversation this has caused 😢. Anyway, assuming the CPP-differences comes back without performance regressions this should be good to go.

MathiasVP · 2020-09-05T18:46:11Z

Sorry for the annoyingly long git conversation this has caused 😢. Anyway, assuming the CPP-differences comes back without performance regressions this should be good to go.

The CPP-difference showed no performance regressions! 🎉

jbj

Otherwise LGTM.

jbj · 2020-09-07T13:35:27Z

cpp/ql/src/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivate.qll

-  override predicate hasLocationInfo(string path, int sl, int sc, int el, int ec) {
-    f.getLocation().hasLocationInfo(path, sl, sc, el, ec)
-  }
+  Field getField() { result = getField(c, startBit, endBit) }


I suggest renaming these two getField predicates to getAField so it's clear to callers there might be multiple results.

Good point. I've done this in 9c1837e. I also changed the qldoc to reflect that there might be multiple results.

jbj · 2020-09-07T13:57:13Z

cpp/ql/test/library-tests/dataflow/taint-tests/taint.cpp


 	sink(x); // clean [FALSE POSITIVE]
-	sink(y); // tainted
+	sink(y); // tainted [NOT DETECTED by IR]


Do we have an internal issue for recovering these cases of lost flow?

I don't think we do. I think that's a great idea to have, though!
Note that this wasn't lost due to this PR. On main we don't catch this flow using the IR, and I just updated the comment to reflect that.

jbj

LGTM. I'll leave the final review/merge to @dbartol.

dbartol

The cleaned-up IR changes look good. LGTM.

jbj · 2020-09-11T10:20:04Z

There's a conflict in a test output file.

MathiasVP · 2020-09-12T11:54:47Z

Note that the only check that's not passing is one that's broken on all PRs. (So this PR should be safe to merge.)

MathiasVP added 3 commits August 26, 2020 15:50

C++: Demonstrate lack of flow when taking the address of a field and …

2a8ee90

…loading it afterwards

C++: Use the information provided by the IR alias analysis to detect …

9d9c78c

…dataflow read and store steps.

C++: Accept test changes

e4807c0

MathiasVP added the C++ label Aug 26, 2020

MathiasVP marked this pull request as ready for review August 26, 2020 14:04

MathiasVP requested a review from a team as a code owner August 26, 2020 14:04

MathiasVP added 3 commits August 26, 2020 16:14

C++: Keep ExplicitFieldStoreQualifierNode private

dd8984d

C++: Accept test changes in query tests

d900a70

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

1221165

jbj reviewed Aug 27, 2020

View reviewed changes

MathiasVP added the depends on internal PR This PR should only be merged in sync with an internal Semmle PR label Aug 27, 2020

MathiasVP added 4 commits September 1, 2020 10:53

C++: Add getUpdatedInterval predicate to ChiInstructions, and getUsed…

1e13a39

…Interval predicate to NonPhiMemoryOperands.

C++: Use the newly added predicates in field flow. This commit also a…

9de570b

…dds a Class column to the FieldContent branch so FieldContent has a pretty toString implementation again.

C++: Accept test changes

3cca74e

C#: Sync identical files

91a2309

MathiasVP closed this Sep 1, 2020

MathiasVP reopened this Sep 1, 2020

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

472363b

MathiasVP requested a review from a team as a code owner September 1, 2020 09:26

MathiasVP added 2 commits September 2, 2020 13:42

C++: Accept test changes

7d00b49

MathiasVP force-pushed the mathiasvp/read-step-without-memory-operands branch from 6412503 to 5546830 Compare September 2, 2020 12:05

MathiasVP added 2 commits September 2, 2020 14:21

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

69c1ead

C++: Make FieldContent private again

7f5f6b1

MathiasVP removed request for a team, felicitymay, hubwriter, jf205, mchammer01 and shati-patel September 4, 2020 15:19

MathiasVP force-pushed the mathiasvp/read-step-without-memory-operands branch from 5e5d64a to c679516 Compare September 4, 2020 15:21

MathiasVP removed JS depends on internal PR This PR should only be merged in sync with an internal Semmle PR documentation labels Sep 4, 2020

github-actions bot added the C# label Sep 4, 2020

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

ed7e499

MathiasVP added 2 commits September 4, 2020 18:04

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

208b85c

C++: Don't import internals.

a5ac8eb

jbj reviewed Sep 7, 2020

View reviewed changes

C++: Rename getField to getAField following review comments.

9c1837e

MathiasVP force-pushed the mathiasvp/read-step-without-memory-operands branch from 2889821 to 9c1837e Compare September 7, 2020 14:52

jbj previously approved these changes Sep 8, 2020

View reviewed changes

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

ad602b8

MathiasVP dismissed jbj’s stale review via ad602b8 September 9, 2020 14:18

dbartol previously approved these changes Sep 9, 2020

View reviewed changes

Merge branch 'main' into mathiasvp/read-step-without-memory-operands

2d57abd

MathiasVP dismissed dbartol’s stale review via 2d57abd September 11, 2020 10:47

jbj approved these changes Sep 14, 2020

View reviewed changes

jbj merged commit 021aa64 into github:main Sep 14, 2020

Conversation

MathiasVP commented Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbj left a comment

Choose a reason for hiding this comment

Uh oh!

jbj Aug 27, 2020

Choose a reason for hiding this comment

Uh oh!

MathiasVP Aug 27, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MathiasVP commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbj commented Sep 1, 2020

Uh oh!

MathiasVP commented Sep 1, 2020

Uh oh!

MathiasVP commented Sep 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Sep 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Sep 5, 2020

Uh oh!

jbj left a comment

Choose a reason for hiding this comment

Uh oh!

jbj Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

MathiasVP Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

jbj Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

MathiasVP Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

jbj left a comment

Choose a reason for hiding this comment

Uh oh!

dbartol left a comment

Choose a reason for hiding this comment

Uh oh!

jbj commented Sep 11, 2020

Uh oh!

MathiasVP commented Sep 12, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MathiasVP commented Aug 26, 2020 •

edited

Loading

MathiasVP commented Aug 26, 2020 •

edited

Loading

MathiasVP commented Sep 1, 2020 •

edited

Loading

MathiasVP commented Sep 2, 2020 •

edited

Loading

MathiasVP commented Sep 2, 2020 •

edited

Loading

MathiasVP commented Sep 4, 2020 •

edited

Loading