C++: Fix more conflation in dataflow #13425

MathiasVP · 2023-06-09T14:36:11Z

This PR fixes two conflation issues that were giving us a bunch of FPs on https://github.com/lief-project/lief.

The first fix is very simple (see 153df2c), and the second fix took a couple of days of debugging (see 7c32721) 😂.

Commit-by-commit review recommended.

MathiasVP · 2023-06-09T14:52:23Z

cpp/ql/test/library-tests/dataflow/dataflow-tests/test.cpp

+void does_not_write_source_to_dereference(int *p) // $ ast-def=p ir-def=*p
+{
+  int x = source();
+  p = &x;
+  *p = 42;
+}
+
+void test_does_not_write_source_to_dereference()
+{
+  int x;
+  does_not_write_source_to_dereference(&x);
+  sink(x); // $ ast,ir=733:7 SPURIOUS: ast,ir=726:11
+}


This was a testcase I came up with as an attempt to further reduce the testcase added in 90ffb45. However, it looks like this is a different conflation problem since test this wasn't fixed by this PR.

To be clear: This isn't a regression caused by this PR. I just tagged it along here to avoid a merge conflict with this PR.

jketema · 2023-06-09T20:35:18Z

We seem to be missing some taint flow now (see missing DCA results).

jketema

Some minor comments

cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivate.qll

cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowUtil.qll

jketema · 2023-06-12T09:12:50Z

DCA nightly suite result changes:

Changes in SAMATE: some results disappear that should have been blocked by isBarrierIn in cpp/cleartext-transmission, but weren't.
The vim results losses for cpp/command-line-injection and cpp/path-injection seem genuine lost results. It's currently not clear to me why these are lost.
The new apache/httpd results look like genuine results.

DCA MCTV suite result changes:

Erlang OTP changes were FPs that now disappeared. Previously, at some point in the path is started confusing the allocated array with a constant one.
vim cpp/constant-array-overflow: we lose all but 69 results. It seems we might have broken something related to global variables?

MathiasVP · 2023-06-12T12:51:12Z

vim cpp/constant-array-overflow: we lose all but 69 results. It seems we might have broken something related to global variables?

That would explain the large performance improvement on vim 😅

jketema · 2023-06-12T12:57:13Z

vim cpp/constant-array-overflow: we lose all but 69 results. It seems we might have broken something related to global variables?

That would explain the large performance improvement on vim 😅

Question is what. The vim dataflow paths are rather horrendous as usual.

MathiasVP · 2023-06-12T13:05:34Z

If you want to get started on debugging this while I'm away I suggest:

Pick the simplest possible path we lose
Make the paths as explicit as possible by setting castnode to any and hidden nodes to none.
do a "binary search" with a partial flow version of the query to see which step is gone. We already have a hypothesis that this is related to global, so maybe you can make this debugging search slightly more clever
Once we know which step is missing we can do a small test case and debug it like we did last week

1-3 is probably a whole day kind of task already. So if you're too busy with other things feel free to leave it for me to do once I'm back

jketema · 2023-06-12T13:34:59Z

So if you're too busy with other things feel free to leave it for me to do once I'm back

I think I might just give it a go.

jketema · 2023-06-13T13:22:47Z

So if you're too busy with other things feel free to leave it for me to do once I'm back

I think I might just give it a go.

Gave it a go, but wasn't able to make any progress. I tried restricting the source of one of the problematic queries as much possible, and tried making is start later on a path that is disappearing. However, even with this partial flow is basically not computable on my machine.

MathiasVP · 2023-06-18T12:16:17Z

Thanks for all the investigations on this Jeroen! I've pushed a reduced testcase demonstrating the missing flow on vim (turns out our model for strncpy has been broken for who-knows-how-long 😬), and we've been saved because of the accidental conflation that we've now fixed.

Luckily, the fix was super easy. I'll start another DCA run to see what the impact of this is.

jketema · 2023-06-18T20:19:11Z

I don't think the strncpy fix is sufficient. Running cpp/constant-array-overflow with the updated branch on vim, I only get 69 results instead of the 1000s that were there before. Or were those all FPs?

MathiasVP · 2023-06-18T21:13:47Z

Good point. Yeah, the strncpy fix was strictly a taintflow fix, and that query uses only dataflow. So we still need to figure out if those lost results are because we broke something, or because we fixed conflation issues

MathiasVP · 2023-06-19T09:47:03Z

Investigation so far: It looks like we're losing all the vim results that come from the uf_name field. These are indeed all FPs (because it's a flexible array member). But I'm still trying to figure out why we lose them.

MathiasVP · 2023-06-19T10:22:31Z

I've done a spot check of some of the lost results and it does look like the it's caused by the now-fixed conflation 🎉. For example, the flow starts here and then moves to:

s here: https://github.com/vim/vim/blob/a5a1ec1826c0e43d0282ba4d35c155a97bab3e27/src/evalfunc.c#L3632
and then to &name here: https://github.com/vim/vim/blob/a5a1ec1826c0e43d0282ba4d35c155a97bab3e27/src/evalfunc.c#L3644
and then here: https://github.com/vim/vim/blob/a5a1ec1826c0e43d0282ba4d35c155a97bab3e27/src/userfunc.c#L2665
and then to start indirection here: https://github.com/vim/vim/blob/a5a1ec1826c0e43d0282ba4d35c155a97bab3e27/src/userfunc.c#L2705

And at this point we're suddenly tracking the indirection now (instead of the pointer). And after this PR this is no longer happening 🎉.

…s to not allow pointer conflation.

…e.qll Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

MathiasVP · 2023-06-22T14:46:32Z

DCA looks great! We got the lost result on vim back, and we're still seeing a 43% performance improvement 😮. I'll add a testcase that demonstrates the effect of 992af55 (via a force-push), but then I think this PR is good to go 🎉

jketema · 2023-06-22T15:04:55Z

I'll add a testcase

There is also a test regression that need to be looked at.

MathiasVP · 2023-06-22T15:05:58Z

Thanks for the heads up. That looks like 992af55 fixed a missing flow, but I'll double check to verify that.

MathiasVP · 2023-06-22T16:43:37Z

@jketema this PR should be all ready now. I've force-pushed a testcase that demonstrates the missing flow we saw on vim, but otherwise nothing has changed since the DCA run.

The query test change was just some changes to path explanations. So nothing major to see there.

…rect parameters.

MathiasVP requested a review from jketema June 9, 2023 14:36

MathiasVP requested a review from a team as a code owner June 9, 2023 14:36

github-actions bot added the C++ label Jun 9, 2023

MathiasVP added the no-change-note-required This PR does not need a change note label Jun 9, 2023

MathiasVP commented Jun 9, 2023

View reviewed changes

jketema reviewed Jun 10, 2023

View reviewed changes

MathiasVP and others added 12 commits June 22, 2023 10:59

C++: Add failing test.

5816f17

C++: Fix conflation bug in 'getIRRepresentationOfIndirectInstruction'.

9e9c811

C++: Change the API for indirect operands and indirection instruction…

6034eb0

…s to not allow pointer conflation.

C++: Adjust the rest of the library to the new API.

3b0a286

C++: Accept test changes.

6543da9

C++: Ensure that 'PrintIR' for dataflow still compiles.

3365ff0

C++: Add another testcase with conflation.

c7cff37

C++: Add QLDoc to 'hasAddressOperandAndIndirectionIndex'.

6528985

Update cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivat…

a8a04c8

…e.qll Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

Update cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowPrivat…

4f1b2c6

…e.qll Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

Update cpp/ql/lib/semmle/code/cpp/ir/dataflow/internal/DataFlowUtil.qll

ff3c76c

Co-authored-by: Jeroen Ketema <93738568+jketema@users.noreply.github.com>

C++: Add testcase demonstrating that the model for 'strncpy' is broken.

273e5bc

C++: Fix strncpy model.

fe97572

MathiasVP force-pushed the fix-more-conflation-in-dataflow branch from 1ec3eb9 to 992af55 Compare June 22, 2023 09:59

MathiasVP force-pushed the fix-more-conflation-in-dataflow branch from 992af55 to aca4716 Compare June 22, 2023 16:40

MathiasVP added 3 commits June 22, 2023 19:33

C++: Add testcase that demonstrate the need for self-flow out of indi…

da54751

…rect parameters.

C++: Allow self-flow through indirect parameters.

0839c1a

C++: Accept test changes.

79fb6a6

MathiasVP force-pushed the fix-more-conflation-in-dataflow branch from aca4716 to 79fb6a6 Compare June 22, 2023 18:35

jketema approved these changes Jun 22, 2023

View reviewed changes

MathiasVP merged commit b615e98 into github:main Jun 22, 2023

MathiasVP mentioned this pull request Jan 17, 2025

C++: Remove pointer/pointee conflation from parameter nodes #18531

Merged

C++: Fix more conflation in dataflow #13425

C++: Fix more conflation in dataflow #13425

Uh oh!

Conversation

MathiasVP commented Jun 9, 2023

Uh oh!

MathiasVP Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jketema commented Jun 9, 2023

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jketema commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Jun 12, 2023

Uh oh!

jketema commented Jun 12, 2023

Uh oh!

MathiasVP commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jketema commented Jun 12, 2023

Uh oh!

jketema commented Jun 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Jun 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jketema commented Jun 18, 2023

Uh oh!

MathiasVP commented Jun 18, 2023

Uh oh!

MathiasVP commented Jun 19, 2023

Uh oh!

MathiasVP commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Jun 22, 2023

Uh oh!

jketema commented Jun 22, 2023

Uh oh!

MathiasVP commented Jun 22, 2023

Uh oh!

MathiasVP commented Jun 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MathiasVP Jun 9, 2023 •

edited

Loading

jketema commented Jun 12, 2023 •

edited

Loading

MathiasVP commented Jun 12, 2023 •

edited

Loading

jketema commented Jun 13, 2023 •

edited

Loading

MathiasVP commented Jun 18, 2023 •

edited

Loading

MathiasVP commented Jun 19, 2023 •

edited

Loading