C++: IR translation for static local initializers by MathiasVP · Pull Request #12900 · github/codeql

MathiasVP · 2023-04-21T13:37:25Z

This PR adds IR for the static initializers of static locals. On main, we already generate IR for runtime initializers static locals. For example, given this code:

int get_my_data();

void foo() {
  static int n = get_my_data();
  use(n);
}

we generate IR corresponding to this snippet:

int get_my_data();
bool n_has_been_initialized;

void foo() {
  static int n;
  if(!n_has_been_initialized) {
    n = get_my_data();
    n_has_been_initialized = true;
  }
  use(n);
}

but for static initialized static locals no IR was generated. This PR adds such initialization. So given code like:

void foo() {
  static int n = 42;
  use(n);
}

we generate IR corresponding to:

static int n;

void initialize_n() {
  n = 42;
}

void foo() {
  use(n);
}

Meaning that static initialized static locals are now handled exactly like we handle initialization of global variables (which are always static initialized).

Once this has been merged, I'll add the necessary dataflow steps to make sure that we track dataflow appropriately.

Wireshark seems to have some very very large static-local initializers (such as this 44k LoC initializer). This is what's causing a large slowdown in IR generation. It's not like there's any bad performance now. We really just generate twice as much IR as before.

I did a QA run across 3000 projects, and no other project seems to suffer from the same problem. We could disable IR generation for static initializers in auto generated files (since the initializer in the file above is very clearly in an autogenerated file), but after discussing this with @rdmarsh2 we decided it wasn't worth adding this heuristic into the IR.

MathiasVP · 2023-04-24T16:32:16Z

I don't think we should add a change note for this until we've added the dataflow component as well.

cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/IRConstruction.qll

jketema · 2023-04-26T14:29:46Z

Wireshark DCA results show quite a significant increase on nonUniqueEnclosingIRFunction. Is that expected?

jketema · 2023-04-26T14:35:42Z

Looking at the DCA results, the performance of kamailio also seems to have regressed somewhat. Do we know why?

MathiasVP · 2023-04-26T14:46:56Z

Wireshark DCA results show quite a significant increase on nonUniqueEnclosingIRFunction. Is that expected?

Ouch. I missed that one completely (in my defence, it's all the way to the right of the window 😂). I'll take a look at this.

MathiasVP · 2023-04-26T15:03:41Z

Looking at the DCA results, the performance of kamailio also seems to have regressed somewhat. Do we know why?

Yeah, I think I know why now 😄: https://gist.github.com/MathiasVP/9a947427b9f4f37ad025efed208554f9. Will fix!

Edit: Fixed in 60aab20

Before: ``` Evaluated non-recursive predicate TranslatedElement#ea057665::TranslatedElement::getInstructionVariable#1#dispred#fff@146210id in 201548ms (size: 3469729). Evaluated relational algebra for predicate TranslatedElement#ea057665::TranslatedElement::getInstructionVariable#1#dispred#fff@146210id with tuple counts: ... 1812768 ~3% {3} r65 = JOIN num#InstructionTag#c9183db3::OnlyInstructionTag#f WITH TranslatedExpr#043317a1::TranslatedNonFieldVariableAccess#ff CARTESIAN PRODUCT OUTPUT Rhs.1, Lhs.0, Rhs.0 1812767 ~0% {4} r66 = JOIN r65 WITH Access#8878f617::Access::getTarget#0#dispred#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.2, Lhs.0 3996407117 ~3% {5} r67 = JOIN r66 WITH TranslatedElement#ea057665::getIRUserVariable#2#fff_102#join_rhs ON FIRST 1 OUTPUT Lhs.3, Rhs.1, Lhs.1, Lhs.2, Rhs.2 1815194 ~0% {3} r68 = JOIN r67 WITH TranslatedExpr#043317a1::getEnclosingDeclaration#1#ff ON FIRST 2 OUTPUT Lhs.3, Lhs.2, Lhs.4 ... ``` After: ``` Evaluated non-recursive predicate TranslatedExpr#043317a1::accessHasEnclosingDeclarationAndVariable#3#fff@665ccb8o in 865ms (size: 2769549). Evaluated relational algebra for predicate TranslatedExpr#043317a1::accessHasEnclosingDeclarationAndVariable#3#fff@665ccb8o with tuple counts: 2769549 ~1% {3} r1 = JOIN Access#8878f617::Access::getTarget#0#dispred#ff WITH TranslatedExpr#043317a1::getEnclosingDeclaration#1#ff ON FIRST 1 OUTPUT Rhs.1, Lhs.1, Lhs.0 return r1 ... Evaluated non-recursive predicate TranslatedElement#ea057665::TranslatedElement::getInstructionVariable#1#dispred#fff@7d4d33to in 805ms (size: 3469729). Evaluated relational algebra for predicate TranslatedElement#ea057665::TranslatedElement::getInstructionVariable#1#dispred#fff@7d4d33to with tuple counts: ... 1963209 ~0% {2} r34 = JOIN TranslatedElement#ea057665::getIRUserVariable#2#fff WITH TranslatedExpr#043317a1::accessHasEnclosingDeclarationAndVariable#3#fff ON FIRST 2 OUTPUT Rhs.2, Lhs.2 1815194 ~2% {2} r35 = JOIN r34 WITH TranslatedExpr#043317a1::TranslatedNonFieldVariableAccess#ff_10#join_rhs ON FIRST 1 OUTPUT Lhs.1, Rhs.1 1815194 ~0% {3} r36 = JOIN r35 WITH num#InstructionTag#c9183db3::OnlyInstructionTag#f CARTESIAN PRODUCT OUTPUT Lhs.1, Rhs.0, Lhs.0 ... ```

MathiasVP · 2023-04-26T16:50:52Z

Wireshark DCA results show quite a significant increase on nonUniqueEnclosingIRFunction. Is that expected?

Ouch. I missed that one completely (in my defence, it's all the way to the right of the window 😂). I'll take a look at this.

I can't say that I understand the issue yet, but they all seem to reduce to the testcase I've added in 1dcac76. The large number of new inconsistencies is simply because it appears so many times in those very large static initializers. The printed IR is also very strange (which I guess is expected given inconsistencies):

 7| void regression_test()
 7|   Block 0
 7|     v7_1(void)    = EnterFunction      :
 7|     m7_2(unknown) = AliasedDefinition  :
 7|     m7_3(unknown) = InitializeNonLocal :
 7|     m7_4(unknown) = Chi                : total:m7_2, partial:m7_3
10|     v10_1(void)   = NoOp               :
 7|     v7_5(void)    = ReturnVoid         :
 7|     v7_6(void)    = AliasedUse         : m7_3
 7|     v7_7(void)    = ExitFunction       :

 9| const void *[] a
 9|   Block 0
 9|     v9_1(void)            = EnterFunction      :
 9|     m9_2(unknown)         = AliasedDefinition  :
 9|     r9_3(glval<void *[]>) = VariableAddress[a] :
 9|     r9_4(int)             = Constant[0]        :
 9|     r9_5(glval<void *>)   = PointerAdd[8]      : r9_3, r9_4
 9|     r9_6(int)             = Constant[0]        :
  |   False -> Block 1

 9|   Block 1
 9|     r9_7(glval<_A[]>)  = VariableAddress[as]        :
 9|     r9_8(_A *)         = Convert                    : r9_7
 9|     r9_9(glval<_A *>)  = VariableAddress[#temp9:31] :
 9|     m9_10(_A *)        = Store[#temp9:31]           : &:r9_9, r9_8
 9|     r9_11(glval<_A *>) = VariableAddress[#temp9:31] :
 9|     r9_12(_A *)        = Load[#temp9:31]            : &:r9_11, m9_10
 9|     r9_13(void *)      = Convert                    : r9_12
 9|     m9_14(void *)      = Store[?]                   : &:r9_5, r9_13
 9|     m9_15(unknown)     = Chi                        : total:m9_2, partial:m9_14
 9|     v9_16(void)        = ReturnVoid                 :
 9|     v9_17(void)        = AliasedUse                 : ~m9_15
 9|     v9_18(void)        = ExitFunction               :

The search continues!

Edit: Fixed in b18e096 🎉

MathiasVP · 2023-04-26T17:03:46Z

cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/TranslatedInitialization.qll

  override TranslatedElement getChild(int id) { none() }

-  override Function getFunction() { result = getParent().getFunction() }
+  override Declaration getFunction() { result = getParent().getFunction() }


This wasn't actually necessary to fix the inconsistency, but it was the only remaining getFunction implementation where:

The return type was specified to be Function, and

where result value wasn't obviously a Function.

cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/TranslatedInitialization.qll

MathiasVP · 2023-04-27T07:59:22Z

@jketema DCA looks good now: a fairly large reduction in inconsistencies, and the slowdown on Kamailio is gone. So now we're just left with the Wireshark slowdown (which we understand, and are okay with).

jketema

LGTM

github-actions bot added C# C++ labels Apr 21, 2023

MathiasVP force-pushed the ir-translate-constant-static-local-vars-2 branch 2 times, most recently from 2c4d90c to 9f22a10 Compare April 24, 2023 16:16

MathiasVP marked this pull request as ready for review April 24, 2023 16:30

MathiasVP requested review from a team as code owners April 24, 2023 16:30

MathiasVP added the no-change-note-required This PR does not need a change note label Apr 24, 2023

MathiasVP force-pushed the ir-translate-constant-static-local-vars-2 branch from 9f22a10 to 7cec237 Compare April 25, 2023 15:42

MathiasVP added 4 commits April 25, 2023 17:51

C++: IR translation for non-runtime-initialized static local variables.

d14ee93

C++: Accept test changes.

9cc4bfe

C++: Fix enclosing functions for static locals.

648c08b

C++: Accept test changes.

3f03cc2

MathiasVP force-pushed the ir-translate-constant-static-local-vars-2 branch from 7cec237 to 3f03cc2 Compare April 25, 2023 16:51

MathiasVP requested a review from rdmarsh2 April 25, 2023 21:07

jketema reviewed Apr 26, 2023

View reviewed changes

cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/IRConstruction.qll Show resolved Hide resolved

C++: Add a test with global variable templates.

3d38133

MathiasVP force-pushed the ir-translate-constant-static-local-vars-2 branch from 8e7648d to 60aab20 Compare April 26, 2023 15:33

C++: Add a weird testcase demonstrating invalid IR.

1dcac76

C++: Fix missing result for 'getFunction' and accept test changes.

b18e096

MathiasVP commented Apr 26, 2023

View reviewed changes

github-advanced-security bot found potential problems Apr 26, 2023

View reviewed changes

cpp/ql/lib/semmle/code/cpp/ir/implementation/raw/internal/TranslatedInitialization.qll Fixed Show fixed Hide fixed

C++: Fix implicit 'this'.

6bfdbef

jketema approved these changes Apr 28, 2023

View reviewed changes

MathiasVP merged commit 5f4d089 into github:main Apr 28, 2023

MathiasVP mentioned this pull request Apr 28, 2023

C++: Dataflow for static local variables #12966

Merged

MathiasVP mentioned this pull request May 17, 2023

C++: Reduce memory pressure from getInstruction #13207

Merged

MathiasVP mentioned this pull request Jun 27, 2023

C++: Add barriers to cpp/invalid-pointer-deref #13559

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C++: IR translation for static local initializers#12900

C++: IR translation for static local initializers#12900
MathiasVP merged 9 commits intogithub:mainfrom
MathiasVP:ir-translate-constant-static-local-vars-2

MathiasVP commented Apr 21, 2023 •

edited

Loading

Uh oh!

MathiasVP commented Apr 24, 2023

Uh oh!

Uh oh!

jketema commented Apr 26, 2023 •

edited

Loading

Uh oh!

jketema commented Apr 26, 2023

Uh oh!

MathiasVP commented Apr 26, 2023

Uh oh!

MathiasVP commented Apr 26, 2023 •

edited

Loading

Uh oh!

MathiasVP commented Apr 26, 2023 •

edited

Loading

Uh oh!

MathiasVP Apr 26, 2023

Uh oh!

Uh oh!

MathiasVP commented Apr 27, 2023

Uh oh!

jketema left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MathiasVP commented Apr 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Apr 24, 2023

Uh oh!

Uh oh!

jketema commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jketema commented Apr 26, 2023

Uh oh!

MathiasVP commented Apr 26, 2023

Uh oh!

MathiasVP commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathiasVP Apr 26, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MathiasVP commented Apr 27, 2023

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MathiasVP commented Apr 21, 2023 •

edited

Loading

jketema commented Apr 26, 2023 •

edited

Loading

MathiasVP commented Apr 26, 2023 •

edited

Loading

MathiasVP commented Apr 26, 2023 •

edited

Loading