C++: Parameterize the semantic range analysis #11728

rdmarsh2 · 2022-12-16T16:00:49Z

This PR makes the semantic range analysis into a parameterized module, with parameters for the type of deltas, the Bound type, the language-specific predicates in RangeAnalysisSpecific.qll, and the utility predicates in RangeUtil.qll. ModulusAnalysis and SignAnalysis are also parameterized, but are instantiated by the main range analysis module. The range analysis is not yet parameterized over the remainder of the Semantic module. This change should be behavior-preserving - the parameterized module is in the new RangeAnalysisStage.qll file, with an instantiation in RangeAnalysis.qll

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

MathiasVP · 2022-12-19T16:28:04Z

This looks great. Looking forward to seeing this PR pulled out of draft!

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysis.qll

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll

MathiasVP

This looks great! If DCA is happy (make sure to run the memory-corruption query pack) then I think this should be pretty safe to merge

MathiasVP · 2022-12-19T18:12:38Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysis.qll

+    SemReason reason
+  ) {
+    not Specific::ignoreExprBound(e) and
+    //not Specific::mayOverflow(e, upper.booleanNot()) and


I'm guessing this is a subtle hint to the next PR? 😄

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

MathiasVP · 2022-12-19T18:12:38Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll

+  predicate mayOverflow(SemExpr e, boolean positively) {
+    exists(IR::Instruction i |
+      i = e and
+      (
+      positively = true and
+      Simple::exprMightOverflowPositively(i.getConvertedResultExpression())
+      or
+      positively = false and
+      Simple::exprMightOverflowNegatively(i.getConvertedResultExpression())
+      )
+    )
+  }
+}


Did you intent to commit this?

MathiasVP · 2022-12-19T18:12:38Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll

@@ -3,86 +3,105 @@
 */

 private import experimental.semmle.code.cpp.semantic.Semantic
+private import RangeAnalysisStage
+private import semmle.code.cpp.ir.IR as IR
+private import semmle.code.cpp.rangeanalysis.SimpleRangeAnalysis as Simple


Did you intent to import SimpleRangeAnalysis here?

MathiasVP · 2022-12-19T18:12:38Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

+  private module SignAnalysisInstantiated = SignAnalysis<D, UtilParam>; // TODO: will this cause reevaluation if it's instantiated with the same DeltaSig and UtilParam multiple times?
+
+  private import SignAnalysisInstantiated
+
+  private module ModulusAnalysisInstantiated = ModulusAnalysis<D, UtilParam>; // TODO: will this cause reevaluation if it's instantiated with the same DeltaSig and UtilParam multiple times?


A note for anyone else reading this: We discussed in our 1:1 that we believe this should not cause re-evaluation because modules are applicative. But we should still make sure that we don't get re-evaluation due to different optimisation decisions.

MathiasVP · 2022-12-19T18:12:38Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

@@ -157,7 +186,7 @@ module RangeStage<RangeSig RangeParam> {
     */
    cached
    predicate possibleReason(SemGuard guard) {
-      guard = boundFlowCond(_, _, _, _, _) or guard = semEqFlowCond(_, _, _, _, _)
+      guard = boundFlowCond(_, _, _, _, _) or guard = UtilParam::semEqFlowCond(_, _, _, _, _)


Could we privately import UtilParam inside the RangeStage module to avoid prefixing with UtilParam? (And similarly for the other module parameters.)

What was the conclusion here? Is this possible, or not? Or is this not something we want?

It's done now - see this commit

Not in ModulusAnalysis, where I'd also expect it.

Looking at this again. I'm somewhat split on this. On the one hand, having the imports makes the code more compact. On the other hand, without the imports it's clearer where things are coming from. So I'm fine with leaving things as-is in the modulus analysis.

MathiasVP · 2022-12-19T18:12:38Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/ModulusAnalysis.qll

@@ -146,7 +146,7 @@ module ModulusAnalysis<DeltaSig D, UtilSig<D> U> {
  private predicate phiSelfModulus(
    SemSsaPhiNode phi, SemSsaVariable inp, SemSsaReadPositionPhiInputEdge edge, int mod
  ) {
-    exists(SemSsaBound phibound, int v, int m |
+    exists(Bounds::SemSsaBound phibound, int v, int m |


Again, it would be awesome if we could just import Bounds to avoid all these prefixes 🙏.

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/SimpleRangeAnalysisWrapper.qll

rdmarsh2 · 2023-01-03T17:31:56Z

Rebased away the overflow-related changes. I'll import the module params as well.

github-code-scanning

CodeQL found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/SignAnalysisCommon.qll

-class UseSignExpr extends FlowSignExpr {
-  SemSsaVariable v;
+  /** A binary expression whose sign is computed from the signs of its operands. */
+  private class BinarySignExpr extends FlowSignExpr {


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/SignAnalysisCommon.qll

-private class UnarySignExpr extends FlowSignExpr {
-  SemUnaryExpr unary;
+  /** A unary expression whose sign is computed from the sign of its operand. */
+  private class UnarySignExpr extends FlowSignExpr {


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/SignAnalysisCommon.qll

+   * A `Convert`, `Box`, or `Unbox` expression, whose sign is computed based on
+   * the sign of its operand and the source and destination types.
+   */
+  abstract private class CastSignExpr extends FlowSignExpr {


jketema · 2023-01-04T14:13:56Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeUtils.qll

+      not Lang::ignoreSsaReadArithmeticExpr(result)
+    )
+    or
+    exists(SemSubExpr sub, float d1, SemConstantIntegerExpr c |


Why is this suddenly float and not D::Delta?

It needs to be either float or int so that we can do the addition on line 29

So when should I expect something from DeltaSig and when should I expect and int or a float? And, what determines the choice between int and float?

The answers to both these questions should probably be documented as part of the code.

Discussed in our sync: prefer delta, but unwrap when needing to do an arithmetic operation. Prefer unwrapping to float, except in the modulus analysis where we unwrap to int (so we can do the modulus analysis).

The answers to both these questions should probably be documented as part of the code.

I don't think this is necessary, but a qldoc string on DeltaSig is still appreciated.

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

jketema · 2023-01-06T14:53:34Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeUtils.qll

+      result = sub and
+      sub.getLeftOperand() = semSsaRead(v, D::fromFloat(d1)) and
+      sub.getRightOperand() = c and
+      delta = D::fromFloat(d1 + c.getIntValue()) and


Discussed in sync: I was somewhat confused here, as the previous case is more "delta"-heavy. To match that I would expect the following here:

delta = D::fromFloat(D::toFloat(d1) + c.getIntValue()) and

(with the declaration of d1 changed accordingly).

jketema

One final question below, otherwise this looks like a pretty straightforward parameterization.

QLDoc checks are currently failing, so that probably still needs to be resolved.

I'm not sure I'm looking at the correct DCA experiment, but the last one seems to show a quite significant slowdown.

jketema · 2023-01-06T15:37:42Z

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/SimpleRangeAnalysisWrapper.qll

@@ -0,0 +1,25 @@
+import cpp


Nothing from this file is being used elsewhere yet. Do I see that correctly?

Oh, didn't mean to include that here.

MathiasVP

A couple of comments. Otherwise this LGTM! We should also do another DCA run to make sure that the crazy performance regressions are gone.

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/FloatDelta.qll

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

This makes the modulus analysis and sign analysis into parameterized modules which are instantiated in the main range analysis module, and makes RangeAnalysisSpecific and RangeUtils into parameters to the main range analysis. Some classes also need to be moved and made into `instanceof` extensions because they'd otherwise be extending across parameterized module boundaries.

These now instantiate their respective parameterized modules. No results change.

MathiasVP · 2023-01-13T09:59:58Z

@rdmarsh2 I tried to fix the merge conflict so that I could start another DCA run on this, but it's not quite trivial enough for me to solve it. Do you mind taking a look at it?

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll

@@ -3,86 +3,91 @@
 */

 private import experimental.semmle.code.cpp.semantic.Semantic
+private import RangeAnalysisStage
+private import experimental.semmle.code.cpp.semantic.analysis.FloatDelta
+private import experimental.semmle.code.cpp.semantic.analysis.IntDelta


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll

@@ -3,86 +3,91 @@
 */

 private import experimental.semmle.code.cpp.semantic.Semantic
+private import RangeAnalysisStage
+private import experimental.semmle.code.cpp.semantic.analysis.FloatDelta
+private import experimental.semmle.code.cpp.semantic.analysis.IntDelta


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

+   * - `upper = false` : `inp >= b + delta`
+   */
+  pragma[noinline]
+  private predicate boundedPhiCand(


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

+            if semNegative(x)
+            then upper = true and delta = D::fromInt(0)
+            else none()


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

+            if semNegative(x)
+            then upper = false and delta = D::fromInt(0)
+            else none()


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

Conflicting change to boundedPhiInp copied to RangeAnalysisStage.qll

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll

+   * - `upper = false` : `inp >= b + delta`
+   */
+  pragma[noinline]
+  private predicate boundedPhiCand(


cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll

…eAnalysisSpecific.qll

MathiasVP · 2023-01-18T14:22:16Z

DCA looks good, and the CI failure is unrelated to this PR (it would be fixed by merging in main). But I think it's time to merge this 🎉.

github-actions bot added the C++ label Dec 16, 2022

github-code-scanning bot found potential problems Dec 16, 2022

View changes

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll Fixed Show fixed Hide fixed

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll Fixed Show fixed Hide fixed

rdmarsh2 force-pushed the rdmarsh2/parameterize-range-analysis branch from 3774db7 to 452d438 Compare Dec 19, 2022

rdmarsh2 marked this pull request as ready for review Dec 19, 2022

rdmarsh2 requested a review from a team as a code owner Dec 19, 2022

github-code-scanning bot found potential problems Dec 19, 2022

View changes

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysis.qll Fixed Show fixed Hide fixed

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll Fixed Show fixed Hide fixed

rdmarsh2 force-pushed the rdmarsh2/parameterize-range-analysis branch from d2d62ad to 35ecf55 Compare Dec 19, 2022

MathiasVP reviewed Dec 19, 2022

View changes

github-code-scanning bot found potential problems Dec 20, 2022

View changes

rdmarsh2 force-pushed the rdmarsh2/parameterize-range-analysis branch from a985ccf to a7e5e55 Compare Jan 3, 2023

github-code-scanning bot found potential problems Jan 3, 2023

View changes

rdmarsh2 force-pushed the rdmarsh2/parameterize-range-analysis branch from be5ff77 to 359b428 Compare Jan 3, 2023

github-code-scanning bot found potential problems Jan 3, 2023

View changes

jketema reviewed Jan 4, 2023

View changes

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll Show resolved Hide resolved

jketema reviewed Jan 6, 2023

View changes

MathiasVP reviewed Jan 11, 2023

View changes

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/FloatDelta.qll Outdated Show resolved Hide resolved

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisStage.qll Outdated Show resolved Hide resolved

rdmarsh2 added 10 commits Jan 12, 2023

Convert RangeAnalysis to trivial parameterized mod

edbe958

C++: rename to RagneAnalysisStage.qll

eebada4

C++: convert RangeAnalysis to float

b8c43d7

C++: fix float binding issue in range analysis

c10733f

C++: move language specific predicates to LangParam

c062d5e

C++: Make RangeAnalysis.qll expose the old API

71b93d1

C++: make SemBound a RangeAnalysis parameter

02f1957

C++: autoformat

6db7281

C++: Make bounds import private to preserve API

2328141

rdmarsh2 added 6 commits Jan 12, 2023

C++: private import for module params

488368e

C++: fix ambiguous import warnings

7586762

C++: update test QL for modulus and sign analysis

938176c

These now instantiate their respective parameterized modules. No results change.

C++: fix a join order in range analysis

31b61b1

C++: use rounding to prevent float wobble in range analysis

b2b4523

C++: cleanup some unneeded code

337a747

rdmarsh2 force-pushed the rdmarsh2/parameterize-range-analysis branch from c93c804 to 337a747 Compare Jan 12, 2023

github-code-scanning bot found potential problems Jan 13, 2023

View changes

Merge branch 'main' into rdmarsh2/parameterize-range-analysis

601b43a

Conflicting change to boundedPhiInp copied to RangeAnalysisStage.qll

rdmarsh2 force-pushed the rdmarsh2/parameterize-range-analysis branch from 1969c9a to 601b43a Compare Jan 13, 2023

github-code-scanning bot found potential problems Jan 13, 2023

View changes

MathiasVP reviewed Jan 17, 2023

View changes

cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/RangeAnalysisSpecific.qll Outdated Show resolved Hide resolved

Update cpp/ql/lib/experimental/semmle/code/cpp/semantic/analysis/Rang…

44ebc77

…eAnalysisSpecific.qll

MathiasVP approved these changes Jan 18, 2023

View changes

MathiasVP merged commit e26e83b into main Jan 18, 2023
12 of 13 checks passed

MathiasVP deleted the rdmarsh2/parameterize-range-analysis branch Jan 18, 2023

C++: Parameterize the semantic range analysis #11728

C++: Parameterize the semantic range analysis #11728

rdmarsh2 commented Dec 16, 2022 •

edited

MathiasVP commented Dec 19, 2022

MathiasVP left a comment

MathiasVP Dec 19, 2022

MathiasVP Dec 19, 2022

MathiasVP Dec 19, 2022

MathiasVP Dec 19, 2022

MathiasVP Dec 19, 2022

jketema Jan 4, 2023 •

edited

rdmarsh2 Jan 4, 2023

jketema Jan 4, 2023

jketema Jan 6, 2023

MathiasVP Dec 19, 2022

rdmarsh2 commented Jan 3, 2023

github-code-scanning bot left a comment

jketema Jan 4, 2023

rdmarsh2 Jan 4, 2023

jketema Jan 5, 2023

jketema Jan 6, 2023

jketema Jan 6, 2023

jketema left a comment •

edited

jketema Jan 6, 2023

rdmarsh2 Jan 12, 2023

MathiasVP left a comment

MathiasVP commented Jan 13, 2023

MathiasVP commented Jan 18, 2023

C++: Parameterize the semantic range analysis #11728

C++: Parameterize the semantic range analysis #11728

Conversation

rdmarsh2 commented Dec 16, 2022 • edited

MathiasVP commented Dec 19, 2022

MathiasVP left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jketema Jan 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdmarsh2 commented Jan 3, 2023

github-code-scanning bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jketema left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MathiasVP left a comment

Choose a reason for hiding this comment

MathiasVP commented Jan 13, 2023

MathiasVP commented Jan 18, 2023

rdmarsh2 commented Dec 16, 2022 •

edited

jketema Jan 4, 2023 •

edited

jketema left a comment •

edited