Go: Make flow configurations use new data flow API #13820

owen-mc · 2023-07-26T09:29:59Z

I highly recommend reviewing commit by commit (and spreading it out over several sessions). All configurations have tests (normally of the related query). There are no changes in results, though sometimes there are changes in exact PathNodes and edges between them.

go/ql/test/library-tests/semmle/go/dataflow/GuardingFunctions/test.ql

owen-mc · 2023-07-27T06:15:57Z

~~I will rebase on main when #13822 is merged, which will fix the sync files check.~~

Removed edges and labels are mostly duplicates. No alerts are lost.

mbg

Good work updating all of this! ✨ The refactoring generally looks reasonable. There are a few specific comments I had while looking over this and two more broad ones:

For queries where the expected test output changed and you haven't done so yet, it would be good to note why it changed in the commit description or in the PR description.
Most/all of the old data/taint flow configurations had docs comments explaining their purpose. However, the new ConfigSig implementations and data/taint flow configurations mostly do not have matching docs comments. In general, it would be nice to have more documentation/comments.

There also seem to a good few ConfigSigs which are basically the same as a result of the [..]Customizations approach along the lines of:

private module Config implements DataFlow::ConfigSig {
    predicate isSource(DataFlow::Node source) { source instanceof Source }

    predicate isSink(DataFlow::Node sink) { sink instanceof Sink }

    predicate isBarrier(DataFlow::Node node) { node instanceof Barrier }
}

Would it be possible (in the future, not as part of this PR) to refactor this so that we have a generic implementation that we can just parameterise over the module that's providing the Source, Sink, and Barrier?

mbg · 2023-08-01T06:38:00Z

go/ql/test/query-tests/Security/CWE-190/AllocationSizeOverflow.expected

For queries where the expected test output has changed, it would be good to include some discussion in the PR description why this has changed.

It generally isn't obvious why. I know one cause is when there were more than one config in scope in the old version. I'm not sure how much value there is in tracking it down. I'm going to bet that in the corresponding PRs for java and C# they have similar changes and they haven't looked into what caused them. (Of course if there was a change in the results it would be different. It's just because the only change is in path nodes and edges.)

mbg · 2023-08-01T06:41:50Z

go/ql/lib/semmle/go/security/AllocationSizeOverflow.qll

+    predicate isBarrier(DataFlow::Node nd) { nd instanceof Sanitizer }
+  }
+
+  private module FindLargeLensFlow = TaintTracking::Global<FindLargeLensConfig>;


I note that this is private while FindLargeLensConfiguration is not. Is that an intentional change? If so, do we think that marking FindLargeLensConfiguration as deprecated and pointing at FindLargeLensFlow as an alternative makes sense?

Also, more generally, we have documentation for the old taint tracking Configuration classes, but we don't currently have any documentation for the new taint tracking modules. Should we have documentation for each of the new modules?

I have tended to make things private when they don't need to be public. An example of when they need to be public: an additional predicate in the config module is used in the .ql file. (Though it could be moved out of the module and made a top-level predicate.)

FindLargeLensFlow is a helper flow that is only used in this file. I think it is appropriate for it to be private. If someone was using FindLargeLensConfiguration then I guess they should make a copy of FindLargeLensFlow and the things it relies on.

It makes sense that these could have been private to begin with. Although we don't know for sure whether e.g. FindLargeLensConfiguration has been used by anyone externally since it is public, we also don't change its visibility. Marking it as deprecated instead seems fair then, as long as the comment doesn't suggest to use the private replacement instead and notes what you noted in your reply.

mbg · 2023-08-01T07:08:21Z

go/ql/test/library-tests/semmle/go/frameworks/Encoding/jsoniter.expected

How come we get the edges here but didn't before?

mbg · 2023-08-01T07:08:57Z

go/ql/test/library-tests/semmle/go/frameworks/Gorestful/gorestful.expected

How come we get the edges here but didn't before?

mbg · 2023-08-01T07:20:05Z

go/ql/test/query-tests/Security/CWE-918/RequestForgery.expected

Why the differences?

mbg · 2023-08-01T10:36:13Z

go/ql/lib/semmle/go/security/IncorrectIntegerConversionLib.qll

@@ -148,6 +150,115 @@ class ConversionWithoutBoundsCheckConfig extends TaintTracking::Configuration {
  }
 }

+/** Flow state for ConversionWithoutBoundsCheckConfig. */
+newtype MyFlowState =


Should this have a less generic name, e.g. IntegerConversionState?

mbg · 2023-08-01T10:44:09Z

go/ql/test/library-tests/semmle/go/frameworks/Spew/TaintFlows.ql

+    source =
+      any(Function f | f.getName() = ["getUntrustedString", "getUntrustedStruct"])
+          .getACall()
+          .getResult()


I note that the test previously ranged over all of UntrustedFlowSource but now is an inlined UntrustedSource. Do we miss out anything important by not having this range over all of UntrustedFlowSource anymore?

I don't think so - we're just trying to match the two functions used as sources in the test files. I can only imagine was written like this to be more like a real source definition.

mbg · 2023-08-01T10:46:41Z

go/ql/test/library-tests/semmle/go/frameworks/Protobuf/TaintFlows.ql

+    exists(Function fn | fn.hasQualifiedName(_, ["getUntrustedString", "getUntrustedBytes"]) |
+      source = fn.getACall().getResult()
+    )


This previously ranged over all of UntrustedFlowSource, but is now more limited. Do we miss out on anything as a result of this change?

As above, I don't think it matters for tests.

mbg · 2023-08-01T10:48:54Z

go/ql/test/library-tests/semmle/go/frameworks/K8sIoApiCoreV1/main.go

@@ -9,86 +9,88 @@ import (
 //go:generate depstubber -vendor k8s.io/apimachinery/pkg/runtime ProtobufMarshaller,ProtobufReverseMarshaller

 func source() interface{} {
-	return make([]byte, 1, 1)
+	return make([]byte, 1)


What's the reason for this change?

My IDE told me it was better to do it this way. Presumably it's a built-in Go linter of some kind.

mbg · 2023-08-01T10:50:46Z

go/ql/test/library-tests/semmle/go/frameworks/EvanphxJsonPatch/TaintFlows.ql

    )
  }
+
+  int fieldFlowBranchLimit() { result = 1000 }


Could you add a comment noting why you added this here for this particular configuration?

It isn't needed. I've removed it. I must have added it by mistake, probably copying from somewhere else.

owen-mc requested a review from a team as a code owner July 26, 2023 09:30

github-actions bot added the Go label Jul 26, 2023

owen-mc force-pushed the go/refactor-data-flow-configurations branch from 85baa81 to 1d2c268 Compare July 26, 2023 09:34

github-code-scanning bot found potential problems Jul 26, 2023

View reviewed changes

go/ql/test/library-tests/semmle/go/dataflow/GuardingFunctions/test.ql Fixed Show fixed Hide fixed

owen-mc added the no-change-note-required This PR does not need a change note label Jul 26, 2023

owen-mc force-pushed the go/refactor-data-flow-configurations branch from 1d2c268 to 028eb9b Compare July 26, 2023 09:51

owen-mc marked this pull request as draft July 26, 2023 10:25

owen-mc force-pushed the go/refactor-data-flow-configurations branch from 028eb9b to 9919221 Compare July 26, 2023 21:18

github-actions bot added C# C++ Java Python Ruby Swift labels Jul 26, 2023

owen-mc force-pushed the go/refactor-data-flow-configurations branch from 9919221 to 355f9c6 Compare July 27, 2023 06:13

github-actions bot removed C# C++ Java Python Ruby Swift labels Jul 27, 2023

owen-mc marked this pull request as ready for review July 27, 2023 06:14

owen-mc mentioned this pull request Aug 1, 2023

Go: Refactor to use new module API for DataFlow configurations #12835

Closed

owen-mc force-pushed the go/refactor-data-flow-configurations branch from 355f9c6 to d8ed7fe Compare August 1, 2023 06:53

mbg and others added 5 commits August 1, 2023 07:59

Make AllocationSizeOverflow use new API

1aedaf0

Make UntrustedDataToExternalAPI use new API

0badc41

Make UntrustedDataToUnknownExternalAPI use new API

c1f6798

Make CommandInjection.qll use new API

6709b42

Make InsecureRandomness use new API

195b290

owen-mc added 28 commits August 1, 2023 08:00

Make StoredXss use new API

6939e70

Make StringBreak use new API

41773cf

Make TaintedPath use new API

210448f

Make UnsafeUnzipSymlink use new API

28ad7af

Make XPathInjection use new API

fc5c46d

Make ZipSlip use new API

5a3f97c

Make HardcodedKeys use new API

086ed3f

Make WeakCryptoAlgorithm use new API

ff97e77

Make SSRF use new API

1cc6fa5

Make EmailInjection use new API

a2efcff

Make LDAPInjection use new API

76fcb44

Make DsnInjection use new API

645db62

Make Timing use new API

4fae957

Make PamAuthBypass use new API

4571d1c

Make DivideByZero use new API

16f1ed0

Make HTMLTemplateEscapingPassthrough use new API

cbba97a

Removed edges and labels are mostly duplicates. No alerts are lost.

Make ConditionalBypass use new API

326f6c9

Make WrongUsageOfUnsafe use new API

0b136d8

Make StackTraceExposure use new API

61b11e4

Make SensitiveConditionBypass use new API

ca6417e

Make CorsMisconfiguration use new API

21adf7a

Make InsecureTLS use new API

3cd2587

Make BadRedirectCheck use new API

cd4a4c1

Make IncorrectIntegerConversion use new API

7d28849

Make taint tracking tests use config from InlineFlowTest

ab81e39

Make taint tracking tests use InlineFlowTest

d22169a

Make taint tracking tests use new API

ba69465

Note deprecation in QLDoc for LogInjection

9376ead

owen-mc force-pushed the go/refactor-data-flow-configurations branch from d8ed7fe to 9376ead Compare August 1, 2023 07:04

mbg reviewed Aug 1, 2023

View reviewed changes

Go: Make flow configurations use new data flow API #13820

Go: Make flow configurations use new data flow API #13820

owen-mc commented Jul 26, 2023 •

edited

owen-mc commented Jul 27, 2023 •

edited

mbg left a comment

mbg Aug 1, 2023

owen-mc Aug 1, 2023

mbg Aug 1, 2023

owen-mc Aug 2, 2023

mbg Aug 3, 2023

mbg Aug 1, 2023

mbg Aug 1, 2023

mbg Aug 1, 2023

mbg Aug 1, 2023

mbg Aug 1, 2023

owen-mc Aug 2, 2023

mbg Aug 1, 2023

owen-mc Aug 2, 2023

mbg Aug 1, 2023

owen-mc Aug 1, 2023

mbg Aug 1, 2023

owen-mc Aug 2, 2023

Go: Make flow configurations use new data flow API #13820

Are you sure you want to change the base?

Go: Make flow configurations use new data flow API #13820

Conversation

owen-mc commented Jul 26, 2023 • edited

owen-mc commented Jul 27, 2023 • edited

mbg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owen-mc commented Jul 26, 2023 •

edited

owen-mc commented Jul 27, 2023 •

edited