⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

@aschackmull @atorralba As discussed. I did not yet add any release notes, as I want to get some feedback from you before I do any further work on this.
I tested this on WebGoat where it yields an additional deserialization result as well as several results for the UntrustedDataToExternalAPI.ql query. I'll investigate those a bit more and post back here. But if you have any other projects on which you would want to test this, please do let me know.

I recommend commit-by-commit review.

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

After initial investigation:

• This fixes a known false negative in our tests for the UrlRedirection check.
• It reveals two new results for WebGoat. See the results here. The 6 closed results were found with the changes in this PR. The first result looks like a true positive. For the second result, I don't have enough experience with SQL injections. Perhaps there is some way to smuggle in the single quote via certain unicode tricks or conversions. @atorralba what do you think?
• To satisfy our response splitting query tests I also had to create a sanitizer. I am not sure whether that's the best way to do this.

I have a feeling that those changes will change our results on several projects. So perhaps it would be wise to further test this. Do we have a larger-scale test harness?

The first result looks like a true positive.

It is.

For the second result, I don't have enough experience with SQL injections. Perhaps there is some way to smuggle in the single quote via certain unicode tricks or conversions.

In this specific case, it seems that the single quote replacement makes the statement safe from further injections (although the logged query that is inserted in the database is already injectable, so it doesn't matter much). I think it's acceptable to have FP like this, because they are very context-dependent. In general, using replace et al to sanitize SQL injections is a really bad idea and prone to bypasses, so I wouldn't add it as a general SQLi sanitizer.

To satisfy our response splitting query tests I also had to create a sanitizer. I am not sure whether that's the best way to do this.

You're discarding any string that uses any negative regex to make a replacement, which I think is a little too generous. Normally, response splitting is sanitized by removing carriage return (%0d and \r) and line break (%0a and \n) so we could look for replace, replaceAll and replaceFirst calls that included those characters in its first argument.

Also, instead of SpringLiteral, we could use CompileTimeConstantExpr::getStringValue() which should take into account a String constant being used in the argument instead of directly the literal.

Wonderful, thanks for the feedback! I'll incorporate those suggestions tomorrow and add a release note.

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

@atorralba I addressed some of your comments, however:

1. You also referred to %0d and %0a as special cases. I presume those are just the url-encoded versions of \r and n. My understanding is that those are fine to add to headers, so I did not address them in the sanitizer. Or do you have any objections here?
2. The current implementation of the sanitizer does not consider replaceFirst(), since in the general case, this function wouldn't replace all instances of line breaks. But if you feel strongly about it, then I am happy to add it.
3. I still allow for a whitelisting-based sanitization approach, since we already had a test case for it.
4. There are two false positives in the test cases. Addressing those, would require a more complex sanitizer implementation, possibly involving taint or data flow analysis. I am not sure it's worth doing that. I also haven't seen this being done before for sanitizers.

Do you mean the chained replace[All] calls? I'd not get into that right now either — although it's an interesting concept for sanitization in general, it seems out of this PR's scope. I'd mark those cases as false positives in the test case though.

Yep, chained or generally scattered. Ok, I'll leave the implementation as is and change the comments.

I'd mark those cases as false positives in the test case though.

Those are false negatives for the query though, that's why I named them as such. I guess they are false positives from the point of view of the sanitizer...

I'd mark those cases as false positives in the test case though.

Those are false negatives for the query though, that's why I named them as such. I guess they are false positives from the point of view of the sanitizer...

Yes. The query defines the sanitizer but it's not enough to filter out those cases, so it's a query FP.

Wait. Are you referring to this:

// GOOD: replace all line breaks
response.setHeader("h", t.replace('\n', ' ').replace('\r', ' '));

or this?

// FALSE NEGATIVE: replace only some line breaks
response.setHeader("h", t.replace('\n', ' '));

I think you refer to the second one, because the first one picks up any of the replace calls as sanitizers even if they aren't complete by themselves.

Ok, now I get it. Yes, false negatives 😄 Nothing to change then.

or this?
// FALSE NEGATIVE: replace only some line breaks
response.setHeader("h", t.replace('\n', ' '));

Yep, that's what I was referring to. It doesn't appear in the expected file, that's why I would have labeled it as FN. It seems that we do this on various occasions and it helped my understand test case breakage when I implemented this PR

Added release notes and rebased.

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

-    Java Standard Library,``java.*``,3,572,130,28,,,7,,,10
+    Java Standard Library,``java.*``,3,580,130,28,,,7,,,10
-    Totals,,217,6433,1474,117,6,10,107,33,1,84
+    Totals,,217,6441,1474,117,6,10,107,33,1,84

- java.lang,13,,58,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,46,12
+ java.lang,13,,66,,,,,,,,,,,8,,,,,4,,,1,,,,,,,,,,,,,,,54,12

I think the addition of the charAt and getBytes taint steps mean that we need a performance check with dca prior to merging. Also, @atorralba we should make sure that the addition of charAt doesn't introduce to much FP flow all over the place - it's slightly special and has been previously excluded on purpose, since a single char of a tainted string typically isn't "large" enough to be considered tainted, but if all the chars are somehow copied individually to some other data container then taint is likely to be preserved, which has also caused us FNs in the past, so I'm unsure whether to include charAt by default.

Also, @atorralba we should make sure that the addition of charAt doesn't introduce to much FP flow all over the place

👍 for the time being I've launched a DCA run, we'll see if too many new alerts pop up.

I feel inclined to stay optimist and think that, when applications only pick one char out of a tainted string, it would get filtered out by our usual default sanitizers, or the sinks would not be of the appropriate type (char or int) anyway, whereas the case "getting one char at a time in a loop and adding it to a string", which is taint-propagating, seems more likely to happen.

The only dangerous case (FP-wise) I can think of would be taking a single char from a tainted string and then concatenating it to another string that ends up reaching a sink — we could even write a query for that and see how often it happens in a big sample.

Do we have any results for this yet? Also, is there are place I can read up on "DCA"? I presume this is some test harness for performance or also for query results?

If there are concerns over this being merged, I am happy to split out the charAt() and getBytes() parts, do let me know.

If there are concerns over this being merged, I am happy to split out the charAt() and getBytes() parts, do let me know.

I'll let @atorralba decide on that - I just wanted to make sure that some conscious thought was given to this.

@atorralba Thanks for taking the time to double-check this!