Java: add Gson support to unsafe-deserialization query by smowton · Pull Request #6463 · github/codeql

smowton · 2021-08-10T16:27:38Z

~~Based on #6456 -- please only review the top two commits here.~~

Since the original by @luchua-bc added models of some Android functions I've ported the existing Android models to CSV and generated tests for them here.

github-actions · 2021-08-10T16:36:57Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Android,``android.*``,18,34,70,,,3,67,,,
+    Android,``android.*``,18,135,70,,,3,67,,,
-    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``groovy.lang``, ``groovy.util``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,251,146,,,,14,18,,
+    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,262,146,,,,14,18,,
-    Totals,,84,2711,398,13,6,6,107,33,1,66
+    Totals,,84,2823,398,13,6,6,107,33,1,66

Changes to framework-coverage-java.csv:

- android.content,8,,4,,,,,,,,,,,,,8,,,,,,4,
+ android.content,8,,33,,,,,,,,,,,,,8,,,,,,33,
+ android.os,,,72,,,,,,,,,,,,,,,,,,,71,1
+ flexjson,,,1,,,,,,,,,,,,,,,,,,,,1
+ jodd.json,,,10,,,,,,,,,,,,,,,,,,,,10

github-actions · 2021-08-11T08:55:56Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Android,``android.*``,18,34,70,,,3,67,,,
+    Android,``android.*``,18,145,70,,,3,67,,,
-    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``groovy.lang``, ``groovy.util``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,251,146,,,,14,18,,
+    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,262,146,,,,14,18,,
-    Totals,,84,2711,398,13,6,6,107,33,1,66
+    Totals,,84,2833,398,13,6,6,107,33,1,66

Changes to framework-coverage-java.csv:

- android.content,8,,4,,,,,,,,,,,,,8,,,,,,4,
+ android.content,8,,33,,,,,,,,,,,,,8,,,,,,33,
+ android.os,,,82,,,,,,,,,,,,,,,,,,,81,1
+ flexjson,,,1,,,,,,,,,,,,,,,,,,,,1
+ jodd.json,,,10,,,,,,,,,,,,,,,,,,,,10

java/ql/src/semmle/code/java/frameworks/android/Android.qll

java/ql/src/semmle/code/java/frameworks/android/Intent.qll

smowton · 2021-08-11T10:35:25Z

On all the extra modelling: I've just ported the existing models here, please feel free to follow up with further modelling

atorralba

The Gson part LGTM after resolving the comments above.

aschackmull · 2021-09-14T12:52:14Z

java/ql/test/library-tests/frameworks/android/flow-steps/test.ql

+import semmle.code.java.dataflow.DataFlow
+import semmle.code.java.dataflow.ExternalFlow
+import semmle.code.java.dataflow.TaintTracking
+import TestUtilities.InlineExpectationsTest


The boilerplate here has been abstracted into java/ql/test/TestUtilities/InlineFlowTest.qll so just import that.

smowton · 2021-09-14T14:39:05Z

Suggest #6599 goes before this PR, because it includes detailed Intent models that could then be removed from this PR

github-actions · 2021-10-07T10:56:14Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Android,``android.*``,45,233,70,,,3,67,,,
+    Android,``android.*``,45,344,70,,,3,67,,,
-    Totals,,143,5125,408,13,6,10,107,33,1,66
+    Totals,,143,5236,408,13,6,10,107,33,1,66

Changes to framework-coverage-java.csv:

- android.content,8,27,61,,,,,,,,,,,,,8,,,,,,27,,4,57
+ android.content,8,27,90,,,,,,,,,,,,,8,,,,,,27,,33,57
- android.os,,,82,,,,,,,,,,,,,,,,,,,,,2,80
+ android.os,,,164,,,,,,,,,,,,,,,,,,,,,83,81

luchua-bc · 2021-10-07T11:12:09Z

Great. We are going to finalize this PR as well:-)

github-actions · 2021-10-07T11:15:57Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Android,``android.*``,45,233,70,,,3,67,,,
+    Android,``android.*``,45,273,70,,,3,67,,,
-    Totals,,143,5125,408,13,6,10,107,33,1,66
+    Totals,,143,5165,408,13,6,10,107,33,1,66

Changes to framework-coverage-java.csv:

- android.os,,,82,,,,,,,,,,,,,,,,,,,,,2,80
+ android.os,,,122,,,,,,,,,,,,,,,,,,,,,41,81

* Account for changed dataflow graph shape using external flow * Account for BaseBundle only existing as of Android 5 * Properly implement Parcelable, which we previously got away with due to a partial stub * Restore an Android 11 function that had been added to the Android 9 Context class (I won't get into enforcing the difference in this PR)

Note these still aren't entirely mechanically generated stubs matching the real Android 9.

smowton · 2021-10-12T11:36:49Z

@aschackmull this is now updated and mergeable AFAIK

github-actions · 2021-10-12T11:37:52Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Android,``android.*``,45,245,70,,,3,67,,,
+    Android,``android.*``,45,285,70,,,3,67,,,
-    Totals,,143,5217,408,13,6,10,107,33,1,66
+    Totals,,143,5257,408,13,6,10,107,33,1,66

Changes to framework-coverage-java.csv:

- android.os,,,82,,,,,,,,,,,,,,,,,,,,,2,80
+ android.os,,,122,,,,,,,,,,,,,,,,,,,,,41,81

aschackmull · 2021-10-12T12:20:38Z

java/ql/lib/semmle/code/java/frameworks/google/Gson.qll

+
+/**
+ * Holds if `intentNode` is an `Intent` used in the context `(T)intentNode.getParcelableExtra(...)` and
+ * `parcelNode` is the corresponding parameter of `Parcelable.Creator<T> { public T createFromParcel(Parcel parcelNode) { }`.


The qldoc should likely be extended with a tidbit like "and T is a concrete type that ..., which explains the role of T` and in particular highlights that it isn't a type variable (which one might otherwise easily assume).

smowton requested a review from a team as a code owner August 10, 2021 16:27

smowton changed the title ~~Smowton/admin/gson unsafe deserialization~~ Java: add Gson support to unsafe-deserialization query Aug 10, 2021

github-actions bot added documentation Java labels Aug 10, 2021

smowton mentioned this pull request Aug 10, 2021

Java: CWE-502 Unsafe JSON deserialization with Gson, Flexjson, Jabsorb and JoddJson #5954

Closed

atorralba reviewed Aug 11, 2021

View reviewed changes

atorralba previously approved these changes Aug 11, 2021

View reviewed changes

smowton dismissed atorralba’s stale review via 5dc0ca5 August 12, 2021 10:23

smowton force-pushed the smowton/admin/gson-unsafe-deserialization branch from 5dc0ca5 to 48c497e Compare September 10, 2021 15:35

aschackmull reviewed Sep 14, 2021

View reviewed changes

smowton force-pushed the smowton/admin/gson-unsafe-deserialization branch from 48c497e to 57771e1 Compare October 7, 2021 10:53

smowton added 11 commits October 12, 2021 12:35

Add Gson support to unsafe deserialization query

cd2c9e9

Add tests for Android flow steps

fc0b18c

Add change note and update qhelp

81c0e66

Fix comments

490168f

Remove now-unnecessary models and tests

9e0b112

Remove redundant import

1afc03b

Introduce TaintInheritingContent

5da392e

Fix bad merge on Uri.java

205b6fe

Improve Android stub fidelity to the point that all relevant tests work

8816aa1

Note these still aren't entirely mechanically generated stubs matching the real Android 9.

Remove duplicate declaration

3c96e62

smowton force-pushed the smowton/admin/gson-unsafe-deserialization branch from 216dc0f to 3c96e62 Compare October 12, 2021 11:35

aschackmull reviewed Oct 12, 2021

View reviewed changes

Update javadoc

83c6406

aschackmull approved these changes Oct 12, 2021

View reviewed changes

aschackmull merged commit 0e5f89a into github:main Oct 12, 2021

Conversation

smowton commented Aug 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 10, 2021

java

Generated file changes for java

Uh oh!

github-actions bot commented Aug 11, 2021

java

Generated file changes for java

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smowton commented Aug 11, 2021

Uh oh!

atorralba left a comment

Choose a reason for hiding this comment

Uh oh!

aschackmull Sep 14, 2021

Choose a reason for hiding this comment

Uh oh!

smowton commented Sep 14, 2021

Uh oh!

github-actions bot commented Oct 7, 2021

java

Generated file changes for java

Uh oh!

luchua-bc commented Oct 7, 2021

Uh oh!

github-actions bot commented Oct 7, 2021

java

Generated file changes for java

Uh oh!

smowton commented Oct 12, 2021

Uh oh!

github-actions bot commented Oct 12, 2021

java

Generated file changes for java

Uh oh!

aschackmull Oct 12, 2021

Choose a reason for hiding this comment

Uh oh!

smowton Oct 12, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smowton commented Aug 10, 2021 •

edited

Loading