Java: add models of JSON-java, aka `org.json` by smowton · Pull Request #6325 · github/codeql

smowton · 2021-07-19T16:59:24Z

Strategy: like javax.json, I have adopted the approach of associated monolithic taint with JSON objects as a whole rather than elements, map-values and so on, on the assumption that 99% of use cases are about de/serialization and not using the objects like containers.

github-actions · 2021-07-19T17:01:41Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.databind``, ``com.unboundid.ldap.sdk``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,12,82,,,,14,18,,
+    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.databind``, ``com.unboundid.ldap.sdk``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,237,82,,,,14,18,,
-    Totals,,84,2428,296,13,6,6,107,33,1,66
+    Totals,,84,2653,296,13,6,6,107,33,1,66

Changes to framework-coverage-java.csv:

+ org.json,,,225,,,,,,,,,,,,,,,195,30

Marcono1234

Great that this pull request covers the complete JSON-java project!

Hopefully the following review comments are helpful.

Marcono1234 · 2021-07-25T12:55:18Z

+        "org.json;XMLXsiTypeConverter;true;convert;;;Argument[0];ReturnValue;taint",
+        "org.json;CDL;false;rowToJSONArray;;;Argument[0];ReturnValue;taint",
+        "org.json;CDL;false;rowToJSONObject;;;Argument[0..1];ReturnValue;taint",
+        "org.json;CDL;false;toJSONArray;;;Argument[0..1];ReturnValue;taint",


This is missing rowToString

Marcono1234 · 2021-07-25T13:01:17Z

+        "org.json;JSONTokener;false;nextTo;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;nextValue;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;syntaxError;;;Argument[0..1];ReturnValue;taint",
+        "org.json;JSONTokener;false;toString;;;Argument[-1];ReturnValue;taint",


These should probably all consider subtypes because JSONTokener has the subclasses HTTPTokener and XMLTokener.

Marcono1234 · 2021-07-25T13:05:07Z

+        "org.json;JSONPointer$Builder;false;append;;;Argument[-1];ReturnValue;value",
+        "org.json;JSONPointer$Builder;false;build;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONStringer;false;toString;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;JSONTokener;;;Argument[0];Argument[-1];taint",


Should this also cover the static method dehexchar?

I left individual chars out as too far-fetched a taint vector

Marcono1234 · 2021-07-25T13:07:30Z

+        "org.json;JSONArray;false;optNumber;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONArray;false;optQuery;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONArray;false;optString;;;Argument[-1];ReturnValue;taint",
+        // Default values that may be returned by the `opt*` functions above:


Really minor, but for Java they are usually called "method":

Suggested change

// Default values that may be returned by the `opt*` functions above:

// Default values that may be returned by the `opt*` methods above:

Marcono1234 · 2021-07-25T13:11:44Z

+        "org.json;JSONArray;false;optLong;;;Argument[1];ReturnValue;value",
+        "org.json;JSONArray;false;optNumber;;;Argument[1];ReturnValue;value",
+        "org.json;JSONArray;false;optString;;;Argument[1];ReturnValue;value",
+        "org.json;JSONArray;false;put;(boolean);;Argument[0];Argument[-1];taint",


Might be missing value flow from put to the return value (since the methods return this).

Marcono1234 · 2021-07-25T13:42:34Z

+        "org.json;JSONTokener;false;nextClean;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;nextString;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;nextTo;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;nextValue;;;Argument[-1];ReturnValue;taint",


Should this also cover skipTo? Maybe as taint flow from argument 0 to the return value?

Marcono1234 · 2021-07-25T13:45:26Z

+        "org.json;JSONTokener;false;nextValue;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONTokener;false;syntaxError;;;Argument[0..1];ReturnValue;taint",
+        "org.json;JSONTokener;false;toString;;;Argument[-1];ReturnValue;taint",
+        "org.json;JSONWriter;true;JSONWriter;;;Argument[-1];Argument[0];taint",


Out of pure interest, is this actually supported by CodeQL? I.e. modeling flow from the to-be-created instance to one of the constructor arguments.

Not yet (Go can do this, Java can't). Will add a comment that this doesn't work yet.

I'd be curious to know more about how this is designed to work in Go. As-is, this line is never going to work so might as well remove it (the input Argument[-1] to a constructor in Java is the result of the implicit malloc, which is passed into the constructor as the value of the this parameter, and there isn't really any way that flow can reach that Node).

The single-step relation (via the FunctionInput helper class) does some backward SSA use -> def walking, here: https://github.com/github/codeql-go/blob/main/ql/src/semmle/go/dataflow/FunctionInputsAndOutputs.qll#L124

For Java I expect this step would be something like: whenever taint should propagate from the result of a constructor (denoted Argument[-1]), introduce an edge from any post-update node concerning any variable that gets assigned the result of the constructor (or of the constructor itself if there is none) to the post-update node of the argument being tainted.

So here if we had

void f(Appendable sink, String source) { JSONWriter w = new JSONWriter(sink); w.write(source);

Then the w.write(source) would propagate taint from source to the post-update node of w as usual, but then the rule for new JSONWriter(sink) would wire that post-update node to sink's post-update node (nb. Go's use of SSA definitions as the post-update nodes of corresponding uses means less SSA graph walking is necessary there).

This is pretty brittle in Go and would be here too, and is designed to pick up the simplest cases of writer-wrapping.

I've put this in an issue for now: #6402

Marcono1234 · 2021-07-25T13:48:04Z

+        "org.json;Property;false;toProperties;;;Argument[0];MapKey of ReturnValue;taint",
+        "org.json;Property;false;toProperties;;;Argument[0];MapValue of ReturnValue;taint",
+        "org.json;Property;false;toJSONObject;;;MapKey of Argument[0];ReturnValue;taint",
+        "org.json;Property;false;toJSONObject;;;MapValue of Argument[0];ReturnValue;taint",


For consistency might be good to switch this because toJSONObject comes before toProperties (alphabetically and in the javadoc).

Marcono1234 · 2021-07-25T13:53:43Z

+        "org.json;Property;false;toJSONObject;;;MapValue of Argument[0];ReturnValue;taint",
+        "org.json;XML;false;escape;;;Argument[0];ReturnValue;taint",
+        "org.json;XML;false;stringToValue;;;Argument[0];ReturnValue;taint",
+        "org.json;XML;false;toJSONObject;;;Argument[0];ReturnValue;taint",


Should this also cover taint from the XMLParserConfiguration argument to the return value? XMLParserConfiguration allows specifying how xsi:type elements are converted, see withXsiTypeMap (then it would also be necessary to model that class).
Though maybe covering this is not worth it.

I'll leave this as relevant taint flow seems very far-fetched

Marcono1234 · 2021-07-25T13:54:47Z

+        "org.json;XML;false;escape;;;Argument[0];ReturnValue;taint",
+        "org.json;XML;false;stringToValue;;;Argument[0];ReturnValue;taint",
+        "org.json;XML;false;toJSONObject;;;Argument[0];ReturnValue;taint",
+        "org.json;XML;false;toString;;;Argument[0];ReturnValue;taint",


Should this also cover the tagName argument of the toString methods?

smowton · 2021-07-27T15:04:39Z

Thanks @Marcono1234 @aschackmull this is ready for re-review

github-actions · 2021-07-27T15:06:24Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.databind``, ``com.unboundid.ldap.sdk``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,12,82,,,,14,18,,
+    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.databind``, ``com.unboundid.ldap.sdk``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,248,82,,,,14,18,,
-    Totals,,84,2465,296,13,6,6,107,33,1,66
+    Totals,,84,2701,296,13,6,6,107,33,1,66

Changes to framework-coverage-java.csv:

+ org.json,,,236,,,,,,,,,,,,,,,198,38

Remove unnecessary import Co-authored-by: Anders Schack-Mulligen <aschackmull@users.noreply.github.com>

github-actions · 2021-08-02T13:49:17Z

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged. The following differences were found:

java

Generated file changes for java

Changes to framework-coverage-java.rst:

-    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.databind``, ``com.unboundid.ldap.sdk``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,12,98,,,,14,18,,
+    Others,"``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.databind``, ``com.unboundid.ldap.sdk``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.json``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,248,98,,,,14,18,,
-    Totals,,84,2465,313,13,6,6,107,33,1,66
+    Totals,,84,2701,313,13,6,6,107,33,1,66

Changes to framework-coverage-java.csv:

+ org.json,,,236,,,,,,,,,,,,,,,,198,38

Add models of JSON-java, aka org.json

34a4b71

smowton requested a review from a team as a code owner July 19, 2021 16:59

github-actions Bot added the Java label Jul 19, 2021

Add change note

8051a7c

smowton changed the title ~~Add models of JSON-java, aka org.json~~ Java: add models of JSON-java, aka org.json Jul 19, 2021

github-actions Bot added the documentation label Jul 19, 2021

aschackmull reviewed Jul 22, 2021

View reviewed changes

Comment thread java/ql/test/library-tests/frameworks/json-java/output.bqrs Outdated

aschackmull reviewed Jul 22, 2021

View reviewed changes

Comment thread java/ql/test/library-tests/frameworks/json-java/output.json Outdated

Remove stubbing script outputs

40173f7

Marcono1234 reviewed Jul 25, 2021

View reviewed changes

Add missing models and other minor improvements per Marcono1234's review

23de085

aschackmull reviewed Jul 28, 2021

View reviewed changes

Comment thread java/ql/test/library-tests/frameworks/json-java/test.ql Outdated

Update java/ql/test/library-tests/frameworks/json-java/test.ql

170bb43

Remove unnecessary import Co-authored-by: Anders Schack-Mulligen <aschackmull@users.noreply.github.com>

aschackmull mentioned this pull request Aug 3, 2021

Java: Consider some support for backwards flow through writer wrappers #6402

Open

aschackmull approved these changes Aug 3, 2021

View reviewed changes

smowton merged commit f83f950 into github:main Aug 3, 2021

	// Default values that may be returned by the `opt*` functions above:
	// Default values that may be returned by the `opt*` methods above:

Conversation

smowton commented Jul 19, 2021

Uh oh!

github-actions Bot commented Jul 19, 2021

java

Generated file changes for java

Uh oh!

Uh oh!

Uh oh!

Marcono1234 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smowton commented Jul 27, 2021

Uh oh!

github-actions Bot commented Jul 27, 2021

java

Generated file changes for java

Uh oh!

Uh oh!

github-actions Bot commented Aug 2, 2021

java

Generated file changes for java

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants