Python/JS: Share sensitive data modeling by RasmusWL · Pull Request #5739 · github/codeql

RasmusWL · 2021-04-21T09:49:07Z

Python had an (outdated) copy of the sensitive data modeling from JS. This PR adds a qll library that is shared between languages.

However, doing so introduces a problem of where to put the language-independent model. For this case, where it's not supposed to be used directly by an end-user, but rather re-exported by an other .qll file we provide, I opted for putting the model in an internal folder. I'm not feeling very strongly about this, so if someone has better ideas, I'm very open to suggestions.

I would strongly recommend reviewing commit-by-commit (I also included info in the commit messages of individual commits). I don't have a very good solution for checking that the final shared library matches what was in the JS files (that is commit 0d08718 is ok) -- guess you just have to go through it predicate-by-predicate 🤷

Creating as draft PR until someone tells me that my change in 08e86fd is semantically OK.

Initially I had called `nameIndicatesSensitiveData` for `maybeSensitiveName`, which made the relationship with `maybeSensitive` and `notSensitive` quite strange -- and therefore I added the more informative `maybeSensitiveRegexp` and `notSensitiveRegexp`. Although I'm no longer using `maybeSensitiveName`, and I no longer have a strong argument for making this name change, I still like it. If someone thinks this is a terrible idea, I'm happy to change it though 👍

which also prompted me to rewrite the QLDoc for `nameIndicatesSensitiveData`

Although there are warnings for the new deprecated classes/predicates, the test in javascript/ql/test/library-tests/SensitiveActions/ passes 👍

I added this predicate mostly because it was nice with an easy shortcut for it, but also since I spotted the `CredentialsFunctionName` not checking agaisnt the regexps in `notSensitive`, which looked suspicious. So the main goal of adding `nameIndicatesSensitiveData` is that you don't accidentially forget to ensure that the name doesn't match against `notSensitve`.

Someone from JS team needs to verify that this is actually OK.

erik-krogh · 2021-04-22T11:19:55Z

javascript/ql/src/semmle/javascript/security/SensitiveActions.qll

-    // match the regexps in `notSensitive`?
-    this.regexpMatch(maybeSensitive(classification))
-  }
+  CredentialsFunctionName() { nameIndicatesSensitiveData(this, classification) }


It looks OK to me.
It effectively just adds not this.regexpMatch(notSensitive()).

But I don't think I've touched this code, so someone else might have another opinion.

I also think it is fine.
In the worst case we would lose results that would be likely FPs.

Thanks 👍 that was my conclusion as well, I just didn't want to take the sole responsibility for this 😅

erik-krogh

JS 👍

Just a single note about a deprecation comment.

javascript/ql/src/semmle/javascript/security/internal/SensitiveDataHeuristics.qll

erik-krogh

JS 👍

yoff

Generally this is quite nice, I like the code sharing and the shortcut and I am happy with the naming of things. I had one question about some of the deprecated code, though.

yoff · 2021-05-04T16:45:46Z

javascript/ql/src/semmle/javascript/security/internal/SensitiveDataHeuristics.qll

   * That is, one of the rexeps from `maybeSensitiveRegexp` matches `name` (with the
   * given classification), and none of the regexps from `notSensitiveRegexp` matches


Is it intentional that both "rexeps" and "regexps" is used? (and would it be more mainstream to use "regexes"?)

just a typo, will fix up 👍

It would probably be more mainstream to use regexes... but I wanted to match the wording we use in CodeQL, which is regexp, for example in regexpMatch

yoff · 2021-05-04T16:56:51Z

javascript/ql/src/semmle/javascript/security/SensitiveActions.qll

+  /** DEPRECATED: Use `SensitiveDataClassification::secret` instead. */
+  deprecated predicate secret = SensitiveDataClassification::secret/0;

-  /** Gets the classification for user names or other account information. */
-  Classification id() { result = "id" }
+  /** DEPRECATED: Use `SensitiveDataClassification::id` instead. */
+  deprecated predicate id = SensitiveDataClassification::id/0;

-  /** Gets the classification for passwords or authorization keys. */
-  Classification password() { result = "password" }
+  /** DEPRECATED: Use `SensitiveDataClassification::password` instead. */
+  deprecated predicate password = SensitiveDataClassification::password/0;

-  /** Gets the classification for certificates. */
-  Classification certificate() { result = "certificate" }
+  /** DEPRECATED: Use `SensitiveDataClassification::certificate` instead. */
+  deprecated predicate certificate = SensitiveDataClassification::certificate/0;


Should these keep their return type?

I would say that they do. They used to have Classification as return type, and Classification is now an alias for SensitiveDataClassification (line 28)... so their return type is still Classification in some sense.

Ah, I just saw that it said predicate secret rather than SensitiveDataClassification secret. I guess I am not used to the predicate reference syntax.

yoff · 2021-05-04T17:01:10Z

javascript/ql/src/semmle/javascript/security/internal/SensitiveDataHeuristics.qll

+   * That is, one of the rexeps from `maybeSensitiveRegexp` matches `name` (with the
+   * given classification), and none of the regexps from `notSensitiveRegexp` matches
+   * `name`.


Is it intentional that both "rexeps" and "regexps" are used? (and would it be more mainstream to use "regexes"?)

I see that this comment shows up twice. I had it on one of the individual commits and did not see it in the view of all commits, so I wrote it again..

yoff

LGTM

RasmusWL added 8 commits April 21, 2021 11:31

Python: Update SensitiveDataHeuristics with newer JS version

775ed41

which also prompted me to rewrite the QLDoc for `nameIndicatesSensitiveData`

JS: Adapt SensitiveActions to use shared lib

0d08718

Although there are warnings for the new deprecated classes/predicates, the test in javascript/ql/test/library-tests/SensitiveActions/ passes 👍

JS: Rewrite to use SensitiveDataClassification

94fec5f

JS: Rewrite to use SensitiveDataClassification::password (and like)

b6f8e50

JS: Rewrite to use notSensitiveRegexp

e977d6e

JS: Make CredentialsFunctionName use nameIndicatesSensitiveData

08e86fd

Someone from JS team needs to verify that this is actually OK.

github-actions bot added JS Python labels Apr 21, 2021

RasmusWL requested a review from a team April 21, 2021 09:51

erik-krogh reviewed Apr 22, 2021

View reviewed changes

RasmusWL marked this pull request as ready for review April 22, 2021 12:40

RasmusWL requested a review from a team as a code owner April 22, 2021 12:40

RasmusWL added the no-change-note-required This PR does not need a change note label Apr 22, 2021

erik-krogh self-assigned this Apr 28, 2021

erik-krogh reviewed Apr 28, 2021

View reviewed changes

javascript/ql/src/semmle/javascript/security/internal/SensitiveDataHeuristics.qll Show resolved Hide resolved

Python/JS: Remove "Only added to aid with internal rewrite"

8b9c5f8

RasmusWL requested a review from erik-krogh April 28, 2021 09:51

erik-krogh previously approved these changes Apr 28, 2021

View reviewed changes

yoff reviewed May 4, 2021

View reviewed changes

Python/JS: Fix typo

dc4a0c1

RasmusWL dismissed erik-krogh’s stale review via dc4a0c1 May 5, 2021 08:19

RasmusWL requested a review from yoff May 5, 2021 08:19

yoff approved these changes May 11, 2021

View reviewed changes

yoff merged commit 549c9ee into github:main May 11, 2021

RasmusWL deleted the share-sensitive-data-modeling branch May 11, 2021 11:11

		* That is, one of the rexeps from `maybeSensitiveRegexp` matches `name` (with the
		* given classification), and none of the regexps from `notSensitiveRegexp` matches

Conversation

RasmusWL commented Apr 21, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erik-krogh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erik-krogh left a comment

Choose a reason for hiding this comment

Uh oh!

yoff left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yoff left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants