Python: Add annotated call-graph tests#3790
Conversation
See the added README for in-depth details
|
On "Class Construction": In the upcoming data flow implementation, we currently use |
|
Actually, looks like you can just use |
Yep, using So it seems like my initial idea of |
tausbn
left a comment
There was a problem hiding this comment.
Overall: really good stuff. I've made a bunch of comments and suggestions, but these are mostly about the documentation. The code itself seems solid. 💪
python/ql/test/experimental/library-tests/CallGraph-xfail/call_edge_xfail.py
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| /** There is an obvious problem with the annotation `name` */ | ||
| predicate name_in_error_state(string name) { |
There was a problem hiding this comment.
Is there a logic to whether the names of these methods are written using underscores or in mixedCase? (Ditto comment_for above. Also maybe some of these should be private.)
There was a problem hiding this comment.
nah, that's just me being inconsistent. I have transformed everything into camelCase now. I guess some of them could be private, but seeing as they're in a path that can't be imported from anywhere (since it contains -), I don't really see the point in thinking too much about this.
python/ql/test/experimental/library-tests/CallGraph/CallGraphTest.qll
Outdated
Show resolved
Hide resolved
python/ql/test/experimental/library-tests/CallGraph/CallGraphTest.qll
Outdated
Show resolved
Hide resolved
| ... | ||
| else: | ||
| ... | ||
| ``` |
There was a problem hiding this comment.
I'm not sure I understand the significance of this example (and by extension, the section in which it appears). 🤔
There was a problem hiding this comment.
yeah, I can see that it's a little empty 😄 tried to fix it up in a new commit :)
Co-authored-by: Taus <tausbn@gmail.com>
Adjusting test setup properly requires some deep thinking, and I don't think I'm ready to do that right now. Added a TODO instead.
tausbn
left a comment
There was a problem hiding this comment.
I think this looks good. Regarding the whole __init__/__new__ issue, it may be better to verify correctness at the level of dataflow rather than just the call graph (i.e. testing that arguments to classes flow to the correct places for these two special methods).
Start off by reading the README added for explanation :)
The tests included here is by no means complete yet. I wanted to get the approval from the rest of the team before investing more time into it. Please let me know @tausbn and @yoff 😊
I have made sure that all the added Python files are runnable, so we can manually inspect that the annotations are correct.
I would at least like to port all call related cases from https://github.com/github/codeql/tree/master/python/ql/test/library-tests/PointsTo/regressions, although not all of them are critical to handle right away, I think this should the right place to track such instances going forwards.
I also want to point out that I added tests that show the error handling works in a good way. I moved this out of the regular tests because it polluted everything. However, getting the error reporting right was not easy, so I wanted to keep it in there to show it actually works 😄
For further discussion:
Going forwards (if you all agree that this is nice), we need to figure out how to handle these 2 cases
1) "magic" methods in Python
How to handle special methods in Python. For example, if we have
obj = MyClass(), thenstr(obj)would call__str__if that is defined onMyClass. This can also happen for accessing/assigning/deleting object properties with@property, and many other cases, such asobj + 1using__add__.Our current setup of
Value.getACall()returning aCallNodedoesn't fully support this (since these aren't calls), so although I have included a test file for this inclass_advanced.py, I have not annotated anything (so nothing is tested).2) Class construction
Currently the edges I consider that points-to can resolve is very simple:
so it doesn't report there being an from
A()->A.__init__(seeclass_simple.py). I thought about adding support for this with the code in the following snippet, but I kinda feel like that is cheating ("if you need to write custom code to make it work, then it's not supported"). I want to know whether you agree.