-
Notifications
You must be signed in to change notification settings - Fork 2k
Python: Port and extend XXE modeling #6112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
RasmusWL
merged 91 commits into
github:main
from
jorgectf:jorgectf/python/deserialization
Mar 14, 2022
Merged
Changes from 1 commit
Commits
Show all changes
91 commits
Select commit
Hold shift + click to select a range
0e61558
Empty commit
jorgectf 78deec8
Upload main structure and initial tests
jorgectf b9fa57f
Move tests to `test/`
jorgectf c3b3bde
Add `XMLParser` concept
jorgectf d475d52
Add partial modeling
jorgectf 11f4c1c
Format tests
jorgectf b5e10b6
Write `(String|Bytes)IO` additional taint step
jorgectf 068150b
Finish modeling
jorgectf 0d2646f
Polish documentation
jorgectf 61e873d
Polish tests
jorgectf b83b31c
Write qldocs
jorgectf 1dd77f1
Fix undetected tests
jorgectf 93c8529
Add `.expected`
jorgectf 48bca5b
Fix references' link anchor
jorgectf 21da603
Update `.qlref`
jorgectf 61a81b6
Extend `.qlref`
jorgectf 67fddda
Merge branch 'main' into jorgectf/python/deserialization
RasmusWL 9c286a1
Python: fix name of `.qhelp` file
RasmusWL e472814
Python: Fix XXE qhelp
RasmusWL 8df3dab
Python: Adjust `.expected` with subpaths
RasmusWL 15dfc6d
Fix `xml_sax_parser.py` good/bad naming
jorgectf 5b66a15
Extend `mayBeDangerous()` QLDoc
jorgectf 320a00b
Delete simple `API::Node`s
jorgectf be42470
Apply suggestions from code review
jorgectf c2046f1
Improve readability for `xmlDom()`
jorgectf f1a73e3
Merge branch 'jorgectf/python/deserialization' of https://github.com/…
jorgectf 58bc110
Merge branch 'main' into jorgectf/python/deserialization
RasmusWL 066b400
Add `lxml.etree.XMLParser` missing `resolve_entities` dangerous case
jorgectf 637901d
Make concepts instances of their ranges
jorgectf cb8e54e
Delete redundant `LXMLParser` dangerous check
jorgectf 9ab6d21
Add forward type tracking test
jorgectf a1f8acc
Merge branch 'github:main' into jorgectf/python/deserialization
jorgectf 080775c
Merge branch 'jorgectf/python/deserialization' of https://github.com/…
jorgectf d96eb01
Merge branch 'github:main' into jorgectf/python/deserialization
jorgectf 43fde35
Merge branch 'jorgectf/python/deserialization' of https://github.com/…
jorgectf 99e14d1
Merge branch 'github:main' into jorgectf/python/deserialization
jorgectf d2f07e4
Merge branch 'jorgectf/python/deserialization' of https://github.com/…
jorgectf 8f9cd16
Update
jorgectf 7c4a6a1
Test polish
jorgectf 01ad25f
Apply `.getALocalSource()` and fix `xmltodict`'s `vulnerable` predicate
jorgectf b00051e
Update `.expected`
jorgectf 85b5ef3
`XmlInjection` -> `XmlEntityInjection`
jorgectf c5f30d9
Create an extendable `AdditionalTaintStep` class in customizations
jorgectf 518e2ae
Merge branch 'main' into jorgectf/python/deserialization
RasmusWL 500e0ac
Python: Rewrite sax XML tests
RasmusWL ee23c05
Python: XML: Expose vuln kind on sink
RasmusWL aaf55b2
Python: Add XMLVulnerabilityKind
RasmusWL 16e482b
Python: Improve QLDoc for XML parsing/parsers
RasmusWL 6dd776b
Python: Only produce one alert per vulnerable XML sink
RasmusWL 7f7758b
Python: rewrite xml sax modeling
RasmusWL 515b824
Python: Add lxml positive test
RasmusWL 661d8bf
Python: Better handling of `resolve_entities` arg in lxml
RasmusWL 52891cb
Python: Add PoC for XML vulns
RasmusWL 3c321dd
Python: Model `lxml.etree.get_default_parser` in own class
RasmusWL 124c03c
Python: Expand lxml tests
RasmusWL e295399
Python: Properly handle `huge_tree` in lxml
RasmusWL 703e3e8
Python: Handle DTD retrieval vuln in lxml
RasmusWL 6129193
Python: Properly model `xml.etree`
RasmusWL 3affa6c
Python: Annotate xmltodict tests
RasmusWL c4d08db
Python: Expand XML PoC with minidom/pulldom/expat
RasmusWL 5a65248
Python: Annotate xml.dom tests
RasmusWL 9406a97
Python: Fix vuln detection for xml.minidom with parser arg
RasmusWL 7cda901
Python: Add separate query for SimpleXMLRPCServer
RasmusWL 4b03f5c
Python: Rename xml.sax test for consistency
RasmusWL faebaee
Python: Use concept tests for XML Parsing
RasmusWL a7134ca
Python: Port xml.dom tests
RasmusWL 5fb4c4d
Python: Port xml.etree tests
RasmusWL 0b12d91
Python: Port xml.sax tests
RasmusWL c739ae4
Python: Port `xmltodict` tests
RasmusWL 2451123
Python: Move XML PoC to new test dir
RasmusWL 3278793
Python: Handle more functions and kw-args
RasmusWL f72f673
Python: Update `XmlEntityInjection.expected`
RasmusWL 33ebcdf
Python: Support feed method of lxml/xml.etree Parsers
RasmusWL 46238d5
Python: Add test for XMLPullParser
RasmusWL de0e67f
Python: Restructure overall XML modeling
RasmusWL a033b71
Python: Align QLdocs of XML modeling
RasmusWL c0a2c25
Python: Restructure modeling of `xml.etree` parsers
RasmusWL c0a6f9f
Python: Restructure lxml modeling
RasmusWL df8e0fc
Python: Minor fixup of qldoc
RasmusWL 837daaa
Python: Remove XMLParser concept
RasmusWL 0d69dc8
Python: Minor qldoc improvement
RasmusWL 3f6c55e
Python: Rename `vulnerable` predicate => `vulnerableTo`
RasmusWL 683c2fa
Apply suggestions from code review
jorgectf 3cd165d
Python: Apply suggestions from code review
RasmusWL d6cbfec
Python: huge_tree tests were wrong
RasmusWL f0131af
Python: Fix `huge_tree` modeling
RasmusWL 1a9620a
Python: Add conditional assignment check for sax parser
RasmusWL ef045a6
Python: Fix typo in set_default_parser
RasmusWL 5552834
Merge pull request #9 from RasmusWL/WIP
jorgectf 6b14c1d
Merge branch 'main' into jorgectf/python/deserialization
RasmusWL 0e9da4a
Python: Resolve name conflict over `XML` module
RasmusWL File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Polish documentation
- Loading branch information
commit 0d2646fd3dade349586b362fb610830867249322
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| from flask import request, Flask | ||
| import lxml.etree | ||
| import xml.etree.ElementTree | ||
|
|
||
|
|
||
| @app.route("/example") | ||
| def example(): | ||
| xml_content = request.args['xml_content'] | ||
|
|
||
| parser = lxml.etree.XMLParser() | ||
| parsed_xml = xml.etree.ElementTree.fromstring(xml_content, parser=parser) | ||
|
|
||
| return parsed_xml.text |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| <!DOCTYPE qhelp PUBLIC | ||
| "-//Semmle//qhelp//EN" | ||
| "qhelp.dtd"> | ||
| <qhelp> | ||
|
|
||
| <overview> | ||
| <p> | ||
| Parsing untrusted XML files with a weakly configured XML parser may lead to an XML External Entity (XXE) attack. | ||
| This type of attack uses external entity references to access arbitrary files on a system, carry out denial of | ||
| service, or server side request forgery. Even when the result of parsing is not returned to the user, out-of-band | ||
| data retrieval techniques may allow attackers to steal sensitive data. Denial of services can also be carried out | ||
| in this situation. | ||
| </p> | ||
| <p> | ||
| There are many XML parsers for Python, and most of them are vulnerable to XXE because their default settings enable | ||
| parsing of external entities. This query currently identifies vulnerable XML parsing from the following parsers: | ||
| <code>xml.etree.ElementTree.XMLParser</code>, <code>lxml.etree.XMLParser</code>, <code>lxml.etree.get_default_parser</code>, | ||
| <code>xml.sax.make_parser</code>. | ||
| </p> | ||
| </overview> | ||
|
|
||
| <recommendation> | ||
| <p> | ||
| The best way to prevent XXE attacks is to disable the parsing of any Document Type Declarations (DTDs) in untrusted data. | ||
|
jorgectf marked this conversation as resolved.
Outdated
|
||
| If this is not possible you should disable the parsing of external general entities and external parameter entities. | ||
| This improves security but the code will still be at risk of denial of service and server side request forgery attacks. | ||
| </p> | ||
| </recommendation> | ||
|
|
||
| <example> | ||
| <p> | ||
| The following example calls <code>xml.etree.ElementTree.fromstring</code> using a parser (<code>lxml.etree.XMLParser</code>) | ||
| that is not safely configured on untrusted data, and is therefore inherently unsafe. | ||
| </p> | ||
| <sample src="XXE.py"/> | ||
| </example> | ||
|
|
||
| <references> | ||
| <li>Python <a href="https://www.edureka.co/blog/python-xml-parser-tutorial/">XML Parsing</a>.</li> | ||
| <li>OWASP vulnerability description: <a href="https://www.owasp.org/index.php/XML_External_Entity_(XXE)_Processing">XML External Entity (XXE) Processing</a>.</li> | ||
| <li>OWASP guidance on parsing xml files: <a href="https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html#java">XXE Prevention Cheat Sheet</a>.</li> | ||
| <li>Paper by Timothy Morgen: <a href="https://research.nccgroup.com/2014/05/19/xml-schema-dtd-and-entity-attacks-a-compendium-of-known-techniques/">XML Schema, DTD, and Entity Attacks</a></li> | ||
| <li>Out-of-band data retrieval: Timur Yunusov & Alexey Osipov, Black hat EU 2013: <a href="https://www.slideshare.net/qqlan/bh-ready-v4">XML Out-Of-Band Data Retrieval</a>.</li> | ||
| <li>Denial of service attack (Billion laughs): <a href="https://en.wikipedia.org/wiki/Billion_laughs">Billion Laughs.</a></li> | ||
| </references> | ||
|
|
||
| </qhelp> | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.