String constant folding (with test) #4083
Conversation
Ensure that when StrNode is a BytesLiteral, that we don't coerce it to unicode Fixes cython#3951
Needed to change the TreePath slightly to allow bytes-to-str comparison
Cython/Compiler/Visitor.py
Outdated
| @@ -834,7 +834,7 @@ def repr_of(self, node): | |||
| result += "(name=\"%s\")" % node.name | |||
| elif isinstance(node, ExprNodes.AttributeNode): | |||
| result += "(type=%s, attribute=\"%s\")" % (repr(node.type), node.attribute) | |||
| elif isinstance(node, ExprNodes.ConstNode): | |||
| elif isinstance(node, (ExprNodes.ConstNode, ExprNodes.StringNode)): | |||
There was a problem hiding this comment.
I also snuck in this slight inprovement to PrintTree
There was a problem hiding this comment.
What about using PyConstNode instead? And maybe using %r for the value instead of "%s"?
Cython/Compiler/Optimize.py
Outdated
| @@ -4424,6 +4424,8 @@ def _multiply_string(self, node, string_node, multiplier_node): | |||
| string_node.unicode_value = encoded_string( | |||
| string_node.unicode_value * multiplier, | |||
| string_node.unicode_value.encoding) | |||
| if not string_node.value.is_unicode: | |||
| build_string = bytes_literal | |||
There was a problem hiding this comment.
I don't think I understand exactly what's needed from these two comments on the other PR. So I haven't made the changes.
There was a problem hiding this comment.
Actually - I think I do understand so I've made the changes
Cython/Compiler/TreePath.py
Outdated
| elif (isinstance(attr_value, bytes) and isinstance(value, str) and | ||
| attr_value.decode("utf-8") == value): |
There was a problem hiding this comment.
Shouldn't you encode value instead of decoding attr_value? (And only if it's unicode and not just str?)
How do you know that attr_value is really encoded in UTF-8 and not something else?
Cython/Compiler/Visitor.py
Outdated
| @@ -834,7 +834,7 @@ def repr_of(self, node): | |||
| result += "(name=\"%s\")" % node.name | |||
| elif isinstance(node, ExprNodes.AttributeNode): | |||
| result += "(type=%s, attribute=\"%s\")" % (repr(node.type), node.attribute) | |||
| elif isinstance(node, ExprNodes.ConstNode): | |||
| elif isinstance(node, (ExprNodes.ConstNode, ExprNodes.StringNode)): | |||
There was a problem hiding this comment.
What about using PyConstNode instead? And maybe using %r for the value instead of "%s"?
|
I've made those changes. I think the failures on Py2.7 are unrelated to this change and possibly to do with a new version of decorator (https://pypi.org/project/decorator/#history) that's incompatible with Python 2.7. But it does mean that this isn't tested on Python 2.7 |
It isn't necessary to show the bug, but it's worth testing anyway.
Cython/Compiler/TreePath.py
Outdated
| elif (isinstance(attr_value, bytes) and isinstance(value, str) and | ||
| attr_value.decode("utf-8") == value): | ||
| attr_value == value.encode()): |
There was a problem hiding this comment.
This is also not going to work correctly in Py2, where str is bytes and .encode() will .decode() it first. If the value has some encoding other than us-ascii, then this implicit decoding will fail. Thus my suggestion that this should only be done for unicode strings and not Py2 str.
|
Test failure seems random. Let's get this in and see if it works then. |
* Handle constant folding for LanguageLevel 2 on Python 3. Ensure that when StrNode is a BytesLiteral, that we don't coerce it to unicode. * Add test for string multiplication bug. Needed to change the TreePath slightly to allow bytes-to-str comparison. Fixes #3951
This just finishes off #3952 by adding a test-case.
It's surprisingly hard to create a test-case for this. I had to allow bytes-to-string comparisons in the TreePath code (allowing bytes constants might be another possibility I think). I've confirmed that it fails on the current master and passes with the patch.