Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

comparison chaining has wrong source positions in most contexts (python 3.11.0rc2) #95921

Closed
15r10nk opened this issue Aug 12, 2022 · 6 comments
Assignees
Labels
type-bug An unexpected behavior, bug, or error

Comments

@15r10nk
Copy link

15r10nk commented Aug 12, 2022

problem

comparison chaining has wrong source positions in most contexts

>>> cnd = True                                            
>>> a = 1 if 1 in 2 == 3 else 0                                                                
Traceback (most recent call last):                                                              
  File "example.py", line 43, in <module>                                                             
    a = 1 if 1 in 2 == 3 else 0                                              
        ^^^^^^^^^^^^^^^^^^^^^^^                                                                  
TypeError: argument of type 'int' is not iterable  

This bug requires two things to happen:

  1. The comparison chain has to contain more than on comparison the type does not matter (==,in,<=,is,...)
  2. The context has to be something other than an expression

no problem

The following examples work:
only one comparison

Traceback (most recent call last):
  File "example.py", line 4, in <module>
    if 5<"5":
       ^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

inside expression

>>> a=False or 4<"5"<6
Traceback (most recent call last):
  File "example.py", line 15, in <module>
    a=False or 4<"5"<6
               ^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

reason

The problem seems to be the positions in the compiled bytecode:

script:

import dis

def foo():
    if 1<2<"no int":
        return 1
    return 0

for p in dis.get_instructions(foo):
    print(p.positions, p.opname,p.argval)

output (Python 3.11.0rc2+):

Positions(lineno=4, end_lineno=4, col_offset=0, end_col_offset=0) RESUME 0
Positions(lineno=5, end_lineno=5, col_offset=7, end_col_offset=8) LOAD_CONST 1
Positions(lineno=5, end_lineno=5, col_offset=9, end_col_offset=10) LOAD_CONST 2
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) SWAP 2
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) COPY 2
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) COMPARE_OP < # range of the whole if
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) POP_JUMP_FORWARD_IF_FALSE 30
Positions(lineno=5, end_lineno=5, col_offset=11, end_col_offset=19) LOAD_CONST no int
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) COMPARE_OP < # range of the whole if
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) POP_JUMP_FORWARD_IF_FALSE 38
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) JUMP_FORWARD 34
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) POP_TOP None
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) JUMP_FORWARD 38
Positions(lineno=6, end_lineno=6, col_offset=15, end_col_offset=16) LOAD_CONST 1
Positions(lineno=6, end_lineno=6, col_offset=15, end_col_offset=16) RETURN_VALUE None
Positions(lineno=7, end_lineno=7, col_offset=11, end_col_offset=12) LOAD_CONST 0
Positions(lineno=7, end_lineno=7, col_offset=11, end_col_offset=12) RETURN_VALUE None

the generated ast looks fine (snippet):

            If(
               test=Compare(
                  left=Constant(
                     value=1,
                     lineno=5,
                     col_offset=7, # correct
                     end_lineno=5,
                     end_col_offset=8),
                  ops=[
                     Lt(),
                     Lt()],
                  comparators=[
                     Constant(
                        value=2,
                        lineno=5,
                        col_offset=9, # correct
                        end_lineno=5,
                        end_col_offset=10),
                     Constant(
                        value='no int',
                        lineno=5,
                        col_offset=11, # correct
                        end_lineno=5,
                        end_col_offset=19)],
                  lineno=5,
                  col_offset=7,
                  end_lineno=5,
                  end_col_offset=19),

other examples

here is a list of all the problems I found so far:

>>> if 4<"5"<6:
...     pass
...
Traceback (most recent call last):
  File "example.py", line 20, in <module>
    if 4<"5"<6:
TypeError: '<' not supported between instances of 'int' and 'str'
>>> while 4<"5"<6:
...     pass
...
Traceback (most recent call last):
  File "example.py", line 24, in <module>
    while 4<"5"<6:
TypeError: '<' not supported between instances of 'int' and 'str'

>>> assert 4<"5"<6
Traceback (most recent call last):
  File "example.py", line 27, in <module>
    assert 4<"5"<6
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a=2+(5 if 4<"5"<6 else 3)
Traceback (most recent call last):
  File "example.py", line 29, in <module>
    a=2+(5 if 4<"5"<6 else 3)
         ^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a=[a for a in [1] if 4<"5"<a ]
Traceback (most recent call last):
  File "example.py", line 34, in <listcomp>
    a=[a for a in [1] if 4<"5"<a ]
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a={a for a in [1] if 4<"5"<a }
Traceback (most recent call last):
  File "example.py", line 35, in <setcomp>
    a={a for a in [1] if 4<"5"<a }
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a={a:a for a in [1] if 4<"5"<a }
Traceback (most recent call last):
  File "example.py", line 36, in <dictcomp>
    a={a:a for a in [1] if 4<"5"<a }
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a=list((a for a in [1] if 4<"5"<a ))
Traceback (most recent call last):
  File "example.py", line 37, in <genexpr>
    a=list((a for a in [1] if 4<"5"<a ))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> cnd = True
>>> a = 1 if 1 in 2 == 3 else 0
Traceback (most recent call last):
  File "example.py", line 43, in <module>
    a = 1 if 1 in 2 == 3 else 0
        ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'int' is not iterable

>>> if True and 4<"5"<6:
...     pass
...
Traceback (most recent call last):
  File "example.py", line 45, in <module>
    if True and 4<"5"<6:
TypeError: '<' not supported between instances of 'int' and 'str'

expected result

The positions should always match the ast-node range:

>>> a=list((a for a in [1] if 4<"5"<a ))
Traceback (most recent call last):
  File "example.py", line 37, in <genexpr>
    a=list((a for a in [1] if 4<"5"<a ))
                              ^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

or, even better only the failing comparison:

>>> a=list((a for a in [1] if 4<"5"<a ))
Traceback (most recent call last):
  File "example.py", line 37, in <genexpr>
    a=list((a for a in [1] if 4<"5"<a ))
                              ^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'
@15r10nk 15r10nk added the type-bug An unexpected behavior, bug, or error label Aug 12, 2022
15r10nk added a commit to 15r10nk/executing that referenced this issue Aug 12, 2022
…sert rewriting

This affects also chained comparisons inside normal assertions because of
python/cpython#95921.
This is also the reason for the changes in sample results.

once that is fixed we could revert the changes in the sample_results
15r10nk added a commit to 15r10nk/executing that referenced this issue Aug 28, 2022
@15r10nk 15r10nk changed the title comparison chaining in assertions produces wrong source positions in bytecode comparison chaining has wrong source positions in most contexts (python 3.11.0rc2) Sep 18, 2022
@15r10nk
Copy link
Author

15r10nk commented Sep 18, 2022

@brandtbucher I think this is related to #93691.
Do you know if there someone already working on this?

Does it makes sense for me to look into it? I have (almost) no cpython core developer experience and it would take me probably far longer than someone else.

@brandtbucher
Copy link
Member

brandtbucher commented Sep 18, 2022

Thanks for the detailed report. This issue seems to stem from the fact that we have two implementations of comparisons in the bytecode compiler: one generic version (compiler_compare) and one optimized for branching conditions (in compiler_jump_if). The latter doesn’t update the location to the current node, which is probably an oversight.

I’ll have a fix up either today or tomorrow. It will probably need to wait for 3.11.1, though.

@brandtbucher brandtbucher self-assigned this Sep 18, 2022
brandtbucher added a commit to brandtbucher/cpython that referenced this issue Sep 20, 2022
brandtbucher added a commit to brandtbucher/cpython that referenced this issue Sep 20, 2022
…thonGH-96968).

(cherry picked from commit dfc73b5)

Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com>
brandtbucher added a commit to brandtbucher/cpython that referenced this issue Sep 20, 2022
…thonGH-96968).

(cherry picked from commit dfc73b5)

Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com>
brandtbucher added a commit to brandtbucher/cpython that referenced this issue Sep 20, 2022
…thonGH-96968).

(cherry picked from commit dfc73b5)

Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com>
miss-islington pushed a commit that referenced this issue Sep 20, 2022
GH-96973)

(cherry picked from commit dfc73b5)

Automerge-Triggered-By: GH:brandtbucher
miss-islington pushed a commit that referenced this issue Sep 20, 2022
GH-96974)

(cherry picked from commit dfc73b5)

Automerge-Triggered-By: GH:brandtbucher
@brandtbucher
Copy link
Member

brandtbucher commented Sep 20, 2022

Thanks again for the great bug report!

@15r10nk
Copy link
Author

15r10nk commented Sep 21, 2022

Big thank you for the quick bugfix @brandtbucher. But the fix caused another bug.

The CALL which creates the AssertionError("msg") gets now incorrect positions.

script:

def func():
    assert a and b<c , "msg"

import dis
for i in dis.get_instructions(func):
    if i.opname in ("COMPARE_OP","CALL"):
        print(i.positions,i.opname,i.argval)

output (with your fix):

Positions(lineno=4, end_lineno=4, col_offset=17, end_col_offset=20) COMPARE_OP <
Positions(lineno=4, end_lineno=4, col_offset=17, end_col_offset=20) CALL 0

You see that the positions of the CALL are equal to the positions of the b<c.

output (the commit before your fix):

Positions(lineno=4, end_lineno=4, col_offset=17, end_col_offset=20) COMPARE_OP <
Positions(lineno=4, end_lineno=4, col_offset=4, end_col_offset=28) CALL 0

The positions of the CALL is here the whole assert statement (which is ok for me, because it is easy to handle).

This did not cause any incorrect Tracebacks (at least I don't know how to produce it), but can cause problems for tools which analyse this positions.

@15r10nk
Copy link
Author

15r10nk commented Sep 21, 2022

It is maybe also important that this is not the only example.

    assert b<c , "msg"
    assert a<b<c , "msg"
    assert a+b<c , "msg"

show the same problems. I looks like everything with one or more < causes this problem.

@brandtbucher
Copy link
Member

brandtbucher commented Sep 22, 2022

Yep, I see the issue. My patch sets the location when recursing into the expression, but doesn't restore it after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

2 participants