Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

socket makefile read-write discards received data #80109

Open
pravn mannequin opened this issue Feb 7, 2019 · 5 comments
Open

socket makefile read-write discards received data #80109

pravn mannequin opened this issue Feb 7, 2019 · 5 comments
Labels
3.7 3.8 expert-IO type-bug An unexpected behavior, bug, or error

Comments

@pravn
Copy link
Mannequin

pravn mannequin commented Feb 7, 2019

BPO 35928
Nosy @vadmium, @serhiy-storchaka, @ammaraskar, @ZackerySpytz, @kctherookie
PRs
  • bpo-35928: textio, set decoded_chars to NULL with check #11878
  • bpo-35928: io.TextIOWrapper drops the internal buffer during write() #22535
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2019-02-07.12:30:32.136>
    labels = ['3.8', 'type-bug', '3.7', 'expert-IO']
    title = 'socket makefile read-write discards received data'
    updated_at = <Date 2020-10-04.05:50:57.985>
    user = 'https://bugs.python.org/pravn'

    bugs.python.org fields:

    activity = <Date 2020-10-04.05:50:57.985>
    actor = 'ZackerySpytz'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['IO']
    creation = <Date 2019-02-07.12:30:32.136>
    creator = 'pravn'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 35928
    keywords = ['patch', '3.6regression']
    message_count = 5.0
    messages = ['335017', '335117', '335196', '335197', '335631']
    nosy_count = 6.0
    nosy_names = ['martin.panter', 'serhiy.storchaka', 'ammar2', 'ZackerySpytz', 'pravn', 'kc']
    pr_nums = ['11878', '22535']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue35928'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    @pravn
    Copy link
    Mannequin Author

    pravn mannequin commented Feb 7, 2019

    Using socket.makefile in read-write mode had a bug introduced between version 3.6.6 and 3.6.7. The same bug is present in version 3.7.x.

    The below code example will behave very differently between 3.6.6 and 3.6.7. It's based on the echo-server example from the docs.

    import socket
    
    HOST = '127.0.0.1'
    PORT = 0
    
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        s.bind((HOST, PORT))
    print(f'Waiting for connection on port {s.getsockname()[1]}')
    s.listen(1)
    
    conn, addr = s.accept()
    print(f'Connected by {addr}')
    
        with conn:
            f = conn.makefile(mode='rw')
    
            while True:
                m = f.readline()
                print(f'msg: {m!r}')
    
                if not m:
                    exit(0)
    
                f.write(m)
                f.flush()

    Python 3.6.7:
    Sending the string "Hello\nYou\n" will only print "Hello\n" and also only return "Hello\n" to the client.
    Removing the lines with f.write(m) and f.flush() and both "Hello\n" and "You\n" will be returned to the client.
    It's like the call to f.write() somehow empties the read buffer.

    Python 3.6.6:
    Sending "Hello\nYou\n" will return "Hello\n" and "You\n" to the client without any modifications to the above code.

    @pravn pravn mannequin added 3.7 expert-IO type-bug An unexpected behavior, bug, or error labels Feb 7, 2019
    @vadmium
    Copy link
    Member

    vadmium commented Feb 8, 2019

    Looking over the changelog, my guess (untested) is this is caused by commit d6a283b for bpo-25862. That change looks like it drops the internal TextIOWrapper decoding buffer for each successful write.

    I don't have the right version of Python to test with, but I expect this to also be broken without using a socket:

    >>> f = TextIOWrapper(BufferedRWPair(BytesIO(b"Hello\nYou\n"), BytesIO()))
    >>> f.readline()
    'Hello\n'
    >>> f.write(_)
    6
    >>> f.readline()  # Does this now return EOF?
    'You\n'

    @pravn
    Copy link
    Mannequin Author

    pravn mannequin commented Feb 11, 2019

    >>> f = TextIOWrapper(BufferedRWPair(BytesIO(b"Hello\nYou\n"), BytesIO()))
    >>> f.readline()
    'Hello\n'
    >>> f.write(_)
    6
    >>> f.readline()  # Returns empty string
    ''

    @ammaraskar
    Copy link
    Member

    ammaraskar commented Feb 11, 2019

    Recreatable on master as well, also Martin your suspicion seems correct, reverting 23db935 fixes it.

    @ammaraskar ammaraskar added the 3.8 label Feb 11, 2019
    @kctherookie
    Copy link
    Mannequin

    kctherookie mannequin commented Feb 15, 2019

    Added PR 11878, this will pass both this bug report and PR 3918 regression, the commit Ammar noted, it is an addition to this change.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 3.8 expert-IO type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants