Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bz2.BZ2File / gzip.GZipFile / lzma.LZMAFile expose misleading fileno method. #100066

Open
Yhg1s opened this issue Dec 6, 2022 · 0 comments
Open
Labels
3.12 type-bug An unexpected behavior, bug, or error

Comments

@Yhg1s
Copy link
Member

Yhg1s commented Dec 6, 2022

The various compressing/decompressing file wrappers (bz2.BZ2File, gzip.GZipFile, lzma.LZMAFile) currently have fileno methods that return the underlying file descriptor:

cpython/Lib/bz2.py

Lines 126 to 129 in 0a4c82d

def fileno(self):
"""Return the file descriptor for the underlying file."""
self._check_not_closed()
return self._fp.fileno()

I imagine this was done because it seemed useful, but I'm not sure what use it is. You can't safely use things like select since the compression/decompression might buffer, and passing it to things that use the file descriptor directly will produce garbage (when reading) or corrupt the file (when writing).

An example how misleading this can be, courtesy of @ericfrederich:

>>> import bz2
>>> import subprocess
>>> with bz2.open('/tmp/out.bz2', 'w') as f:
...   subprocess.check_call(['echo', '-n', "Why doesn't this work?"], stdout=f)
...
0
>>> bz2.open('/tmp/out.bz2', 'r').read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/bz2.py", line 178, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.7/_compression.py", line 103, in read
    data = self._decompressor.decompress(rawblock, size)
OSError: Invalid data stream
>>> open('/tmp/out.bz2', 'rb').read()
b"Why doesn't this work?BZh9\x17rE8P\x90\x00\x00\x00\x00"

Note the (empty) bz2 data after the data written by the subprocess.

Am I missing a situation where this is actually useful? If there isn't one, can we consider adding a warning for the confusing behaviour?

@Yhg1s Yhg1s added type-bug An unexpected behavior, bug, or error 3.12 labels Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

1 participant