Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for os.PathLike filename in zipfile.ZipFile.write() #84686

Open
domragusa mannequin opened this issue May 5, 2020 · 5 comments
Open

add support for os.PathLike filename in zipfile.ZipFile.write() #84686

domragusa mannequin opened this issue May 5, 2020 · 5 comments
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@domragusa
Copy link
Mannequin

domragusa mannequin commented May 5, 2020

BPO 40506
Nosy @Yhg1s, @ethanfurman, @serhiy-storchaka, @domragusa
Files
  • pathlike_writestr.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2020-05-05.02:12:03.387>
    labels = ['type-feature', 'library', '3.10']
    title = 'add support for os.Pathlike filenames in zipfile.ZipFile.writestr'
    updated_at = <Date 2021-09-08.00:49:22.380>
    user = 'https://github.com/domragusa'

    bugs.python.org fields:

    activity = <Date 2021-09-08.00:49:22.380>
    actor = 'ethan.furman'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2020-05-05.02:12:03.387>
    creator = 'd.ragusa'
    dependencies = []
    files = ['49132']
    hgrepos = []
    issue_num = 40506
    keywords = ['patch']
    message_count = 2.0
    messages = ['368098', '368289']
    nosy_count = 4.0
    nosy_names = ['twouters', 'ethan.furman', 'serhiy.storchaka', 'd.ragusa']
    pr_nums = ['20002']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue40506'
    versions = ['Python 3.10']

    Linked PRs

    @domragusa
    Copy link
    Mannequin Author

    domragusa mannequin commented May 5, 2020

    ZipFile seems to support Pathlike objects pretty well, except in ZipFile.writestr.
    For example:

    >>> a = ZipFile(Path('test.zip'), 'w') # this works ok
    >>> a.write(Path('./foo.jpeg'), arcname=PurePath('/some/thing.jpeg')) # this works as well
    >>> a.writestr(PurePath('/test.txt'), 'idk') # this doesn't
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python3.8/zipfile.py", line 1788, in writestr
        zinfo = ZipInfo(filename=zinfo_or_arcname,
      File "/usr/lib/python3.8/zipfile.py", line 349, in __init__
        null_byte = filename.find(chr(0))
    AttributeError: 'PurePosixPath' object has no attribute 'find'

    I think it would be more consistent if it accepted any kind of paths, it would suffice to call os.fspath in ZipInfo.__init__ when the filename is a Pathlike-object, it's just 2 lines (+ tests, of course).

    Can I go ahead and prepare a patch for this?

    @domragusa domragusa mannequin added 3.9 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 5, 2020
    @domragusa
    Copy link
    Mannequin Author

    domragusa mannequin commented May 6, 2020

    Here's a small patch to do this. Everything seems to work fine.
    I don't know if where I placed the test (in OtherTests) is the most appropriate.

    On Tue, May 5, 2020 at 4:12 AM Domenico Ragusa <report@bugs.python.org> wrote:
    >
    >
    > New submission from Domenico Ragusa <domenicoragusa@gmail.com>:
    >
    > ZipFile seems to support Pathlike objects pretty well, except in ZipFile.writestr.
    > For example:
    >
    > >>> a = ZipFile(Path('test.zip'), 'w') # this works ok
    > >>> a.write(Path('./foo.jpeg'), arcname=PurePath('/some/thing.jpeg')) # this works as well
    > >>> a.writestr(PurePath('/test.txt'), 'idk') # this doesn't
    > Traceback (most recent call last):
    >   File "<stdin>", line 1, in <module>
    >   File "/usr/lib/python3.8/zipfile.py", line 1788, in writestr
    >     zinfo = ZipInfo(filename=zinfo_or_arcname,
    >   File "/usr/lib/python3.8/zipfile.py", line 349, in __init__
    >     null_byte = filename.find(chr(0))
    > AttributeError: 'PurePosixPath' object has no attribute 'find'
    >
    > I think it would be more consistent if it accepted any kind of paths, it would suffice to call os.fspath in ZipInfo.__init__ when the filename is a Pathlike-object, it's just 2 lines (+ tests, of course).
    >
    > Can I go ahead and prepare a patch for this?
    >
    > 

    components: Library (Lib)
    messages: 368098
    nosy: d.ragusa
    priority: normal
    severity: normal
    status: open
    title: add support for os.Pathlike filenames in zipfile.ZipFile.writestr
    type: enhancement
    versions: Python 3.9


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue40506\>


    @csabella csabella added 3.10 only security fixes and removed 3.9 only security fixes labels May 22, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @TBSenseo
    Copy link

    I have encountered the same issue with a different zipfile function: The zipfile.ZipFile.getinfo function throws a KeyError when using a pathlib.Path object containing the valid path to an existing zip-file member. I am actually using the zipfile.ZipFile.extract function in my case. This function calls the getinfo function, that in turn throws a key error:

    traceback.print_exc()
    Traceback (most recent call last):
      File "<my script file>", line 501, in extractTestFilesZipFolderItem
        return Path(zf.extract(zipFileItemPath, path = destinationFolderPath))
      File "C:\Progs\Python\Python310\lib\zipfile.py", line 1630, in extract
        return self._extract_member(member, path, pwd)
      File "C:\Progs\Python\Python310\lib\zipfile.py", line 1669, in _extract_member
        member = self.getinfo(member)
      File "C:\Progs\Python\Python310\lib\zipfile.py", line 1443, in getinfo
        raise KeyError(
    KeyError: "There is no item named WindowsPath('<Zip file member path>') in the archive"

    The code of the function is:

    from typing import Union
    import os
    from pathlib import Path
    import zipfile
    import bdb
    from breakOnException import setBreakOnException, getBreakOnException
    
    def extractTestFilesZipFolderItem(zipFilePath : Union[str, os.PathLike], zipFileItemPath : Union[str, os.PathLike], destinationFolderPath : Union[None, str, os.PathLike] = None) -> Path:
    	# Parameter zipFilePath
    	if not (isinstance(zipFilePath, str) or isinstance(zipFilePath, os.PathLike)):
    		raise TypeError('Parameter zipFilePath must be a string or another path-like instance')
    	# Parameter zipFileItemPath
    	if not (isinstance(zipFileItemPath, str) or isinstance(zipFileItemPath, os.PathLike)):
    		raise TypeError('Parameter zipFileItemPath must be a string or another path-like instance')
    	# Parameter destinationFolderPath
    	if not (destinationFolderPath is None or isinstance(destinationFolderPath, str) or isinstance(destinationFolderPath, os.PathLike)):
    		raise TypeError('Parameter destinationFolderPath must be None, a string or another path-like instance')
    	# Normal code
    	try:
    		with zipfile.ZipFile(zipFilePath) as zf:
    			return Path(zf.extract(zipFileItemPath, path = destinationFolderPath))
    	except Exception as ex:
    		if type(ex) not in [KeyboardInterrupt, bdb.BdbQuit] and getBreakOnException():
    			breakpoint()
    		raise

    The getBreakOnException function is just a function from a self-made library breakOnException that globally tracks whether I want a breakpoint to be triggered when an exception occurs to be able to investigate the values of the variables that may have caused the exception.

    When I put 'str(' and ')' around zipFileItemPath like zf.extract(str(zipFileItemPath), path = destinationFolderPath), then it works as intended.

    In my opinion, the zipfile functions expecting paths (including file names) should be able to accept any string and path-like object.
    I am using:

    • Spyder version: 5.4.3 (pip)
    • Python version: 3.10.11 64-bit
    • Qt version: 5.15.2
    • PyQt5 version: 5.15.9
    • Operating System: Windows 10

    @serhiy-storchaka
    Copy link
    Member

    It was discussed previously (maybe several times), and I opposed it. Path objects should only be supported for external paths (see #72418). Internal ZIP file paths do not represent real files, they are simply string keys. Using a Path object to access an entry in the ZIP file is a programming error, and we should not make easier to do such errors.

    @gpshead
    Copy link
    Member

    gpshead commented Dec 22, 2023

    Thanks, in general I think I agree. Though the zipfile module has APIs that conflate the two concepts, writestr is not one of those.

    ZipFile.write and ZipInfo.from_file are both APIs that take a local filesystem path in order to read & store it in the zip file. Those specific two being made to properly handle PathLike for their filename parameters make sense.

    Otherwise as far as supporting it in all situations, what I'm finding as I explore in my draft PR is that it gets a bit messy to support them in "lookup/update" scenarios. I had to use more conditional _sanitize_filename calls in too many places to be comfortable - and I still have pathname bugs when a *WindowsPath is involved (see the CI) - and even then callers need to ensure they're using relative paths for lookups.

    The original simple PR (#20002) put up by @domragusa even used pathlib.PurePosixPath within its test as a way to avoid the need for that, but exposes a bit of the smell: zip files are not posix. that they happened to settle on forward slashes internally for paths is an implementation detail but asking users to use PurePosixPath if they want pathlib types for use as keys for lookups from a zip file seem awkward.

    @gpshead gpshead changed the title add support for os.Pathlike filenames in zipfile.ZipFile.writestr add support for os.PathLike filename= in zipfile.ZipFile.write Dec 22, 2023
    @gpshead gpshead changed the title add support for os.PathLike filename= in zipfile.ZipFile.write add support for os.PathLike filename= in zipfile.ZipFile.write() Dec 22, 2023
    @gpshead gpshead changed the title add support for os.PathLike filename= in zipfile.ZipFile.write() add support for os.PathLike filename in zipfile.ZipFile.write() Dec 22, 2023
    @gpshead gpshead removed their assignment Dec 22, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    Status: No status
    Development

    No branches or pull requests

    4 participants