Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update pip, setuptools, wheel, and parse requirements from METADATA #263

Open
wants to merge 1 commit into
base: master
from

Conversation

@samschlegel
Copy link

@samschlegel samschlegel commented Dec 4, 2019

This updates pip, setuptools, and wheel, and switches to parsing METADATA instead of metadata.json

pip==19.3.1
setuptools==42.0.2
wheel==0.33.6

Due to changes in pypa/wheel#195, wheel no longer includes a metadata.json, so I've also switched to using pkg_resources to parse the package metadata and resolve dependencies and extras. This was most obvious in packages where wheels were built locally, but after making the change it was obvious that this should also fix some metadata related bugs.

As it is currently, this might cause issues with extras for packages that only include the extras in metadata.json, but I wasn't able to find any packages that did this.

Fixes #126, #253, #95, #70, #185

@samschlegel samschlegel requested review from brandjon and lberki as code owners Dec 4, 2019
@samschlegel
Copy link
Author

@samschlegel samschlegel commented Dec 5, 2019

Looks like this fails due to missing googleapis-common-protos[grpc]


(02:10:36) ERROR: /private/var/tmp/_bazel_buildkite/824a4042afe4624418ffa6a61b765b34/external/examples_extras/requirements.bzl:243:5: Traceback (most recent call last):
--
  | File "/Users/buildkite/builds/bk-imacpro-10/bazel/rules-python-python/examples/extras/BUILD", line 21
  | py_test(name = "extras_test", <3 more arguments>)
  | File "/Users/buildkite/builds/bk-imacpro-10/bazel/rules-python-python/examples/extras/BUILD", line 28, in py_test
  | requirement(<1 more arguments>)
  | File "/private/var/tmp/_bazel_buildkite/824a4042afe4624418ffa6a61b765b34/external/examples_extras/requirements.bzl", line 243, in requirement
  | fail(<1 more arguments>)
  | Could not find pip-provided dependency: 'googleapis-common-protos[grpc]'
@samschlegel
Copy link
Author

@samschlegel samschlegel commented Dec 5, 2019

This looks to be due to the wheel version upgrade removing support for metadata.json pypa/wheel#195

@samschlegel samschlegel changed the title Update to pip 19.3.1 Update pip, setuptools, and wheel Dec 5, 2019
@samschlegel samschlegel changed the title Update pip, setuptools, and wheel Update pip, setuptools, wheel, and parse requirements from METADATA Dec 5, 2019
@samschlegel samschlegel force-pushed the discord:update-to-pip-19-3-1 branch from 6cde0d8 to f641e95 Dec 6, 2019
@jmcfallargo
Copy link

@jmcfallargo jmcfallargo commented Dec 6, 2019

This PR fixed my issues with Tensorflow and didn't introduce any regressions fwiw. How do we get a reviewer?

@kjson
Copy link

@kjson kjson commented Jan 14, 2020

Any update on this? This PR fixes several packages for us. Currently unable to use rules_python without this.

@gertvdijk
Copy link

@gertvdijk gertvdijk commented Feb 6, 2020

How does this pass the test on version check? This PR does not touch the line to assert for Pip version 9.0.3:

self.assertEqual(pip.__version__, '9.0.3')

yet CI says:

//examples/version:version_test (cached) PASSED in 0.2s

@aaliddell
Copy link

@aaliddell aaliddell commented Mar 5, 2020

Since there seems to be some activity ongoing on this repo again, could one of the PRs be merged that supports METADATA parsing? This fix solves a lot of issues that are persistently raised around rules_python.
@brandjon @lberki

@ddlees
Copy link

@ddlees ddlees commented Apr 8, 2020

@brandjon @samschlegel

Would you be able to provide some timeline for when you might be able to resolve conflicts / review and merge this PR? I'm content w/ the prescribed workaround but it would be awesome if I didn't have to list out all the transitive deps of a package. Cheers! 🍻

@samschlegel samschlegel force-pushed the discord:update-to-pip-19-3-1 branch from f641e95 to 25141f3 Apr 9, 2020
* Also updates setuptools and wheel
* Use pkg_resources to parse METADATA
@samschlegel samschlegel force-pushed the discord:update-to-pip-19-3-1 branch from 25141f3 to 7ed8458 Apr 9, 2020
@samschlegel
Copy link
Author

@samschlegel samschlegel commented Apr 9, 2020

I've rebased! For what it's worth we ended up moving to https://github.com/dillon-giacoppo/rules_python_external

@dayfine
Copy link

@dayfine dayfine commented Apr 21, 2020

So will this be merged? Or am I supposed to use rules_python_external?

  • I don't (really) want to rely on dillon-giacoppo/rules_python_external for a while and then have to switch back
  • If dillon-giacoppo/rules_python_external is gonna be the solution, then README.md should be updated.

On a side note, do I still need this change if I simply always have a transitively-closed requirements.txt?

@thundergolfer
Copy link
Collaborator

@thundergolfer thundergolfer commented Apr 21, 2020

Hey @dayfine,

I maintain rules_python_external (alongside @dillon-giacoppo) and have recently become a maintainer here.

The relationship between rules_python and rules_python_external is of immediate concern to the community, but the recently added maintainers need some time build a working relationship and discuss next steps.

If rules_python_external solves an immediate problem you're having, I'd recommend using it. Where it is the solution will be answered ASAP.

On a side note, do I still need this...

This change is affecting how the requirement(X) targets get created, so I think this would affect whether extras work even if your requirements.txt is transitively closed.

@ddlees
Copy link

@ddlees ddlees commented May 13, 2020

@thundergolfer Semi-related to this issue is the dependency on the system installed version of python. Regardless of where these rules will be maintained, I put together a small rules project to address that bit of it. (https://github.com/digital-plumbers-union/rules_pyenv)

For this issue, my thought is that bazel controlled versions of python would aid the bits around pip, wheels and setuptools; ensuring those too are hermetic/ bazel controlled. Anyway, I wanted to update this post with that link while I was thinking about it. Maybe you guys will find it useful too! ¯_(ツ)_/¯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

9 participants
You can’t perform that action at this time.