Provide a way to better connect pip_import to the Python toolchain #257
Labels
Comments
This was referenced Nov 12, 2019
kku1993
added a commit
to kku1993/rules_python
that referenced
this issue
May 16, 2020
This allows users to use a custom python interpreter that is built by another repository rule instead of using a pre-built interpreter binary that is checked-in. This tangentially addresses bazelbuild#257 since a common setup is to use the custom built interpreter in the python toolchain. For example, see: https://github.com/kku1993/bazel-hermetic-python
kku1993
added a commit
to kku1993/rules_python
that referenced
this issue
May 16, 2020
This allows users to use a custom python interpreter that is built by another repository rule instead of using a pre-built interpreter binary that is checked-in. This tangentially addresses bazelbuild#257 since a common setup is to use the custom built interpreter in the python toolchain. For example, see: https://github.com/kku1993/bazel-hermetic-python
kku1993
added a commit
to kku1993/rules_python
that referenced
this issue
May 16, 2020
This allows users to use a custom python interpreter that is built by another repository rule instead of using a pre-built interpreter binary that is checked-in. This tangentially addresses bazelbuild#257 since a common setup is to use the custom built interpreter in the python toolchain. For example, see: https://github.com/kku1993/bazel-hermetic-python
andyscott
added a commit
that referenced
this issue
Jun 24, 2020
* Support python interpreter target in pip_import. This allows users to use a custom python interpreter that is built by another repository rule instead of using a pre-built interpreter binary that is checked-in. This tangentially addresses #257 since a common setup is to use the custom built interpreter in the python toolchain. For example, see: https://github.com/kku1993/bazel-hermetic-python * Actually use interpreter path. Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>
thundergolfer
added a commit
to gmweaver/rules_python
that referenced
this issue
Jul 31, 2020
add zipextended.py to BUILD fix import path use pip internal unzip use pip internal unzip tools remove custom zipfile implementation Add documentation of community / Bazel team ownership (bazelbuild#308) This adds a more nuanced CODEOWNERS and explains its purpose in CONTRIBUTING.md. Fixes bazelbuild#291. point README readers to new 0.0.2 release (bazelbuild#302) update version in version.bzl (bazelbuild#303) Fix for when there are so many file arguments it creates the Command To Long error (bazelbuild#320) Fix failing build on CI by specifying pip package version (bazelbuild#329) `bazel build //...` was failing due to "googleapis-common-protos[grpc]" pip package being unavailable. It seems to be caused by latest googleapis-common-protos release. Specify googleapis-common-protos in requirements.txt to be in the previous version (1.51.0) to fix this. Fixes bazelbuild#321. "Skylark" is an outdated name of the language, please use "starlark" instead (bazelbuild#327) Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> Support python interpreter target in pip_import. (bazelbuild#312) * Support python interpreter target in pip_import. This allows users to use a custom python interpreter that is built by another repository rule instead of using a pre-built interpreter binary that is checked-in. This tangentially addresses bazelbuild#257 since a common setup is to use the custom built interpreter in the python toolchain. For example, see: https://github.com/kku1993/bazel-hermetic-python * Actually use interpreter path. Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> Address bazelbuild#289 (bazelbuild#328) Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> Fix errors with incompatible_disallow_empty_glob (bazelbuild#315) Allow py_library sources to be empty. If --incompatible_disallow_empty_glob is set then generated py_library targets will fail if there are no .py files. Examples are pymssql==2.1.4 and cx-Oracle==7.2.3. Set `allow_empty = True` for glob(). Bazel issue for incompatible_disallow_empty_glob: bazelbuild/bazel#8195 Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> rebuild piptool and whltool update filename, add links to pip function definitions remove typing checks chore: github setup improvements (bazelbuild#334) Remove mention and usage of Bazel Federation (bazelbuild#339) It's currently a stalled project so it's not useful for us to direct new users there in our README. Separately it is harder to develop on rules_python since it is currently not self-contained. For example it's hard to find or adjust the version of rules_pkg without looking/editing in the federation repo. Tony says this is an okay change: bazelbuild/bazel-federation@63f9746#commitcomment-40577834 leaner implementation of pip unzip remove unzip.py temp test with tools 2nd test with tools final test with tools replace with master tools merge original tools feat(examples): move examples to a nested WORKSPACE (bazelbuild#337) This lets users understand the example in isolation. They can copy/paste the example directory and it works correctly. This refactors the existing examples which are quite weak, only really demonstrating pip usage. This makes room for examples demonstrating other features (like protocol buffers) or package managers (like poetry). In a later commit I'll add bazel-integration-testing so we get a test target that confirms the examples build (including their WORKSPACE being self-contained) warn against putting .par file changes in PR. (bazelbuild#342)
thundergolfer
added a commit
to gmweaver/rules_python
that referenced
this issue
Jul 31, 2020
add zipextended.py to BUILD fix import path use pip internal unzip use pip internal unzip tools remove custom zipfile implementation Add documentation of community / Bazel team ownership (bazelbuild#308) This adds a more nuanced CODEOWNERS and explains its purpose in CONTRIBUTING.md. Fixes bazelbuild#291. point README readers to new 0.0.2 release (bazelbuild#302) update version in version.bzl (bazelbuild#303) Fix for when there are so many file arguments it creates the Command To Long error (bazelbuild#320) Fix failing build on CI by specifying pip package version (bazelbuild#329) `bazel build //...` was failing due to "googleapis-common-protos[grpc]" pip package being unavailable. It seems to be caused by latest googleapis-common-protos release. Specify googleapis-common-protos in requirements.txt to be in the previous version (1.51.0) to fix this. Fixes bazelbuild#321. "Skylark" is an outdated name of the language, please use "starlark" instead (bazelbuild#327) Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> Support python interpreter target in pip_import. (bazelbuild#312) * Support python interpreter target in pip_import. This allows users to use a custom python interpreter that is built by another repository rule instead of using a pre-built interpreter binary that is checked-in. This tangentially addresses bazelbuild#257 since a common setup is to use the custom built interpreter in the python toolchain. For example, see: https://github.com/kku1993/bazel-hermetic-python * Actually use interpreter path. Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> Address bazelbuild#289 (bazelbuild#328) Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> Fix errors with incompatible_disallow_empty_glob (bazelbuild#315) Allow py_library sources to be empty. If --incompatible_disallow_empty_glob is set then generated py_library targets will fail if there are no .py files. Examples are pymssql==2.1.4 and cx-Oracle==7.2.3. Set `allow_empty = True` for glob(). Bazel issue for incompatible_disallow_empty_glob: bazelbuild/bazel#8195 Co-authored-by: Andy Scott <andyscott@users.noreply.github.com> rebuild piptool and whltool update filename, add links to pip function definitions remove typing checks chore: github setup improvements (bazelbuild#334) Remove mention and usage of Bazel Federation (bazelbuild#339) It's currently a stalled project so it's not useful for us to direct new users there in our README. Separately it is harder to develop on rules_python since it is currently not self-contained. For example it's hard to find or adjust the version of rules_pkg without looking/editing in the federation repo. Tony says this is an okay change: bazelbuild/bazel-federation@63f9746#commitcomment-40577834 leaner implementation of pip unzip remove unzip.py temp test with tools 2nd test with tools final test with tools replace with master tools merge original tools feat(examples): move examples to a nested WORKSPACE (bazelbuild#337) This lets users understand the example in isolation. They can copy/paste the example directory and it works correctly. This refactors the existing examples which are quite weak, only really demonstrating pip usage. This makes room for examples demonstrating other features (like protocol buffers) or package managers (like poetry). In a later commit I'll add bazel-integration-testing so we get a test target that confirms the examples build (including their WORKSPACE being self-contained) warn against putting .par file changes in PR. (bazelbuild#342)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(Issue forked from #37.)
This issue tracks the fact that the Python interpreter used by
pip_import(at WORKSPACE evaluation time) is not necessarily related in any way to the Python interpreter used by the toolchain (at analysis time). This is a consequence of the fact that toolchain information is not generally available at WORKSPACE evaluation time.This can lead to confusion when the wrong major version of Python is used (before #249, it's hard to avoid this problem). It can be even more confusing when the discrepancy is only in minor version.
For major version, we could consider a modification to
whl_librarythat setssrcs_version = "PY[2|3]ONLY". This would prevent accidental mixing of pip dependencies between PY2 vs PY3, at the expense of prohibiting the same pip package from being used for both when its sources support that use case. (We could also make this an optional feature, controlled by an attribute ofpip_import.)To address the stronger problem, where you want to guarantee that exactly the same interpreter gets used at both WORKSPACE eval time and analysis, it's not clear what we can do. Perhaps we could have a repo macro/rule that generates the toolchain in such a way that it's the same system interpreter as is passed to
pip_import. Then at least users who use this construct wouldn't have a problem.Note that it doesn't make sense to use the same interpreter for WORKSPACE eval and analysis if the interpreter is not a prebuilt binary, since Bazel can't build its own tools at WORKSPACE eval time.
The text was updated successfully, but these errors were encountered: