Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a way to better connect pip_import to the Python toolchain #257

Open
brandjon opened this issue Nov 12, 2019 · 0 comments
Open

Provide a way to better connect pip_import to the Python toolchain #257

brandjon opened this issue Nov 12, 2019 · 0 comments

Comments

@brandjon
Copy link
Member

@brandjon brandjon commented Nov 12, 2019

(Issue forked from #37.)

This issue tracks the fact that the Python interpreter used by pip_import (at WORKSPACE evaluation time) is not necessarily related in any way to the Python interpreter used by the toolchain (at analysis time). This is a consequence of the fact that toolchain information is not generally available at WORKSPACE evaluation time.

This can lead to confusion when the wrong major version of Python is used (before #249, it's hard to avoid this problem). It can be even more confusing when the discrepancy is only in minor version.

For major version, we could consider a modification to whl_library that sets srcs_version = "PY[2|3]ONLY". This would prevent accidental mixing of pip dependencies between PY2 vs PY3, at the expense of prohibiting the same pip package from being used for both when its sources support that use case. (We could also make this an optional feature, controlled by an attribute of pip_import.)

To address the stronger problem, where you want to guarantee that exactly the same interpreter gets used at both WORKSPACE eval time and analysis, it's not clear what we can do. Perhaps we could have a repo macro/rule that generates the toolchain in such a way that it's the same system interpreter as is passed to pip_import. Then at least users who use this construct wouldn't have a problem.

Note that it doesn't make sense to use the same interpreter for WORKSPACE eval and analysis if the interpreter is not a prebuilt binary, since Bazel can't build its own tools at WORKSPACE eval time.

kku1993 added a commit to kku1993/rules_python that referenced this issue May 16, 2020
This allows users to use a custom python interpreter that is built by
another repository rule instead of using a pre-built interpreter binary
that is checked-in.

This tangentially addresses bazelbuild#257 since a common setup is to use the
custom built interpreter in the python toolchain.

For example, see: https://github.com/kku1993/bazel-hermetic-python
kku1993 added a commit to kku1993/rules_python that referenced this issue May 16, 2020
This allows users to use a custom python interpreter that is built by
another repository rule instead of using a pre-built interpreter binary
that is checked-in.

This tangentially addresses bazelbuild#257 since a common setup is to use the
custom built interpreter in the python toolchain.

For example, see: https://github.com/kku1993/bazel-hermetic-python
kku1993 added a commit to kku1993/rules_python that referenced this issue May 16, 2020
This allows users to use a custom python interpreter that is built by
another repository rule instead of using a pre-built interpreter binary
that is checked-in.

This tangentially addresses bazelbuild#257 since a common setup is to use the
custom built interpreter in the python toolchain.

For example, see: https://github.com/kku1993/bazel-hermetic-python
andyscott added a commit that referenced this issue Jun 24, 2020
* Support python interpreter target in pip_import.

This allows users to use a custom python interpreter that is built by
another repository rule instead of using a pre-built interpreter binary
that is checked-in.

This tangentially addresses #257 since a common setup is to use the
custom built interpreter in the python toolchain.

For example, see: https://github.com/kku1993/bazel-hermetic-python

* Actually use interpreter path.

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>
thundergolfer added a commit to gmweaver/rules_python that referenced this issue Jul 31, 2020
add zipextended.py to BUILD

fix import path

use pip internal unzip

use pip internal unzip tools

remove custom zipfile implementation

Add documentation of community / Bazel team ownership (bazelbuild#308)

This adds a more nuanced CODEOWNERS and explains its purpose in CONTRIBUTING.md.

Fixes bazelbuild#291.

point README readers to new 0.0.2 release (bazelbuild#302)

update version in version.bzl (bazelbuild#303)

Fix for when there are so many file arguments it creates the Command To Long error (bazelbuild#320)

Fix failing build on CI by specifying pip package version (bazelbuild#329)

`bazel build //...` was failing due to "googleapis-common-protos[grpc]"
pip package being unavailable.

It seems to be caused by latest googleapis-common-protos release.
Specify googleapis-common-protos in requirements.txt to be in the previous
version (1.51.0) to fix this.

Fixes bazelbuild#321.

"Skylark" is an outdated name of the language, please use "starlark" instead (bazelbuild#327)

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

Support python interpreter target in pip_import. (bazelbuild#312)

* Support python interpreter target in pip_import.

This allows users to use a custom python interpreter that is built by
another repository rule instead of using a pre-built interpreter binary
that is checked-in.

This tangentially addresses bazelbuild#257 since a common setup is to use the
custom built interpreter in the python toolchain.

For example, see: https://github.com/kku1993/bazel-hermetic-python

* Actually use interpreter path.

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

Address bazelbuild#289 (bazelbuild#328)

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

Fix errors with incompatible_disallow_empty_glob (bazelbuild#315)

Allow py_library sources to be empty.
If --incompatible_disallow_empty_glob is set then generated py_library
targets will fail if there are no .py files.

Examples are pymssql==2.1.4 and cx-Oracle==7.2.3.

Set `allow_empty = True` for glob().

Bazel issue for incompatible_disallow_empty_glob:
bazelbuild/bazel#8195

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

rebuild piptool and whltool

update filename, add links to pip function definitions

remove typing checks

chore: github setup improvements (bazelbuild#334)

Remove mention and usage of Bazel Federation (bazelbuild#339)

It's currently a stalled project so it's not useful for us to direct new users there in our README.
Separately it is harder to develop on rules_python since it is currently not self-contained.
For example it's hard to find or adjust the version of rules_pkg without looking/editing in the federation repo.
Tony says this is an okay change: bazelbuild/bazel-federation@63f9746#commitcomment-40577834

leaner implementation of pip unzip

remove unzip.py

temp test with tools

2nd test with tools

final test with tools

replace with master tools

merge original tools

feat(examples): move examples to a nested WORKSPACE (bazelbuild#337)

This lets users understand the example in isolation. They can copy/paste the example directory
and it works correctly.

This refactors the existing examples which are quite weak, only really demonstrating pip usage.
This makes room for examples demonstrating other features (like protocol buffers) or package
managers (like poetry).

In a later commit I'll add bazel-integration-testing so we get a test target that confirms
the examples build (including their WORKSPACE being self-contained)

warn against putting .par file changes in PR. (bazelbuild#342)
thundergolfer added a commit to gmweaver/rules_python that referenced this issue Jul 31, 2020
add zipextended.py to BUILD

fix import path

use pip internal unzip

use pip internal unzip tools

remove custom zipfile implementation

Add documentation of community / Bazel team ownership (bazelbuild#308)

This adds a more nuanced CODEOWNERS and explains its purpose in CONTRIBUTING.md.

Fixes bazelbuild#291.

point README readers to new 0.0.2 release (bazelbuild#302)

update version in version.bzl (bazelbuild#303)

Fix for when there are so many file arguments it creates the Command To Long error (bazelbuild#320)

Fix failing build on CI by specifying pip package version (bazelbuild#329)

`bazel build //...` was failing due to "googleapis-common-protos[grpc]"
pip package being unavailable.

It seems to be caused by latest googleapis-common-protos release.
Specify googleapis-common-protos in requirements.txt to be in the previous
version (1.51.0) to fix this.

Fixes bazelbuild#321.

"Skylark" is an outdated name of the language, please use "starlark" instead (bazelbuild#327)

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

Support python interpreter target in pip_import. (bazelbuild#312)

* Support python interpreter target in pip_import.

This allows users to use a custom python interpreter that is built by
another repository rule instead of using a pre-built interpreter binary
that is checked-in.

This tangentially addresses bazelbuild#257 since a common setup is to use the
custom built interpreter in the python toolchain.

For example, see: https://github.com/kku1993/bazel-hermetic-python

* Actually use interpreter path.

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

Address bazelbuild#289 (bazelbuild#328)

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

Fix errors with incompatible_disallow_empty_glob (bazelbuild#315)

Allow py_library sources to be empty.
If --incompatible_disallow_empty_glob is set then generated py_library
targets will fail if there are no .py files.

Examples are pymssql==2.1.4 and cx-Oracle==7.2.3.

Set `allow_empty = True` for glob().

Bazel issue for incompatible_disallow_empty_glob:
bazelbuild/bazel#8195

Co-authored-by: Andy Scott <andyscott@users.noreply.github.com>

rebuild piptool and whltool

update filename, add links to pip function definitions

remove typing checks

chore: github setup improvements (bazelbuild#334)

Remove mention and usage of Bazel Federation (bazelbuild#339)

It's currently a stalled project so it's not useful for us to direct new users there in our README.
Separately it is harder to develop on rules_python since it is currently not self-contained.
For example it's hard to find or adjust the version of rules_pkg without looking/editing in the federation repo.
Tony says this is an okay change: bazelbuild/bazel-federation@63f9746#commitcomment-40577834

leaner implementation of pip unzip

remove unzip.py

temp test with tools

2nd test with tools

final test with tools

replace with master tools

merge original tools

feat(examples): move examples to a nested WORKSPACE (bazelbuild#337)

This lets users understand the example in isolation. They can copy/paste the example directory
and it works correctly.

This refactors the existing examples which are quite weak, only really demonstrating pip usage.
This makes room for examples demonstrating other features (like protocol buffers) or package
managers (like poetry).

In a later commit I'll add bazel-integration-testing so we get a test target that confirms
the examples build (including their WORKSPACE being self-contained)

warn against putting .par file changes in PR. (bazelbuild#342)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.