@duggelz Do you have any suggestions on the best way to download and invoke pip without using get-pip.py to install it as a system-wide dependency? This seems to be why the OSX build is failing.

@sebgoa You'd asked for this at one point, if you are still interested I'd love your feedback.

I moved whl.sh into a little Python script that I can invoke directly, but also reasonably unit test.

I have not moved pip.sh yet, and thinking about it left me with an idea. Considering:

• Shelling out to pip from Python is a bit silly,
• @damienmg's trick won't work if I have a pip library dependency,
• I still haven't solved the pip fetching problem.

What if we build / publish a PAR (similar to the process rules_docker uses), which links the pip library, and removes the need to get-pip.py somehow? WDYT?

I still haven't found a clean process for doing this from a single repo (google/containerregistry and bazelbuild/rules_docker are separate repos), but that seems like a smaller problem to solve.

I found this, which may be relevant to why certs aren't working properly from pip in a PAR file. It is similar to the trick we do for httplib2 in google/containerregistry tools.

@damienmg Can we add a buildifier check to the Jenkins job, or do I have to set up Travis for that?

Not right now unfortunately. I want to add it (bazelbuild/continuous-integration#13) but we kind of need bazelbuild/continuous-integration#100.

Design question for you:

Assumption: Bazel isn't doing some super-clever multi-level hierarchical WORKSPACE namespacing that I'm not aware of.

Scenario setup:

Workspace A depends on pip package P1 at version V1. Workspace B depends on pip package P1 at version V2. Workspace C depends on A and B via git_repository() rules in c/WORKSPACE file. A and B have a_install() and b_install() functions that are also called in c/WORKSPACE. A and B have py_xxx targets that C will use in some manner.

Scenario 1: V1 = V2

Result: Bazel rejects the configuration because a_install and b_install create duplicate targets.

Question: Can B detect that it doesn't need to create P1 target?

Scenario 2: V1 != V2

Result: Same as Scenario 1

Scenario 3: V1 != V2, and we only create P1 once

Result: Build succeeds, but Python programs in C might get V1 or V2 depending on the order of things in c/WORKSPACE.

Scenario 4: V1 != V2, and we incorporate versions into target labels like mock_2_0_0 instead of mock, so we create two P1 sets of targets.

Now assume a py_binary in C depends on a py_library from A and another py_library from B.

Result: Build succeeds, but Python programs in C can depend on P1.V1 and P1.V2 without Bazel realizing there is a clash. Python import semantics mean that the program might get files from V1, V2, or both, at runtime, in fairly hard to predict ways.

Note that current Google repos that manually depend on pip packages in WORKSPACE via http_repository and similar, use both naming conventions (@mock and @mock_2_0_0), so we actually get all of the problems described above.

Scenario 5: We require that V1 = V2, and make it work correctly.

Result: A, B and C are now version-deadlocked. C can never update unless A and B happen to both update to the same V3. Now multiply by the combinatorial explosion of dependencies and versions that we find in the real world.

Suggestion: I don't think this can be completely solved without making new py_library() and py_binary() rules that delay resolving py_library() deps until the final py_binary() step.

In the meantime: I prefer baking version numbers into package labels (e.g. @mock_2_0_0// instead of @mock) so we don't have deadlock. The missing piece is something in Bazel that looks at deps, or runtimes, or I'm not sure what, and complains loudly if it finds @foo_anything and @foo_anything_else in the same target.

@duggelz I think that's a very good question, and I think it breaks down where you have:

A and B have a_install() and b_install() functions that are also called in c/WORKSPACE

As you are clearly driving at, we have 3 sets of requirements that are being merged after the constraints have been solved, which I think is a losing proposition.

I think that my expectation is that instead of a_install and b_install we would pip_import a blended requirements.txt from A, B, C, and just abc_install.

This story is only half-baked because:

1. We don't have a rule for combining @A//:requirements.txt and @B//:requirements.txt (but I suspect this is pretty easy).
2. A, B, and C may each references this collection of dependencies with different repository names (but I believe this is doable by aliasing repositories with bind). At one point, with this exact problem in mind, I debated removing users capacity to name the repository itself, which would effectively guarantee a single resolution step, but I thought this was a bit extreme (e.g. what if I want to write an invoked tool that has its own requirements.txt, that's ok).

I think the crux of this issue is that WORKSPACE currently serves two roles (I hint at this above):

1. Resolving dependencies based on fuzzy constraints (incl. transitive resolution).
2. Fetching dependencies based on concrete specification.

The analogs to this outside of Bazel are things like [frozen] requirements.txt or package[-lock].json.

I think the guidance should be for users not to mix independently resolved external dependencies, but I'm not 100% sure if/how we can enforce it. Perhaps we can do this through Bazel's constraint system, but I have not looked any further at how we might express this.

Is what I'm describing clear? Does my logic make sense?

@mattmoor I agree with what you wrote. Some additional points:

1. Bazel has a native.existing_rules that make fixing my points 1 and 2 easy.
2. Looking at various repository rules, there is a nice omit_foo pattern that allows C to override A and B as desired.
3. The conflict between "fuzzy constraints" and "concrete specification" is real. Python aims to address this with Pipfile vs Pipfile.lock, but meanwhile we muddle through.

I don't think we need to make a tool that tries to merge A and B automatically or anything. I think it's reasonable to say that C must provide a requirements.txt that satisfies every need that A and B have.

After reading some discussions about Java/Maven, I do think that we should put the names in a canonical format that we document and agree on. Specifically, I think we should have the literal prefix pypi_ on names that would otherwise be very short and ambigous. E.g. @pypi__package_name_here//:package_name or @pypi__package_name_here__version_here//:package_name

@duggelz Given that things are hidden behind package(...) in the expected usage today, I think this is completely fine. The main change is that we wouldn't namespace packages based on the enclosing pip_import, which I think is fine.

I've gone with pypi__{distribution}_{version}, used existing_rules to avoid multiple imports, but each pip_import will still exposes the library through package() even if it didn't import it. I included a section in the README.md covering canonical naming.

@duggelz Hooray, green CI :)