bpo-41538: Make python interpreter customizable for EnvBuilder #21854

abbec · 2020-08-13T11:35:15Z

This adds a new keyword parameter python_interpreter to
venv.EnvBuilder that lets the user customize which python to use when
creating the virtualenv, much like the virtualenv command's -p flag.

The reason that this is needed is that for some embedded cases (Blender
being one example), sys._base_executable is not set to a valid Python
executable.

https://bugs.python.org/issue41538

the-knights-who-say-ni · 2020-08-13T11:35:18Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@abbec

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

vsajip

Not sure python_interpreter is the best name - no other sort of interpreter would be valid here, right? Perhaps you could use executable or venv_executable as the keyword argument and attribute name, as that's more commonly used to refer to a path to a Python interpreter executable (for example, sys.executable for the running interpreter).

Also, a test should be implemented for this (which would only be run if an alternative Python was available in the system path, and skipped otherwise).

bedevere-bot · 2020-08-15T18:53:48Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be put in the comfy chair!

abbec · 2020-08-18T06:43:09Z

Hi, thanks for the feedback! I have made the requested changes; please review again.

bedevere-bot · 2020-08-18T06:43:12Z

Thanks for making the requested changes!

@vsajip: please review the changes made to this pull request.

vsajip · 2020-08-19T21:22:40Z

Lib/test/test_venv.py

@@ -33,6 +33,12 @@
    or sys._base_executable != sys.executable,
    'cannot run venv.create from within a venv on this platform')

+requireAlternativePythonInPath = unittest.skipUnless(
+        shutil.which("python") is not None


This won't work if python resolves to a Python 2.x interpreter, as they don't know about Python's built-in venv (which appeared in Python 3.3). See comment below on how test should be structured.

vsajip · 2020-08-19T21:47:52Z

Lib/test/test_venv.py

+
+        # make sure it can be created with the alternative executable
+        rmtree(self.env_dir)
+        self.run_with_capture(venv.create, self.env_dir)


This won't work because venv.create() won't call the builder you just created. Which makes me think:

venv.create() should also have the executable keyword argument, and pass it through to the builder that it creates.

The main function should parse an additional --executable argument to allow an executable to be passed in from the command line, and pass that to the builder it creates.

Also, I would suggest structuring the test as follows:

Remove the requireAlternativePython decorator and function.

Instead, do something along the lines of

# create temp dir to hold venv found_a_python = False for minor in range(3, 11): executable = 'python3.%s' % minor alt_python = shutil.which(executable) if alt_python and alt_python != sys._base_executable: found_a_python = True # create builder with executable=alt_python # remove venv dir # create venv using the builder # check that venv creation appeared to work correctly # check that venv creation used the correct python interpreter by invoking it with # <venv_dir>/bin/python --version and checking what comes back is what's expected if not found_a_python: raise unittest.SkipTest('No alternative Python interpreters found for venv executable test')

Yes, of course, will fix!

Possibly also add a "python3" just in case there's no "python3.X" available.

bedevere-bot · 2020-08-19T21:48:11Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vsajip · 2020-08-19T21:54:23Z

Actually, instead of range(3, 11) you should perhaps use range(3, sys.version_info[1] + 1). This line then shouldn't need revisiting for the rest of the 3.X series.

abbec · 2020-08-20T09:40:12Z

Actually, instead of range(3, 11) you should perhaps use range(3, sys.version_info[1] + 1). This line then shouldn't need revisiting for the rest of the 3.X series.

Yeah, will do that, thanks for the feedback.

This adds a new keyword parameter `python_interpreter` to `venv.EnvBuilder` that lets the user customize which python to use when creating the virtualenv, much like the virtualenv command's `-p` flag. The reason that this is needed is that for some embedded cases (Blender being one example), `sys._base_executable` is not set to a valid Python executable.

This test case verifies that if you set a custom executable, it is actually being used and the virtualenv can be created using it. Note that this test is skipped if an alternative python can not be found in path.

abbec · 2021-01-18T10:25:43Z

Hey, sorry for the long delay but I have made the requested changes; please review again :)

bedevere-bot · 2021-01-18T10:25:47Z

Thanks for making the requested changes!

@vsajip: please review the changes made to this pull request.

abbec · 2021-01-18T10:26:28Z

Also, I can rebase the branch and squash the commit if you want

abbec · 2021-01-19T19:20:46Z

Will look at the build failures

Also, update the test.

abbec requested a review from vsajip as a code owner August 13, 2020 11:35

the-knights-who-say-ni added the CLA not signed label Aug 13, 2020

bedevere-bot added the awaiting review label Aug 13, 2020

abbec force-pushed the venv-python-interpreter-issue-41538 branch from b82b341 to e7c65fd Compare August 13, 2020 15:03

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Aug 13, 2020

vsajip requested changes Aug 15, 2020

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Aug 15, 2020

bedevere-bot added awaiting change review and removed awaiting changes labels Aug 18, 2020

bedevere-bot requested a review from vsajip August 18, 2020 06:43

vsajip requested changes Aug 19, 2020

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting change review labels Aug 19, 2020

abbec added 3 commits January 18, 2021 08:26

Rename python_interpreter to executable

163526b

Add venv custom executable test case

de0a283

This test case verifies that if you set a custom executable, it is actually being used and the virtualenv can be created using it. Note that this test is skipped if an alternative python can not be found in path.

abbec force-pushed the venv-python-interpreter-issue-41538 branch from 64900d4 to 9d0a5d1 Compare January 18, 2021 10:25

bedevere-bot added awaiting change review and removed awaiting changes labels Jan 18, 2021

bedevere-bot requested a review from vsajip January 18, 2021 10:25

abbec force-pushed the venv-python-interpreter-issue-41538 branch from 9d0a5d1 to f6d39bf Compare January 19, 2021 19:28

Add executable argument to venv.create and CLI

728781f

Also, update the test.

abbec force-pushed the venv-python-interpreter-issue-41538 branch from f6d39bf to 728781f Compare January 19, 2021 19:34

ezio-melotti removed the CLA signed label Jul 13, 2022

bpo-41538: Make python interpreter customizable for EnvBuilder #21854

bpo-41538: Make python interpreter customizable for EnvBuilder #21854

abbec commented Aug 13, 2020 •

edited by bedevere-bot

the-knights-who-say-ni commented Aug 13, 2020

vsajip left a comment

bedevere-bot commented Aug 15, 2020

abbec commented Aug 18, 2020

bedevere-bot commented Aug 18, 2020

vsajip Aug 19, 2020

vsajip Aug 19, 2020

abbec Aug 20, 2020

vsajip Aug 20, 2020

bedevere-bot commented Aug 19, 2020

vsajip commented Aug 19, 2020

abbec commented Aug 20, 2020

abbec commented Jan 18, 2021

bedevere-bot commented Jan 18, 2021

abbec commented Jan 18, 2021

abbec commented Jan 19, 2021

bpo-41538: Make python interpreter customizable for EnvBuilder #21854

Are you sure you want to change the base?

bpo-41538: Make python interpreter customizable for EnvBuilder #21854

Conversation

abbec commented Aug 13, 2020 • edited by bedevere-bot

the-knights-who-say-ni commented Aug 13, 2020

CLA Missing

vsajip left a comment

Choose a reason for hiding this comment

bedevere-bot commented Aug 15, 2020

abbec commented Aug 18, 2020

bedevere-bot commented Aug 18, 2020

vsajip Aug 19, 2020

Choose a reason for hiding this comment

vsajip Aug 19, 2020

Choose a reason for hiding this comment

abbec Aug 20, 2020

Choose a reason for hiding this comment

vsajip Aug 20, 2020

Choose a reason for hiding this comment

bedevere-bot commented Aug 19, 2020

vsajip commented Aug 19, 2020

abbec commented Aug 20, 2020

abbec commented Jan 18, 2021

bedevere-bot commented Jan 18, 2021

abbec commented Jan 18, 2021

abbec commented Jan 19, 2021

abbec commented Aug 13, 2020 •

edited by bedevere-bot