Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regr_test.py: Allow non-types dependencies #9382

Merged
merged 14 commits into from Dec 23, 2022

Conversation

AlexWaygood
Copy link
Member

@AlexWaygood AlexWaygood commented Dec 18, 2022

Work towards #5768

This PR reworks tests/regr_test.py so that the script will still succeed if test cases are added for a stubs package that has non-types dependencies. In the process, it undertakes a lot of the refactoring that will be needed to allow non-types dependencies in mypy_test.py.

The approach is very similar to the approach we take in stubtest_third_party.py: dynamically create virtual environments on the fly in order to test stubs in isolation. Unlike stubtest_third_party.py, however, we only create the venv if it's a package that has non-types dependencies listed. Creating a venv for each package with test cases, even if it's a package that didn't have non-types dependencies, would make the script unacceptably slow.

Cc. @Avasam

@AlexWaygood AlexWaygood marked this pull request as ready for review Dec 18, 2022
@AlexWaygood
Copy link
Member Author

AlexWaygood commented Dec 18, 2022

  • The CI run on commit c60c379 shows what happens if we add non-types dependencies to stubs packages that have test cases
  • The CI run on commit 9cb0b22 shows what happens if we use a non-types dependency in a test case
  • The CI run on commit 4800d60 shows that the test cases still fail, as expected, if there's something bad in them (i.e., this refactoring hasn't broken anything).

tests/utils.py Outdated
assert dependency.startswith("types-"), f"unrecognized dependency {dependency!r}"
dependencies.append(dependency[6:].split("<")[0])
return tuple(dependencies)
if dependency.startswith("types-"):
Copy link
Collaborator

@hauntsaninja hauntsaninja Dec 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a problem here, but there are two cases where this logic may do something slightly surprising / different from stub_uploader which will consider something a typeshed stub iff it's something it's uploaded. So maybe worth adding a comment:

This function may consider things to be typeshed stubs even if they haven't been uploaded to PyPI. If a typeshed stub is removed, this function will consider it to be an external dependency.

Copy link
Member Author

@AlexWaygood AlexWaygood Dec 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your comment made me realise that there is also an edge case that this function doesn't currently handle: stubs which are uploaded with nonstandard names to PyPI. It'll break, for example, if we ever get a stubs package that depends on types-pika-ts.

This is unlikely to come up in real life, but I may as well fix it.

Copy link
Member Author

@AlexWaygood AlexWaygood Dec 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does 42f722f look?

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Dec 20, 2022

I realised that if types-requests started depending on urllib3 instead of types-urllib3 (for example), we'd end up creating 15 different venvs and pip installing urllib3 15 times, just to run the test cases for types-requests. That would be very silly, and very slow. e948601 significantly optimizes for that. We now only create one tempdir for each test-case directory; the full python-version/platform matrix is now run inside the same directory.

9823065 adds much better logging configurability. You can now run --verbosity VERBOSE to see exactly when tempdirs and venvs are being created, when things are being pip installed, exactly what subprocess commands are being run, etc.

tests/regr_test.py Outdated Show resolved Hide resolved
@AlexWaygood
Copy link
Member Author

AlexWaygood commented Dec 23, 2022

I'm planning on merging this in a few days, if there are no objections from anybody, as I'd like to move on to working on mypy_test.py :)

tests/regr_test.py Outdated Show resolved Hide resolved
@JelleZijlstra JelleZijlstra merged commit 8671fc5 into python:main Dec 23, 2022
57 checks passed
@AlexWaygood AlexWaygood deleted the regr-test-non-types branch Dec 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants