Skip to content

TST: Run test_large_zip in a child process#15893

Merged
mattip merged 4 commits into
numpy:masterfrom
anirudh2290:move_test_io
Apr 2, 2020
Merged

TST: Run test_large_zip in a child process#15893
mattip merged 4 commits into
numpy:masterfrom
anirudh2290:move_test_io

Conversation

@anirudh2290
Copy link
Copy Markdown
Member

  1. Moved the test to its child process as per discussion on BUG: saving a huge object array with np.savez leaks memory on PyPy #15775 .
  2. Removed the skip for pypy

@mattip

@mattip
Copy link
Copy Markdown
Member

mattip commented Apr 1, 2020

Could you temporarily remove the @slow mark from the test so it runs on all platforms, just to make sure it passes? Otherwise we could miss that it could be failing

@anirudh2290
Copy link
Copy Markdown
Member Author

Looks like there is some download step on windows failing. Is there a way to retrigger the CI manually or do I need to push an empty commit ?

@mattip
Copy link
Copy Markdown
Member

mattip commented Apr 2, 2020

Looks like there is some download step on windows failing.

The goal of 2da3d4f was to check that PyPy passes the new format of the test, which it does. So if you can revert that commit I think this is ready. It would be nice to add a comment to the effect of "run in a subprocess to ensure memory is released on PyPy"

Looking at the log for PyPy, there are some tests that could be marked slow_pypy if we had such a mark. But that is too much for this PR.

@anirudh2290
Copy link
Copy Markdown
Member Author

Looking at the log for PyPy, there are some tests that could be marked slow_pypy if we had such a mark. But that is too much for this PR.

Good point! I can take a look at this and open another PR. I am wondering if we need to change the required memory to probably 8 or 9 GB for this particular test, maybe we can monitor this for a few CI runs and change if required.

@mattip mattip merged commit 081c723 into numpy:master Apr 2, 2020
@mattip
Copy link
Copy Markdown
Member

mattip commented Apr 2, 2020

Thanks @anirudh2290

@anirudh2290 anirudh2290 deleted the move_test_io branch April 4, 2020 02:48
antlarr-suse added a commit to antlarr-suse/numpy that referenced this pull request Jul 17, 2020
Since numpy#15893, test_large_zip's actual test is run in a child process,
so when this test raises a MemoryError exception, the exception is
lost and the @requires_memory decorator can't catch it to return
an xfail.

This commit uses a boolean variable in shared memory to flag if
the exception was raised, and in that case, re-raise it in the
parent process.

Fixes numpy#16889
seberg pushed a commit that referenced this pull request Jul 18, 2020
…h-16890)

Since #15893, test_large_zip's actual test is run in a child process,
so when this test raises a MemoryError exception, the exception is
lost and the @requires_memory decorator can't catch it to return
an xfail.

This commit uses a boolean variable in shared memory to flag if
the exception was raised, and in that case, re-raise it in the
parent process.

Fixes #16889
charris pushed a commit to charris/numpy that referenced this pull request Jul 19, 2020
…umpygh-16890)

Since numpy#15893, test_large_zip's actual test is run in a child process,
so when this test raises a MemoryError exception, the exception is
lost and the @requires_memory decorator can't catch it to return
an xfail.

This commit uses a boolean variable in shared memory to flag if
the exception was raised, and in that case, re-raise it in the
parent process.

Fixes numpy#16889
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants