https://docs.python.org/3/c-api/init.html#c.Py_EndInterpreter states that "Py_FinalizeEx() will destroy all sub-interpreters that haven’t been explicitly destroyed at that point."

As discussed in https://github.com/hexchat/hexchat/issues/2237, Python 3.7+ doesn't currently do that - it calls Py_FatalError instead.

That change came from https://github.com/python/cpython/pull/1728, which was based on my initial PEP 432 refactoring work, and I didn't realise that implicitly cleaning up lingering subinterpreters was a documented behaviour.

So I think we should just fix it to behave as documented, and add a new regression test to make sure it doesn't get broken again in the future.

I have been wondering where the regression to test this can be put..in test__xxsubinterpreters.py may be?

Joannah, yes, that looks like a good place. Eric Snow might have more info; he wrote that module.

As for testing Py_FatalError, there's an assert_python_failure function in test.support.script_helper.

Interestingly, I noticed this independently today. :)

Here's what I wrote in #36477 (which I've closed as a duplicate):

When using subinterpreters, any that exist when Py_FinalizeEx() is called do not appear to get cleaned up during runtime finalization.  Maybe I've been looking at the code too much and I'm missing something. :)

This really isn't a problem except for embedders that use subinterpreters (where we're leaking memory).  However, even with the "python" executable it can have an impact because the subinterpreters' non-daemon threads will exit later than expected. (see #36469 & #36476)

The solution would be to finalize all subinterpreters at the beginning of Py_FinalizeEx(), right before the call to wait_for_thread_shutdown().  This means calling Py_EndInterpreter() for all the runtime's interpreters (except the main one).  It would also mean setting a flag (_PyRuntime.interpreters.finalizing?) right before that to disallow creation of any more subinterptreters.

test__xxsubinterpreters is a great place for tests that exercise use of subinterpreters, including most lifecycle operations.  There are also one or two subinterpreter-related tests in test_embed.  However, for this issue the interplay with runtime finalization means tests should probably stay with other tests that exercise Py_FinalizeEx().

I think test_embed would be the right home for this, as there's an existing test case there for subinterpreter lifecycles and repeated init/finalize cycles: https://github.com/python/cpython/blob/ddbb978e1065dde21d1662386b26ded359f4b16e/Programs/_testembed.c#L43

The test case here would be similar, but it wouldn't need the outer loop - it would just create a handful of subinterpreters, but instead of ending each one before creating the next one the way the existing test does, what it would instead do is:

* setup as per the existing test case
* create a pair of subinterpeters, using a copy of the existing loop, but omitting the `Py_EndInterpreter` call
* switch back to the main interpreter
* create a second pair of subinterpeters
* switch back to the main interpreter
* call Py_Finalize

It also occurs to me that we don't currently have a test case for what happens if you call Py_Finalize from a subinterpreter rather than the main interpreter.

Ping.  It was marked as a 3.7regression but perhaps it should now be just targeted for 3.8.

I am investigating this but in the meantime.

> It also occurs to me that we don't currently have a test case for what happens if you call Py_Finalize from a sub-interpreter rather than the main interpreter.

@ncoghlan Am moving this test request in a new issue. So that this issue only focuses on fixing the lingering sub-interpreters.

The test request is moved to issue37776.

I remember julien wanting to check this out during a discussion we had at the sprints hence the loop in.