This C code:

#include <Python.h>
int main(int argc, char *argv[])
{
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
}

Produces this output:
[7438 refs]
[7499 refs]
[7550 refs]
[7601 refs]
[7652 refs]
[7703 refs]
[7754 refs]

A similar program configured to call the Py_Initialize()/Py_Finalize() 1000 times ends up with:
...
[58295 refs]
[58346 refs]
[58397 refs]

This is with a fresh debug build of Python 2.5.0 on Windows XP, using Visual C++ 2003.

Does the title of this issue accurately reflect the current status of the Python interpreter?

Yes, some objects are not cleaned in finalization.
This is not a problem in usual cases though, when the interpreter is
started only once.

> Does the title of this issue accurately reflect the current status of the Python interpreter?

Yes, here is the running result on current 3.3 latest code:
[37182 refs]
[39415 refs]
[41607 refs]
[43799 refs]
[45991 refs]
[48183 refs]
[50375 refs]


This seems to be a known bug that Py_Finalize() doesn't free all objects according doc http://docs.python.org/dev/c-api/init.html?highlight=py_finalize#Py_Finalize

Interestingly enough, some of the leaked memory came from the finalize routine itself! Here's one example:

0:004> !heap -p -a 0x000000DB144346F0
    address 000000db144346f0 found in
    _HEAP @ db0cae0000
              HEAP_ENTRY Size Prev Flags            UserPtr UserSize - state
        000000db14434690 030a 0000  [00]   000000db144346c0    03074 - (busy)
        7ffc55628b04 ntdll!RtlpCallInterceptRoutine+0x0000000000000040
        7ffc555f9f36 ntdll!RtlAllocateHeap+0x0000000000079836
        7ffc2a60c4da ucrtbased!calloc_base+0x000000000000123a
        7ffc2a60c27d ucrtbased!calloc_base+0x0000000000000fdd
        7ffc2a60f34f ucrtbased!malloc_dbg+0x000000000000002f
        7ffc2a60fdde ucrtbased!malloc+0x000000000000001e
        5a5e6ef9 python36_d!_PyMem_RawMalloc+0x0000000000000029
        5a5e78c7 python36_d!_PyMem_DebugAlloc+0x0000000000000087
        5a5e5e6f python36_d!_PyMem_DebugMalloc+0x000000000000001f
        5a5e7230 python36_d!PyMem_Malloc+0x0000000000000030
        5a582047 python36_d!new_keys_object+0x0000000000000077
        5a57f7c5 python36_d!dictresize+0x0000000000000085
        5a57a4b2 python36_d!PyDict_Merge+0x0000000000000112
        5a57bf33 python36_d!PyDict_Update+0x0000000000000023
        5a75fb1d python36_d!PyImport_Cleanup+0x000000000000045d
        5a778f9e python36_d!Py_Finalize+0x000000000000005e

I tested on the master branch of Python:
---
#include <Python.h>

void func()
{
    Py_Initialize(); Py_Finalize();
    Py_ssize_t cnt = _Py_GetRefTotal();
    printf("sys.gettotalrefcount(): %zd\n", cnt);
}

int main(int argc, char *argv[])
{
    Py_SetProgramName(L"./_testembed");
    for (int i=0; i < 10; i++) {
        func();
    }
}
---

Each iteration leaks around 5,000 Python objects:
---
sys.gettotalrefcount(): 15113
sys.gettotalrefcount(): 19527
sys.gettotalrefcount(): 23941
sys.gettotalrefcount(): 28355
sys.gettotalrefcount(): 32769
sys.gettotalrefcount(): 37183
sys.gettotalrefcount(): 41597
sys.gettotalrefcount(): 46011
sys.gettotalrefcount(): 50425
sys.gettotalrefcount(): 54839
---

I marked bpo-6741 as a duplicate of this issue.

I marked bpo-26888 as a duplicate of this issue.

I marked bpo-21387 as a duplicate of this issue.

One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP 489 "Multi-phase extension module initialization".

I marked bpo-32026 as a duplicate of this issue.

One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP 489 "Multi-phase extension module initialization".
> I try to port _json extension module to multiphase initialization module, but the baseline(using victor's code) in my vm not changed~

Compare to _Py_ForgetReference(), _Py_INC_REFTOTAL in _Py_NewReference() looks redundant.

REF: https://github.com/python/cpython/blob/master/Include/object.h#L442

master brach baseline in my vm:
```
sys.gettotalrefcount(): 18049
sys.gettotalrefcount(): 22463
```

after PR17883
```
sys.gettotalrefcount(): 17589
sys.gettotalrefcount(): 22000
```

FWIW, i counted the difference of each file's refs after `Py_Finalize()`.

[('Objects/dictobject.c', 21434), ('Python/marshal.c', 8135), ('Objects/codeobject.c', 6245), ('Objects/listobject.c', 6037), ('Objects/tupleobject.c', 4169), ('Objects/boolobject.c', 2433), ('Objects/object.c', 2364), ('Objects/unicodeobject.c', 1541), ('Objects/longobject.c', 1387), ('Objects/funcobject.c', 528), ('Objects/classobject.c', 528), ('Objects/abstract.c', 463), ('Python/structmember.c', 369), ('./Include/objimpl.h', 277), ('Objects/stringlib/partition.h', 273), ('Python/import.c', 259), ('Python/codecs.c', 197), ('./Modules/signalmodule.c', 61), ('./Modules/_threadmodule.c', 59), ('Objects/exceptions.c', 15), ('Objects/bytesobject.c', 5), ('./Modules/_weakref.c', 4), ('Python/_warnings.c', 3), ('./Modules/timemodule.c', 1), ('./Modules/_codecsmodule.c', 1), ('Objects/bytearrayobject.c', 1), ('Python/compile.c', 1), ('Objects/sliceobject.c', 0), ('Objects/memoryobject.c', 0), ('Python/context.c', -1), ('Objects/clinic/longobject.c.h', -1), ('Objects/enumobject.c', -1), ('Modules/gcmodule.c', -1), ('Objects/namespaceobject.c', -1), ('Objects/stringlib/unicode_format.h', -2), ('Objects/rangeobject.c', -3), ('Python/pystate.c', -4), ('Objects/fileobject.c', -14), ('./Modules/_io/clinic/bufferedio.c.h', -17), ('./Modules/_io/iobase.c', -21), ('Python/modsupport.c', -28), ('./Modules/_io/fileio.c', -28), ('Python/pylifecycle.c', -37), ('./Modules/_io/textio.c', -39), ('Objects/genobject.c', -53), ('Objects/weakrefobject.c', -54), ('./Modules/_io/bufferedio.c', -56), ('./Python/sysmodule.c', -68), ('./Modules/_io/_iomodule.c', -82), ('Python/errors.c', -90), ('Objects/descrobject.c', -110), ('Objects/structseq.c', -113), ('Python/bltinmodule.c', -118), ('Objects/setobject.c', -339), ('Objects/moduleobject.c', -454), ('./Modules/posixmodule.c', -614), ('./Modules/_abc.c', -664), ('Objects/call.c', -755), ('Objects/typeobject.c', -2035), ('Objects/frameobject.c', -6538), ('Python/ceval.c', -7857), ('./Include/object.h', -48292)]

New changeset ed154c387efc5f978ec97900ec9e0ec6631d5498 by Victor Stinner (Hai Shi) in branch 'master':
bpo-1635741: Port _json extension module to multiphase initialization (PEP 489) (GH-17835)
https://github.com/python/cpython/commit/ed154c387efc5f978ec97900ec9e0ec6631d5498

i thinkt that not checking `PyModule_AddObject()`'s result may cause this probleam too.

1) python-ast.c have one question, i fix it in PR18358.
2) most of the questions in extension module, for example: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L2019-L2022

update the above info:
1) python-ast.c have one question, i fix it in PR18365.

> 1) python-ast.c have one question, i fix it in PR18365.
> 2) most of the questions in extension module, for example: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L2019-L2022

brandt does relevant work already in PR17276、PR38823.

New changeset 1ea45ae257971ee7b648e3b031603a31fc059f81 by Hai Shi in branch 'master':
bpo-1635741: Port _codecs extension module to multiphase initialization (PEP 489) (GH-18065)
https://github.com/python/cpython/commit/1ea45ae257971ee7b648e3b031603a31fc059f81

Leave a note for myself:
I check the remaining object roughly(though dump_refs function), most of remaining object is 'str', such as:
'0x7f779cf88880 [13] str'->'0x7f779cf88880 [26] str'

So far, I don't know which file and fileno create those object. MAYBE I need find a hack way to sign this mallocing operation?(not sure)

New changeset b2b6e27bcab44e914d0a0b170e915d6f1604a76d by Hai Shi in branch 'master':
bpo-1635741: Port _crypt extension module to multiphase initialization (PEP 489) (GH-18404)
https://github.com/python/cpython/commit/b2b6e27bcab44e914d0a0b170e915d6f1604a76d

New changeset 7d7956833cc37a9d42807cbfeb7dcc041970f579 by Hai Shi in branch 'master':
bpo-1635741: Port _contextvars module to multiphase initialization (PEP 489) (GH-18374)
https://github.com/python/cpython/commit/7d7956833cc37a9d42807cbfeb7dcc041970f579

New changeset 4c1b6a6f4fc46add0097efb3026cf3f0c89f88a2 by Hai Shi in branch 'master':
bpo-1635741: Port _abc extension to multiphase initialization (PEP 489) (GH-18030)
https://github.com/python/cpython/commit/4c1b6a6f4fc46add0097efb3026cf3f0c89f88a2

New changeset 5d38517aa1836542a5417b724c093bcb245f0f47 by Hai Shi in branch 'master':
bpo-1635741: Port _bz2 extension module to multiphase initialization(PEP 489) (GH-18050)
https://github.com/python/cpython/commit/5d38517aa1836542a5417b724c093bcb245f0f47

New changeset a158168a787e82c4b7b18f6833153188e93627a5 by Hai Shi in branch 'master':
bpo-1635741: Port _locale extension module to multiphase initialization (PEP 489) (GH-18358)
https://github.com/python/cpython/commit/a158168a787e82c4b7b18f6833153188e93627a5

New changeset 41fbf865a35d4fb64f047f98dc24690cb0c170fd by Hai Shi in branch 'master':
bpo-1635741: Port audioop extension module to multiphase initialization (PEP 489) (GH-18608)
https://github.com/python/cpython/commit/41fbf865a35d4fb64f047f98dc24690cb0c170fd

New changeset aa0c0808efbfdee813d2829e49030c667da44e72 by Hai Shi in branch 'master':
bpo-1635741: Fix potential refleaks in binascii module (GH-18613)
https://github.com/python/cpython/commit/aa0c0808efbfdee813d2829e49030c667da44e72

Thanks Hai Shi for your 3 latest PRs, I merged them.

New changeset 196f1eb6adcfc6a7239330ef508b8bf9dff9940f by Hai Shi in branch 'master':
bpo-1635741: Fix refleaks of time module error handling (GH-18486)
https://github.com/python/cpython/commit/196f1eb6adcfc6a7239330ef508b8bf9dff9940f

hundreds of encoding names can not be released in Py_Finalize().
for example: 
```
0x7ff482f589e0 [1] 'iso_8859_1_1987'
0x7ff482f58970 [1] 'iso_8859_1'
```
-->
```
0x7ff482f589e0 [2] 'iso_8859_1_1987'
0x7ff482f58970 [2] 'iso_8859_1'
```

New changeset 356c878fbf2a97aa3ab7951fd7456d219ff0b466 by Dong-hee Na in branch 'master':
bpo-1635741: Port _statistics module to multiphase initialization (GH-19015)
https://github.com/python/cpython/commit/356c878fbf2a97aa3ab7951fd7456d219ff0b466

New changeset 2037502613471a0a0a0262085cc50adb378ebbad by Hai Shi in branch 'master':
bpo-1635741: Port  _ctypes_test extension to multiphase initialization (PEP 489) (GH-19012)
https://github.com/python/cpython/commit/2037502613471a0a0a0262085cc50adb378ebbad

New changeset 514c469719f149e1722a91a9d0c63bf89dfefb2a by Dong-hee Na in branch 'master':
bpo-1635741: Port itertools module to multiphase initialization (GH-19044)
https://github.com/python/cpython/commit/514c469719f149e1722a91a9d0c63bf89dfefb2a

New changeset 4657a8a0d006c76699ba3d1d4d21a04860bb2586 by Dong-hee Na in branch 'master':
bpo-1635741: Port _heapq module to multiphase initialization (GH19057)
https://github.com/python/cpython/commit/4657a8a0d006c76699ba3d1d4d21a04860bb2586

New changeset 77248a28896d39cae0a7e084965b9ffc2624b7f4 by Dong-hee Na in branch 'master':
bpo-1635741: Port _collections module to multiphase initialization (GH-19074)
https://github.com/python/cpython/commit/77248a28896d39cae0a7e084965b9ffc2624b7f4

New changeset 8334f30a74abcf7e469b901afc307887aa85a888 by Hai Shi in branch 'master':
bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084)
https://github.com/python/cpython/commit/8334f30a74abcf7e469b901afc307887aa85a888

About half of the remaining refs are related to encodings. I noticed that caches on Lib/encodings/__init__.py and codec_search_cach of PyInterpreterState are the places holding the refs. I removed those caches and number went do to:

Before: 4382 refs left
After : 2344 refs left (-46%)

The way to destroy codec_search_cache was recently changed on #36854 and $38962.

(Not proposing to merge this, but my changes are at https://github.com/python/cpython/compare/master...phsilva:remove-codec-caches).

> I noticed that caches on Lib/encodings/__init__.py and codec_search_cach of PyInterpreterState are the places holding the refs. I removed those caches and number went do to.

Good Catch, Paulo.
IMHO, caches is useful in codecs(it's improve the search efficiency).

I have two humble idea:
1. Clean all item of codec_search_xxx in `Py_Finalize()`;
2. change the refcount mechanism(in this case, refcount+1 or refcount+2 make no differenct);

The last merged pull request, GH-GH-19084, causes refleaks in importlib tests. Stable buildbots are failing, I can reproduce on macOS Catalina.

You can test yourself by running:
$ ./python.exe -E -Wd -m test -uall,-gui -l -L -R: test_importlib

Master at 2de7ac9798 does not fail while the next commit, 8334f30a74, introduces the failure.

> The last merged pull request, GH-GH-19084, causes refleaks in importlib tests. Stable buildbots are failing, I can reproduce on macOS Catalina.

thanks, Łukasz.
I catched this problem in my vm of centos too. I don't the broken reason temporarily.

New changeset bd409bb5b78e7ccac5fcda9ab4cec770552f3090 by Paulo Henrique Silva in branch 'master':
bpo-1635741: Port time module to multiphase initialization (PEP 489) (GH-19107)
https://github.com/python/cpython/commit/bd409bb5b78e7ccac5fcda9ab4cec770552f3090

> The last merged pull request, GH-GH-19084, causes refleaks in importlib tests. Stable buildbots are failing, I can reproduce on macOS Catalina.

I expect that the bug is non-trivial, so I prefer to open a separated issue: bpo-40050 "test_importlib leaked [6303, 6299, 6303] references".

New changeset 188078c39dec24aa5b3f2073bdc9a68ebaae42de by Victor Stinner in branch 'master':
Revert "bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084)" (#19128)
https://github.com/python/cpython/commit/188078c39dec24aa5b3f2073bdc9a68ebaae42de

New changeset 93460d097f50db0870161a63911d61ce3c5f4583 by Victor Stinner in branch 'master':
bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19140)
https://github.com/python/cpython/commit/93460d097f50db0870161a63911d61ce3c5f4583

I managed to identify bpo-40050 (test_importlib reference leak) root issue and to fix it, so I reapplied Hai Shi's change for _weakref.

Updating on my findings on msg364833.

It looks like encodings module is not being destoyed at all and keeping all the encoding refs alive. Looks like some cycle but I am not sure yet how to solve it.

To validate this, I:
 - removed codec_search_cach of PyInterpreterState.
 - Py_DECREFd(encodings) after loading it on codecs.c.

Before: 4376 refs left (37fcbb65d4)
After :  352 refs left (-92%)

I've updated the changes at https://github.com/python/cpython/compare/master...phsilva:remove-codec-caches (not a proposed patch, just to validate the idea)

New changeset f3d5ac47720045a72f7ef5af13046d9531e6007b by Paulo Henrique Silva in branch 'master':
bpo-1635741: Port operator module to multiphase initialization (PEP 489) (GH-19150)
https://github.com/python/cpython/commit/f3d5ac47720045a72f7ef5af13046d9531e6007b

New changeset 7dd549eb08939e1927fba818116f5202e76f8d73 by Paulo Henrique Silva in branch 'master':
bpo-1635741: Port _functools module to multiphase initialization (PEP 489) (GH-19151)
https://github.com/python/cpython/commit/7dd549eb08939e1927fba818116f5202e76f8d73

Hum, some clarification is needed here.

"Port xxx extension module to multiphase initialization (PEP 489)" changes are helping to fix "Py_Finalize() doesn't clear all Python objects at exit", but alone they don't fix all issues.

--

For example, if a module still uses globals using "static ..." in C, these globals will not be cleared magically. Example with _datetimemodule.c:

static PyObject *us_per_hour = NULL;    /* 1e6 * 3600 as Python int */
static PyObject *us_per_day = NULL;     /* 1e6 * 3600 * 24 as Python int */
static PyObject *us_per_week = NULL;    /* 1e6*3600*24*7 as Python int */

These variables initialized once in PyInit__datetime():

    us_per_hour = PyLong_FromDouble(3600000000.0);
    us_per_day = PyLong_FromDouble(86400000000.0);
    us_per_week = PyLong_FromDouble(604800000000.0);

Converting the module to multiphase initialization will not magically clear these variables at exit. The _datetime module should be modified to store these variables in a module state: this module could be cleared at exit.

The binascii is a good example: it has a module state, traverse, clear and free methods, and it uses the multiphase initialization. This module can be fully unloaded at exit.

It's a "simple" module: it doesn't define types for example.

--

Another issue is that converting a module to the multiphase initialization doesn't magically fully isolate two instances of the module. For exmaple, the _abc module still uses a type defined statically:

static PyTypeObject _abc_data_type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "_abc_data",                        /*tp_name*/
    sizeof(_abc_data),                  /*tp_basicsize*/
    .tp_dealloc = (destructor)abc_data_dealloc,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_alloc = PyType_GenericAlloc,
    .tp_new = abc_data_new,
};

Example:

vstinner@apu$ ./python
Python 3.9.0a5+ (heads/pr/19122:0ac3031a80, Mar 25 2020, 02:25:19) 
>>> import _abc
>>> class Bla: pass
... 
>>> _abc._abc_init(Bla)
>>> type(Bla._abc_impl)
<class '_abc_data'>

# load a second instance of the module
>>> import sys; del sys.modules['_abc']
>>> import _abc as _abc2
>>> class Bla2: pass
... 
>>> _abc._abc_init(Bla2)

>>> type(Bla2._abc_impl)
<class '_abc_data'>

# _abc and _abc2 have exactly the same type,
# they are not fully isolated
>>> type(Bla2._abc_impl) is type(Bla._abc_impl)
True


That's more an issue for subinterpreters: each interpreter should have its own fully isolated instance of an C extension module.

Thanks for the clarifications. I will keep looking for simple modules, no state and easy to migrate but also dedicate more time to work on the more complex like datetime. I'm working on PR19122 corrections.

> Thanks for the clarifications. I will keep looking for simple modules, no state and easy to migrate but also dedicate more time to work on the more complex like datetime. I'm working on PR19122 corrections.

I like changes which convert C extension modules to multiphase initialization API since they fix the error path: they implicitly ensures that the module is properly destroyed if something goes wrong.

Moreover, it will ease the work to fix the other issues that I listed.

Sorry for the noise, but I just wanted to say thanks to the people working on this issue 13 years after I reported it. :)  Far too many open-source projects arbitrarily close bugs just because they don't have time to fix them and they never get fixed, so I'm glad this wasn't the case here.

>Sorry for the noise, but I just wanted to say thanks to the people working on this issue 13 years after I reported it. :)  Far too many open-source projects arbitrarily close bugs just because they don't have time to fix them and they never get fixed, so I'm glad this wasn't the case here.

cpython is a big family ;)

> bpo-1635741: Port _functools module to multiphase initialization (PEP 489) (GH-19151)
> https://github.com/python/cpython/commit/7dd549eb08939e1927fba818116f5202e76f8d73

This change introduced a regression: bpo-40071 "test__xxsubinterpreters  leaked [1, 1, 1] references: test_ids_global()".

New changeset 1cb763b8808745b9a368c1158fda19d329f63f6f by Dong-hee Na in branch 'master':
bpo-1635741: Port _uuid module to multiphase initialization (GH-19242)
https://github.com/python/cpython/commit/1cb763b8808745b9a368c1158fda19d329f63f6f

New changeset 5be8241392453751beea21d2e32096c15a8d47db by Dong-hee Na in branch 'master':
bpo-1635741: Port math module to multiphase initialization (GH-19243)
https://github.com/python/cpython/commit/5be8241392453751beea21d2e32096c15a8d47db

I created bpo-40137: TODO list when PEP 573 "Module State Access from C Extension Methods" will be implemented.

It tracks code that should be fixed once PEP 573 will be implemented, like _functools and _abc modules.

New changeset 45f7008a66a30cdf749ec03e580bd2692be9a8df by Hai Shi in branch 'master':
bpo-1635741: Port resource extension module to multiphase initialization (PEP 489) (GH-19252)
https://github.com/python/cpython/commit/45f7008a66a30cdf749ec03e580bd2692be9a8df

New changeset 7a6f3bcc43ed729f8038524528c0b326b5610506 by Hai Shi in branch 'master':
bpo-1635741: Fix refleak in _locale init error handling (GH-19307)
https://github.com/python/cpython/commit/7a6f3bcc43ed729f8038524528c0b326b5610506

New changeset 84724dd239c30043616487812f6a710b1d70cd4b by Dong-hee Na in branch 'master':
bpo-1635741: Port _stat module to multiphase initialization (GH-19798)
https://github.com/python/cpython/commit/84724dd239c30043616487812f6a710b1d70cd4b

New changeset b66c0ff8af0c1a4adc6908897b2d05afc78cc27e by Victor Stinner in branch 'master':
bpo-1635741: Fix compiler warning in _stat.c (GH-19822)
https://github.com/python/cpython/commit/b66c0ff8af0c1a4adc6908897b2d05afc78cc27e

New changeset 92a98ed97513c6e365ce8765550ea65d0ddc8cd7 by Dong-hee Na in branch 'master':
bpo-1635741: Port syslog module to multiphase initialization (GH-19907)
https://github.com/python/cpython/commit/92a98ed97513c6e365ce8765550ea65d0ddc8cd7

New changeset 3466922320d54a922cfe6d6d44e89e1cea4023ef by Dong-hee Na in branch 'master':
bpo-1635741: Port errno module to multiphase initialization (GH-19923)
https://github.com/python/cpython/commit/3466922320d54a922cfe6d6d44e89e1cea4023ef

New changeset e9684fac5a158be9806304a676e619857520a4dc by Dong-hee Na in branch 'master':
bpo-1635741: Port fcntl module to multiphase initialization (GH-20540)
https://github.com/python/cpython/commit/e9684fac5a158be9806304a676e619857520a4dc

New changeset 3ad52e366fea37b02a3f619e6b7cffa7dfbdfa2e by Dong-hee Na in branch 'master':
bpo-1635741: Port mmap module to multiphase initialization (GH-19459)
https://github.com/python/cpython/commit/3ad52e366fea37b02a3f619e6b7cffa7dfbdfa2e

I wrote PR 20763 to "finalize" static types in Py_Finalize(). It mostly works, but "./Programs/_testembed test_forced_io_encoding" crash. This program calls Py_Initialize() and Py_Finalize() multiple times in a loop.

It doesn't look to be safe to clear static types. Many functions rely on the fact that static types are "always there" and are never finalized. Also, only a few static types are cleared by my PR: many static types are left unchanged. For example, static types of the _io module.

It seems like a safer approach is to continue the work on bpo-40077: "Convert static types to PyType_FromSpec()".

New changeset bf69a8f99f1b0e19a59509c6c4d7015a31d881a1 by Dong-hee Na in branch 'master':
bpo-1635741: Port _dbm module to multiphase initialization (GH-20848)
https://github.com/python/cpython/commit/bf69a8f99f1b0e19a59509c6c4d7015a31d881a1

New changeset c4862e333ab405dd5789b4061222db1982147de4 by Dong-hee Na in branch 'master':
bpo-1635741: Port _gdbm module to multiphase initialization (GH-20920)
https://github.com/python/cpython/commit/c4862e333ab405dd5789b4061222db1982147de4

See also the 48 open issues with "pep3121" keyword:

https://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=13&priority=&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40group=&%40pagesize=50&%40startwith=0&%40sortdir=on&%40queryname=&%40old-queryname=&%40action=search

(yeah enjoy the ugly Roundup URL...)

See meta bpo-15787 "PEP 3121, 384 Refactoring" which tracks all these issues as dependencies.

New changeset 1937edd376274cb26090d71253191502a9de32d6 by Dong-hee Na in branch 'master':
bpo-1635741: Port _lzma module to multiphase initialization  (GH-19382)
https://github.com/python/cpython/commit/1937edd376274cb26090d71253191502a9de32d6

Current status: 60 stdlib extensions (51%) are ported to PEP 489 multiphase initialization, 58 stdlib extensions (49%) are still using the legacy API:

$ grep -E '\<PyModule_Create\>' $(find Modules/ -name "*.c")|wc -l
60

$ grep -E '\<PyModuleDef_Init\>' $(find Modules/ -name "*.c")|wc -l
58

List of the 58 C extensions using the legacy API:

_asynciomodule.c
_bisectmodule.c
_blake2/blake2module.c
cjkcodecs/multibytecodec.c
cmathmodule.c
_csv.c
_ctypes/_ctypes.c
_cursesmodule.c
_curses_panel.c
_datetimemodule.c
_decimal/_decimal.c
_elementtree.c
faulthandler.c
gcmodule.c
grpmodule.c
_hashopenssl.c
_io/_iomodule.c
_lsprof.c
md5module.c
_multiprocessing/multiprocessing.c
_multiprocessing/posixshmem.c
_opcode.c
ossaudiodev.c
overlapped.c
_pickle.c
_posixsubprocess.c
pwdmodule.c
pyexpat.c
_queuemodule.c
_randommodule.c
readline.c
_scproxy.c
selectmodule.c
sha1module.c
sha256module.c
_sha3/sha3module.c
sha512module.c
signalmodule.c
socketmodule.c
spwdmodule.c
_sqlite/module.c
_sre.c
_ssl.c
_struct.c
symtablemodule.c
termios.c
_testbuffer.c
_testcapimodule.c
_testimportmultiple.c
_testinternalcapi.c
_threadmodule.c
_tkinter.c
_tracemalloc.c
unicodedata.c
_winapi.c
_xxsubinterpretersmodule.c
_xxtestfuzz/_xxtestfuzz.c
zlibmodule.c

New changeset 666ecfb0957a2fa0df5e2bd03804195de74bdfbf by Victor Stinner in branch 'master':
bpo-1635741: Release Unicode interned strings at exit (GH-21269)
https://github.com/python/cpython/commit/666ecfb0957a2fa0df5e2bd03804195de74bdfbf

New changeset 9d006977d7ff5a45d6e7d696c1713fdf2dd308b7 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port sha256 module to multiphase init (PEP 489) (GH-21189)
https://github.com/python/cpython/commit/9d006977d7ff5a45d6e7d696c1713fdf2dd308b7

New changeset 3549ca313a6103a3adb281ef3a849298b7d7f72c by Victor Stinner in branch 'master':
bpo-1635741: Fix unicode_dealloc() for mortal interned string (GH-21270)
https://github.com/python/cpython/commit/3549ca313a6103a3adb281ef3a849298b7d7f72c

New changeset c0b214bc08f0da89e5b2e4b8cc9f07783833d6b8 by Dong-hee Na in branch 'master':
bpo-1635741: Port faulthandler module to multiphase initialization (GH-21294)
https://github.com/python/cpython/commit/c0b214bc08f0da89e5b2e4b8cc9f07783833d6b8

New changeset 1d541c25c8019f7a0b80b3e1b437abe171e40b65 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port multiprocessing ext to multiphase init (GH-21378)
https://github.com/python/cpython/commit/1d541c25c8019f7a0b80b3e1b437abe171e40b65

There will have many unicode strs releaks when we calling `Py_Initialize()` again after `Py_Finalize()`: interned will be cleared in `Py_Finalize()`, but those unicodes str will still alive all the time.

Q: How to solve the probleam?
A: MAYBE we need share the interned of unicodeobject.c all the time and don't care how many times we will calling `Py_Initialize(); Py_Finalize();`

> There will have many unicode strs releaks when we calling `Py_Initialize()` again after `Py_Finalize()`: interned will be cleared in `Py_Finalize()`, but those unicodes str will still alive all the time.

Py_Finalize() calls _PyUnicode_ClearInterned() which clears interned strings. Which strings are still alive after Py_Finalize()?

New changeset 8ecc0c4d390d03de5cd2344aa44b69ed02ffe470 by Hai Shi in branch 'master':
bpo-1635741: Clean sysdict and builtins of interpreter at exit (GH-21605)
https://github.com/python/cpython/commit/8ecc0c4d390d03de5cd2344aa44b69ed02ffe470

> Q: How to solve the problem?

Making sure that the "total reference count" is zero after Py_Finalize() is a long term project which requires to solve many subproblems:

* Convert static types to heap types: bpo-40077
* Somehow related, convert extension modules to multiphase initialization (PEP 489): this issue
* Identify remaining global variables and either clear them explicitly, or move them to a structure which is cleared at exit

To convert extension modules to multiphase init, one practical problem is that the PEP 573 doesn't cover slots and a few other corner cases. The PEP 573 should be extended:
https://mail.python.org/archives/list/capi-sig@python.org/thread/6CGIIZVMJRYHWZDJLNWCLPSYYAVRRVCC/

There are likely a bunch of other misc corner cases which should be fixed as well.

> Py_Finalize() calls _PyUnicode_ClearInterned() which clears interned strings. Which strings are still alive after Py_Finalize()?

Yes.especially those encodings, interpreter leaks much encodings refcount after Py_Finalize(). I am not sure they are corner cases or not.
Maybe we could check it again afer we have done all convert works.

No matter how many times `Py_Initialize(); Py_Finalize();` have called, Holding a same interned of unicodeobject.c all the time sound like a stupied idea.

New changeset e087f7cd43dfa4223c55a8ecd71f4a7d685178e4 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _winapi ext to multi-stage init (GH-21371)
https://github.com/python/cpython/commit/e087f7cd43dfa4223c55a8ecd71f4a7d685178e4

New changeset 8aa163eea6b0fb4693f6c0a314d4f2ccada51d70 by Hai Shi in branch 'master':
bpo-1635741: Explict GC collect after PyInterpreterState_Clear() (GH-21902)
https://github.com/python/cpython/commit/8aa163eea6b0fb4693f6c0a314d4f2ccada51d70

New changeset a7f026870d2dab7015a94e287bec6dd46cdbf604 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _blake2 module to multi-phase init (GH-21856)
https://github.com/python/cpython/commit/a7f026870d2dab7015a94e287bec6dd46cdbf604

New changeset 93d50a6a8d0c5d332c11aef267e66573a09765ac by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _sha3 module to multi-phase init (GH-21855)
https://github.com/python/cpython/commit/93d50a6a8d0c5d332c11aef267e66573a09765ac

I created bpo-41692: "Deprecate immortal interned strings: PyUnicode_InternImmortal()".

New changeset 71d1bd9569c8a497e279f2fea6fe47cd70a87ea3 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _signal module to multi-phase init (PEP 489) (GH-22049)
https://github.com/python/cpython/commit/71d1bd9569c8a497e279f2fea6fe47cd70a87ea3

> bpo-1635741: Port _signal module to multi-phase init (PEP 489) (GH-22049)

This change is causing new issues: bpo-41713 "_signal module leak: test_interpreters leaked [1424, 1422, 1424] references". So I partially reverted it: PR 22087.

New changeset 63f102fe079ecb5cb7b921a1cf8bce4077a9d7e2 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _sha1, _sha512, _md5 to multiphase init (GH-21818)
https://github.com/python/cpython/commit/63f102fe079ecb5cb7b921a1cf8bce4077a9d7e2

New changeset 1aaa21ff818b08af2a68862b552b7ba0857492eb by Mohamed Koubaa in branch 'master':
bpo-1635741 port zlib module to multi-phase init (GH-21995)
https://github.com/python/cpython/commit/1aaa21ff818b08af2a68862b552b7ba0857492eb

New changeset 426f2b4f13f392875e7861dbd7f34735731eff17 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _opcode module to multi-phase init (PEP 489) (GH-22050)
https://github.com/python/cpython/commit/426f2b4f13f392875e7861dbd7f34735731eff17

New changeset 2aabc3200bf03d2ec1aa987e1e20db704948111e by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _overlapped module to multi-phase init (GH-22051)
https://github.com/python/cpython/commit/2aabc3200bf03d2ec1aa987e1e20db704948111e

I closed the following issues as duplicates of this issue:

* bpo-15686: "PEP 3121, 384 Refactoring applied to md5 module"
* bpo-15733: "PEP 3121, 384 Refactoring applied to winapi module"
* bpo-15680: "PEP 3121 refactoring applied to audioop module"
* bpo-15706: "PEP 3121, 384 Refactoring applied to sha512 module"
* bpo-15704: "PEP 3121, 384 Refactoring applied to sha1 module"
* bpo-15681: "PEP 3121 refactoring applied to binascii module"
* bpo-15662: "PEP 3121 refactoring applied to locale module"

and also: bpo-31862: "Port the standard library to PEP 489 multiphase initialization" (binascii).

I also marked bpo-15682 "PEP 3121 refactoring applied to fpectl module" and bpo-15684 "PEP 3121 refactoring applied to fpetest module" as duplicates of bpo-29137: the fpectl module was removed in Python 3.7.

Note: _sha256 module is not ported yet (bpo-15705).

New changeset 1baf030a902392fe92d934ed0fb6a385cf7d8869 by Mohamed Koubaa in branch 'master':
bpo-1635741 port _curses_panel to multi-phase init (PEP 489) (GH-21986)
https://github.com/python/cpython/commit/1baf030a902392fe92d934ed0fb6a385cf7d8869

New changeset 15dcdb211366e0788e831fc2a1f785e6a5ca2749 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port the termios to multi-phase init (PEP 489) (GH-22139)
https://github.com/python/cpython/commit/15dcdb211366e0788e831fc2a1f785e6a5ca2749

I marked bpo-15709 "PEP 3121, 384 Refactoring applied to termios module" as a duplicate of this issue.

I rejected the PR 19122 which tries to port the _datetime module to the multi-phase initialization API (PEP 489). We need first to enhance the PyCapsule C API to pass the module instance to C functions, somehow.

Extension modules calling PyCapsule_New() in their init function:

* _curses
* _datetime
* _decimal
* _socket
* pyexpat
* unicodedata

New changeset 52a2df135c0470b1dbf889edc51b7c556ae4bc80 by Mohamed Koubaa in branch 'master':
bpo-1635741: Convert _sha256 types to heap types (GH-22134)
https://github.com/python/cpython/commit/52a2df135c0470b1dbf889edc51b7c556ae4bc80

2020-06-22 16:58: "List of the 58 C extensions using the legacy API (...)"

UPDATE: 56% of Modules/*.c modules are converted (48 on 85).

Since June, 21 extension modules have been converted. There are now 37 remaining extensions which still use PyModule_Create():

$ grep -l  '\<PyModule_Create\> *(' Modules/*.c|wc -l
37

Modules/*.c:

* 48 files use PyModuleDef_Init()
* 37 files use PyModule_Create()
* 9 files are not modules:

Modules/config.c
Modules/getaddrinfo.c
Modules/getbuildinfo.c
Modules/getnameinfo.c
Modules/getpath.c
Modules/main.c
Modules/_math.c
Modules/rotatingtree.c
Modules/tkappinit.c

--

Outside the Modules/ directory, there are 6 modules using PyModule_Create:

$ grep -l  '\<PyModule_Create\> *(' */*.c|grep -v Modules

Objects/unicodeobject.c
PC/_msi.c
PC/msvcrtmodule.c
PC/winreg.c
PC/winsound.c
Python/import.c
Python/marshal.c
Python/_warnings.c

> Python/_warnings.c

I'm not sure if it's a good idea to convert the _warnings module to multi-phase init, since it uses a state stored in the interpreter:

/* Given a module object, get its per-module state. */
static WarningsState *
warnings_get_state(void)
{
    PyInterpreterState *interp = _PyInterpreterState_GET();
    if (interp == NULL) {
        PyErr_SetString(PyExc_RuntimeError,
                        "warnings_get_state: could not identify "
                        "current interpreter");
        return NULL;
    }
    return &interp->warnings;
}

For example, two _warnings instance would share the _warnings.filters list. Maybe it's ok, I don't know.

New changeset bb083d33f7ffe93cee9e1f63d1e526dc81a6e34f by Victor Stinner in branch 'master':
bpo-1635741: Port _string module to multi-phase init (GH-22148)
https://github.com/python/cpython/commit/bb083d33f7ffe93cee9e1f63d1e526dc81a6e34f

New changeset f315142ddc61e54a59028db562aec5f62db783e1 by Victor Stinner in branch 'master':
bpo-1635741: Port mashal module to multi-phase init (#22149)
https://github.com/python/cpython/commit/f315142ddc61e54a59028db562aec5f62db783e1

New changeset 3ff6975e2c0af0399467f234b2e307cc76efcfa9 by Mohamed Koubaa in branch 'master':
bpo-1635741: port scproxy to multi-phase init (GH-22164)
https://github.com/python/cpython/commit/3ff6975e2c0af0399467f234b2e307cc76efcfa9

New changeset f76d894dc5d5facce1a6c1b71637f6a2b3f9fd2b by Mohamed Koubaa in branch 'master':
bpo-1635741: Port cmath to multi-phase init (PEP 489) (GH-22165)
https://github.com/python/cpython/commit/f76d894dc5d5facce1a6c1b71637f6a2b3f9fd2b

> I rejected the PR 19122 which tries to port the _datetime module to the multi-phase initialization API (PEP 489). We need first to enhance the PyCapsule C API to pass the module instance to C functions, somehow.

I created bpo-41798: [C API] Revisit usage of the PyCapsule C API with multi-phase initialization API.

New changeset 1b328ea9a7d15de4a8c9d0eb8aee94f6c75c1b46 by Mohamed Koubaa in branch 'master':
bpo-1635741: Convert an  _lsprof method to argument clinic (GH-22240)
https://github.com/python/cpython/commit/1b328ea9a7d15de4a8c9d0eb8aee94f6c75c1b46

New changeset 83de110dce94a9196dccc01d526628615714e362 by Mohamed Koubaa in branch 'master':
bpo-1635741: Port _lsprof extension to multi-phase init (PEP 489) (GH-22220)
https://github.com/python/cpython/commit/83de110dce94a9196dccc01d526628615714e362

New changeset ddc0dd001a4224274ba6f83568b45a1dd88c6fc6 by Mohamed Koubaa in branch 'master':
bpo-1635741, unicodedata: add ucd_type parameter to UCD_Check() macro (GH-22328)
https://github.com/python/cpython/commit/ddc0dd001a4224274ba6f83568b45a1dd88c6fc6