Leaking the PyTypeObject structure in the C API indirectly causes ABI issue (especially for statically allocated types), cause practical issues when old fields are removed and new fields are added (ex: tp_vectorcall addition and tp_print removal caused a lot of troubles with C code generated by Cython: see bpo-37250), prevents us to add feature and experiment optimization.

I don't expect that we will be able to make PyTypeObject opaque soon. The purpose of this issue is to track the work done towards this goal.

I propose to slowly prepare the Python code base, the C API and third party code (especially Cython) to make PyTypeObject structure opaque.

We have to identify most common code patterns which access directly PyTypeObject fields, provide helper functions, and ease the migration to solutions which don't access directly PyTypeObject.

See also bpo-39573: "Make PyObject an opaque structure in the limited C API" and bpo-39947 "Make the PyThreadState structure opaque (move it to the internal C API)".

Longer rationale about making structures of the C API opaque:

* https://pythoncapi.readthedocs.io/
* https://pythoncapi.readthedocs.io/bad_api.html
* https://pythoncapi.readthedocs.io/optimization_ideas.html

--

Multiple practical issues are preventing us to make PyTypeObject opaque right now.


(*) Many C extension modules are still using statically allocated types: there is an on-going effort in bpo-40077 to convert C extension modules one by one to PyType_FromSpec().


(*) Py_TYPE(obj)->tp_name is commonly accessed to format an error message. Example:

        PyErr_Format(PyExc_TypeError, "exec() globals must be a dict, not %.100s",
                     Py_TYPE(globals)->tp_name);

I worked on bpo-34595 and started a discussion on python-dev to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format():
https://mail.python.org/archives/list/python-dev@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53

Sadly, we failed to reach a consensus and I gave up on this idea. We should reconsider this idea.

We need to agree on how types should be formatted:

* just the name without any dot "type_name",
* qualified name "something.type_name",
* fully qualified name "module.something.type_name"

There is also the question of breaking applications which rely on the current exact error message. And the question of removing legacy "%.100s" which was used before Python was able to allocate a buffer large enough to arbitrary string length. When an error is formatted in pure Python, names are never truncated.


(*) Call the function of the parent type when a method is overriden in a subclass. Example with PyTypeObject.tp_free called in a deallocator:

static void
abc_data_dealloc(_abc_data *self)
{
    PyTypeObject *tp = Py_TYPE(self);
    ...
    tp->tp_free(self);
    Py_DECREF(tp);
}


(*) The PEP 384 provides the most generic PyType_GetSlot() but it's not convenient to use: need to handle error (NULL), need to cast the void* into the expected type (error prone cast), etc.

We should slowly add more and more helper functions for most common use cases. We can try to convert a few C extension modules of the Python stdlib to see which use cases are the most common.

Hopefully, many use cases are already abstracted by widely used functions like PyNumber_Add(), PySequence_Size(), etc.


(*) Likely other issues that I forgot.

Macros and static inline functions of the public C API which access directly PyTypeObject fields. There may be more.

#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )

static inline vectorcallfunc
PyVectorcall_Function(PyObject *callable)
{
    ...
    tp = Py_TYPE(callable);
    offset = tp->tp_vectorcall_offset;
    ...
}

#define PyObject_CheckBuffer(obj) \
    ((Py_TYPE(obj)->tp_as_buffer != NULL) &&  \
     (Py_TYPE(obj)->tp_as_buffer->bf_getbuffer != NULL))

#define PyIndex_Check(obj)                              \
    (Py_TYPE(obj)->tp_as_number != NULL &&            \
     Py_TYPE(obj)->tp_as_number->nb_index != NULL)

#define PyObject_GET_WEAKREFS_LISTPTR(o) \
    ((PyObject **) (((char *) (o)) + Py_TYPE(o)->tp_weaklistoffset))

static inline int
PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
#ifdef Py_LIMITED_API
    return ((PyType_GetFlags(type) & feature) != 0);
#else
    return ((type->tp_flags & feature) != 0);
#endif
}

#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )

New changeset 38aefc585f60a77d66f4fbe5a37594a488b53474 by Victor Stinner in branch 'master':
bpo-40170: PyObject_GET_WEAKREFS_LISTPTR() becomes a function (GH-19377)
https://github.com/python/cpython/commit/38aefc585f60a77d66f4fbe5a37594a488b53474

Just manually verified that PR19377, when compiled against xlc - crashes during make:

	rm -f libpython3.9d.a
	ar rcs libpython3.9d.a Modules/getbuildinfo.o  Parser/acceler.o  Parser/grammar1.o  Parser/listnode.o  Parser/node.o  Parser/parser.o  Parser/token.o   Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o  Objects/abstract.o  Objects/accu.o  Objects/boolobject.o  Objects/bytes_methods.o  Objects/bytearrayobject.o  Objects/bytesobject.o  Objects/call.o  Objects/capsule.o  Objects/cellobject.o  Objects/classobject.o  Objects/codeobject.o  Objects/complexobject.o  Objects/descrobject.o  Objects/enumobject.o  Objects/exceptions.o  Objects/genobject.o  Objects/fileobject.o  Objects/floatobject.o  Objects/frameobject.o  Objects/funcobject.o  Objects/interpreteridobject.o  Objects/iterobject.o  Objects/listobject.o  Objects/longobject.o  Objects/dictobject.o  Objects/odictobject.o  Objects/memoryobject.o  Objects/methodobject.o  Objects/moduleobject.o  Objects/namespaceobject.o  Objects/object.o  Objects/obmalloc.o  Objects/picklebufobject.o  Objects/rangeobject.o  Objects/setobject.o  Objects/sliceobject.o  Objects/structseq.o  Objects/tupleobject.o  Objects/typeobject.o  Objects/unicodeobject.o  Objects/unicodectype.o  Objects/weakrefobject.o  Python/_warnings.o  Python/Python-ast.o  Python/asdl.o  Python/ast.o  Python/ast_opt.o  Python/ast_unparse.o  Python/bltinmodule.o  Python/ceval.o  Python/codecs.o  Python/compile.o  Python/context.o  Python/dynamic_annotations.o  Python/errors.o  Python/frozenmain.o  Python/future.o  Python/getargs.o  Python/getcompiler.o  Python/getcopyright.o  Python/getplatform.o  Python/getversion.o  Python/graminit.o  Python/hamt.o  Python/import.o  Python/importdl.o  Python/initconfig.o  Python/marshal.o  Python/modsupport.o  Python/mysnprintf.o  Python/mystrtoul.o  Python/pathconfig.o  Python/peephole.o  Python/preconfig.o  Python/pyarena.o  Python/pyctype.o  Python/pyfpe.o  Python/pyhash.o  Python/pylifecycle.o  Python/pymath.o  Python/pystate.o  Python/pythonrun.o  Python/pytime.o  Python/bootstrap_hash.o  Python/structmember.o  Python/symtable.o  Python/sysmodule.o  Python/thread.o  Python/traceback.o  Python/getopt.o  Python/pystrcmp.o  Python/pystrtod.o  Python/pystrhex.o  Python/dtoa.o  Python/formatter_unicode.o  Python/fileutils.o  Python/dynload_shlib.o        Modules/config.o  Modules/getpath.o  Modules/main.o  Modules/gcmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o  Python/frozen.o
	./Modules/makexp_aix Modules/python.exp . libpython3.9d.a;  xlc_r     -Wl,-bE:Modules/python.exp -lld -o python Programs/python.o libpython3.9d.a -lintl -ldl  -lm   -lm 
	 ./python -E -S -m sysconfig --generate-posix-vars ; if test $? -ne 0 ; then  echo "generate-posix-vars failed" ;  rm -f ./pybuilddir.txt ;  exit 1 ;  fi
Objects/genobject.c:127: _PyObject_GC_TRACK: Assertion "!(((PyGC_Head *)(op)-1)->_gc_next != 0)" failed: object already tracked by the garbage collector
Enable tracemalloc to get the memory block allocation traceback
object address  : 30084150
object refcount : 0
object type     : 20013aa8
object type name: generator
object repr     : <refcnt 0 at 30084150>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: core initialized
Current thread 0x00000001 (most recent call first):
  File "<frozen importlib._bootstrap_external>", line 1593 in _setup
  File "<frozen importlib._bootstrap_external>", line 1634 in _install
  File "<frozen importlib._bootstrap>", line 1189 in _install_external_importers
/bin/sh: 24117648 IOT/Abort trap(coredump)
make: 1254-004 The error code from the last command is 134.
Stop.

FYI: about two hours ago I verified that xlc and 08050e959e6c40839cd2c9e5f6a4fd1513e3d605 : bpo-40147: Fix a compiler warning on Windows in Python/compile.c (GH-19389)

all was green.

Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 have any issues building.

Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc:

#define Py_TRASHCAN_BEGIN(op, dealloc) \
    Py_TRASHCAN_BEGIN_CONDITION(op, \
        Py_TYPE(op)->tp_dealloc == (destructor)(dealloc))

It should use PyType_GetSlot() or a new getter function (to read PyTypeObject.tp_dealloc) should be added.

> It should use PyType_GetSlot()

Oh. It seems like currently, PyType_GetSlot() can only be used on a heap allocated types :-( The function starts with:

    if (!PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE) || slot < 0) {
        PyErr_BadInternalCall();
        return NULL;
    }

> Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 have any issues building.

That sounds like an AIX specific issue. Please open a separated issue.

New changeset 9205520d8c43488696d66cbdd9aefbb21871c508 by Victor Stinner in branch 'master':
bpo-40170: PyObject_NEW() becomes an alias to PyObject_New() (GH-19379)
https://github.com/python/cpython/commit/9205520d8c43488696d66cbdd9aefbb21871c508

New changeset ef5c615f5ae72c4f6979159c94da46afefbfab9a by Victor Stinner in branch 'master':
bpo-40170: Convert PyObject_CheckBuffer() macro to a function (GH-19376)
https://github.com/python/cpython/commit/ef5c615f5ae72c4f6979159c94da46afefbfab9a

New changeset 45ec5b99aefa54552947049086e87ec01bc2fc9a by Victor Stinner in branch 'master':
bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)
https://github.com/python/cpython/commit/45ec5b99aefa54552947049086e87ec01bc2fc9a

New changeset a15e260b708a98edaba86a2aa663c3f6b2abc964 by Victor Stinner in branch 'master':
bpo-40170: Add _PyIndex_Check() internal function (GH-19426)
https://github.com/python/cpython/commit/a15e260b708a98edaba86a2aa663c3f6b2abc964

Hum, once most changes will land, maybe it would be worth it to document them at:
https://docs.python.org/dev/whatsnew/3.9.html#build-and-c-api-changes

New changeset 307b9d0144e719b016a47fcc43397c070615e01e by Victor Stinner in branch 'master':
bpo-40170: Remove PyIndex_Check() macro (GH-19428)
https://github.com/python/cpython/commit/307b9d0144e719b016a47fcc43397c070615e01e

/* Test if an object has a GC head */
#define PyObject_IS_GC(o) \
    (PyType_IS_GC(Py_TYPE(o)) \
     && (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))

This macro should be converted to an opaque function.

> This macro should be converted to an opaque function
Can I try it?