[PEP 573] bpo-38787: Module State Access from C Extension Methods #17145

ghost · 2019-11-13T14:34:08Z

This is a WIP pull request for PEP 573.

The purpose of this PEP is to provide a way to access per-module state from methods belonging to classes defined in extension modules.

https://bugs.python.org/issue38787

scoder · 2019-11-13T20:08:53Z

Objects/typeobject.c

@@ -2912,6 +2922,11 @@ PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
    Py_INCREF(res->ht_qualname);
    type->tp_name = spec->name;

+    if (module) {
+        Py_INCREF(module);


Py_XINCREF()?

scoder · 2019-11-13T20:13:09Z

Objects/typeobject.c

+            break;
+        }
+
+        type = (PyTypeObject *)PyTuple_GetItem(mro, pos);


What is the guarantee that prevents the two tuple item accesses (here and below) from running out of bounds?
Could that be worth a comment, at least?

A for loop might be better: first use PyTuple_Size to also check the type, then use the faster PyTuple_GET_ITEM.

encukou

Lots of comments on first glance:

encukou · 2019-11-19T13:07:08Z

Include/object.h

+typedef struct {
+    Py_ssize_t dict;
+    Py_ssize_t weaklist;
+} PyType_offsets;
+


This should be unnecessary.

encukou · 2019-11-19T13:17:28Z

Include/methodobject.h


-#define PyCFunction_Check(op) (Py_TYPE(op) == &PyCFunction_Type)
+#define PyCFunction_Check(op) ((Py_TYPE(op) == &PyCFunction_Type) || (PyType_IsSubtype(Py_TYPE(op), &PyCFunction_Type)))


Can we use the existing PyCFunction_Type for methods?
The METH_METHOD flag should signal that the underlying struct is PyCMethodObject; you'll also want to set tp_itemsize and the use PyObject_GC_NewVar.

encukou · 2019-11-19T13:19:42Z

Include/methodobject.h


 PyAPI_FUNC(PyCFunction) PyCFunction_GetFunction(PyObject *);
 PyAPI_FUNC(PyObject *) PyCFunction_GetSelf(PyObject *);
 PyAPI_FUNC(int) PyCFunction_GetFlags(PyObject *);
+PyAPI_FUNC(PyTypeObject *) PyCMethod_GetClass(PyObject *);


This looks unnecessary; the PEP only mentions PyCFunction_GET_CLASS.

encukou · 2019-11-19T13:22:09Z

Include/object.h

@@ -197,6 +202,18 @@ PyAPI_FUNC(PyObject*) PyType_FromSpecWithBases(PyType_Spec*, PyObject*);
 #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03040000
 PyAPI_FUNC(void*) PyType_GetSlot(struct _typeobject*, int);
 #endif
+#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03080000
+PyAPI_FUNC(PyObject*) PyType_FromModuleAndSpec(PyObject *, PyType_Spec *, PyObject *);


Oops, this is missing from the PEP's summary.

encukou · 2019-11-19T13:23:25Z

Include/object.h

+/* access macro to the members which are floating "behind" the object */
+#define PyHeapType_GET_MEMBERS(etype) \
+    ((PyMemberDef *)(((char *)etype) + Py_TYPE(etype)->tp_basicsize))
+#endif


This is duplicates the definition in Include/internal/object.h.

encukou · 2019-11-19T13:38:39Z

Objects/descrobject.c

+    if (descr->d_method->ml_flags & METH_METHOD) {
+        return PyCMethod_New(descr->d_method, type, NULL, descr->d_common.d_type);
+    } else {
+        return PyCFunction_NewEx(descr->d_method, type, NULL);
+    }


This should be much simpler, since PyCFunction_NewEx(...) is defined as PyCMethod_New(..., NULL).

encukou · 2019-11-19T15:08:20Z

Objects/methodobject.c

+        if (ml->ml_flags & (METH_NOARGS | METH_O | METH_CLASS | METH_STATIC)) {
+            PyErr_SetString(PyExc_SystemError,
+                            "METH_METHOD cannot be used with METH_NOARGS, "
+                            "METH_O, METH_CLASS, nor METH_STATIC");
+            return NULL;
+        }


This looks like it would allow (METH_VARARGS | METH_METHOD) or METH_FASTCALL | METH_METHOD

encukou · 2019-11-19T15:12:02Z

Objects/typeobject.c

+            break;
+        }
+
+        type = (PyTypeObject *)PyTuple_GetItem(mro, pos);


A for loop might be better: first use PyTuple_Size to also check the type, then use the faster PyTuple_GET_ITEM.

encukou · 2019-11-19T15:19:26Z

Modules/_testmultiphase.c

+
+/*[clinic input]
+_testmultiphase.StateAccessType.__init__
+    cls: defining_class


Could you also test with another slot, not only __init__?

encukou · 2019-11-19T15:24:56Z

Tools/clinic/clinic.py

+                fields.insert(0, normalize_snippet("""
+                    PyTypeObject *cls;
+
+                    cls = PyType_DefiningTypeFromSlotFunc(Py_TYPE(self),


Is it always named cls?

the-knights-who-say-ni added the CLA signed label Nov 13, 2019

bedevere-bot added the awaiting review label Nov 13, 2019

scoder reviewed Nov 13, 2019

View reviewed changes

terryjreedy added the DO-NOT-MERGE label Nov 15, 2019

encukou reviewed Nov 19, 2019

View reviewed changes

Marcel Plch and others added 22 commits January 21, 2020 11:08

add ht_module to heap types

6026bc1

Implement PyCMethod

f2d45a4

static exception prototype

e85b726

METH_METHOD clinic support

e593a86

slot method per-module state access

9f9f81f

port io module

625a961

port _csv module

6ad36fb

clinic support for METH_METHOD and heaptype state access

a526fc5

port _elementtree

085cab7

consoleio

b95d6a4

Helper for getting per-module state from type

1f6c04d

Correct MRO walking algorithm

41fe314

Limited APIs - rewrite from 3.7 to 3.8

40275e2

EuroPython Sprint Changes

a84e9a2

recover clinic after rebase

7f78843

Sanitize the implementation

387267a

Update method_enter_call() call

f284861

Revert Objects/typeslots.py changes.

4422099

Remove MRO walker.

06c13dd

Reflect PR comments

a54f720

Add news entry

a6cd004

Reflect also hidden comments

f77edf4

ghost closed this Jan 22, 2020

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[PEP 573] bpo-38787: Module State Access from C Extension Methods #17145

[PEP 573] bpo-38787: Module State Access from C Extension Methods #17145

Uh oh!

ghost commented Nov 13, 2019 •

edited by bedevere-bot

Loading

Uh oh!

scoder Nov 13, 2019

Uh oh!

scoder Nov 13, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou left a comment

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

encukou Nov 19, 2019

Uh oh!

Uh oh!


		#define PyCFunction_Check(op) (Py_TYPE(op) == &PyCFunction_Type)
		#define PyCFunction_Check(op) ((Py_TYPE(op) == &PyCFunction_Type) \|\| (PyType_IsSubtype(Py_TYPE(op), &PyCFunction_Type)))

Uh oh!

[PEP 573] bpo-38787: Module State Access from C Extension Methods #17145

[PEP 573] bpo-38787: Module State Access from C Extension Methods #17145

Uh oh!

Conversation

ghost commented Nov 13, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ghost commented Nov 13, 2019 •

edited by bedevere-bot

Loading