Parent

Thecc_parentfield (accessed for example by a__parent__ or__objclass__descriptor from Python code) can be any Python object, or NULL. Custom classes are free to setcc_parentto whatever they want. It is only used by the C call protocol if the CCALL_OBJCLASSflag is set.

For methods of extension types,cc_parentpoints to the class that defines the method (which may be a superclass oftype(self)). This is currently non-trivial to retrieve from a method’s code. In the future, this can be used to access the module state via the defining class. See the rationale ofPEP 573for details.

When the flagCCALL_OBJCLASSis set (as it will be for methods of extension types),cc_parentis used for type checks like the following:

>>>list.append({},"x")
Traceback (most recent call last):
File"<stdin>",line1,in<module>
TypeError:descriptor 'append' requires a 'list' object but received a 'dict'

For functions of modules,cc_parentis set to the module. Currently, this is exactly the same as__self__. However, using__self__for the module is a quirk of the current implementation: in the future, we want to allow functions which use__self__ in the normal way, for implementing methods. Such functions can still usecc_parentinstead to refer to the module.

The parent would also typically be used to implement__qualname__. The new C API functionPyCCall_GenericGetQualname()does exactly that.

Using tp_print

We propose to replace the existing unused fieldtp_print bytp_ccalloffset. SincePy_TPFLAGS_HAVE_CCALLwouldnotbe added to Py_TPFLAGS_DEFAULT,this ensures full backwards compatibility for existing extension modules settingtp_print. It also means that we can require thattp_ccalloffsetis a valid offset whenPy_TPFLAGS_HAVE_CCALLis specified: we do not need to checktp_ccalloffset!=0. In future Python versions, we may decide thattp_print becomestp_ccalloffsetunconditionally, drop thePy_TPFLAGS_HAVE_CCALLflag and instead check for tp_ccalloffset!=0.

NOTE:the exact layout ofPyTypeObjectis not part of thestable ABI). Therefore, changing thetp_printfield from aprintfunc(a function pointer) to aPy_ssize_tshould not be a problem, even if this changes the memory layout of thePyTypeObjectstructure. Moreover, on all systems for which binaries are commonly built (Windows, Linux, macOS), the size ofprintfuncandPy_ssize_tare the same, so the issue of binary compatibility will not come up anyway.

Checking __objclass__

If theCCALL_OBJCLASSflag is set and ifcr_selfis NULL (this is the case for unbound methods of extension types), then a type check is done: the function must be called with at least one positional argument and the first (typically calledself) must be an instance of cc_parent(which must be a class). If not, aTypeErroris raised.

Self slicing

Ifcr_selfis not NULL or if the flagCCALL_SELFARG is not set incc_flags,then the argument passed asself is simplycr_self.

Ifcr_selfis NULL and the flagCCALL_SELFARGis set, then the first positional argument is removed from argsand instead passed asselfargument to the C function. Effectively, the first positional argument is treated as__self__. If there are no positional arguments,TypeErroris raised.

This process is called “self slicing” and a function is said to have self slicing ifcr_selfis NULL andCCALL_SELFARGis set.

Note that aCCALL_NOARGSfunction with self slicing effectively has one argument, namelyself. Analogously, aCCALL_Ofunction with self slicing has two arguments.

Descriptor behavior

Classes supporting the C call protocol must implement the descriptor protocol in a specific way.

This is required for an efficient implementation of bound methods: if other code can make assumptions on what__get__does, it enables optimizations which would not be possible otherwise. In particular, we want to allow sharing thePyCCallDefstructure between bound and unbound methods. We also need a correct implementation of_PyObject_GetMethod which is used by theLOAD_METHOD/CALL_METHODoptimization.

First of all, iffuncsupports the C call protocol, thenfunc.__set__andfunc.__delete__must not be implemented.

Second,func.__get__must behave as follows:

• Ifcr_selfis not NULL, then__get__must be a no-op in the sense thatfunc.__get__(obj,cls)(*args,**kwds) behaves exactly the same asfunc(*args,**kwds). It is also allowed for__get__to be not implemented at all.
• Ifcr_selfis NULL, thenfunc.__get__(obj,cls)(*args,**kwds) (withobjnot None) must be equivalent tofunc(obj,*args,**kwds). In particular,__get__must be implemented in this case. This is unrelated toself slicing:objmay be passed asselfargument to the C function or it may be the first positional argument.
• Ifcr_selfis NULL, thenfunc.__get__(None,cls)(*args,**kwds) must be equivalent tofunc(*args,**kwds).

There are no restrictions on the objectfunc.__get__(obj,cls). The latter is not required to implement the C call protocol for example. We only specify whatfunc.__get__(obj,cls).__call__does.

For classes that do not care about__self__and__get__at all, the easiest solution is to assigncr_self=Py_None (or any other non-NULL value).

The __name__ attribute

The C call protocol requires that the function has a__name__ attribute which is of typestr(not a subclass).

Furthermore, the object returned by__name__must be stored somewhere; it cannot be a temporary object. This is required becausePyEval_GetFuncName uses a borrowed reference to the__name__attribute (see also[2]).

Generic API functions

This section lists the new public API functions or macros dealing with the C call protocol.

• intPyCCall_Check(PyObject*op): return true ifopimplements the C call protocol.

All the functions and macros below apply to any instance supporting the C call protocol. In other words,PyCCall_Check(func)must be true.

• PyObject*PyCCall_Call(PyObject*func,PyObject*args,PyObject*kwds): callfuncwith positional argumentsargs and keyword argumentskwds(kwdsmay be NULL). This function is meant to be put in thetp_callslot.
• PyObject*PyCCall_FastCall(PyObject*func,PyObject*const*args,Py_ssize_tnargs,PyObject*kwds): callfuncwithnargspositional arguments given byargs[0],…,args[nargs-1]. The parameterkwdscan be NULL (no keyword arguments), a dict withname:valueitems or a tuple with keyword names. In the latter case, the keyword values are stored in theargs array, starting atargs[nargs].

Macros to access thePyCCallRootandPyCCallDefstructures:

• constPyCCallRoot*PyCCall_CCALLROOT(PyObject*func): pointer to thePyCCallRootstructure insidefunc.
• constPyCCallDef*PyCCall_CCALLDEF(PyObject*func): shorthand forPyCCall_CCALLROOT(func)->cr_ccall.
• uint32_tPyCCall_FLAGS(PyObject*func): shorthand forPyCCall_CCALLROOT(func)->cr_ccall->cc_flags.
• PyObject*PyCCall_SELF(PyOject*func): shorthand forPyCCall_CCALLROOT(func)->cr_self.

Generic getters, meant to be put into thetp_getsetarray:

• PyObject*PyCCall_GenericGetParent(PyObject*func,void*closure): returncc_parent. RaiseAttributeErrorifcc_parentis NULL.
• PyObject*PyCCall_GenericGetQualname(PyObject*func,void*closure): return a string suitable for using as__qualname__. This uses the__qualname__ofcc_parentif possible. It also uses the__name__attribute.

Profiling

The profiling events c_call,c_returnandc_exceptionare only generated when calling actual instances ofbuiltin_function_or_methodormethod_descriptor. This is done for simplicity and also for backwards compatibility (such that the profile function does not receive objects that it does not recognize). In a future PEP, we may extend C-level profiling to arbitrary classes implementing the C call protocol.

C API functions

The following function is added (also to thestable ABI):

• PyObject*PyCFunction_ClsNew(PyTypeObject*cls,PyMethodDef*ml,PyObject*self,PyObject*module,PyObject*parent): create a new object with object structurePyCFunctionObjectand classcls. The entries of thePyMethodDefstructure are used to construct the new object, but the pointer to thePyMethodDefstructure is not stored. The flags for the C call protocol are automatically determined in terms ofml->ml_flags,selfandparent.

The existing functionsPyCFunction_New,PyCFunction_NewExand PyDescr_NewMethodare implemented in terms ofPyCFunction_ClsNew.

The undocumented functionsPyCFunction_GetFlags andPyCFunction_GET_FLAGSare deprecated. They are still artificially supported by storing the originalMETH_... flags in a bitfield insidecc_flags. Despite the fact thatPyCFunction_GetFlagsis technically part of thestable ABI, it is highly unlikely to be used that way: first of all, it is not even documented. Second, the flagMETH_FASTCALL is not part of the stable ABI but it is very common (because of Argument Clinic). So, if one cannot supportMETH_FASTCALL, it is hard to imagine a use case forPyCFunction_GetFlags. The fact thatPyCFunction_GET_FLAGSandPyCFunction_GetFlags are not used at all by CPython outside ofObjects/call.c further shows that these functions are not particularly useful.

Why is this better than PEP 575?

One of the major complaints ofPEP 575was that is was coupling functionality (the calling and introspection protocol) with the class hierarchy: a class could only benefit from the new features if it was a subclass ofbase_function. It may be difficult for existing classes to do that because they may have other constraints on the layout of the C object structure, coming from an existing base class or implementation details. For example,functools.lru_cachecannot implementPEP 575as-is.

It also complicated the implementation precisely because changes were needed both in the implementation details and in the class hierarchy.

The current PEP does not have these problems.

Why store the function pointer in the instance?

The actual information needed for calling an object is stored in the instance (in thePyCCallDefstructure) instead of the class. This is different from thetp_callslot or earlier attempts at implementing atp_fastcallslot[1].

The main use case is built-in functions and methods. For those, the C function to be called does depend on the instance.

Note that the current protocol makes it easy to support the case where the same C function is called for all instances: just use a single staticPyCCallDefstructure for every instance.

Why CCALL_OBJCLASS?

The flagCCALL_OBJCLASSis meant to support various cases where the class of aselfargument must be checked, such as:

>>>list.append({},None)
Traceback (most recent call last):
File"<stdin>",line1,in<module>
TypeError:append() requires a 'list' object but received a 'dict'

>>>list.__len__({})
Traceback (most recent call last):
File"<stdin>",line1,in<module>
TypeError:descriptor '__len__' requires a 'list' object but received a 'dict'

>>>float.__dict__["fromhex"](list,"0xff")
Traceback (most recent call last):
File"<stdin>",line1,in<module>
TypeError:descriptor 'fromhex' for type 'float' doesn't apply to type 'list'

In the reference implementation, only the first of these uses the new code. The other examples show that these kind of checks appear in multiple places, so it makes sense to add generic support for them.

Why CCALL_SELFARG?

The flagCCALL_SELFARGand the concept of self slicing are needed to support methods: the C function should not care whether it is called as unbound method or as bound method. In both cases, there should be aselfargument and this is simply the first positional argument of an unbound method call.

For example,list.appendis aMETH_Omethod. Both the callslist.append([],42)and[].append(42)should translate to the C calllist_append([],42).

Thanks to the proposed C call protocol, we can support this in such a way that both the unbound and the bound method share aPyCCallDef structure (with theCCALL_SELFARGflag set).

So,CCALL_SELFARGhas two advantages: there is no extra layer of indirection for calling methods and constructing bound methods does not require setting up aPyCCallDefstructure.

Another minor advantage is that we could make the error messages for a wrong call signature more uniform between Python methods and built-in methods. In the following example, Python is undecided whether a method takes 1 or 2 arguments:

>>>classList(list):
...defmyappend(self,item):
...self.append(item)
>>>List().myappend(1,2)
Traceback (most recent call last):
File"<stdin>",line1,in<module>
TypeError:myappend() takes 2 positional arguments but 3 were given
>>>List().append(1,2)
Traceback (most recent call last):
File"<stdin>",line1,in<module>
TypeError:append() takes exactly one argument (2 given)

It is currently impossible forPyCFunction_Call to know the actual number of user-visible arguments since it cannot distinguish at runtime between a function (withoutselfargument) and a bound method (withselfargument). TheCCALL_SELFARGflag makes this difference explicit.

Why CCALL_DEFARG?

The flagCCALL_DEFARGgives the callee access to thePyCCallDef*. There are various use cases for this:

1. The callee can use thecc_parentfield, which is useful forPEP 573.
2. Applications are free to extend thePyCCallDefstructure with user-defined fields, which can then be accessed analogously.
3. In the case where thePyCCallDefstructure is part of the object structure (this is true for example forPyCFunctionObject), an appropriate offset can be subtracted from thePyCCallDefpointer to get a pointer to the callable object defining thatPyCCallDef.

An earlier version of this PEP defined a flagCCALL_FUNCARG instead ofCCALL_DEFARGwhich would pass the callable object to the callee. This had similar use cases, but there was some ambiguity for bound methods: should the “callable object” be the bound method object or the original function wrapped by the method? By passing thePyCCallDef*instead, this ambiguity is gone since the bound method uses thePyCCallDef*from the wrapped function.

Replacing tp_print

We repurposetp_printastp_ccalloffsetbecause this makes it easier for external projects to backport the C call protocol to earlier Python versions. In particular, the Cython project has shown interest in doing that (seehttps://mail.python.org/pipermail/python-dev/2018-June/153927.html).