PEP 575 – Unifying function/method classes
- Author:
- Jeroen Demeyer <J.Demeyer at UGent.be>
- Status:
- Withdrawn
- Type:
- Standards Track
- Created:
- 27-Mar-2018
- Python-Version:
- 3.8
- Post-History:
- 31-Mar-2018, 12-Apr-2018, 27-Apr-2018, 05-May-2018
Table of Contents
- Withdrawal notice
- Abstract
- Motivation
- New classes
- Calling base_function instances
- Automatic creation of built-in functions
- Further changes
- Non-CPython implementations
- Rationale
- Why not simply change existing classes?
- Why __text_signature__ is not a solution
- defined_function versus function
- Scope of this PEP: which classes are involved?
- Not treating METH_STATIC and METH_CLASS
- __self__ in base_function
- Two implementations of __doc__
- Subclassing
- Replacing tp_call: METH_PASS_FUNCTION and METH_CALL_UNBOUND
- Backwards compatibility
- Two-phase Implementation
- Reference Implementation
- Appendix: current situation
- References
- Copyright
Withdrawal notice
SeePEP 580for a better solution to allowing fast calling of custom classes.
SeePEP 579for a broader discussion of some of the other issues from this PEP.
Abstract
Reorganize the class hierarchy for functions and methods with the goal of reducing the difference between built-in functions (implemented in C) and Python functions. Mainly, make built-in functions behave more like Python functions without sacrificing performance.
A new base classbase_function
is introduced and the various function
classes, as well asmethod
(renamed tobound_method
), inherit from it.
We also allow subclassing the Pythonfunction
class.
Motivation
Currently, CPython has two different function classes:
the first is Python functions, which is what you get
when defining a function withdef
orlambda
.
The second is built-in functions such aslen
,isinstance
ornumpy.dot
.
These are implemented in C.
These two classes are implemented completely independently and have different functionality.
In particular, it is currently not possible to implement a function efficiently in C
(only built-in functions can do that)
while still allowing introspection likeinspect.signature
orinspect.getsourcefile
(only Python functions can do that).
This is a problem for projects like Cython[1]that want to do exactly that.
In Cython, this was worked around by inventing a new function class calledcyfunction
.
Unfortunately, a new function class creates problems:
theinspect
module does not recognize such functions as being functions[2]
and the performance is worse
(CPython has specific optimizations for calling built-in functions).
A second motivation is more generally making built-in functions and methods
behave more like Python functions and methods.
For example, Python unbound methods are just functions but
unbound methods of extension types (e.g.dict.get
) are a distinct class.
Bound methods of Python classes have a__func__
attribute,
bound methods of extension types do not.
Third, this PEP allows great customization of functions.
Thefunction
class becomes subclassable and custom function
subclasses are also allowed for functions implemented in C.
In the latter case, this can be done with the same performance
as true built-in functions.
All functions can access the function object
(theself
in__call__
), paving the way forPEP 573.
New classes
This is the new class hierarchy for functions and methods:
object
|
|
base_function
/|\
/|\
/|defined_function
/|\
cfunction(*)|\
|function
|
bound_method(*)
The two classes marked with (*) donotallow subclassing; the others do.
There is no difference between functions and unbound methods,
while bound methods are instances ofbound_method
.
base_function
The classbase_function
becomes a new base class for all function types.
It is based on the existingbuiltin_function_or_method
class,
but with the following differences and new features:
- It acts as a descriptor implementing
__get__
to turn a function into a method ifm_self
isNULL
. Ifm_self
is notNULL
, then this is a no-op: the existing function is returned instead. - A new read-only attribute
__parent__
,represented in the C structure asm_parent
. If this attribute exists, it represents the defining object. For methods of extension types, this is the defining class (__class__
in plain Python) and for functions of a module, this is the defining module. In general, it can be any Python object. If__parent__
is a class, it carries special semantics: in that case, the function must be called withself
being an instance of that class. Finally,__qualname__
and__reduce__
will use__parent__
as namespace (instead of__self__
before). - A new attribute
__objclass__
which equals__parent__
if__parent__
is a class. Otherwise, accessing__objclass__
raisesAttributeError
. This is meant to be backwards compatible withmethod_descriptor
. - The field
ml_doc
and the attributes__doc__
and__text_signature__
(seeArgument Clinic) are not supported. - A new flag
METH_PASS_FUNCTION
forml_flags
. If this flag is set, the C function stored inml_meth
is called with an additional first argument equal to the function object. - A new flag
METH_BINDING
forml_flags
which only applies to functions of a module (not methods of a class). If this flag is set, thenm_self
is set toNULL
instead of the module. This allows the function to behave more like a Python function as it enables__get__
. - A new flag
METH_CALL_UNBOUND
to disableself slicing. - A new flag
METH_PYTHON
forml_flags
. This flag indicates that this function should be treated as Python function. Ideally, use of this flag should be avoided because it goes against the duck typing philosophy. It is still needed in a few places though, for exampleprofiling.
The goal ofbase_function
is that it supports all different ways
of calling functions and methods in just one structure.
For example, the new flagMETH_PASS_FUNCTION
will be used by the implementation of methods.
It is not possible to directly create instances ofbase_function
(tp_new
isNULL
).
However, it is legal for C code to manually create instances.
These are the relevant C structures:
PyTypeObjectPyBaseFunction_Type;
typedefstruct{
PyObject_HEAD
PyCFunctionDef*m_ml;/*DescriptionoftheCfunctiontocall*/
PyObject*m_self;/*__self__:anything,canbeNULL;readonly*/
PyObject*m_module;/*__module__:anything(typicallystr)*/
PyObject*m_parent;/*__parent__:anything,canbeNULL;readonly*/
PyObject*m_weakreflist;/*Listofweakreferences*/
}PyBaseFunctionObject;
typedefstruct{
constchar*ml_name;/*Thenameofthebuilt-infunction/method*/
PyCFunctionml_meth;/*TheCfunctionthatimplementsit*/
intml_flags;/*CombinationofMETH_xxxflags,whichmostly
describetheargsexpectedbytheCfunc*/
}PyCFunctionDef;
Subclasses may extendPyCFunctionDef
with extra fields.
The Python attribute__self__
returnsm_self
,
except ifMETH_STATIC
is set.
In that case or ifm_self
isNULL
,
then there is no__self__
attribute at all.
For that reason, we write eitherm_self
or__self__
in this PEP
with slightly different meanings.
cfunction
This is the new version of the oldbuiltin_function_or_method
class.
The namecfunction
was chosen to avoid confusion with “built-in”
in the sense of “something in thebuiltins
module”.
It also fits better with the C API which use thePyCFunction
prefix.
The classcfunction
is a copy ofbase_function
,with the following differences:
m_ml
points to aPyMethodDef
structure, extendingPyCFunctionDef
with an additionalml_doc
field to implement__doc__
and__text_signature__
as read-only attributes:typedefstruct{ constchar*ml_name; PyCFunctionml_meth; intml_flags; constchar*ml_doc; }PyMethodDef;
Note that
PyMethodDef
is part of thePython Stable ABI and it is used by practically all extension modules, so we absolutely cannot change this structure.- Argument Clinicis supported.
__self__
always exists. In the cases wherebase_function.__self__
would raiseAttributeError
,insteadNone
is returned.
The type object isPyTypeObjectPyCFunction_Type
and we definePyCFunctionObject
as alias ofPyBaseFunctionObject
(except for the type ofm_ml
).
defined_function
The classdefined_function
is an abstract base class meant
to indicate that the function has introspection support.
Instances ofdefined_function
are required to support all attributes
that Python functions have, namely
__code__
,__globals__
,__doc__
,
__defaults__
,__kwdefaults__
,__closure__
and__annotations__
.
There is also a__dict__
to support attributes added by the user.
None of these is required to be meaningful.
In particular,__code__
may not be a working code object,
possibly only a few fields may be filled in.
This PEP does not dictate how the various attributes are implemented.
They may be simple struct members or more complicated descriptors.
Only read-only support is required, none of the attributes is required to be writable.
The classdefined_function
is mainly meant for auto-generated C code,
for example produced by Cython[1].
There is no API to create instances of it.
The C structure is the following:
PyTypeObjectPyDefinedFunction_Type;
typedefstruct{
PyBaseFunctionObjectbase;
PyObject*func_dict;/*__dict__:dictorNULL*/
}PyDefinedFunctionObject;
TODO:maybe find a better name fordefined_function
.
Other proposals:inspect_function
(anything that satisfiesinspect.isfunction
),
builtout_function
(a function that is better built out; pun on builtin),
generic_function
(original proposal but conflicts withfunctools.singledispatch
generic functions),
user_function
(defined by the user as opposed to CPython).
function
This is the class meant for functions implemented in Python.
Unlike the other function types,
instances offunction
can be created from Python code.
This is not changed, so we do not describe the details in this PEP.
The layout of the C structure is the following:
PyTypeObjectPyFunction_Type;
typedefstruct{
PyBaseFunctionObjectbase;
PyObject*func_dict;/*__dict__:dictorNULL*/
PyObject*func_code;/*__code__:code*/
PyObject*func_globals;/*__globals__:dict;readonly*/
PyObject*func_name;/*__name__:string*/
PyObject*func_qualname;/*__qualname__:string*/
PyObject*func_doc;/*__doc__:canbeanythingorNULL*/
PyObject*func_defaults;/*__defaults__:tupleorNULL*/
PyObject*func_kwdefaults;/*__kwdefaults__:dictorNULL*/
PyObject*func_closure;/*__closure__:tupleofcellobjectsorNULL;readonly*/
PyObject*func_annotations;/*__annotations__:dictorNULL*/
PyCFunctionDef_ml;/*Storageforbase.m_ml*/
}PyFunctionObject;
The descriptor__name__
returnsfunc_name
.
When setting__name__
,alsobase.m_ml->ml_name
is updated
with the UTF-8 encoded name.
The_ml
field reserves space to be used bybase.m_ml
.
Abase_function
instance must have the flagMETH_PYTHON
set
if and only if it is an instance offunction
.
When constructing an instance offunction
fromcode
andglobals
,
an instance is created withbase.m_ml=&_ml
,
base.m_self=NULL
.
To make subclassing easier, we also add a copy constructor:
iff
is an instance offunction
,thentypes.FunctionType(f)
copiesf
.
This conveniently allows using a custom function type as decorator:
>>>fromtypesimportFunctionType
>>>classCustomFunction(FunctionType):
...pass
>>>@CustomFunction
...deff(x):
...returnx
>>>type(f)
<class '__main__.CustomFunction'>
This also removes many use cases offunctools.wraps
:
wrappers can be replaced by subclasses offunction
.
bound_method
The classbound_method
is used for all bound methods,
regardless of the class of the underlying function.
It adds one new attribute on top ofbase_function
:
__func__
points to that function.
bound_method
replaces the oldmethod
class
which was used only for Python functions bound as method.
There is a complication because we want to allow
constructing a method from an arbitrary callable.
This may be an already-bound method or simply not an instance ofbase_function
.
Therefore, in practice there are two kinds of methods:
- For arbitrary callables, we use a single fixed
PyCFunctionDef
structure with theMETH_PASS_FUNCTION
flag set. - For methods which bind instances of
base_function
(more precisely, which have thePy_TPFLAGS_BASEFUNCTION
flag set) that haveself slicing, we instead use thePyCFunctionDef
from the original function. This way, we don’t lose any performance when calling bound methods. In this case, the__func__
attribute is only used to implement various attributes but not for calling the method.
When constructing a new method from abase_function
,
we check that theself
object is an instance of__objclass__
(if a class was specified as parent) and raise aTypeError
otherwise.
The C structure is:
PyTypeObjectPyMethod_Type;
typedefstruct{
PyBaseFunctionObjectbase;
PyObject*im_func;/*__func__:functionimplementingthemethod;readonly*/
}PyMethodObject;
Calling base_function instances
We specify the implementation of__call__
for instances ofbase_function
.
Checking __objclass__
First of all, a type check is done if the__parent__
of the function
is a class
(recall that__objclass__
then becomes an alias of__parent__
):
ifm_self
isNULL
(this is the case for unbound methods of extension types),
then the function must be called with at least one positional argument
and the first (typically calledself
) must be an instance of__objclass__
.
If not, aTypeError
is raised.
Note that bound methods havem_self!=NULL
,so the__objclass__
is not checked.
Instead, the__objclass__
check is done when constructing the method.
Flags
For convenience, we define a new constant:
METH_CALLFLAGS
combines all flags fromPyCFunctionDef.ml_flags
which specify the signature of the C function to be called.
It is equal to
METH_VARARGS|METH_FASTCALL|METH_NOARGS|METH_O|METH_KEYWORDS|METH_PASS_FUNCTION
Exactly one of the first four flags above must be set
and onlyMETH_VARARGS
andMETH_FASTCALL
may be combined withMETH_KEYWORDS
.
Violating these rules is undefined behaviour.
There are one new flags which affects calling functions,
namelyMETH_PASS_FUNCTION
andMETH_CALL_UNBOUND
.
Some flags are already documented in[5].
We explain the others below.
Self slicing
If the function hasm_self==NULL
and the flagMETH_CALL_UNBOUND
is not set, then the first positional argument (if any)
is removed from*args
and instead passed as first argument to the C function.
Effectively, the first positional argument is treated as__self__
.
This is meant to support unbound methods
such that the C function does not see the difference
between bound and unbound method calls.
This does not affect keyword arguments in any way.
This process is calledself slicingand a function is said to
have self slicingifm_self==NULL
andMETH_CALL_UNBOUND
is not set.
Note that aMETH_NOARGS
function which has self slicing
effectively has one argument, namelyself
.
Analogously, aMETH_O
function with self slicing has two arguments.
METH_PASS_FUNCTION
If this flag is set, then the C function is called with an
additional first argument, namely the function itself
(thebase_function
instance).
As special case, if the function is abound_method
,
then the underlying function of the method is passed
(but not recursively: if abound_method
wraps abound_method
,
then__func__
is only applied once).
For example, an ordinaryMETH_VARARGS
function has signature
(PyObject*self,PyObject*args)
.
WithMETH_VARARGS|METH_PASS_FUNCTION
,this becomes
(PyObject*func,PyObject*self,PyObject*args)
.
METH_FASTCALL
This is an existing but undocumented flag. We suggest to officially support and document it.
If the flagMETH_FASTCALL
is set withoutMETH_KEYWORDS
,
then theml_meth
field is of typePyCFunctionFast
which takes the arguments(PyObject*self,PyObject*const*args,Py_ssize_tnargs)
.
Such a function takes only positional arguments and they are passed as plain C array
args
of lengthnargs
.
If the flagsMETH_FASTCALL|METH_KEYWORDS
are set,
then theml_meth
field is of typePyCFunctionFastKeywords
which takes the arguments(PyObject*self,PyObject*const*args,Py_ssize_tnargs,PyObject*kwnames)
.
The positional arguments are passed as C arrayargs
of lengthnargs
.
Thevaluesof the keyword arguments follow in that array,
starting at positionnargs
.
Thekeys(names) of the keyword arguments are passed as atuple
inkwnames
.
As an example, assume that 3 positional and 2 keyword arguments are given.
Thenargs
is an array of length 3 + 2 = 5,nargs
equals 3 andkwnames
is a 2-tuple.
Automatic creation of built-in functions
Python automatically generates instances ofcfunction
for extension types (using thePyTypeObject.tp_methods
field) and modules
(using thePyModuleDef.m_methods
field).
The arraysPyTypeObject.tp_methods
andPyModuleDef.m_methods
must be arrays ofPyMethodDef
structures.
Unbound methods of extension types
The type of unbound methods changes frommethod_descriptor
tocfunction
.
The object which appears as unbound method is the same object which
appears in the class__dict__
.
Python automatically sets the__parent__
attribute to the defining class.
Built-in functions of a module
For the case of functions of a module,
__parent__
will be set to the module.
Unless the flagMETH_BINDING
is given, also__self__
will be set to the module (for backwards compatibility).
An important consequence is that such functions by default
do not become methods when used as attribute
(base_function.__get__
only does that ifm_self
wasNULL
).
One could consider this a bug, but this was done for backwards compatibility reasons:
in an initial post on Python -ideas[6]the consensus was to keep this
misfeature of built-in functions.
However, to allow this anyway for specific or newly implemented
built-in functions, theMETH_BINDING
flag prevents setting__self__
.
Further changes
New type flag
A newPyTypeObject
flag (fortp_flags
) is added:
Py_TPFLAGS_BASEFUNCTION
to indicate that instances of this type are
functions which can be called and bound as method like abase_function
.
This is different from flags likePy_TPFLAGS_LIST_SUBCLASS
because it indicates more than just a subclass:
it also indicates a default implementation of__call__
and__get__
.
In particular, such subclasses ofbase_function
must follow the implementation from the sectionCalling base_function instances.
This flag is automatically set for extension types which
inherit thetp_call
andtp_descr_get
implementation frombase_function
.
Extension types can explicitly specify it if they
override__call__
or__get__
in a compatible way.
The flagPy_TPFLAGS_BASEFUNCTION
must never be set for a heap type
because that would not be safe (heap types can be changed dynamically).
C API functions
We list some relevant Python/C API macros and functions. Some of these are existing (possibly changed) functions, some are new:
intPyBaseFunction_CheckFast(PyObject*op)
:return true ifop
is an instance of a class with thePy_TPFLAGS_BASEFUNCTION
set. This is the function that you need to use to determine whether it is meaningful to access thebase_function
internals.intPyBaseFunction_Check(PyObject*op)
:return true ifop
is an instance ofbase_function
.PyObject*PyBaseFunction_New(PyTypeObject*cls,PyCFunctionDef*ml,PyObject*self,PyObject*module,PyObject*parent)
: create a new instance ofcls
(which must be a subclass ofbase_function
) from the given data.intPyCFunction_Check(PyObject*op)
:return true ifop
is an instance ofcfunction
.intPyCFunction_NewEx(PyMethodDef*ml,PyObject*self,PyObject*module)
: create a new instance ofcfunction
. As special case, ifself
isNULL
, then setself=Py_None
instead (for backwards compatibility). Ifself
is a module, then__parent__
is set toself
. Otherwise,__parent__
isNULL
.- For many existing
PyCFunction_...
andPyMethod_
functions, we define a new functionPyBaseFunction_...
acting onbase_function
instances. The old functions are kept as aliases of the new functions. intPyFunction_Check(PyObject*op)
:return true ifop
is an instance ofbase_function
with theMETH_PYTHON
flag set (this is equivalent to checking whetherop
is an instance offunction
).intPyFunction_CheckFast(PyObject*op)
:equivalent toPyFunction_Check(op)&&PyBaseFunction_CheckFast(op)
.intPyFunction_CheckExact(PyObject*op)
:return true if the type ofop
isfunction
.PyObject*PyFunction_NewPython(PyTypeObject*cls,PyObject*code,PyObject*globals,PyObject*name,PyObject*qualname)
: create a new instance ofcls
(which must be a subclass offunction
) from the given data.PyObject*PyFunction_New(PyObject*code,PyObject*globals)
: create a new instance offunction
.PyObject*PyFunction_NewWithQualName(PyObject*code,PyObject*globals,PyObject*qualname)
: create a new instance offunction
.PyObject*PyFunction_Copy(PyTypeObject*cls,PyObject*func)
: create a new instance ofcls
(which must be a subclass offunction
) by copying a givenfunction
.
Changes to the types module
Two types are added:types.BaseFunctionType
corresponding to
base_function
andtypes.DefinedFunctionType
corresponding to
defined_function
.
Apart from that, no changes to thetypes
module are made.
In particular,types.FunctionType
refers tofunction
.
However, the actual types will change:
in particular,types.BuiltinFunctionType
will no longer be the same
astypes.BuiltinMethodType
.
Changes to the inspect module
The new functioninspect.isbasefunction
checks for an instance ofbase_function
.
inspect.isfunction
checks for an instance ofdefined_function
.
inspect.isbuiltin
checks for an instance ofcfunction
.
inspect.isroutine
checksisbasefunction
orismethoddescriptor
.
NOTE:bpo-33261[3]should be fixed first.
Profiling
Currently,sys.setprofile
supportsc_call
,c_return
andc_exception
events for built-in functions.
These events are generated when calling or returning from a built-in function.
By contrast, thecall
andreturn
events are generated by the function itself.
So nothing needs to change for thecall
andreturn
events.
Since we no longer make a difference between C functions and Python functions,
we need to prevent thec_*
events for Python functions.
This is done by not generating those events if the
METH_PYTHON
flag inml_flags
is set.
Non-CPython implementations
Most of this PEP is only relevant to CPython.
For other implementations of Python,
the two changes that are required are thebase_function
base class
and the fact thatfunction
can be subclassed.
The classescfunction
anddefined_function
are not required.
We requirebase_function
for consistency but we put no requirements on it:
it is acceptable if this is just a copy ofobject
.
Support for the new__parent__
(and__objclass__
) attribute is not required.
If there is nodefined_function
class,
thentypes.DefinedFunctionType
should be an alias oftypes.FunctionType
.
Rationale
Why not simply change existing classes?
One could try to solve the problem by keeping the existing classes
without introducing a newbase_function
class.
That might look like a simpler solution but it is not:
it would require introspection support for 3 distinct classes:
function
,builtin_function_or_method
andmethod_descriptor
.
For the latter two classes, “introspection support” would mean
at a minimum allowing subclassing.
But we don’t want to lose performance, so we want fast subclass checks.
This would require two new flags intp_flags
.
And we want subclasses to allow__get__
for built-in functions,
so we should implement theLOAD_METHOD
opcode for built-in functions too.
More generally, a lot of functionality would need to be duplicated
and the end result would be far more complex code.
It is also not clear how the introspection of built-in function subclasses
would interact with__text_signature__
.
Having two independent kinds ofinspect.signature
support on the same
class sounds like asking for problems.
And this would not fix some of the other differences between built-in functions and Python functions that were mentioned in themotivation.
Why __text_signature__ is not a solution
Built-in functions have an attribute__text_signature__
,
which gives the signature of the function as plain text.
The default values are evaluated byast.literal_eval
.
Because of this, it supports only a small number of standard Python classes
and not arbitrary Python objects.
And even if__text_signature__
would allow arbitrary signatures somehow,
that is only one piece of introspection:
it does not help withinspect.getsourcefile
for example.
defined_function versus function
In many places, a decision needs to be made whether the oldfunction
class
should be replaced bydefined_function
or the newfunction
class.
This is done by thinking of the most likely use case:
types.FunctionType
refers tofunction
because that type might be used to construct instances usingtypes.FunctionType(...)
.inspect.isfunction()
refers todefined_function
because this is the class where introspection is supported.- The C API functions must refer to
function
because we do not specify how the various attributes ofdefined_function
are implemented. We expect that this is not a problem since there is typically no reason for introspection to be done by C extensions.
Scope of this PEP: which classes are involved?
The main motivation of this PEP is fi xing function classes,
so we certainly want to unify the existing classes
builtin_function_or_method
andfunction
.
Since built-in functions and methods have the same class, it seems natural to include bound methods too. And since there are no “unbound methods” for Python functions, it makes sense to get rid of unbound methods for extension types.
For now, no changes are made to the classesstaticmethod
,
classmethod
andclassmethod_descriptor
.
It would certainly make sense to put these in thebase_function
class hierarchy and unifyclassmethod
andclassmethod_descriptor
.
However, this PEP is already big enough
and this is left as a possible future improvement.
Slot wrappers for extension types like__init__
or__eq__
are quite different from normal methods.
They are also typically not called directly because you would normally
writefoo[i]
instead offoo.__getitem__(i)
.
So these are left outside the scope of this PEP.
Python also has aninstancemethod
class,
which seems to be a relic from Python 2,
where it was used for bound and unbound methods.
It is not clear whether there is still a use case for it.
In any case, there is no reason to deal with it in this PEP.
TODO:shouldinstancemethod
be deprecated?
It doesn’t seem used at all within CPython 3.7,
but maybe external packages use it?
Not treating METH_STATIC and METH_CLASS
Almost nothing in this PEP refers to the flagsMETH_STATIC
andMETH_CLASS
.
These flags are checked only by theautomatic creation of built-in functions.
When astaticmethod
,classmethod
orclassmethod_descriptor
is bound (i.e.__get__
is called),
abase_function
instance is created withm_self!=NULL
.
For aclassmethod
,this is obvious sincem_self
is the class that the method is bound to.
For astaticmethod
,one can take an arbitrary Python object form_self
.
For backwards compatibility, we choosem_self=__parent__
for static methods
of extension types.
__self__ in base_function
It may look strange at first sight to add the__self__
slot
inbase_function
as opposed tobound_method
.
We took this idea from the existingbuiltin_function_or_method
class.
It allows us to have a single general implementation of__call__
and__get__
for the various function classes discussed in this PEP.
It also makes it easy to support existing built-in functions
which set__self__
to the module (for example,sys.exit.__self__
issys
).
Two implementations of __doc__
base_function
does not support function docstrings.
Instead, the classescfunction
andfunction
each have their own way of dealing with docstrings
(andbound_method
just takes the__doc__
from the wrapped function).
Forcfunction
,the docstring is stored (together with the text signature)
as C string in the read-onlyml_doc
field of aPyMethodDef
.
Forfunction
,the docstring is stored as a writable Python object
and it does not actually need to be a string.
It looks hard to unify these two very different ways of dealing with__doc__
.
For backwards compatibility, we keep the existing implementations.
Fordefined_function
,we require__doc__
to be implemented
but we do not specify how. A subclass can implement__doc__
the
same way ascfunction
or using a struct member or some other way.
Subclassing
We disallow subclassing ofcfunction
andbound_method
to enable fast type checks forPyCFunction_Check
andPyMethod_Check
.
We allow subclassing of the other classes because there is no reason to disallow it.
For Python modules, the only relevant class to subclass is
function
because the others cannot be instantiated anyway.
Replacing tp_call: METH_PASS_FUNCTION and METH_CALL_UNBOUND
The new flagsMETH_PASS_FUNCTION
andMETH_CALL_UNBOUND
are meant to support cases where formerly a customtp_call
was used.
It reduces the number of special fast paths inPython/ceval.c
for calling objects:
instead of treating Python functions, built-in functions and method descriptors
separately, there would only be a single check.
The signature oftp_call
is essentially the signature
ofPyBaseFunctionObject.m_ml.ml_meth
with flags
METH_VARARGS|METH_KEYWORDS|METH_PASS_FUNCTION|METH_CALL_UNBOUND
(the only difference is an addedself
argument).
Therefore, it should be easy to change existingtp_call
slots
to use thebase_function
implementation instead.
It also makes sense to useMETH_PASS_FUNCTION
withoutMETH_CALL_UNBOUND
in cases where the C function simply needs access to additional metadata
from the function, such as the__parent__
.
This is for example needed to supportPEP 573.
Converting existing methods to useMETH_PASS_FUNCTION
is trivial:
it only requires adding an extra argument to the C function.
Backwards compatibility
While designing this PEP, great care was taken to not break backwards compatibility too much. Most of the potentially incompatible changes are changes to CPython implementation details which are different anyway in other Python interpreters. In particular, Python code which correctly runs on PyPy will very likely continue to work with this PEP.
The standard classes and functions like
staticmethod
,functools.partial
oroperator.methodcaller
do not need to change at all.
Changes to types and inspect
The proposed changes totypes
andinspect
are meant to minimize changes in behaviour.
However, it is unavoidable that some things change
and this can cause code which usestypes
orinspect
to break.
In the Python standard library for example,
changes are needed in thedoctest
module because of this.
Also, tools which take various kinds of functions as input will need to deal with the new function hierarchy and the possibility of custom function classes.
Python functions
For Python functions, essentially nothing changes. The attributes that existed before still exist and Python functions can be initialized, called and turned into methods as before.
The namefunction
is kept for backwards compatibility.
While it might make sense to change the name to something more
specific likePython _function
,
that would require a lot of annoying changes in documentation and testsuites.
Built-in functions of a module
Also for built-in functions, nothing changes.
We keep the old behaviour that such functions do not bind as methods.
This is a consequence of the fact that__self__
is set to the module.
Built-in bound and unbound methods
The types of built-in bound and unbound methods will change.
However, this does not affect calling such methods
because the protocol inbase_function.__call__
(in particular the handling of__objclass__
and self slicing)
was specifically designed to be backwards compatible.
All attributes which existed before (like__objclass__
and__self__
)
still exist.
New attributes
Some objects get new special double-underscore attributes.
For example, the new attribute__parent__
appears on
all built-in functions and all methods get a__func__
attribute.
The fact that__self__
is now a special read-only attribute
for Python functions caused trouble in[4].
Generally, we expect that not much will break though.
method_descriptor and PyDescr_NewMethod
The classmethod_descriptor
and the constructorPyDescr_NewMethod
should be deprecated.
They are no longer used by CPython itself but are still supported.
Two-phase Implementation
TODO:this section is optional. If this PEP is accepted, it should be decided whether to apply this two-phase implementation or not.
As mentioned above, thechanges to types and inspectcan break some existing code. In order to further minimize breakage, this PEP could be implemented in two phases.
Phase one: keep existing classes but add base classes
Initially, implement thebase_function
class
and use it as common base class but otherwise keep the existing classes
(but not their implementation).
In this proposal, the class hierarchy would become:
object
|
|
base_function
/|\
/|\
/|\
cfunction|defined_function
|||\
||bound_method\
||\
|method_descriptorfunction
|
builtin_function_or_method
The leaf classesbuiltin_function_or_method
,method_descriptor
,
bound_method
andfunction
correspond to the existing classes
(withmethod
renamed tobound_method
).
Automatically created functions created in modules become instances
ofbuiltin_function_or_method
.
Unbound methods of extension types become instances ofmethod_descriptor
.
The classmethod_descriptor
is a copy ofcfunction
except
that__get__
returns abuiltin_function_or_method
instead of a
bound_method
.
The classbuiltin_function_or_method
has the same C structure as a
bound_method
,but it inherits fromcfunction
.
The__func__
attribute is not mandatory:
it is only defined when binding amethod_descriptor
.
We keep the implementation of theinspect
functions as they are.
Because of this and because the existing classes are kept,
backwards compatibility is ensured for code doing type checks.
Since showing an actualDeprecationWarning
would affect a lot
of correctly-functioning code,
any deprecations would only appear in the documentation.
Another reason is that it is hard to show warnings for callingisinstance(x,t)
(but it could be done using__instancecheck__
hacking)
and impossible fortype(x)ist
.
Phase two
Phase two is what is actually described in the rest of this PEP. In terms of implementation, it would be a relatively small change compared to phase one.
Reference Implementation
Most of this PEP has been implemented for CPython at https://github /jdemeyer/c Python /tree/pep575
There are four steps, corresponding to the commits on that branch. After each step, CPython is in a mostly working state.
- Add the
base_function
class and make it a subclass forcfunction
. This is by far the biggest step as the complete__call__
protocol is implemented in this step. - Rename
method
tobound_method
and make it a subclass ofbase_function
. Change unbound methods of extension types to be instances ofcfunction
such that bound methods of extension types are also instances ofbound_method
. - Implement
defined_function
andfunction
. - Changes to other parts of Python, such as the standard library and testsuite.
Appendix: current situation
NOTE: This section is more useful during the draft period of the PEP, so feel free to remove this once the PEP has been accepted.
For reference, we describe in detail the relevant existing classes in CPython 3.7.
Each of the classes involved is an “orphan” class (no non-trivial subclasses nor superclasses).
builtin_function_or_method: built-in functions and bound methods
These are of typePyCFunction_Type with structurePyCFunctionObject:
typedefstruct{
PyObject_HEAD
PyMethodDef*m_ml;/*DescriptionoftheCfunctiontocall*/
PyObject*m_self;/*Passedas'self'argtotheCfunc,canbeNULL*/
PyObject*m_module;/*The__module__attribute,canbeanything*/
PyObject*m_weakreflist;/*Listofweakreferences*/
}PyCFunctionObject;
structPyMethodDef{
constchar*ml_name;/*Thenameofthebuilt-infunction/method*/
PyCFunctionml_meth;/*TheCfunctionthatimplementsit*/
intml_flags;/*CombinationofMETH_xxxflags,whichmostly
describetheargsexpectedbytheCfunc*/
constchar*ml_doc;/*The__doc__attribute,orNULL*/
};
wherePyCFunction
is a C function pointer (there are various forms of this, the most basic
takes two arguments forself
and*args
).
This class is used both for functions and bound methods:
for a method, them_self
slot points to the object:
>>>dict(foo=42).get
<built-in method get of dict object at 0x...>
>>>dict(foo=42).get.__self__
{'foo': 42}
In some cases, a function is considered a “method” of the module defining it:
>>>importos
>>>os.kill
<built-in function kill>
>>>os.kill.__self__
<module 'posix' (built-in)>
method_descriptor: built-in unbound methods
These are of typePyMethodDescr_Type with structurePyMethodDescrObject:
typedefstruct{
PyDescrObjectd_common;
PyMethodDef*d_method;
}PyMethodDescrObject;
typedefstruct{
PyObject_HEAD
PyTypeObject*d_type;
PyObject*d_name;
PyObject*d_qualname;
}PyDescrObject;
function: Python functions
These are of typePyFunction_Type with structurePyFunctionObject:
typedefstruct{
PyObject_HEAD
PyObject*func_code;/*Acodeobject,the__code__attribute*/
PyObject*func_globals;/*Adictionary(othermappingswon't do) */
PyObject*func_defaults;/*NULLoratuple*/
PyObject*func_kwdefaults;/*NULLoradict*/
PyObject*func_closure;/*NULLoratupleofcellobjects*/
PyObject*func_doc;/*The__doc__attribute,canbeanything*/
PyObject*func_name;/*The__name__attribute,astringobject*/
PyObject*func_dict;/*The__dict__attribute,adictorNULL*/
PyObject*func_weakreflist;/*Listofweakreferences*/
PyObject*func_module;/*The__module__attribute,canbeanything*/
PyObject*func_annotations;/*Annotations,adictorNULL*/
PyObject*func_qualname;/*Thequalifiedname*/
/*Invariant:
*func_closurecontainsthebindingsforfunc_code->co_freevars,so
*PyTuple_Size(func_closure)==PyCode_GetNumFree(func_code)
*(func_closuremaybeNULLifPyCode_GetNumFree(func_code)==0).
*/
}PyFunctionObject;
In Python 3, there is no “unbound method” class: an unbound method is just a plain function.
method: Python bound methods
These are of typePyMethod_Type with structurePyMethodObject:
typedefstruct{
PyObject_HEAD
PyObject*im_func;/*Thecallableobjectimplementingthemethod*/
PyObject*im_self;/*Theinstanceitisboundto*/
PyObject*im_weakreflist;/*Listofweakreferences*/
}PyMethodObject;
References
Copyright
This document has been placed in the public domain.
Source:https://github / Python /peps/blob/main/peps/pep-0575.rst
Last modified:2023-09-09 17:39:29 GMT