Introduction

This PEP describes theaugmented assignmentproposal for Python 2.0. This PEP tracks the status and ownership of this feature, slated for introduction in Python 2.0. It contains a description of the feature and outlines changes necessary to support the feature. This PEP summarizes discussions held in mailing list forums[1],and provides URLs for further information where appropriate. The CVS revision history of this file contains the definitive historical record.

Proposed Semantics

The proposed patch that adds augmented assignment to Python introduces the following new operators:

+=-=*=/=%=**=<<=>>=&=^=|=

They implement the same operator as their normal binary form, except that the operation is donein-placewhen the left-hand side object supports it, and that the left-hand side is only evaluated once.

They truly behave as augmented assignment, in that they perform all of the normal load and store operations, in addition to the binary operation they are intended to do. So, given the expression:

x+=y

The objectxis loaded, thenyis added to it, and the resulting object is stored back in the original place. The precise action performed on the two arguments depends on the type ofx,and possibly ofy.

The idea behind augmented assignment in Python is that it isn’t just an easier way to write the common practice of storing the result of a binary operation in its left-hand operand, but also a way for the left-hand operand in question to know that it should operateon itself,rather than creating a modified copy of itself.

To make this possible, a number of newhooksare added to Python classes and C extension types, which are called when the object in question is used as the left hand side of an augmented assignment operation. If the class or type does not implement thein-placehooks, the normal hooks for the particular binary operation are used.

So, given an instance objectx,the expression:

x+=y

tries to callx.__iadd__(y),which is thein-placevariant of __add__.If__iadd__is not present,x.__add__(y)is attempted, and finallyy.__radd__(x)if__add__is missing too. There is no right-hand-sidevariant of__iadd__,because that would require for yto know how to in-place modifyx,which is unsafe to say the least. The__iadd__hook should behave similar to__add__,returning the result of the operation (which could beself) which is to be assigned to the variablex.

For C extension types, thehooksare members of thePyNumberMethodsand PySequenceMethodsstructures. Some special semantics apply to make the use of these methods, and the mi xing of Python instance objects and C types, as unsurprising as possible.

In the generic case ofx<augop>y(or a similar case using the PyNumber_InPlaceAPI functions) the principal object being operated on is x.This differs from normal binary operations, wherexandy could be consideredco-operating,because unlike in binary operations, the operands in an in-place operation cannot be swapped. However, in-place operations do fall back to normal binary operations when in-place modification is not supported, resulting in the following rules:

• If the left-hand object (x) is an instance object, and it has a __coerce__method, call that function withyas the argument. If coercion succeeds, and the resulting left-hand object is a different object thanx,stop processing it as in-place and call the appropriate function for the normal binary operation, with the coercedxandyas arguments. The result of the operation is whatever that function returns.

  If coercion does not yield a different object forx,orxdoes not define a__coerce__method, andxhas the appropriate__ihook__ for this operation, call that method withyas the argument, and the result of the operation is whatever that method returns.

• Otherwise, if the left-hand object is not an instance object, but its type does define the in-place function for this operation, call that function withxandyas the arguments, and the result of the operation is whatever that function returns.

  Note that no coercion on eitherxoryis done in this case, and it’s perfectly valid for a C type to receive an instance object as the second argument; that is something that cannot happen with normal binary operations.

• Otherwise, process it exactly as a normal binary operation (not in-place), including argument coercion. In short, if either argument is an instance object, resolve the operation through__coerce__,__hook__and __rhook__.Otherwise, both objects are C types, and they are coerced and passed to the appropriate function.
• If no way to process the operation can be found, raise aTypeErrorwith an error message specific to the operation.
• Some special casing exists to account for the case of+and*, which have a special meaning for sequences: for+,sequence concatenation, no coercion what so ever is done if a C type defines sq_concatorsq_inplace_concat.For*,sequence repeating, yis converted to a C integer before calling either sq_inplace_repeatandsq_repeat.This is done even ifyis an instance, though not ifxis an instance.

The in-place function should always return a new reference, either to the oldxobject if the operation was indeed performed in-place, or to a new object.

Rationale

There are two main reasons for adding this feature to Python: simplicity of expression, and support for in-place operations. The end result is a tradeoff between simplicity of syntax and simplicity of expression; like most new features, augmented assignment doesn’t add anything that was previously impossible. It merely makes these things easier to do.

Adding augmented assignment will make Python’s syntax more complex. Instead of a single assignment operation, there are now twelve assignment operations, eleven of which also perform a binary operation. However, these eleven new forms of assignment are easy to understand as the coupling between assignment and the binary operation, and they require no large conceptual leap to understand. Furthermore, languages that do have augmented assignment have shown that they are a popular, much used feature. Expressions of the form:

<x>=<x><operator><y>

are common enough in those languages to make the extra syntax worthwhile, and Python does not have significantly fewer of those expressions. Quite the opposite, in fact, since in Python you can also concatenate lists with a binary operator, something that is done quite frequently. Writing the above expression as:

<x><operator>=<y>

is both more readable and less error prone, because it is instantly obvious to the reader that it is<x>that is being changed, and not<x>that is being replaced by something almost, but not quite, entirely unlike<x>.

The new in-place operations are especially useful to matrix calculation and other applications that require large objects. In order to efficiently deal with the available program memory, such packages cannot blindly use the current binary operations. Because these operations always create a new object, adding a single item to an existing (large) object would result in copying the entire object (which may cause the application to run out of memory), add the single item, and then possibly delete the original object, depending on reference count.

To work around this problem, the packages currently have to use methods or functions to modify an object in-place, which is definitely less readable than an augmented assignment expression. Augmented assignment won’t solve all the problems for these packages, since some operations cannot be expressed in the limited set of binary operators to start with, but it is a start.PEP 211 is looking at adding new operators.

New methods

The proposed implementation adds the following 11 possiblehookswhich Python classes can implement to overload the augmented assignment operations:

__iadd__
__isub__
__imul__
__idiv__
__imod__
__ipow__
__ilshift__
__irshift__
__iand__
__ixor__
__ior__

Theiin__iadd__stands forin-place.

For C extension types, the following struct members are added.

ToPyNumberMethods:

binaryfuncnb_inplace_add;
binaryfuncnb_inplace_subtract;
binaryfuncnb_inplace_multiply;
binaryfuncnb_inplace_divide;
binaryfuncnb_inplace_remainder;
binaryfuncnb_inplace_power;
binaryfuncnb_inplace_lshift;
binaryfuncnb_inplace_rshift;
binaryfuncnb_inplace_and;
binaryfuncnb_inplace_xor;
binaryfuncnb_inplace_or;

ToPySequenceMethods:

binaryfuncsq_inplace_concat;
intargfuncsq_inplace_repeat;

In order to keep binary compatibility, thetp_flagsTypeObject member is used to determine whether the TypeObject in question has allocated room for these slots. Until a clean break in binary compatibility is made (which may or may not happen before 2.0) code that wants to use one of the new struct members must first check that they are available with the PyType_HasFeature()macro:

if(PyType_HasFeature(x->ob_type,Py_TPFLAGS_HAVE_INPLACE_OPS)&&
x->ob_type->tp_as_number&&x->ob_type->tp_as_number->nb_inplace_add){
/*...*/

This check must be made even before testing the method slots forNULL values! The macro only tests whether the slots are available, not whether they are filled with methods or not.

Implementation

The current implementation of augmented assignment[2]adds, in addition to the methods and slots already covered, 13 new bytecodes and 13 new API functions.

The API functions are simply in-place versions of the current binary-operation API functions:

PyNumber_InPlaceAdd(PyObject*o1,PyObject*o2);
PyNumber_InPlaceSubtract(PyObject*o1,PyObject*o2);
PyNumber_InPlaceMultiply(PyObject*o1,PyObject*o2);
PyNumber_InPlaceDivide(PyObject*o1,PyObject*o2);
PyNumber_InPlaceRemainder(PyObject*o1,PyObject*o2);
PyNumber_InPlacePower(PyObject*o1,PyObject*o2);
PyNumber_InPlaceLshift(PyObject*o1,PyObject*o2);
PyNumber_InPlaceRshift(PyObject*o1,PyObject*o2);
PyNumber_InPlaceAnd(PyObject*o1,PyObject*o2);
PyNumber_InPlaceXor(PyObject*o1,PyObject*o2);
PyNumber_InPlaceOr(PyObject*o1,PyObject*o2);
PySequence_InPlaceConcat(PyObject*o1,PyObject*o2);
PySequence_InPlaceRepeat(PyObject*o,intcount);

They call either the Python class hooks (if either of the objects is a Python class instance) or the C type’s number or sequence methods.

The new bytecodes are:

INPLACE_ADD
INPLACE_SUBTRACT
INPLACE_MULTIPLY
INPLACE_DIVIDE
INPLACE_REMAINDER
INPLACE_POWER
INPLACE_LEFTSHIFT
INPLACE_RIGHTSHIFT
INPLACE_AND
INPLACE_XOR
INPLACE_OR
ROT_FOUR
DUP_TOPX

TheINPLACE_*bytecodes mirror theBINARY_*bytecodes, except that they are implemented as calls to theInPlaceAPI functions. The other two bytecodes areutilitybytecodes:ROT_FOURbehaves likeROT_THREE except that the four topmost stack items are rotated.

DUP_TOPXis a bytecode that takes a single argument, which should be an integer between 1 and 5 (inclusive) which is the number of items to duplicate in one block. Given a stack like this (where the right side of the list is thetopof the stack):

[1,2,3,4,5]

DUP_TOPX3would duplicate the top 3 items, resulting in this stack:

[1,2,3,4,5,3,4,5]

DUP_TOPXwith an argument of 1 is the same asDUP_TOP.The limit of 5 is purely an implementation limit. The implementation of augmented assignment requires onlyDUP_TOPXwith an argument of 2 and 3, and could do without this new opcode at the cost of a fair number ofDUP_TOPand ROT_*.

Open Issues

ThePyNumber_InPlaceAPI is only a subset of the normalPyNumberAPI: only those functions that are required to support the augmented assignment syntax are included. If other in-place API functions are needed, they can be added later.

TheDUP_TOPXbytecode is a conveniency bytecode, and is not actually necessary. It should be considered whether this bytecode is worth having. There seems to be no other possible use for this bytecode at this time.

Copyright

This document has been placed in the public domain.

References