PEP 728 – TypedDict with Typed Extra Items
- Author:
- Zixuan James Li <p359101898 at gmail >
- Sponsor:
- Jelle Zijlstra <jelle.zijlstra at gmail >
- Discussions-To:
- Discourse thread
- Status:
- Draft
- Type:
- Standards Track
- Topic:
- Typing
- Created:
- 12-Sep-2023
- Python-Version:
- 3.13
- Post-History:
- 09-Feb-2024
Abstract
This PEP proposes a way to limit extra items forTypedDict
using aclosed
argument and to type them with the special__extra_items__
key. This addresses the need to define closed TypedDict type or to type a subset
of keys that might appear in adict
while permitting additional items of a
specified type.
Motivation
Atyping.TypedDict
type can annotate the value type of each known
item in a dictionary. However, due to structural subtyping, a TypedDict can have
extra items that are not visible through its type. There is currently no way to
restrict the types of items that might be present in the TypedDict type’s
structural subtypes.
Defining a Closed TypedDict Type
The current behavior of TypedDict prevents users from defining a closed TypedDict type when it is expected that the type contains no additional items.
Due to the possible presence of extra items, type checkers cannot infer more
precise return types for.items()
and.values()
on a TypedDict. This can
also be resolved by
defining a closed TypedDict type.
Another possible use case for this is a sound way to
enable type narrowingwith the
in
check:
classMovie(TypedDict):
name:str
director:str
classBook(TypedDict):
name:str
author:str
deffun(entry:Movie|Book)->None:
if"author"inentry:
reveal_type(entry)# Revealed type is 'Movie | Book'
Nothing prevents adict
that is structurally compatible withMovie
to
have theauthor
key, and under the current specification it would be
incorrect for the type checker to narrow its type.
Allowing Extra Items of a Certain Type
For supporting API interfaces or legacy codebase where only a subset of possible keys are known, it would be useful to explicitly expect additional keys of certain value types.
However, the typing spec is more restrictive on type checking the construction of a TypedDict,preventing users from doing this:
classMovieBase(TypedDict):
name:str
deffun(movie:MovieBase)->None:
# movie can have extra items that are not visible through MovieBase
...
movie:MovieBase={"name":"Blade Runner","year":1982}# Not OK
fun({"name":"Blade Runner","year":1982})# Not OK
While the restriction is enforced when constructing a TypedDict, due to structural subtyping, the TypedDict may have extra items that are not visible through its type. For example:
classMovie(MovieBase):
year:int
movie:Movie={"name":"Blade Runner","year":1982}
fun(movie)# OK
It is not possible to acknowledge the existence of the extra items through
in
checks and access them without breaking type safety, even though they
might exist from arbitrary structural subtypes ofMovieBase
:
defg(movie:MovieBase)->None:
if"year"inmovie:
reveal_type(movie["year"])# Error: TypedDict 'MovieBase' has no key 'year'
Some workarounds have already been implemented in response to the need to allow
extra keys, but none of them is ideal. For mypy,
--disable-error-code=typeddict-unknown-key
suppresses type checking error
specifically for unknown keys on TypedDict. This sacrifices type safety over
flexibility, and it does not offer a way to specify that the TypedDict type
expects additional keys compatible with a certain type.
Support Additional Keys forUnpack
PEP 692adds a way to precisely annotate the types of individual keyword
arguments represented by**kwargs
using TypedDict withUnpack
.However,
because TypedDict cannot be defined to accept arbitrary extra items, it is not
possible to
allow additional keyword arguments
that are not known at the time the TypedDict is defined.
Given the usage of pre-PEP 692type annotation for**kwargs
in existing
codebases, it will be valuable to accept and type extra items on TypedDict so
that the old typing behavior can be supported in combination with the new
Unpack
construct.
Rationale
A type that allows extra items of typestr
on a TypedDict can be loosely
described as the intersection between the TypedDict andMapping[str,str]
.
Index Signatures in TypeScript achieve this:
typeFoo={
a:string
[key:string]:string
}
This proposal aims to support a similar feature without introducing general intersection of types or syntax changes, offering a natural extension to the existing type consistency rules.
We propose that we add an argumentclosed
to TypedDict. Similar to
total
,only a literalTrue
orFalse
value is allowed. When
closed=True
is used in the TypedDict type definition, we give the dunder
attribute__extra_items__
a special meaning: extra items are allowed, and
their types should be compatible with the value type of__extra_items__
.
Ifclosed=True
is set, but there is no__extra_items__
key, the
TypedDict is treated as if it contained an item__extra_items__:Never
.
Note that__extra_items__
on the same TypedDict type definition will remain
as a regular item ifclosed=True
is not used.
Different from index signatures, the types of the known items do not need to be
consistent with the value type of__extra_items__
.
There are some advantages to this approach:
- Inheritance works naturally.
__extra_items__
defined on a TypedDict will also be available to its subclasses. - We can build on top of thetype consistency rules defined in the typing spec.
__extra_items__
can be treated as a pseudo-item in terms of type consistency. - There is no need to introduce a grammar change to specify the type of the extra items.
- We can precisely type the extra items without making
__extra_items__
the union of known items. - We do not lose backwards compatibility as
__extra_items__
still can be used as a regular key.
Specification
This specification is structured to parallelPEP 589to highlight changes to the original TypedDict specification.
Ifclosed=True
is specified, extra items are treated as non-required items
having the same type of__extra_items__
whose keys are allowed when
determining
supported and unsupported operations.
Using TypedDict Types
Assuming thatclosed=True
is used in the TypedDict type definition.
For a TypedDict type that has the special__extra_items__
key, during
construction, the value type of each unknown item is expected to be non-required
and compatible with the value type of__extra_items__
.For example:
classMovie(TypedDict,closed=True):
name:str
__extra_items__:bool
a:Movie={"name":"Blade Runner","novel_adaptation":True}# OK
b:Movie={
"name":"Blade Runner",
"year":1982,# Not OK. 'int' is incompatible with 'bool'
}
In this example,__extra_items__:bool
does not mean thatMovie
has a
required string key"__extra_items__"
whose value type isbool
.Instead,
it specifies that keys other than “name” have a value type ofbool
and are
non-required.
The alternative inline syntax is also supported:
Movie=TypedDict("Movie",{"name":str,"__extra_items__":bool},closed=True)
Accessing extra keys is allowed. Type checkers must infer its value type from
the value type of__extra_items__
:
deff(movie:Movie)->None:
reveal_type(movie["name"])# Revealed type is 'str'
reveal_type(movie["novel_adaptation"])# Revealed type is 'bool'
When a TypedDict type defines__extra_items__
withoutclosed=True
,
closed
defaults toFalse
and the key is assumed to be a regular key:
classMovie(TypedDict):
name:str
__extra_items__:bool
a:Movie={"name":"Blade Runner","novel_adaptation":True}# Not OK. Unexpected key 'novel_adaptation'
b:Movie={
"name":"Blade Runner",
"__extra_items__":True,# OK
}
For such non-closed TypedDict types, it is assumed that they allow non-required
extra items of value typeReadOnly[object]
during inheritance or type
consistency checks. However, extra keys found during construction should still
be rejected by the type checker.
closed
is not inherited through subclassing:
classMovieBase(TypedDict,closed=True):
name:str
__extra_items__:ReadOnly[str|None]
classMovie(MovieBase):
__extra_items__:str# A regular key
a:Movie={"name":"Blade Runner","__extra_items__":None}# Not OK. 'None' is incompatible with 'str'
b:Movie={
"name":"Blade Runner",
"__extra_items__":"A required regular key",
"other_extra_key":None,
}# OK
Here,"__extra_items__"
ina
is a regular key defined onMovie
where
its value type is narrowed fromReadOnly[str|None]
tostr
,
"other_extra_key"
inb
is an extra key whose value type must be
consistent with the value type of"__extra_items__"
defined on
MovieBase
.
Interaction with Totality
It is an error to useRequired[]
orNotRequired[]
with the special
__extra_items__
item.total=False
andtotal=True
have no effect on
__extra_items__
itself.
The extra items are non-required, regardless of the totality of the TypedDict.
Operations that are available toNotRequired
items should also be available
to the extra items:
classMovie(TypedDict,closed=True):
name:str
__extra_items__:int
deff(movie:Movie)->None:
delmovie["name"]# Not OK
delmovie["year"]# OK
Interaction withUnpack
For type checking purposes,Unpack[TypedDict]
with extra items should be
treated as its equivalent in regular parameters, and the existing rules for
function parameters still apply:
classMovie(TypedDict,closed=True):
name:str
__extra_items__:int
deff(**kwargs:Unpack[Movie])->None:...
# Should be equivalent to
deff(*,name:str,**kwargs:int)->None:...
Interaction with PEP 705
When the special__extra_items__
item is annotated withReadOnly[]
,the
extra items on the TypedDict have the properties of read-only items. This
interacts with inheritance rules specified inPEP 705.
Notably, if the TypedDict type declares__extra_items__
to be read-only, a
subclass of the TypedDict type may redeclare__extra_items__
’s value type or
additional non-extra items’ value type.
Because a non-closed TypedDict type implicitly allows non-required extra items
of value typeReadOnly[object]
,its subclass can override the special
__extra_items__
with more specific types.
More details are discussed in the later sections.
Inheritance
When the TypedDict type is defined asclosed=False
(the default),
__extra_items__
should behave and be inherited the same way a regular key
would. A regular__extra_items__
key can coexist with the special
__extra_items__
and both should be inherited when subclassing.
We assume thatclosed=True
whenever__extra_items__
is mentioned for the
rest of this section.
__extra_items__
is inherited the same way as a regularkey:value_type
item. As with the other keys, the same rules from
the typing spec
andPEP 705apply. We interpret the existing rules in the
context of__extra_items__
.
We need to reinterpret the following rule to define how__extra_items__
interacts with it:
- Changing a field type of a parent TypedDict class in a subclass is not allowed.
First, it is not allowed to change the value type of__extra_items__
in a subclass
unless it is declared to beReadOnly
in the superclass:
classParent(TypedDict,closed=True):
__extra_items__:int|None
classChild(Parent,closed=True):
__extra_items__:int# Not OK. Like any other TypedDict item, __extra_items__'s type cannot be changed
Second,__extra_items__:T
effectively defines the value type of any unnamed
items accepted to the TypedDict and marks them as non-required. Thus, the above
restriction applies to any additional items defined in a subclass. For each item
added in a subclass, all of the following conditions should apply:
- If
__extra_items__
is read-only- The item can be either required or non-required
- The item’s value type is consistent with
T
- If
__extra_items__
is not read-only- The item is non-required
- The item’s value type is consistent with
T
T
is consistent with the item’s value type
- If
__extra_items__
is not redeclared, the subclass inherits it as-is.
For example:
classMovieBase(TypedDict,closed=True):
name:str
__extra_items__:int|None
classAdaptedMovie(MovieBase):# Not OK. 'bool' is not consistent with 'int | None'
adapted_from_novel:bool
classMovieRequiredYear(MovieBase):# Not OK. Required key 'year' is not known to 'Parent'
year:int|None
classMovieNotRequiredYear(MovieBase):# Not OK. 'int | None' is not consistent with 'int'
year:NotRequired[int]
classMovieWithYear(MovieBase):# OK
year:NotRequired[int|None]
Due to this nature, an important side effect allows us to define a TypedDict type that disallows additional items:
classMovieFinal(TypedDict,closed=True):
name:str
__extra_items__:Never
Here, annotating__extra_items__
withtyping.Never
specifies that
there can be no other keys inMovieFinal
other than the known ones.
Because of its potential common use, this is equivalent to:
classMovieFinal(TypedDict,closed=True):
name:str
where we implicitly assume the__extra_items__:Never
field by default
if onlyclosed=True
is specified.
Type Consistency
In addition to the setS
of keys of the explicitly defined items, a
TypedDict type that has the item__extra_items__:T
is considered to have an
infinite set of items that all satisfy the following conditions:
- If
__extra_items__
is read-only- The key’s value type is consistent with
T
- The key is not in
S
.
- The key’s value type is consistent with
- If
__extra_items__
is not read-only- The key is non-required
- The key’s value type is consistent with
T
T
is consistent with the key’s value type- The key is not in
S
.
For type checking purposes, let__extra_items__
be a non-required pseudo-item to
be included whenever “for each… item/key” is stated in
the existing type consistency rules from PEP 705,
and we modify it as follows:
A TypedDict typeA
is consistent with TypedDictB
ifA
is structurally compatible withB
.This is true if and only if all of the following are satisfied:
- For each item in
B
,A
has the corresponding key, unless the item inB
is read-only, not required, and of top value type (ReadOnly[NotRequired[object]]
).[Edit: Otherwise, if the corresponding key with the same name cannot be found in ``A``, “__extra_items__” is considered the corresponding key.]- For each item in
B
,ifA
has the corresponding key[Edit: or “__extra_items__” ],the corresponding value type inA
is consistent with the value type inB
.- For each non-read-only item in
B
,its value type is consistent with the corresponding value type inA
.[Edit: if the corresponding key with the same name cannot be found in ``A``, “__extra_items__” is considered the corresponding key.]- For each required key in
B
,the corresponding key is required inA
. For each non-required key inB
,if the item is not read-only inB
, the corresponding key is not required inA
. [Edit: if the corresponding key with the same name cannot be found in ``A``, “__extra_items__” is considered to be non-required as the corresponding key.]
The following examples illustrate these checks in action.
__extra_items__
puts various restrictions on additional items for type
consistency checks:
classMovie(TypedDict,closed=True):
name:str
__extra_items__:int|None
classMovieDetails(TypedDict,closed=True):
name:str
year:NotRequired[int]
__extra_items__:int|None
details:MovieDetails={"name":"Kill Bill Vol. 1","year":2003}
movie:Movie=details# Not OK. While 'int' is consistent with 'int | None',
# 'int | None' is not consistent with 'int'
classMovieWithYear(TypedDict,closed=True):
name:str
year:int|None
__extra_items__:int|None
details:MovieWithYear={"name":"Kill Bill Vol. 1","year":2003}
movie:Movie=details# Not OK. 'year' is not required in 'Movie',
# so it shouldn't be required in 'MovieWithYear' either
Because “year” is absent inMovie
,__extra_items__
is considered the
corresponding key."year"
being required violates the rule “For each
required key inB
,the corresponding key is required inA
”.
When__extra_items__
is defined to be read-only in a TypedDict type, it is possible
for an item to have a narrower type than__extra_items__
’s value type:
classMovie(TypedDict,closed=True):
name:str
__extra_items__:ReadOnly[str|int]
classMovieDetails(TypedDict,closed=True):
name:str
year:NotRequired[int]
__extra_items__:int
details:MovieDetails={"name":"Kill Bill Vol. 2","year":2004}
movie:Movie=details# OK. 'int' is consistent with 'str | int'.
This behaves the same way asPEP 705specified ifyear:ReadOnly[str|int]
is an item defined inMovie
.
__extra_items__
as a pseudo-item follows the same rules that other items have, so
when both TypedDicts contain__extra_items__
,this check is naturally enforced:
classMovieExtraInt(TypedDict,closed=True):
name:str
__extra_items__:int
classMovieExtraStr(TypedDict,closed=True):
name:str
__extra_items__:str
extra_int:MovieExtraInt={"name":"No Country for Old Men","year":2007}
extra_str:MovieExtraStr={"name":"No Country for Old Men","description":""}
extra_int=extra_str# Not OK. 'str' is inconsistent with 'int' for item '__extra_items__'
extra_str=extra_int# Not OK. 'int' is inconsistent with 'str' for item '__extra_items__'
A non-closed TypedDict type implicitly allows non-required extra keys of value
typeReadOnly[object]
.This allows to apply the type consistency rules
between this type and a closed TypedDict type:
classMovieNotClosed(TypedDict):
name:str
extra_int:MovieExtraInt={"name":"No Country for Old Men","year":2007}
not_closed:MovieNotClosed={"name":"No Country for Old Men"}
extra_int=not_closed# Not OK. 'ReadOnly[object]' implicitly on 'MovieNotClosed' is not consistent with 'int' for item '__extra_items__'
not_closed=extra_int# OK
Interaction with Constructors
TypedDicts that allow extra items of typeT
also allow arbitrary keyword
arguments of this type when constructed by calling the class object:
classOpenMovie(TypedDict):
name:str
OpenMovie(name="No Country for Old Men")# OK
OpenMovie(name="No Country for Old Men",year=2007)# Not OK. Unrecognized key
classExtraMovie(TypedDict,closed=True):
name:str
__extra_items__:int
ExtraMovie(name="No Country for Old Men")# OK
ExtraMovie(name="No Country for Old Men",year=2007)# OK
ExtraMovie(
name="No Country for Old Men",
language="English",
)# Not OK. Wrong type for extra key
# This implies '__extra_items__: Never',
# so extra keyword arguments produce an error
classClosedMovie(TypedDict,closed=True):
name:str
ClosedMovie(name="No Country for Old Men")# OK
ClosedMovie(
name="No Country for Old Men",
year=2007,
)# Not OK. Extra items not allowed
Interaction with Mapping[KT, VT]
A TypedDict type can be consistent withMapping[KT,VT]
types other than
Mapping[str,object]
as long as the union of value types on the TypedDict
type is consistent withVT
.It is an extension of this rule from the typing
spec:
- A TypedDict with all
int
values is not consistent withMapping[str,int]
,since there may be additional non-int
values not visible through the type, due to structural subtyping. These can be accessed using thevalues()
anditems()
methods inMapping
For example:
classMovieExtraStr(TypedDict,closed=True):
name:str
__extra_items__:str
extra_str:MovieExtraStr={"name":"Blade Runner","summary":""}
str_mapping:Mapping[str,str]=extra_str# OK
int_mapping:Mapping[str,int]=extra_int# Not OK. 'int | str' is not consistent with 'int'
int_str_mapping:Mapping[str,int|str]=extra_int# OK
Furthermore, type checkers should be able to infer the precise return types of
values()
anditems()
on such TypedDict types:
deffun(movie:MovieExtraStr)->None:
reveal_type(movie.items())# Revealed type is 'dict_items[str, str]'
reveal_type(movie.values())# Revealed type is 'dict_values[str, str]'
Interaction with dict[KT, VT]
Note that because the presence of__extra_items__
on a closed TypedDict type
prohibits additional required keys in its structural subtypes, we can determine
if the TypedDict type and its structural subtypes will ever have any required
key during static analysis.
The TypedDict type is consistent withdict[str,VT]
if all items on the
TypedDict type satisfy the following conditions:
VT
is consistent with the value type of the item- The value type of the item is consistent with
VT
- The item is not read-only.
- The item is not required.
For example:
classIntDict(TypedDict,closed=True):
__extra_items__:int
classIntDictWithNum(IntDict):
num:NotRequired[int]
deff(x:IntDict)->None:
v:dict[str,int]=x# OK
v.clear()# OK
not_required_num:IntDictWithNum={"num":1,"bar":2}
regular_dict:dict[str,int]=not_required_num# OK
f(not_required_num)# OK
In this case, methods that are previously unavailable on a TypedDict are allowed:
not_required_num.clear()# OK
reveal_type(not_required_num.popitem())# OK. Revealed type is tuple[str, int]
However,dict[str,VT]
is not necessarily consistent with a TypedDict type,
because such dict can be a subtype of dict:
classCustomDict(dict[str,int]):
...
not_a_regular_dict:CustomDict={"num":1}
int_dict:IntDict=not_a_regular_dict# Not OK
How to Teach This
The choice of the spelling"__extra_items__"
is intended to make this
feature more understandable to new users compared to shorter alternatives like
"__extra__"
.
Details of this should be documented in both the typing spec and the
typing
documentation.
Backwards Compatibility
Because__extra_items__
remains as a regular key ifclosed=True
is not
specified, no existing codebase will break due to this change.
If the proposal is accepted, none of__required_keys__
,
__optional_keys__
,__readonly_keys__
and__mutable_keys__
should
include"__extra_items__"
defined on the same TypedDict type when
closed=True
is specified.
Note thatclosed
as a keyword argument does not collide with the keyword
arguments alternative to define keys with the functional syntax that allows
things likeTD=TypedDict( "TD",foo=str,bar=int)
,because it is scheduled
to be removed in Python 3.13.
Because this is a type-checking feature, it can be made available to older versions as long as the type checker supports it.
Rejected Ideas
Allowing Extra Items without Specifying the Type
extra=True
was originally proposed for defining a TypedDict that accepts extra
items regardless of the type, like howtotal=True
works:
classTypedDict(extra=True):
pass
Because it did not offer a way to specify the type of the extra items, the type
checkers will need to assume that the type of the extra items isAny
,which
compromises type safety. Furthermore, the current behavior of TypedDict already
allows untyped extra items to be present in runtime, due to structural
subtyping.closed=True
plays a similar role in the current proposal.
SupportingTypedDict(extra=type)
During the discussion of the PEP, there were strong objections against adding another place where types are passed as values instead of annotations from some authors of type checkers. While this design is potentially viable, there are also several partially addressable concerns to consider.
- Usability of forward reference
As in the functional syntax, using a quoted type or a type alias will be
required when SomeType is a forward reference. This is already a requirement
for the functional syntax, so implementations can potentially reuse that piece
of logic, but this is still extra work that the
closed=True
proposal doesn’t have. - Concerns about using type as a value Whatever is not allowed as the value type in the functional syntax should not be allowed as the argument for extra either. While type checkers might be able to reuse this check, it still needs to be somehow special-cased for the class-based syntax.
- How to teach
Notably, the
extra=type
often gets brought up due to it being an intuitive solution for the use case, so it is potentially simpler to learn than the less obvious solution. However, the more common used case only requiresclosed=True
,and the other drawbacks mentioned earlier outweigh what is need to teach the usage of the special key.
Support Extra Items with Intersection
Supporting intersections in Python’s type system requires a lot of careful consideration, and it can take a long time for the community to reach a consensus on a reasonable design.
Ideally, extra items in TypedDict should not be blocked by work on intersections, nor does it necessarily need to be supported through intersections.
Moreover, the intersection betweenMapping[...]
andTypedDict
is not
equivalent to a TypedDict type with the proposed__extra_items__
special
item, as the value type of all known items inTypedDict
needs to satisfy the
is-subtype-of relation with the value type ofMapping[...]
.
Requiring Type Compatibility of the Known Items with__extra_items__
__extra_items__
restricts the value type for keys that areunknownto the
TypedDict type. So the value type of anyknownitem is not necessarily
consistent with__extra_items__
’s type, and__extra_items__
’s type is
not necessarily consistent with the value types of all known items.
This differs from TypeScript’sIndex Signatures syntax, which requires all properties’ types to match the string index’s type. For example:
interfaceMovieWithExtraNumber{
name:string// Property 'name' of type 'string' is not assignable to 'string' index type 'number'.
[index:string]:number
}
interfaceMovieWithExtraNumberOrString{
name:string// OK
[index:string]:number|string
}
This is a known limitation discussed inTypeScript’s issue tracker,
where it is suggested that there should be a way to exclude the defined keys
from the index signature so that it is possible to define a type like
MovieWithExtraNumber
.
Reference Implementation
This proposal is supported inpyright 1.1.352,andpyanalyze 0.12.0.
Acknowledgments
Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedback, Eric Traut whoproposed the original design this PEP iterates on, and Alice Purcell for offering their perspective as the author ofPEP 705.
Copyright
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source:https://github / Python /peps/blob/main/peps/pep-0728.rst
Last modified:2024-03-16 13:29:41 GMT