Initialization, Finalization, and Threads¶

SeePython Initialization Configurationfor details on how to configure the interpreter prior to initialization.

Before Python Initialization¶

In an application embedding Python, thePy_Initialize()function must be called before using any other Python/C API functions; with the exception of a few functions and theglobal configuration variables.

The following functions can be safely called before Python is initialized:

Functions that initialize the interpreter:
- Py_Initialize()
- Py_InitializeEx()
- Py_InitializeFromConfig()
- Py_BytesMain()
- Py_Main()
- the runtime pre-initialization functions covered inPython Initialization Configuration
Configuration functions:
- PyImport_AppendInittab()
- PyImport_ExtendInittab()
- PyInitFrozenExtensions()
- PyMem_SetAllocator()
- PyMem_SetupDebugHooks()
- PyObject_SetArenaAllocator()
- Py_SetProgramName()
- Py_SetPythonHome()
- PySys_ResetWarnOptions()
- the configuration functions covered inPython Initialization Configuration
Informative functions:
Utilities:
- Py_DecodeLocale()
- the status reporting and utility functions covered inPython Initialization Configuration
Memory allocators:
Synchronization:
- PyMutex_Lock()
- PyMutex_Unlock()

Note

Despite their apparent similarity to some of the functions listed above, the following functionsshould not be calledbefore the interpreter has been initialized:Py_EncodeLocale(),Py_GetPath(), Py_GetPrefix(),Py_GetExecPrefix(), Py_GetProgramFullPath(),Py_GetPythonHome(), Py_GetProgramName(),PyEval_InitThreads(),and Py_RunMain().

Global configuration variables¶

Python has variables for the global configuration to control different features and options. By default, these flags are controlled bycommand line options.

When a flag is set by an option, the value of the flag is the number of times that the option was set. For example,-bsetsPy_BytesWarningFlag to 1 and-bbsetsPy_BytesWarningFlagto 2.

intPy_BytesWarningFlag¶

This API is kept for backward compatibility: setting PyConfig.bytes_warningshould be used instead, seePython Initialization Configuration.

Issue a warning when comparingbytesorbytearraywith strorbyteswithint.Issue an error if greater or equal to2.

Set by the-boption.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_DebugFlag¶

This API is kept for backward compatibility: setting PyConfig.parser_debugshould be used instead, seePython Initialization Configuration.

Turn on parser debugging output (for expert only, depending on compilation options).

Set by the-doption and thePYTHONDEBUGenvironment variable.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_DontWriteBytecodeFlag¶

This API is kept for backward compatibility: setting PyConfig.write_bytecodeshould be used instead, seePython Initialization Configuration.

If set to non-zero, Python won’t try to write.pycfiles on the import of source modules.

Set by the-Boption and thePYTHONDONTWRITEBYTECODE environment variable.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_FrozenFlag¶

This API is kept for backward compatibility: setting PyConfig.pathconfig_warningsshould be used instead, see Python Initialization Configuration.

Suppress error messages when calculating the module search path in Py_GetPath().

Private flag used by_freeze_moduleandfrozenmainprograms.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_HashRandomizationFlag¶

This API is kept for backward compatibility: setting PyConfig.hash_seedandPyConfig.use_hash_seedshould be used instead, seePython Initialization Configuration.

Set to1if thePYTHONHASHSEEDenvironment variable is set to a non-empty string.

If the flag is non-zero, read thePYTHONHASHSEEDenvironment variable to initialize the secret hash seed.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_IgnoreEnvironmentFlag¶

This API is kept for backward compatibility: setting PyConfig.use_environmentshould be used instead, see Python Initialization Configuration.

Ignore allPYTHON*environment variables, e.g. PYTHONPATHandPYTHONHOME,that might be set.

Set by the-Eand-Ioptions.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_InspectFlag¶

This API is kept for backward compatibility: setting PyConfig.inspectshould be used instead, see Python Initialization Configuration.

When a script is passed as first argument or the-coption is used, enter interactive mode after executing the script or the command, even when sys.stdindoes not appear to be a terminal.

Set by the-ioption and thePYTHONINSPECTenvironment variable.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_InteractiveFlag¶

This API is kept for backward compatibility: setting PyConfig.interactiveshould be used instead, see Python Initialization Configuration.

Set by the-ioption.

Deprecated since version 3.12.

intPy_IsolatedFlag¶

This API is kept for backward compatibility: setting PyConfig.isolatedshould be used instead, see Python Initialization Configuration.

Run Python in isolated mode. In isolated modesys.pathcontains neither the script’s directory nor the user’s site-packages directory.

Set by the-Ioption.

Added in version 3.4.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_LegacyWindowsFSEncodingFlag¶

This API is kept for backward compatibility: setting PyPreConfig.legacy_windows_fs_encodingshould be used instead, see Python Initialization Configuration.

If the flag is non-zero, use thembcsencoding withreplaceerror handler, instead of the UTF-8 encoding withsurrogatepasserror handler, for thefilesystem encoding and error handler.

Set to1if thePYTHONLEGACYWINDOWSFSENCODINGenvironment variable is set to a non-empty string.

SeePEP 529for more details.

Availability:Windows.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_LegacyWindowsStdioFlag¶

This API is kept for backward compatibility: setting PyConfig.legacy_windows_stdioshould be used instead, see Python Initialization Configuration.

If the flag is non-zero, useio.FileIOinstead of io._WindowsConsoleIOforsysstandard streams.

Set to1if thePYTHONLEGACYWINDOWSSTDIOenvironment variable is set to a non-empty string.

SeePEP 528for more details.

Availability:Windows.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_NoSiteFlag¶

This API is kept for backward compatibility: setting PyConfig.site_importshould be used instead, see Python Initialization Configuration.

Disable the import of the modulesiteand the site-dependent manipulations ofsys.paththat it entails. Also disable these manipulations ifsiteis explicitly imported later (call site.main()if you want them to be triggered).

Set by the-Soption.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_NoUserSiteDirectory¶

This API is kept for backward compatibility: setting PyConfig.user_site_directoryshould be used instead, see Python Initialization Configuration.

Don’t add theusersite-packagesdirectoryto sys.path.

Set by the-sand-Ioptions, and the PYTHONNOUSERSITEenvironment variable.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_OptimizeFlag¶

This API is kept for backward compatibility: setting PyConfig.optimization_levelshould be used instead, see Python Initialization Configuration.

Set by the-Ooption and thePYTHONOPTIMIZEenvironment variable.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_QuietFlag¶

This API is kept for backward compatibility: setting PyConfig.quietshould be used instead, seePython Initialization Configuration.

Don’t display the copyright and version messages even in interactive mode.

Set by the-qoption.

Added in version 3.2.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_UnbufferedStdioFlag¶

This API is kept for backward compatibility: setting PyConfig.buffered_stdioshould be used instead, seePython Initialization Configuration.

Force the stdout and stderr streams to be unbuffered.

Set by the-uoption and thePYTHONUNBUFFERED environment variable.

Deprecated since version 3.12, will be removed in version 3.14.

intPy_VerboseFlag¶

This API is kept for backward compatibility: setting PyConfig.verboseshould be used instead, seePython Initialization Configuration.

Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded. If greater or equal to2,print a message for each file that is checked for when searching for a module. Also provides information on module cleanup at exit.

Set by the-voption and thePYTHONVERBOSEenvironment variable.

Deprecated since version 3.12, will be removed in version 3.14.

Initializing and finalizing the interpreter¶

voidPy_Initialize()¶

Part of theStable ABI.

Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; see Before Python Initializationfor the few exceptions.

This initializes the table of loaded modules (sys.modules), and creates the fundamental modulesbuiltins,__main__andsys. It also initializes the module search path (sys.path). It does not set sys.argv;use thePython Initialization Configuration API for that. This is a no-op when called for a second time (without calling Py_FinalizeEx()first). There is no return value; it is a fatal error if the initialization fails.

UsePy_InitializeFromConfig()to customize the Python Initialization Configuration.

Note

On Windows, changes the console mode fromO_TEXTtoO_BINARY, which will also affect non-Python uses of the console using the C Runtime.

voidPy_InitializeEx(intinitsigs)¶

Part of theStable ABI.

This function works likePy_Initialize()ifinitsigsis1.If initsigsis0,it skips initialization registration of signal handlers, which may be useful when CPython is embedded as part of a larger application.

UsePy_InitializeFromConfig()to customize the Python Initialization Configuration.

PyStatusPy_InitializeFromConfig(constPyConfig*config)¶

Initialize Python fromconfigconfiguration, as described in Initialization with PyConfig.

See thePython Initialization Configurationsection for details on pre-initializing the interpreter, populating the runtime configuration structure, and querying the returned status structure.

intPy_IsInitialized()¶: Part of theStable ABI.
Return true (nonzero) when the Python interpreter has been initialized, false (zero) if not. AfterPy_FinalizeEx()is called, this returns false until Py_Initialize()is called again.

intPy_IsFinalizing()¶: Part of theStable ABIsince version 3.13.
Return true (non-zero) if the main Python interpreter is shutting down.Return false (zero) otherwise.

Added in version 3.13.

intPy_FinalizeEx()¶

Part of theStable ABIsince version 3.6.

Undo all initializations made byPy_Initialize()and subsequent use of Python/C API functions, and destroy all sub-interpreters (see Py_NewInterpreter()below) that were created and not yet destroyed since the last call toPy_Initialize().Ideally, this frees all memory allocated by the Python interpreter. This is a no-op when called for a second time (without callingPy_Initialize()again first).

Since this is the reverse ofPy_Initialize(),it should be called in the same thread with the same interpreter active. That means the main thread and the main interpreter. This should never be called whilePy_RunMain()is running.

Normally the return value is0. If there were errors during finalization (flushing buffered data), -1is returned.

This function is provided for a number of reasons. An embedding application might want to restart Python without having to restart the application itself. An application that has loaded the Python interpreter from a dynamically loadable library (or DLL) might want to free all memory allocated by Python before unloading the DLL. During a hunt for memory leaks in an application a developer might want to free all memory allocated by Python before exiting from the application.

Bugs and caveats:The destruction of modules and objects in modules is done in random order; this may cause destructors (__del__()methods) to fail when they depend on other objects (even functions) or modules. Dynamically loaded extension modules loaded by Python are not unloaded. Small amounts of memory allocated by the Python interpreter may not be freed (if you find a leak, please report it). Memory tied up in circular references between objects is not freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application callsPy_Initialize()and Py_FinalizeEx()more than once.

Raises anauditing eventc Python._PySys_ClearAuditHookswith no arguments.

Added in version 3.6.

voidPy_Finalize()¶: Part of theStable ABI.
This is a backwards-compatible version ofPy_FinalizeEx()that disregards the return value.

intPy_BytesMain(intargc,char**argv)¶: Part of theStable ABIsince version 3.8.
Similar toPy_Main()butargvis an array of bytes strings, allowing the calling application to delegate the text decoding step to the CPython runtime.

Added in version 3.8.

intPy_Main(intargc,wchar_t**argv)¶

Part of theStable ABI.

The main program for the standard interpreter, encapsulating a full initialization/finalization cycle, as well as additional behaviour to implement reading configurations settings from the environment and command line, and then executing__main__in accordance with Command line.

This is made available for programs which wish to support the full CPython command line interface, rather than just embedding a Python runtime in a larger application.

Theargcandargvparameters are similar to those which are passed to a C program’smain()function, except that theargventries are first converted towchar_tusingPy_DecodeLocale().It is also important to note that the argument list entries may be modified to point to strings other than those passed in (however, the contents of the strings pointed to by the argument list are not modified).

The return value will be0if the interpreter exits normally (i.e., without an exception),1if the interpreter exits due to an exception, or2if the argument list does not represent a valid Python command line.

Note that if an otherwise unhandledSystemExitis raised, this function will not return1,but exit the process, as long as Py_InspectFlagis not set. IfPy_InspectFlagis set, execution will drop into the interactive Python prompt, at which point a second otherwise unhandledSystemExitwill still exit the process, while any other means of exiting will set the return value as described above.

In terms of the CPython runtime configuration APIs documented in the runtime configurationsection (and without accounting for error handling),Py_Mainis approximately equivalent to:

PyConfigconfig;
PyConfig_InitPythonConfig(&config);
PyConfig_SetArgv(&config,argc,argv);
Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);

Py_RunMain();

In normal usage, an embedding application will call this function insteadof callingPy_Initialize(),Py_InitializeEx()or Py_InitializeFromConfig()directly, and all settings will be applied as described elsewhere in this documentation. If this function is instead calledaftera preceding runtime initialization API call, then exactly which environmental and command line configuration settings will be updated is version dependent (as it depends on which settings correctly support being modified after they have already been set once when the runtime was first initialized).

intPy_RunMain(void)¶

Executes the main module in a fully configured CPython runtime.

Executes the command (PyConfig.run_command), the script (PyConfig.run_filename) or the module (PyConfig.run_module) specified on the command line or in the configuration. If none of these values are set, runs the interactive Python prompt (REPL) using the__main__module’s global namespace.

IfPyConfig.inspectis not set (the default), the return value will be0if the interpreter exits normally (that is, without raising an exception), or1if the interpreter exits due to an exception. If an otherwise unhandledSystemExitis raised, the function will immediately exit the process instead of returning1.

IfPyConfig.inspectis set (such as when the-ioption is used), rather than returning when the interpreter exits, execution will instead resume in an interactive Python prompt (REPL) using the__main__ module’s global namespace. If the interpreter exited with an exception, it is immediately raised in the REPL session. The function return value is then determined by the way theREPL sessionterminates: returning0 if the session terminates without raising an unhandled exception, exiting immediately for an unhandledSystemExit,and returning1for any other unhandled exception.

This function always finalizes the Python interpreter regardless of whether it returns a value or immediately exits the process due to an unhandled SystemExitexception.

SeePython Configurationfor an example of a customized Python that always runs in isolated mode using Py_RunMain().

Process-wide parameters¶

voidPy_SetProgramName(constwchar_t*name)¶

Part of theStable ABI.

This API is kept for backward compatibility: setting PyConfig.program_nameshould be used instead, seePython Initialization Configuration.

This function should be called beforePy_Initialize()is called for the first time, if it is called at all. It tells the interpreter the value of theargv[0]argument to themain()function of the program (converted to wide characters). This is used byPy_GetPath()and some other functions below to find the Python run-time libraries relative to the interpreter executable. The default value is' Python '.The argument should point to a zero-terminated wide character string in static storage whose contents will not change for the duration of the program’s execution. No code in the Python interpreter will change the contents of this storage.

UsePy_DecodeLocale()to decode a bytes string to get a wchar_*string.

Deprecated since version 3.11.

wchar_t*Py_GetProgramName()¶

Part of theStable ABI.

Return the program name set withPyConfig.program_name,or the default. The returned string points into static storage; the caller should not modify its value.

This function should not be called beforePy_Initialize(),otherwise it returnsNULL.

Changed in version 3.10:It now returnsNULLif called beforePy_Initialize().

Deprecated since version 3.13, will be removed in version 3.15:Getsys.executableinstead.

wchar_t*Py_GetPrefix()¶

Part of theStable ABI.

Return theprefixfor installed platform-independent files. This is derived through a number of complicated rules from the program name set with PyConfig.program_nameand some environment variables; for example, if the program name is'/usr/local/bin/ Python ',the prefix is'/usr/local'.The returned string points into static storage; the caller should not modify its value. This corresponds to theprefixvariable in the top-level Makefileand the--prefixargument to theconfigure script at build time. The value is available to Python code assys.base_prefix. It is only useful on Unix. See also the next function.

This function should not be called beforePy_Initialize(),otherwise it returnsNULL.

Changed in version 3.10:It now returnsNULLif called beforePy_Initialize().

Deprecated since version 3.13, will be removed in version 3.15:Getsys.base_prefixinstead, orsys.prefixif virtual environmentsneed to be handled.

wchar_t*Py_GetExecPrefix()¶

Part of theStable ABI.

Return theexec-prefixfor installed platform-dependentfiles. This is derived through a number of complicated rules from the program name set with PyConfig.program_nameand some environment variables; for example, if the program name is'/usr/local/bin/ Python ',the exec-prefix is '/usr/local'.The returned string points into static storage; the caller should not modify its value. This corresponds to theexec_prefix variable in the top-levelMakefileand the--exec-prefix argument to theconfigurescript at build time. The value is available to Python code assys.base_exec_prefix.It is only useful on Unix.

Background: The exec-prefix differs from the prefix when platform dependent files (such as executables and shared libraries) are installed in a different directory tree. In a typical installation, platform dependent files may be installed in the/usr/local/platsubtree while platform independent may be installed in/usr/local.

Generally speaking, a platform is a combination of hardware and software families, e.g. Sparc machines running the Solaris 2.x operating system are considered the same platform, but Intel machines running Solaris 2.x are another platform, and Intel machines running Linux are yet another platform. Different major revisions of the same operating system generally also form different platforms. Non-Unix operating systems are a different story; the installation strategies on those systems are so different that the prefix and exec-prefix are meaningless, and set to the empty string. Note that compiled Python bytecode files are platform independent (but not independent from the Python version by which they were compiled!).

System administrators will know how to configure themountor automountprograms to share/usr/localbetween platforms while having/usr/local/platbe a different filesystem for each platform.

This function should not be called beforePy_Initialize(),otherwise it returnsNULL.

Changed in version 3.10:It now returnsNULLif called beforePy_Initialize().

Deprecated since version 3.13, will be removed in version 3.15:Getsys.base_exec_prefixinstead, orsys.exec_prefixif virtual environmentsneed to be handled.

wchar_t*Py_GetProgramFullPath()¶

Part of theStable ABI.

Return the full program name of the Python executable; this is computed as a side-effect of deriving the default module search path from the program name (set byPyConfig.program_name). The returned string points into static storage; the caller should not modify its value. The value is available to Python code assys.executable.

This function should not be called beforePy_Initialize(),otherwise it returnsNULL.

Changed in version 3.10:It now returnsNULLif called beforePy_Initialize().

Deprecated since version 3.13, will be removed in version 3.15:Getsys.executableinstead.

wchar_t*Py_GetPath()¶

Part of theStable ABI.

Return the default module search path; this is computed from the program name (set byPyConfig.program_name) and some environment variables. The returned string consists of a series of directory names separated by a platform dependent delimiter character. The delimiter character is':' on Unix and macOS,';'on Windows. The returned string points into static storage; the caller should not modify its value. The list sys.pathis initialized with this value on interpreter startup; it can be (and usually is) modified later to change the search path for loading modules.

This function should not be called beforePy_Initialize(),otherwise it returnsNULL.

Changed in version 3.10:It now returnsNULLif called beforePy_Initialize().

Deprecated since version 3.13, will be removed in version 3.15:Getsys.pathinstead.

constchar*Py_GetVersion()¶

Part of theStable ABI.

Return the version of this Python interpreter. This is a string that looks something like

"3.0a5+ (py3k:63103M, May 12 2008, 00:53:55)\n[GCC 4.2.3] "

The first word (up to the first space character) is the current Python version; the first characters are the major and minor version separated by a period. The returned string points into static storage; the caller should not modify its value. The value is available to Python code assys.version.

Thread State and the Global Interpreter Lock¶

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called theglobal interpreter lockorGIL,that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program: for example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice.

Therefore, the rule exists that only the thread that has acquired the GILmay operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly tries to switch threads (seesys.setswitchinterval()). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

The Python interpreter keeps some thread-specific bookkeeping information inside a data structure calledPyThreadState.There’s also one global variable pointing to the currentPyThreadState:it can be retrieved usingPyThreadState_Get().

Releasing the GIL from extension code¶

Most extension code manipulating theGILhas the following simple structure:

Savethethreadstateinalocalvariable.
Releasetheglobalinterpreterlock.
...DosomeblockingI/Ooperation...
Reacquiretheglobalinterpreterlock.
Restorethethreadstatefromthelocalvariable.

This is so common that a pair of macros exists to simplify it:

Py_BEGIN_ALLOW_THREADS
...DosomeblockingI/Ooperation...
Py_END_ALLOW_THREADS

ThePy_BEGIN_ALLOW_THREADSmacro opens a new block and declares a hidden local variable; thePy_END_ALLOW_THREADSmacro closes the block.

The block above expands to the following code:

PyThreadState*_save;

_save=PyEval_SaveThread();
...DosomeblockingI/Ooperation...
PyEval_RestoreThread(_save);

Here is how these functions work: the global interpreter lock is used to protect the pointer to the current thread state. When releasing the lock and saving the thread state, the current thread state pointer must be retrieved before the lock is released (since another thread could immediately acquire the lock and store its own thread state in the global variable). Conversely, when acquiring the lock and restoring the thread state, the lock must be acquired before storing the thread state pointer.

Note

Calling system I/O functions is the most common use case for releasing the GIL, but it can also be useful before calling long-running computations which don’t need access to Python objects, such as compression or cryptographic functions operating over memory buffers. For example, the standardzlibandhashlibmodules release the GIL when compressing or hashing data.

Non-Python created threads¶

When threads are created using the dedicated Python APIs (such as the threadingmodule), a thread state is automatically associated to them and the code showed above is therefore correct. However, when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.

If you need to call Python code from these threads (often this will be part of a callback API provided by the aforementioned third-party library), you must first register these threads with the interpreter by creating a thread state data structure, then acquiring the GIL, and finally storing their thread state pointer, before you can start using the Python/C API. When you are done, you should reset the thread state pointer, release the GIL, and finally free the thread state data structure.

ThePyGILState_Ensure()andPyGILState_Release()functions do all of the above automatically. The typical idiom for calling into Python from a C thread is:

PyGILState_STATEgstate;
gstate=PyGILState_Ensure();

/* Perform Python actions here. */
result=CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

Note that thePyGILState_*functions assume there is only one global interpreter (created automatically byPy_Initialize()). Python supports the creation of additional interpreters (using Py_NewInterpreter()), but mi xing multiple interpreters and the PyGILState_*API is unsupported.

Cautions about fork()¶

Another important thing to note about threads is their behaviour in the face of the Cfork()call. On most systems withfork(),after a process forks only the thread that issued the fork will exist. This has a concrete impact both on how locks must be handled and on all stored state in CPython’s runtime.

The fact that only the “current” thread remains means any locks held by other threads will never be released. Python solves this foros.fork()by acquiring the locks it uses internally before the fork, and releasing them afterwards. In addition, it resets any Lock Objectsin the child. When extending or embedding Python, there is no way to inform Python of additional (non-Python) locks that need to be acquired before or reset after a fork. OS facilities such as pthread_atfork()would need to be used to accomplish the same thing. Additionally, when extending or embedding Python, callingfork() directly rather than throughos.fork()(and returning to or calling into Python) may result in a deadlock by one of Python’s internal locks being held by a thread that is defunct after the fork. PyOS_AfterFork_Child()tries to reset the necessary locks, but is not always able to.

The fact that all other threads go away also means that CPython’s runtime state there must be cleaned up properly, whichos.fork() does. This means finalizing all otherPyThreadStateobjects belonging to the current interpreter and all other PyInterpreterStateobjects. Due to this and the special nature of the“main” interpreter, fork()should only be called in that interpreter’s “main” thread, where the CPython global runtime was originally initialized. The only exception is ifexec()will be called immediately after.

High-level API¶

These are the most commonly used types and functions when writing C extension code, or when embedding the Python interpreter:

typePyInterpreterState¶

Part of theLimited API(as an opaque struct).

This data structure represents the state shared by a number of cooperating threads. Threads belonging to the same interpreter share their module administration and a few other internal items. There are no public members in this structure.

Threads belonging to different interpreters initially share nothing, except process state like available memory, open file descriptors and such. The global interpreter lock is also shared by all threads, regardless of to which interpreter they belong.

typePyThreadState¶

Part of theLimited API(as an opaque struct).

This data structure represents the state of a single thread. The only public data member is:

PyInterpreterState*interp¶: This thread’s interpreter state.

voidPyEval_InitThreads()¶

Part of theStable ABI.

Deprecated function which does nothing.

In Python 3.6 and older, this function created the GIL if it didn’t exist.

Changed in version 3.9:The function now does nothing.

Changed in version 3.7:This function is now called byPy_Initialize(),so you don’t have to call it yourself anymore.

Changed in version 3.2:This function cannot be called beforePy_Initialize()anymore.

Deprecated since version 3.9.

PyThreadState*PyEval_SaveThread()¶: Part of theStable ABI.
Release the global interpreter lock (if it has been created) and reset the thread state toNULL,returning the previous thread state (which is not NULL). If the lock has been created, the current thread must have acquired it.

voidPyEval_RestoreThread(PyThreadState*tstate)¶: Part of theStable ABI.
Acquire the global interpreter lock (if it has been created) and set the thread state totstate,which must not beNULL.If the lock has been created, the current thread must not have acquired it, otherwise deadlock ensues.

Note

Calling this function from a thread when the runtime is finalizing will terminate the thread, even if the thread was not created by Python. You can usePy_IsFinalizing()orsys.is_finalizing()to check if the interpreter is in process of being finalized before calling this function to avoid unwanted termination.

PyThreadState*PyThreadState_Get()¶

Part of theStable ABI.

Return the current thread state. The global interpreter lock must be held. When the current thread state isNULL,this issues a fatal error (so that the caller needn’t check forNULL).

Low-level API¶

All of the following functions must be called afterPy_Initialize().

Changed in version 3.7:Py_Initialize()now initializes theGIL.

PyInterpreterState*PyInterpreterState_New()¶

Part of theStable ABI.

Create a new interpreter state object. The global interpreter lock need not be held, but may be held if it is necessary to serialize calls to this function.

Raises anauditing eventc Python.PyInterpreterState_Newwith no arguments.

voidPyInterpreterState_Clear(PyInterpreterState*interp)¶

Part of theStable ABI.

Reset all information in an interpreter state object. The global interpreter lock must be held.

Raises anauditing eventc Python.PyInterpreterState_Clearwith no arguments.

voidPyInterpreterState_Delete(PyInterpreterState*interp)¶: Part of theStable ABI.
Destroy an interpreter state object. The global interpreter lock need not be held. The interpreter state must have been reset with a previous call to PyInterpreterState_Clear().

PyThreadState*PyThreadState_New(PyInterpreterState*interp)¶: Part of theStable ABI.
Create a new thread state object belonging to the given interpreter object. The global interpreter lock need not be held, but may be held if it is necessary to serialize calls to this function.

voidPyThreadState_Clear(PyThreadState*tstate)¶: Part of theStable ABI.
Reset all information in a thread state object. The global interpreter lock must be held.

Changed in version 3.9:This function now calls thePyThreadState.on_deletecallback. Previously, that happened inPyThreadState_Delete().

voidPyThreadState_Delete(PyThreadState*tstate)¶: Part of theStable ABI.
Destroy a thread state object. The global interpreter lock need not be held. The thread state must have been reset with a previous call to PyThreadState_Clear().

voidPyThreadState_DeleteCurrent(void)¶: Destroy the current thread state and release the global interpreter lock. LikePyThreadState_Delete(),the global interpreter lock must be held. The thread state must have been reset with a previous call toPyThreadState_Clear().

PyFrameObject*PyThreadState_GetFrame(PyThreadState*tstate)¶

Part of theStable ABIsince version 3.10.

Get the current frame of the Python thread statetstate.

Return astrong reference.ReturnNULLif no frame is currently executing.

Sub-interpreter support¶

While in most uses, you will only embed a single Python interpreter, there are cases where you need to create several independent interpreters in the same process and perhaps even in the same thread. Sub-interpreters allow you to do that.

The “main” interpreter is the first one created when the runtime initializes. It is usually the only Python interpreter in a process. Unlike sub-interpreters, the main interpreter has unique process-global responsibilities like signal handling. It is also responsible for execution during runtime initialization and is usually the active interpreter during runtime finalization. The PyInterpreterState_Main()function returns a pointer to its state.

You can switch between sub-interpreters using thePyThreadState_Swap() function. You can create and destroy them using the following functions:

typePyInterpreterConfig¶

Structure containing most parameters to configure a sub-interpreter. Its values are used only inPy_NewInterpreterFromConfig()and never modified by the runtime.

Added in version 3.12.

Structure fields:

intuse_main_obmalloc¶

If this is0then the sub-interpreter will use its own “object” allocator state. Otherwise it will use (share) the main interpreter’s.

If this is0then check_multi_interp_extensions must be1(non-zero). If this is1thengil must not bePyInterpreterConfig_OWN_GIL.

intallow_fork¶

If this is0then the runtime will not support forking the process in any thread where the sub-interpreter is currently active. Otherwise fork is unrestricted.

Note that thesubprocessmodule still works when fork is disallowed.

intallow_exec¶

If this is0then the runtime will not support replacing the current process via exec (e.g.os.execv()) in any thread where the sub-interpreter is currently active. Otherwise exec is unrestricted.

Note that thesubprocessmodule still works when exec is disallowed.

intallow_threads¶: If this is0then the sub-interpreter’sthreadingmodule won’t create threads. Otherwise threads are allowed.

intallow_daemon_threads¶: If this is0then the sub-interpreter’sthreadingmodule won’t create daemon threads. Otherwise daemon threads are allowed (as long as allow_threadsis non-zero).

intcheck_multi_interp_extensions¶

If this is0then all extension modules may be imported, including legacy (single-phase init) modules, in any thread where the sub-interpreter is currently active. Otherwise only multi-phase init extension modules (seePEP 489) may be imported. (Also seePy_mod_multiple_interpreters.)

This must be1(non-zero) if use_main_obmallocis0.

intgil¶

This determines the operation of the GIL for the sub-interpreter. It may be one of the following:

PyInterpreterConfig_DEFAULT_GIL¶: Use the default selection (PyInterpreterConfig_SHARED_GIL).

PyInterpreterConfig_SHARED_GIL¶: Use (share) the main interpreter’s GIL.

PyInterpreterConfig_OWN_GIL¶: Use the sub-interpreter’s own GIL.

If this isPyInterpreterConfig_OWN_GILthen PyInterpreterConfig.use_main_obmallocmust be0.

PyStatusPy_NewInterpreterFromConfig(PyThreadState**tstate_p,constPyInterpreterConfig*config)¶

Create a new sub-interpreter. This is an (almost) totally separate environment for the execution of Python code. In particular, the new interpreter has separate, independent versions of all imported modules, including the fundamental modulesbuiltins,__main__andsys.The table of loaded modules (sys.modules) and the module search path (sys.path) are also separate. The new environment has nosys.argv variable. It has new standard I/O stream file objectssys.stdin, sys.stdoutandsys.stderr(however these refer to the same underlying file descriptors).

The givenconfigcontrols the options with which the interpreter is initialized.

Upon success,tstate_pwill be set to the first thread state created in the new sub-interpreter. This thread state is made in the current thread state. Note that no actual thread is created; see the discussion of thread states below. If creation of the new interpreter is unsuccessful, tstate_pis set toNULL; no exception is set since the exception state is stored in the current thread state and there may not be a current thread state.

Like all other Python/C API functions, the global interpreter lock must be held before calling this function and is still held when it returns. Likewise a current thread state must be set on entry. On success, the returned thread state will be set as current. If the sub-interpreter is created with its own GIL then the GIL of the calling interpreter will be released. When the function returns, the new interpreter’s GIL will be held by the current thread and the previously interpreter’s GIL will remain released here.

Added in version 3.12.

Sub-interpreters are most effective when isolated from each other, with certain functionality restricted:

PyInterpreterConfigconfig={
.use_main_obmalloc=0,
.allow_fork=0,
.allow_exec=0,
.allow_threads=1,
.allow_daemon_threads=0,
.check_multi_interp_extensions=1,
.gil=PyInterpreterConfig_OWN_GIL,
};
PyThreadState*tstate=Py_NewInterpreterFromConfig(&config);

Note that the config is used only briefly and does not get modified. During initialization the config’s values are converted into various PyInterpreterStatevalues. A read-only copy of the config may be stored internally on thePyInterpreterState.

Extension modules are shared between (sub-)interpreters as follows:

For modules using multi-phase initialization, e.g.PyModule_FromDefAndSpec(),a separate module object is created and initialized for each interpreter. Only C-level static and global variables are shared between these module objects.
For modules using single-phase initialization, e.g.PyModule_Create(),the first time a particular extension is imported, it is initialized normally, and a (shallow) copy of its module’s dictionary is squirreled away. When the same extension is imported by another (sub-)interpreter, a new module is initialized and filled with the contents of this copy; the extension’sinitfunction is not called. Objects in the module’s dictionary thus end up shared across (sub-)interpreters, which might cause unwanted behavior (see Bugs and caveatsbelow).

Note that this is different from what happens when an extension is imported after the interpreter has been completely re-initialized by callingPy_FinalizeEx()andPy_Initialize();in that case, the extension’sinitmodulefunctioniscalled again. As with multi-phase initialization, this means that only C-level static and global variables are shared between these modules.

PyThreadState*Py_NewInterpreter(void)¶: Part of theStable ABI.
Create a new sub-interpreter. This is essentially just a wrapper aroundPy_NewInterpreterFromConfig()with a config that preserves the existing behavior. The result is an unisolated sub-interpreter that shares the main interpreter’s GIL, allows fork/exec, allows daemon threads, and allows single-phase init modules.

voidPy_EndInterpreter(PyThreadState*tstate)¶

Part of theStable ABI.

Destroy the (sub-)interpreter represented by the given thread state. The given thread state must be the current thread state. See the discussion of thread states below. When the call returns, the current thread state isNULL.All thread states associated with this interpreter are destroyed. The global interpreter lock used by the target interpreter must be held before calling this function. No GIL is held when it returns.

Py_FinalizeEx()will destroy all sub-interpreters that haven’t been explicitly destroyed at that point.

A Per-Interpreter GIL¶

UsingPy_NewInterpreterFromConfig()you can create a sub-interpreter that is completely isolated from other interpreters, including having its own GIL. The most important benefit of this isolation is that such an interpreter can execute Python code without being blocked by other interpreters or blocking any others. Thus a single Python process can truly take advantage of multiple CPU cores when running Python code. The isolation also encourages a different approach to concurrency than that of just using threads. (SeePEP 554.)

Using an isolated interpreter requires vigilance in preserving that isolation. That especially means not sharing any objects or mutable state without guarantees about thread-safety. Even objects that are otherwise immutable (e.g.None,(1,5)) can’t normally be shared because of the refcount. One simple but less-efficient approach around this is to use a global lock around all use of some state (or object). Alternately, effectively immutable objects (like integers or strings) can be made safe in spite of their refcounts by making themimmortal. In fact, this has been done for the builtin singletons, small integers, and a number of other builtin objects.

If you preserve isolation then you will have access to proper multi-core computing without the complications that come with free-threading. Failure to preserve isolation will expose you to the full consequences of free-threading, including races and hard-to-debug crashes.

Aside from that, one of the main challenges of using multiple isolated interpreters is how to communicate between them safely (not break isolation) and efficiently. The runtime and stdlib do not provide any standard approach to this yet. A future stdlib module would help mitigate the effort of preserving isolation and expose effective tools for communicating (and sharing) data between interpreters.

Added in version 3.12.

Bugs and caveats¶

Because sub-interpreters (and the main interpreter) are part of the same process, the insulation between them isn’t perfect — for example, using low-level file operations likeos.close()they can (accidentally or maliciously) affect each other’s open files. Because of the way extensions are shared between (sub-)interpreters, some extensions may not work properly; this is especially likely when using single-phase initialization or (static) global variables. It is possible to insert objects created in one sub-interpreter into a namespace of another (sub-)interpreter; this should be avoided if possible.

Special care should be taken to avoid sharing user-defined functions, methods, instances or classes between sub-interpreters, since import operations executed by such objects may affect the wrong (sub-)interpreter’s dictionary of loaded modules. It is equally important to avoid sharing objects from which the above are reachable.

Also note that combining this functionality withPyGILState_*APIs is delicate, because these APIs assume a bijection between Python thread states and OS-level threads, an assumption broken by the presence of sub-interpreters. It is highly recommended that you don’t switch sub-interpreters between a pair of matchingPyGILState_Ensure()andPyGILState_Release()calls. Furthermore, extensions (such asctypes) using these APIs to allow calling of Python code from non-Python created threads will probably be broken when using sub-interpreters.

Asynchronous Notifications¶

A mechanism is provided to make asynchronous notifications to the main interpreter thread. These notifications take the form of a function pointer and a void pointer argument.

intPy_AddPendingCall(int(*func)(void*),void*arg)¶

Part of theStable ABI.

Schedule a function to be called from the main interpreter thread. On success,0is returned andfuncis queued for being called in the main thread. On failure,-1is returned without setting any exception.

When successfully queued,funcwill beeventuallycalled from the main interpreter thread with the argumentarg.It will be called asynchronously with respect to normally running Python code, but with both these conditions met:

on abytecodeboundary;
with the main thread holding theglobal interpreter lock (funccan therefore use the full C API).

funcmust return0on success, or-1on failure with an exception set.funcwon’t be interrupted to perform another asynchronous notification recursively, but it can still be interrupted to switch threads if the global interpreter lock is released.

This function doesn’t need a current thread state to run, and it doesn’t need the global interpreter lock.

To call this function in a subinterpreter, the caller must hold the GIL. Otherwise, the functionfunccan be scheduled to be called from the wrong interpreter.

Warning

This is a low-level function, only useful for very special cases. There is no guarantee thatfuncwill be called as quick as possible. If the main thread is busy executing a system call, funcwon’t be called before the system call returns. This function is generallynotsuitable for calling Python code from arbitrary C threads. Instead, use thePyGILState API.

Added in version 3.1.

Changed in version 3.9:If this function is called in a subinterpreter, the functionfuncis now scheduled to be called from the subinterpreter, rather than being called from the main interpreter. Each subinterpreter now has its own list of scheduled calls.

Profiling and Tracing¶

The Python interpreter provides some low-level support for attaching profiling and execution tracing facilities. These are used for profiling, debugging, and coverage analysis tools.

This C interface allows the profiling or tracing code to avoid the overhead of calling through Python-level callable objects, making a direct C function call instead. The essential attributes of the facility have not changed; the interface allows trace functions to be installed per-thread, and the basic events reported to the trace function are the same as had been reported to the Python-level trace functions in previous versions.

typedefint(*Py_tracefunc)(PyObject*obj,PyFrameObject*frame,intwhat,PyObject*arg)¶

The type of the trace function registered usingPyEval_SetProfile()and PyEval_SetTrace().The first parameter is the object passed to the registration function asobj,frameis the frame object to which the event pertains,whatis one of the constantsPyTrace_CALL, PyTrace_EXCEPTION,PyTrace_LINE,PyTrace_RETURN, PyTrace_C_CALL,PyTrace_C_EXCEPTION,PyTrace_C_RETURN, orPyTrace_OPCODE,andargdepends on the value ofwhat:

Value ofwhat	Meaning ofarg
`PyTrace_CALL`	Always`Py_None`.
`PyTrace_EXCEPTION`	Exception information as returned by `sys.exc_info()`.
`PyTrace_LINE`	Always`Py_None`.
`PyTrace_RETURN`	Value being returned to the caller, or`NULL`if caused by an exception.
`PyTrace_C_CALL`	Function object being called.
`PyTrace_C_EXCEPTION`	Function object being called.
`PyTrace_C_RETURN`	Function object being called.
`PyTrace_OPCODE`	Always`Py_None`.

intPyTrace_CALL¶: The value of thewhatparameter to aPy_tracefuncfunction when a new call to a function or method is being reported, or a new entry into a generator. Note that the creation of the iterator for a generator function is not reported as there is no control transfer to the Python bytecode in the corresponding frame.

intPyTrace_EXCEPTION¶: The value of thewhatparameter to aPy_tracefuncfunction when an exception has been raised. The callback function is called with this value for whatwhen after any bytecode is processed after which the exception becomes set within the frame being executed. The effect of this is that as exception propagation causes the Python stack to unwind, the callback is called upon return to each frame as the exception propagates. Only trace functions receives these events; they are not needed by the profiler.

intPyTrace_LINE¶: The value passed as thewhatparameter to aPy_tracefuncfunction (but not a profiling function) when a line-number event is being reported. It may be disabled for a frame by settingf_trace_linesto 0on that frame.

intPyTrace_RETURN¶: The value for thewhatparameter toPy_tracefuncfunctions when a call is about to return.

intPyTrace_C_CALL¶: The value for thewhatparameter toPy_tracefuncfunctions when a C function is about to be called.

intPyTrace_C_EXCEPTION¶: The value for thewhatparameter toPy_tracefuncfunctions when a C function has raised an exception.

intPyTrace_C_RETURN¶: The value for thewhatparameter toPy_tracefuncfunctions when a C function has returned.

intPyTrace_OPCODE¶: The value for thewhatparameter toPy_tracefuncfunctions (but not profiling functions) when a new opcode is about to be executed. This event is not emitted by default: it must be explicitly requested by setting f_trace_opcodesto1on the frame.

voidPyEval_SetProfile(Py_tracefuncfunc,PyObject*obj)¶

Set the profiler function tofunc.Theobjparameter is passed to the function as its first parameter, and may be any Python object, orNULL.If the profile function needs to maintain state, using a different value forobj for each thread provides a convenient and thread-safe place to store it. The profile function is called for all monitored events exceptPyTrace_LINE PyTrace_OPCODEandPyTrace_EXCEPTION.

Reference tracing¶

Added in version 3.13.

typedefint(*PyRefTracer)(PyObject*,intevent,void*data)¶: The type of the trace function registered usingPyRefTracer_SetTracer(). The first parameter is a Python object that has been just created (whenevent is set toPyRefTracer_CREATE) or about to be destroyed (whenevent is set toPyRefTracer_DESTROY). Thedataargument is the opaque pointer that was provided whenPyRefTracer_SetTracer()was called.

Added in version 3.13.

intPyRefTracer_CREATE¶: The value for theeventparameter toPyRefTracerfunctions when a Python object has been created.

intPyRefTracer_DESTROY¶: The value for theeventparameter toPyRefTracerfunctions when a Python object has been destroyed.

intPyRefTracer_SetTracer(PyRefTracertracer,void*data)¶

Register a reference tracer function. The function will be called when a new Python has been created or when an object is going to be destroyed. If datais provided it must be an opaque pointer that will be provided when the tracer function is called. Return0on success. Set an exception and return-1on error.

Not that tracer functionsmust notcreate Python objects inside or otherwise the call will be re-entrant. The tracer alsomust notclear any existing exception or set an exception. The GIL will be held every time the tracer function is called.

The GIL must be held when calling this function.

Added in version 3.13.

PyRefTracerPyRefTracer_GetTracer(void**data)¶

Get the registered reference tracer function and the value of the opaque data pointer that was registered whenPyRefTracer_SetTracer()was called. If no tracer was registered this function will return NULL and will set the datapointer to NULL.

The GIL must be held when calling this function.

Added in version 3.13.

Advanced Debugger Support¶

These functions are only intended to be used by advanced debugging tools.

PyInterpreterState*PyInterpreterState_Head()¶: Return the interpreter state object at the head of the list of all such objects.

PyInterpreterState*PyInterpreterState_Main()¶: Return the main interpreter state object.

PyInterpreterState*PyInterpreterState_Next(PyInterpreterState*interp)¶: Return the next interpreter state object afterinterpfrom the list of all such objects.

PyThreadState*PyInterpreterState_ThreadHead(PyInterpreterState*interp)¶: Return the pointer to the firstPyThreadStateobject in the list of threads associated with the interpreterinterp.

PyThreadState*PyThreadState_Next(PyThreadState*tstate)¶: Return the next thread state object aftertstatefrom the list of all such objects belonging to the samePyInterpreterStateobject.

Thread Local Storage Support¶

The Python interpreter provides low-level support for thread-local storage (TLS) which wraps the underlying native TLS implementation to support the Python-level thread local storage API (threading.local). The CPython C level APIs are similar to those offered by pthreads and Windows: use a thread key and functions to associate avoid*value per thread.

The GIL doesnotneed to be held when calling these functions; they supply their own locking.

Note thatPython.hdoes not include the declaration of the TLS APIs, you need to includepythread.hto use thread-local storage.

Note

None of these API functions handle memory management on behalf of the void*values. You need to allocate and deallocate them yourself. If thevoid*values happen to bePyObject*,these functions don’t do refcount operations on them either.

Thread Specific Storage (TSS) API¶

TSS API is introduced to supersede the use of the existing TLS API within the CPython interpreter. This API uses a new typePy_tss_tinstead of intto represent thread keys.

Added in version 3.7.

Dynamic Allocation¶

Dynamic allocation of thePy_tss_t,required in extension modules built withPy_LIMITED_API,where static allocation of this type is not possible due to its implementation being opaque at build time.

Py_tss_t*PyThread_tss_alloc()¶: Part of theStable ABIsince version 3.7.
Return a value which is the same state as a value initialized with Py_tss_NEEDS_INIT,orNULLin the case of dynamic allocation failure.

voidPyThread_tss_free(Py_tss_t*key)¶: Part of theStable ABIsince version 3.7.
Free the givenkeyallocated byPyThread_tss_alloc(),after first callingPyThread_tss_delete()to ensure any associated thread locals have been unassigned. This is a no-op if thekey argument isNULL.

Note

A freed key becomes a dangling pointer. You should reset the key to NULL.

Methods¶

The parameterkeyof these functions must not beNULL.Moreover, the behaviors ofPyThread_tss_set()andPyThread_tss_get()are undefined if the givenPy_tss_thas not been initialized by PyThread_tss_create().

intPyThread_tss_is_created(Py_tss_t*key)¶: Part of theStable ABIsince version 3.7.
Return a non-zero value if the givenPy_tss_thas been initialized byPyThread_tss_create().

intPyThread_tss_create(Py_tss_t*key)¶: Part of theStable ABIsince version 3.7.
Return a zero value on successful initialization of a TSS key. The behavior is undefined if the value pointed to by thekeyargument is not initialized byPy_tss_NEEDS_INIT.This function can be called repeatedly on the same key – calling it on an already initialized key is a no-op and immediately returns success.

voidPyThread_tss_delete(Py_tss_t*key)¶: Part of theStable ABIsince version 3.7.
Destroy a TSS key to forget the values associated with the key across all threads, and change the key’s initialization state to uninitialized. A destroyed key is able to be initialized again by PyThread_tss_create().This function can be called repeatedly on the same key – calling it on an already destroyed key is a no-op.

intPyThread_tss_set(Py_tss_t*key,void*value)¶: Part of theStable ABIsince version 3.7.
Return a zero value to indicate successfully associating avoid* value with a TSS key in the current thread. Each thread has a distinct mapping of the key to avoid*value.

void*PyThread_tss_get(Py_tss_t*key)¶: Part of theStable ABIsince version 3.7.
Return thevoid*value associated with a TSS key in the current thread. This returnsNULLif no value is associated with the key in the current thread.

Thread Local Storage (TLS) API¶

Deprecated since version 3.7:This API is superseded by Thread Specific Storage (TSS) API.

Note

This version of the API does not support platforms where the native TLS key is defined in a way that cannot be safely cast toint.On such platforms, PyThread_create_key()will return immediately with a failure status, and the other TLS functions will all be no-ops on such platforms.

Due to the compatibility problem noted above, this version of the API should not be used in new code.

intPyThread_create_key()¶: Part of theStable ABI.

voidPyThread_delete_key(intkey)¶: Part of theStable ABI.

intPyThread_set_key_value(intkey,void*value)¶: Part of theStable ABI.

void*PyThread_get_key_value(intkey)¶: Part of theStable ABI.

voidPyThread_delete_key_value(intkey)¶: Part of theStable ABI.

voidPyThread_ReInitTLS()¶: Part of theStable ABI.

Synchronization Primitives¶

The C-API provides a basic mutual exclusion lock.

typePyMutex¶

A mutual exclusion lock. ThePyMutexshould be initialized to zero to represent the unlocked state. For example:

PyMutexmutex={0};

Instances ofPyMutexshould not be copied or moved. Both the contents and address of aPyMutexare meaningful, and it must remain at a fixed, writable location in memory.

Note

APyMutexcurrently occupies one byte, but the size should be considered unstable. The size may change in future Python releases without a deprecation period.

Added in version 3.13.

voidPyMutex_Lock(PyMutex*m)¶: Lock mutexm.If another thread has already locked it, the calling thread will block until the mutex is unlocked. While blocked, the thread will temporarily release theGILif it is held.

Added in version 3.13.

voidPyMutex_Unlock(PyMutex*m)¶: Unlock mutexm.The mutex must be locked — otherwise, the function will issue a fatal error.

Added in version 3.13.

Python Critical Section API¶

The critical section API provides a deadlock avoidance layer on top of per-object locks forfree-threadedCPython. They are intended to replace reliance on theglobal interpreter lock,and are no-ops in versions of Python with the global interpreter lock.

Critical sections avoid deadlocks by implicitly suspending active critical sections and releasing the locks during calls toPyEval_SaveThread(). WhenPyEval_RestoreThread()is called, the most recent critical section is resumed, and its locks reacquired. This means the critical section API provides weaker guarantees than traditional locks – they are useful because their behavior is similar to theGIL.

The functions and structs used by the macros are exposed for cases where C macros are not available. They should only be used as in the given macro expansions. Note that the sizes and contents of the structures may change in future Python versions.

Note

Operations that need to lock two objects at once must use Py_BEGIN_CRITICAL_SECTION2.Youcannotuse nested critical sections to lock more than one object at once, because the inner critical section may suspend the outer critical sections. This API does not provide a way to lock more than two objects at once.

Example usage:

staticPyObject*
set_field(MyObject*self,PyObject*value)
{
Py_BEGIN_CRITICAL_SECTION(self);
Py_SETREF(self->field,Py_XNewRef(value));
Py_END_CRITICAL_SECTION();
Py_RETURN_NONE;
}

In the above example,Py_SETREFcallsPy_DECREF,which can call arbitrary code through an object’s deallocation function. The critical section API avoids potential deadlocks due to reentrancy and lock ordering by allowing the runtime to temporarily suspend the critical section if the code triggered by the finalizer blocks and callsPyEval_SaveThread().

Py_BEGIN_CRITICAL_SECTION(op)¶

Acquires the per-object lock for the objectopand begins a critical section.

In the free-threaded build, this macro expands to:

{
PyCriticalSection_py_cs;
PyCriticalSection_Begin(&_py_cs,(PyObject*)(op))

In the default build, this macro expands to{.

Added in version 3.13.

Py_END_CRITICAL_SECTION()¶

Ends the critical section and releases the per-object lock.

In the free-threaded build, this macro expands to:

PyCriticalSection_End(&_py_cs);
}

In the default build, this macro expands to}.

Added in version 3.13.

Py_BEGIN_CRITICAL_SECTION2(a,b)¶

Acquires the per-objects locks for the objectsaandband begins a critical section. The locks are acquired in a consistent order (lowest address first) to avoid lock ordering deadlocks.

In the free-threaded build, this macro expands to:

{
PyCriticalSection2_py_cs2;
PyCriticalSection_Begin2(&_py_cs2,(PyObject*)(a),(PyObject*)(b))

In the default build, this macro expands to{.

Added in version 3.13.

Py_END_CRITICAL_SECTION2()¶

Ends the critical section and releases the per-object locks.

In the free-threaded build, this macro expands to:

PyCriticalSection_End2(&_py_cs2);
}

In the default build, this macro expands to}.

Added in version 3.13.

Initialization, Finalization, and Threads¶

Before Python Initialization¶

Global configuration variables¶

Initializing and finalizing the interpreter¶

Process-wide parameters¶

Thread State and the Global Interpreter Lock¶

Releasing the GIL from extension code¶

Non-Python created threads¶

Cautions about fork()¶

High-level API¶

Low-level API¶

Sub-interpreter support¶

A Per-Interpreter GIL¶

Bugs and caveats¶

Asynchronous Notifications¶

Profiling and Tracing¶

Reference tracing¶

Advanced Debugger Support¶

Thread Local Storage Support¶

Thread Specific Storage (TSS) API¶

Dynamic Allocation¶

Methods¶

Thread Local Storage (TLS) API¶

Synchronization Primitives¶

Python Critical Section API¶

Table of Contents

Previous topic

Next topic

This Page