Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-99113: A Per-Interpreter GIL! #104210

Merged

Conversation

ericsnowcurrently
Copy link
Member

@ericsnowcurrently ericsnowcurrently commented May 5, 2023

This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)!

Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since the subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.

(FYI, I've split up the original gh-99114 into multiple PRs, with this one being the final one in the chain.)

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 289d050 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 6, 2023
@ericsnowcurrently ericsnowcurrently marked this pull request as ready for review May 6, 2023 22:03
@ericsnowcurrently
Copy link
Member Author

FYI, there are plenty of little things to clean up after this lands, particularly for PEP 684. There are also some corner-case crashes I need to chase down, related to extreme races.

Also, I still can't believe I've finally reached this point with this project!!! 😭

Copy link
Contributor

@erlend-aasland erlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎉 BTW, I think the massive top-level comment in Python/ceval_gil.c needs an overhaul now.

Copy link
Contributor

@kumaraditya303 kumaraditya303 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM is short for the amount of effort you put into it.

@ericsnowcurrently ericsnowcurrently merged commit 5c9ee49 into python:main May 8, 2023
@ericsnowcurrently
Copy link
Member Author

Thanks for all the great work and support, everyone!

There are plenty of small things to wrap up, but this is definitely the climax.

carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
* main: (47 commits)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  pythongh-104273: Remove redundant len() calls in argparse function (python#104274)
  pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)
  pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)
  pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)
  pythongh-103650: Fix perf maps address format (python#103651)
  pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
* main: (29 commits)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
* main: (156 commits)
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
* main: (35 commits)
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  ...
@aotto1968
Copy link

well , i changed my library to use this, but end up in ...

I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my
library. The library is thread safe and supports many languages
-> C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php
other languages with thread support
-> C C++ JAVA C# TCL GO
works fine.
even python works fine without thread support.

Task: now I started thread support in python, I had to change to Multi-phase initialization
isolate the python data on an per-thread-level and finally gent an easy thread example working
-> ok

    static PyInterpreterConfig config = {
        .use_main_obmalloc = 0,
        .allow_fork = 0,
        .allow_exec = 0,
        .allow_threads = 1,
        .allow_daemon_threads = 0,
        .check_multi_interp_extensions = 1,
        .gil = PyInterpreterConfig_OWN_GIL,
    };

    PyThreadState *tstate = NULL;
    PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);

NOW I start with an more complex example and this stuff fails with a python INTERNAL core
that I can't debug. The stack-trace comes from my MqDisasterSignal handler.

-> question: what can I do to debug this issue?
-> debugging with valgrind is not possible because with valgrind this example works fine.


Debug memory block at address p=0x7f11884def70: API '^@'
    18302063728033398269 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x00 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0x00 *** OUCH
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0xfdfe7d0f864bed6d are X> {PRINT               :pid(60831):tid(0x7f118819c700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1cb99b8     ):MqDisasterSignal              }: BackTrace {
:pid(60831):tid(0x7f118819c700):   [ library              : filename                       : lineno ] function
:pid(60831):tid(0x7f118819c700):   [ -------              : --------                       : ------ ] --------
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 705    ] MqDisasterSignal
:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2408   ] _PyObject_DebugDumpAddress
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2326   ] _PyMem_DebugCheckAddress
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2159   ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 685    ] PyMem_RawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 1853   ] _PyObject_Free
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2163   ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2296   ] _PyMem_DebugFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 830    ] PyObject_Free
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1569   ] dictresize
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1194   ] insertion_resize
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1261   ] insertdict
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1865   ] _PyDict_SetItem_Take2
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1883   ] PyDict_SetItem
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7618   ] add_subclass
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7352   ] type_ready_add_subclasses
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7515   ] type_ready
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7553   ] PyType_Ready
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3795   ] type_new_impl
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3929   ] type_new
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 1664   ] type_call
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 240    ] _PyObject_MakeTpCall
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3949   ] type_vectorcall
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 133    ] _PyObject_FastCallDictTstate
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 157    ] PyObject_VectorcallDict
:pid(60831):tid(0x7f118819c700):   [ system               : Python/bltinmodule.c           : 208    ] builtin___build_class__
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/methodobject.c         : 438    ] cfunction_vectorcall_FASTCALL_KEYWORDS
:pid(60831):tid(0x7f118819c700):   [ system               : Include/internal/pycore_call.h : 92     ] _PyObject_VectorcallTstate
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 325    ] PyObject_Vectorcall
:pid(60831):tid(0x7f118819c700):   [ system               : Python/bytecodes.c             : 2714   ] _PyEval_EvalFrameDefault
:pid(60831):tid(0x7f118819c700):   [ system               : nclude/internal/pycore_ceval.h : 89     ] _PyEval_EvalFrame
:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 1683   ] _PyEval_Vector
:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 578    ] PyEval_EvalCode
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1722   ] run_eval_code_obj
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1743   ] run_mod
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1643   ] pyrun_file
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 433    ] _PyRun_SimpleFileObject
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 466    ] PyRun_SimpleFileExFlags
:pid(60831):tid(0x7f118819c700):   [ theLink              : py/MqFactoryC_py.c             : 194    ] py_mqmsgque_sFactoryCTor
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/MqFactoryS_mq.c              : 289    ] MqFactoryInvoke_RT
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 269    ] MqSysServerThreadMain
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 324    ] sSysServerThreadInit
:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown
:pid(60831):tid(0x7f118819c700): }
:pid(60831):tid(0x7f118819c700): ^[[1;36mglobal data -----------------------------------------------------^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mruntime  : MkRuntimeRLS=0x1cb93b0, MqRuntimeRLS=0x7f11879539c8^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mgeDat    : geDat=0x7f1188499de0, geMax=16, geCur=2^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mevent    : event=0x7f1187954610, DataLNum=16, DataLCur=0, geNum=1, fdmax=0^[[0;m
.../NHI1/bin/Nhi1Exec: line 870: 60831 Aborted                 '.../x86_64-suse-linux-gnu/debug/bin/python3' '.../NHI1/example/py/server.py' '--thread' '--uds' '--file' '/tmp/test.uds'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants