Improve Py_mod_multiple_interpreters and Py_mod_gil Usability

[ericsnowcurrently]

Feature or enhancement

Proposal:

Here are some deficiencies:

• the slot name "Py_mod_multiple_interpreters" is a bit long
• the "Py_MOD_MULTIPLE_INTERPRETERS_*" slot value names are a bit long
• the slot name "Py_mod_gil" is overly coupled to the GIL (it's really about thread safety)
• the difference between Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED and Py_MOD_PER_INTERPRETER_GIL_SUPPORTED is unclear
• Py_MOD_PER_INTERPRETER_GIL_SUPPORTED applies to the vast majority of extensions and should be the default
• Py_mod_gil doesn't have an option that explicitly covers the case where an external dependency is not thread-safe
• currently Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED implies an external dependency is not thread-safe and/or isolated to each interpreter; it should be the other way around (and be a synonym for Py_MOD_PER_INTERPRETER_GIL_SUPPORTED), which would introduce a need for a Py_MOD_PER_INTERPRETER_GIL_NOT_SUPPORTED

We could take a simple approach, working with the existing slots:

• (maybe) shorten the names
• add Py_MOD_PER_INTERPRETER_GIL_NOT_SUPPORTED
• add something like Py_MOD_GIL_USED_FOR_EXTERNAL

I think we could do better, by replacing the existing slots with new ones that focus explicitly on what we care about:

• new module def slot: Py_mod_isolation
  • Py_MOD_NOT_ISOLATED - the module has process-global state and/or objects (e.g. static types)
  • Py_MOD_ISOLATED (default) - the module's state/objects have been isolated to the module (i.e interpreter)
• new module def slot: Py_mod_threadsafe
  • Py_MOD_NOT_THREADSAFE (default) - the module's own state/code is not thread-safe
  • Py_MOD_THREADSAFE - the module's state/code are thread-safe; external dependencies are covered by Py_mod_external_libs
• new module def slot: Py_mod_external_libs
  • Py_MOD_EXTERN_NOT_THREADSAFE - at least one external dependency is not thread-safe (may be used without the GIL held)
  • Py_MOD_EXTERN_NOT_ISOLATED - at least one external dependency is not isolated to the module object (i.e. interpreter)
  • Py_MOD_EXTERN_ISOLATED (default) - all external dependencies are isolated and using them is thread-safe
• for Py_mod_isolation and Py_mod_threadsafe, external dependencies are covered by Py_mod_external_libs
• deprecate Py_mod_multiple_interpreters slot
• deprecate Py_mod_gil slot

Regarding the defaults, for Py_mod_isolation, multi-phase init implies isolation (per PEP 489). For Py_mod_external_libs, we assume that the vast majority of modules will not have problematic dependencies.

Equivalents to current usage:

Py_mod_threadsafe	Py_mod_external_libs		Py_mod_gil
Py_MOD_NOT_THREADSAFE	*		Py_MOD_GIL_USED
Py_MOD_THREADSAFE	Py_MOD_EXTERNAL_NOT_THREADSAFE		Py_MOD_GIL_USED
Py_MOD_THREADSAFE	Py_MOD_EXTERNAL_NOT_ISOLATED		Py_MOD_GIL_NOT_USED
Py_MOD_THREADSAFE	Py_MOD_EXTERNAL_ISOLATED		Py_MOD_GIL_NOT_USED

Py_mod_isolation	Py_mod_external_libs		Py_mod_multiple_interpreters
Py_MOD_NOT_ISOLATED	*		Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED
Py_MOD_ISOLATED	Py_MOD_EXTERNAL_NOT_THREADSAFE		Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED
Py_MOD_ISOLATED	Py_MOD_EXTERNAL_NOT_ISOLATED		Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED
Py_MOD_ISOLATED	Py_MOD_EXTERNAL_ISOLATED		Py_MOD_PER_INTERPRETER_GIL_SUPPORTED

CC @encukou

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[encukou]

I'd love to combine this with a proposal in https://discuss.python.org/t/stable-abi-limited-api-for-free-threaded-builds/86458:

I’d like to add a module slot that indicates the version of Python used to build the extension, the Py_LIMITED_API value used (if any), and some flags [...]. This would be checked for compatibility, so we no longer rely only on wheel/.soname tags, and so we can add deprecation warnings for upcoming incompatibilities.

Can it wait until 3.15, or are we in a rush?

[AA-Turner]

By my count, 614 of the top 15,000 PyPI packages (representing 99%+ of PyPI downloads) need to migrate to multi-phase init. This is then 447 of the top 10k, 268 of the top 5k, 72 of the top 1,000, and 11 of the top 100.

We've seen pushback from maintainers on migrating to multi-phase with the current state of affairs (partly due to documentation/marketing, which we're actively improving). My slight concern with changing all of the slots now is that maintainers will question the state of flux and somewhat reasonably ask to delay any change until things have settled down.

As a sketch, the below might be needed for a module bridging Python 3.10--3.15:

static PyModuleDef_Slot spam_slots[] = {
    {Py_mod_exec, spam_exec},
#if PY_VERSION_HEX >= 0x030F00F0  // Python 3.15+
    {Py_mod_isolation, Py_MOD_ISOLATED},
    {Py_mod_threadsafe, Py_MOD_THREADSAFE},
    {Py_mod_external_libs, Py_MOD_EXTERNAL_NOT_ISOLATED},
#else
#  if PY_VERSION_HEX >= 0x030C00F0  // Python 3.12+
    {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
#  endif
#  if PY_VERSION_HEX >= 0x030D00F0  // Python 3.13+
    {Py_mod_gil, Py_MOD_GIL_NOT_USED},
#  endif
#endif
    {0, NULL}
};

The above looks on the surface far more complex and harder to maintain than a 'simple' call to PyModule_Create(). This is more of a request to consider the migration case, as extra friction in adoption makes it harder to convince maintainers to switch.

(maybe) shorten the names

I don't think the names are a problem, if anything longer and more explicit is useful, and these generally only appear once & at the end of extension modules.

A

[encukou]

IMO, the current slots focus on what CPython does, the proposed ones focus on describing the module.

It would help to document how CPython would react to these options (and their combinations), both currently and when future plans are in place.

Going into details:

• "thread-safe" can have different meanings; we'd need to pick the relevant one.
• I read "external dependencies" as coming from a different repo and potentially being under a different licence; that's not what we care about. Should this be "code that runs without attached thread state", or am I misunderstanding it entirely?

I'd love to try going the other way and reducing the number of slots, even if the result is a bit less granular.
For example, using the existing names, could we make Py_MOD_PER_INTERPRETER_GIL_SUPPORTED the default if the user sets Py_MOD_GIL_NOT_USED?