Compare commits

...

464 Commits

Author SHA1 Message Date
Stefano Rivera
197e4c0628
Avoid TestNullDlsym hanging on HP-PA (#141011)
glibc has no support for IFUNC on HP PA RISC yet. Rather than waiting
for an "OK" that we'll never get, skip the test.

See: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/hppa/dl-irel.h;h=770dcb3ea3a6ce232bb11bf59315cfec4c543f93;hb=HEAD
2026-01-27 12:39:03 +01:00
Nathan Goldbaum
f2b5c206c7
gh-143883: Use named parameters in PyModExport-related declarations & docs (GH-143884) 2026-01-27 10:28:28 +00:00
Stefano Rivera
17d447e993
gh-126014: test_makefile_test_folders: Ignore basically-empty directories (#140466)
The code in test_makefile was attempting to ignore any
non-interesting files, but missed some corners:

1. There is never a *file* called `__pycache__`.
2. A directory containing only a `__pycache__` subdirectory should be
   ignored.
3. A directory containing only hidden files should be ignored.

Simplify this all into a couple of filters that let us check for empty
lists.
2026-01-27 11:05:09 +01:00
Bartosz Sławecki
487bd2dea5
gh-142119: Clarify that one contextvars.Token can only reset once in a lifetime (GH-143693) 2026-01-27 10:49:05 +01:00
Nathan Goldbaum
3e9a5b022f
gh-144257: document return values of PyModule_SetDocString (GH-144258)
Co-authored-by: sobolevn <mail@sobolevn.me>
2026-01-27 02:15:16 -05:00
Serhiy Storchaka
2520617361
gh-142037: Fix a refleak introduced in GH-142081 (GH-144256) 2026-01-26 21:15:21 +00:00
Serhiy Storchaka
7febbe6b60
gh-144001: Support ignorechars in binascii.a2b_base64() and base64.b64decode() (GH-144024) 2026-01-26 20:11:40 +02:00
Jeong, YunWon
9181d776da
gh-142966: Make ctypes.POINTER.set_type also reset format (GH-142967)
Make the deprecated set_type method resets the format, using the
same code as in type initialization.

Implementation note: this was done in PyCPointerType_init 
after calling PyCPointerType_SetProto, but was forgotten
after in PyCPointerType_set_type_impl's call to
PyCPointerType_SetProto.
With this change, setting the format is conceptually part of
setting proto (i.e. the pointed-to type).

Co-authored-by: AN Long <aisk@users.noreply.github.com>
2026-01-26 17:40:56 +01:00
Victor Stinner
933540e332
gh-101888: Add function.__builtins__ to ref documentation (#144174) 2026-01-26 17:15:47 +01:00
Bartosz Sławecki
04d497c284
gh-142119: Clarify context manager protocol documentation on ContextVar.set and Token (GH-143694) 2026-01-26 16:50:27 +01:00
Jake Lishman
19de10d3d8
gh-140557: Force alignment of empty bytearray and array.array buffers (GH-140559)
This ensures the buffers used by the empty `bytearray` and `array.array`
are aligned the same as a pointer returned by the allocator.  This is a
more convenient default for interop with other languages that have
stricter requirements of type-safe buffers (e.g. Rust's `&[T]` type)
even when empty.
2026-01-26 16:45:17 +01:00
FooIbar
1f55caf97e
gh-144212: Add image/jxl to mimetypes (GH-144213) 2026-01-26 15:26:55 +01:00
Yashraj
923d9d2ac2
gh-143928: Remove outdated comparison between pickle and marshal regarding recursion (#144025) 2026-01-26 14:38:23 +01:00
VanshAgarwal24036
8f459255eb
gh-144100: Fix crash for POINTER(str) used in ctypes argtypes (#144108)
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-26 13:27:42 +01:00
Priyanshu Singh
decb25e8f0
gh-144128: Fix crash in array.fromlist with reentrant __index__ (#144138)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-26 13:13:49 +01:00
Solomon Ritzow
9982147433
gh-144233: Fix typo in os.eventfd documentation (#144234) 2026-01-26 10:07:56 +02:00
Jelle Zijlstra
639c1ad4f1
gh-144169: Fix three crashes in AST objects with non-str kwargs (#144178)
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-25 17:28:58 -08:00
benediktjohannes
76d3ae71ba
gh-144217: Add dicom support (medical imaging) to mimetypes (GH-144218)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
2026-01-25 22:22:43 +00:00
AN Long
6e55337f8a
gh-143995: Eliminate redundant refcounting in the JIT from LOAD_ATTR_MODULE (GH-143996) 2026-01-25 18:24:44 +00:00
Yashraj
a51bf70f95
gh-143504: Expose CELL status of a symbol in symtable (#143549) 2026-01-25 15:21:27 +00:00
Yi Yang
979d92fefc
gh-144140: Optimize len for string constants in optimizer (GH-144142) 2026-01-24 16:09:29 +00:00
mdehoon
27246c3482
Remove stray typedef in _tkinter.c (GH-142924)
Co-authored-by: Michiel de Hoon <mdehoon@tkx288.genome.gsc.riken.jp>
2026-01-24 17:20:14 +02:00
Serhiy Storchaka
012c498035
gh-142037: Improve error messages for printf-style formatting (GH-142081)
This affects string formatting as well as bytes and bytearray formatting.

* For errors in the format string, always include the position of the
  start of the format unit.
* For errors related to the formatted arguments, always include the number
  or the name of the formatted argument.
* Suggest more probable causes of errors in the format string (stray %,
  unsupported format, unexpected character).
* Provide more information when the number of arguments does not match
  the number of format units.
* Raise more specific errors when access of arguments by name is mixed with
  sequential access and when * is used with a mapping.
* Add tests for some uncovered cases.
2026-01-24 11:13:50 +00:00
Marc Mueller
5f736a0432
gh-142913: Add generated test files to gitattributes (GH-144209) 2026-01-24 10:19:21 +00:00
reiden
6d972e0104
gh-130415: Narrow types to constants in branches involving specialized comparisons with a constant (GH-144150) 2026-01-24 10:02:08 +00:00
Hai Zhu
29840247ff
gh-144068: fix JIT tracer memory leak when daemon thread exits (GH-144077) 2026-01-24 09:43:01 +00:00
Yongtao Huang
6f579147e3
Misc: remove duplicate instr_frame assignment in _PyJit_TryInitializeTracing (GH-144155)
Remove duplicate instr_frame assignment in optimizer
2026-01-24 09:37:45 +00:00
Ken Jin
ca99bfdefb
gh-144016: Fix bad stack assert in the JIT optimizer (GH-144019) 2026-01-24 09:36:40 +00:00
AN Long
4e10fa993a
gh-144007: Eliminate redundant refcounting in the JIT for BINARY_OP (GH-144011) 2026-01-24 09:35:32 +00:00
Terry Jan Reedy
29f1e778fa
gh-135142: Initial 3.15 IDLE News3.txt entries (#144200) 2026-01-23 21:34:41 -05:00
Zachary Ware
58ccf21cbb
gh-74902: Avoid hitting unicode.org for test data (GH-144195)
Use our own pythontest.net instead.
2026-01-23 15:07:27 -06:00
Serhiy Storchaka
25a10b60b0
gh-144027: Fix documentation for ignorechars in base64.a85decode() (GH-144028)
It does not support an ASCII string.

Also add more tests.
2026-01-23 20:55:48 +02:00
AN Long
2f42f83344
gh-144005: Eliminate redundant refcounting in the JIT for BINARY_OP_EXTEND (#144006) 2026-01-23 17:19:01 +00:00
Petr Viktorin
70e67f579e
gh-141376: smelly.py: Print only smelly symbols, or all of them with --verbose (GH-141394)
Instead of long and uninteresting output for all checked libraries, only print found issues by default.

Add a new -v/--verbose option to list all symbols (useful for checking that the script finds the symbols).
2026-01-23 17:51:29 +01:00
Bartosz Sławecki
03e651d601
Programming FAQ: fix some punctuaction typos (GH-144058)
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
2026-01-23 17:34:21 +01:00
Seth Michael Larson
052e55e7d4
gh-144125: email: verify headers are sound in BytesGenerator
Co-authored-by: Denis Ledoux <dle@odoo.com>
Co-authored-by: Denis Ledoux <5822488+beledouxdenis@users.noreply.github.com>
Co-authored-by: Petr Viktorin <302922+encukou@users.noreply.github.com>
Co-authored-by: Bas Bloemsaat <1586868+basbloemsaat@users.noreply.github.com>
2026-01-23 14:59:35 +00:00
Yongtao Huang
f3dd0cae6c
Doc: fix incorrect reference in isasyncgenfunction docs (GH-144099)
Fix incorrect reference in isasyncgenfunction docs
2026-01-23 09:06:47 +01:00
Alex Willmer
f8262b84f5
gh-143513: Remove importlib.abc documentation for removed ABCs (#143605)
In 3.11 ResourceReader, Traversable, & TraversableResources moved from importlib.abc to importlib.resources.abc (commit e712a5b277866a71c195f38c1b5d87d9126dba3e).

In 3.12 old import locations were deprecated (commit 71848c960927af801656026203371c41ad139b5a).

In 3.14 backwards-compat support was removed (commit 0751511d24295c39fdf2f5b2255e3fa3d796ce4d).

Co-authored-by: Brett Cannon <brett@python.org>
2026-01-22 14:00:37 -08:00
Dimma Don't
5b2d49b7da
Add source links to documentation for Windows-specific modules (GH-130244) 2026-01-22 21:30:13 +00:00
Sam Gross
ee4e14aa4c
gh-143756: Avoid borrowed reference in SSL code (gh-143816)
GET_SOCKET() returned a borrowed reference, which was potentially
unsafe. Also, refactor out some common code.
2026-01-22 14:02:48 -05:00
Sam Gross
bcf9cb0217
gh-143756: Fix potential data race in SSLContext.load_cert_chain (gh-143818)
Concurrent calls to `load_cert_chain` caused data races in OpenSSL code.
2026-01-22 14:02:30 -05:00
Gregory P. Smith
a966d94e76
gh-144157: Optimize bytes.translate() by deferring change detection (GH-144158)
Optimize bytes.translate() by deferring change detection

Move the equality check out of the hot loop to allow better compiler
optimization. Instead of checking each byte during translation, perform
a single memcmp at the end to determine if the input can be returned
unchanged.

This allows compilers to unroll and pipeline the loops, resulting in ~2x
throughput improvement for medium-to-large inputs (tested on an AMD zen2).
No change observed on small inputs.

It will also be faster for bytes subclasses as those do not need change
detection.
2026-01-22 09:21:07 -08:00
stratakis
77bf4ba732
gh-142779: Initialize reserved field for proper padding (#142780)
The jitdump specification specifies a reserved field for padding.

Initialize it so no garbage data is embedded in the jitdump files.
2026-01-22 18:06:36 +01:00
Serhiy Storchaka
c5cfcdf16a
gh-67041: Allow to distinguish between empty and not defined URI components (GH-123305)
Changes in the urllib.parse module:

* Add option missing_as_none in urlparse(), urlsplit() and urldefrag(). If
  it is true, represent not defined components as None instead of an
  empty string.
* Add option keep_empty in urlunparse() and urlunsplit(). If it is
  true, keep empty non-None components in the resulting string.
2026-01-22 14:29:13 +02:00
Jeong, YunWon
67535ab2d2
gh-143001: Add @cpython_only to test_sys.test_current_frames() (#144004) 2026-01-22 12:08:08 +01:00
Mark Shannon
d77aaa7311
GH-139109: Partial reworking of JIT data structures (GH-144105)
* Halve size of buffers by reusing combined trace + optimizer buffers for TOS caching
* Add simple buffer struct for more maintainable handling of buffers
* Decouple JIT structs from thread state struct
* Ensure terminator is added to trace, when optimizer gives up
2026-01-22 10:55:49 +00:00
Petr Viktorin
fb690c38ca
gh-141004: Mark up constants for PyOS_double_to_string (GH-143867)
This ensures they show up as C macros in search and the Sphinx inventory.
2026-01-22 11:53:17 +01:00
Jeong, YunWon
c447d1bc14
gh-143959: Make _datetime optional for test_sys (GH-144003) 2026-01-22 12:52:16 +02:00
Rafael Weingartner-Ortner
0b5f8359c5
gh-143993: Document ways to disable remote debugging support (#143994)
Although PEP 768 mentions how to disable the mechanism of
remote debugging, it is not documented in the Python docs.

This change adds a note on how to disable remote debugging support
in a Python interpreter to the remote debugging how-to.

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2026-01-22 11:41:20 +01:00
Petr Viktorin
3e66171efa
gh-142913: Revert adding test_replaced_interpreter (GH-144110)
This partially reverts commit 4d5a676aa0811563ea78ae58ef89cdc0295bf7ed
(GH-142911)

The removed test fails when re-run in --huntrleaks mode.
2026-01-22 10:40:34 +01:00
reiden
0b08438ea6
gh-130415: Narrowing to constants in branches involving is comparisons with a constant (GH-143895) 2026-01-22 09:37:45 +00:00
Brett Cannon
6181b69970
GH-143941: Move WASI-related files to Platforms/WASI (GH-143942)
Along the way, leave a deprecated Tools/wasm/wasi/__main__.py behind for backwards-compatibility.

Co-authored-by: Zachary Ware <zachary.ware@gmail.com>
2026-01-21 14:31:58 -08:00
Serhiy Storchaka
cf71e34940
gh-77188: Add pickle tests for objects with slots (GH-144116) 2026-01-21 19:21:02 +02:00
Mark Shannon
9eab67d507
GH-138245: Perform boolean guards by testing a single bit, rather than a full pointer comparison. (GH-143810) 2026-01-21 15:58:27 +00:00
Steve Dower
f52af86cba
Update install manager docs (python/pymanager#227) (GH-144079) 2026-01-21 15:33:05 +00:00
CF Bolz-Tereick
5db331a561
gh-144030: Add check that argument is callable to Python version of functools.lru_cache (#144031)
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: AN Long <aisk@users.noreply.github.com>
2026-01-21 15:19:19 +01:00
Sergey B Kirpichev
4c7ec78092
gh-143869: Add PEP 757 functions to the limited API (#143906)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2026-01-21 14:47:14 +01:00
Yongtao Huang
4ef30a5871
gh-144085: Add {gi,cr}_suspended to inspect comments and generator/coroutine tests (#144086) 2026-01-21 13:33:56 +03:00
Serhiy Storchaka
9060b4abbe
gh-125346: Deprecate accepting standard Base64 alphabet when alternative alphabet is used (GH-141128)
Emit a warning in base64.urlsafe_b64decode() and base64.b64decode() when
the "+" or "/" characters occur in the Base64 data with alternative
alphabet if they are not the part of the alternative alphabet.

It is a DeprecationWarning in the strict mode (will be error) and
a FutureWarning in non-strict mode (will be ignored).
2026-01-21 09:41:58 +02:00
Tim Peters
48795b6460
GH-143948: Explain graphlib's cycle-finding code (#143950)
Explain topsort's cycle-finding algorithm, and why it's written that way.

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2026-01-20 19:28:48 -06:00
Seth Michael Larson
95746b3a13
gh-143919: Reject control characters in http cookies
Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com>
Co-authored-by: sobolevn <mail@sobolevn.me>
2026-01-20 21:23:42 +00:00
Seth Michael Larson
b234a2b675
gh-143923: Reject control characters in POP3 commands 2026-01-20 20:46:32 +00:00
Seth Michael Larson
f25509e78e
gh-143925: Reject control characters in data: URL mediatypes 2026-01-20 20:45:58 +00:00
Seth Michael Larson
6262704b13
gh-143921: Reject control characters in IMAP commands 2026-01-20 20:45:42 +00:00
Donghee Na
27a7160b8b
gh-141504: Move PYTHON_UOPS_OPTIMIZE to policy object (GH-144082) 2026-01-20 18:47:38 +00:00
Neil Schemenauer
48b6866047
gh-144054: no deferred refcount for untracked (gh-144081)
This reverts gh-144055 and fixes the bug in a different way.  Deferred
reference counting relies on the object being tracked by the GC,
otherwise the object will live until interpreter shutdown.  So, take
care that we do not enable deferred reference counting for objects that
are untracked.  Also, if a tuple has deferred reference counting
enabled, don't untrack it.
2026-01-20 10:01:09 -08:00
Sam Gross
43bb6300b3
gh-143939: Fix assignment to _PyThreadStateImpl.generator_return_kind (gh-143951)
The assignment to generator_return_kind has to be after any potentially
escaping calls to ensure that it's not overwritten.
2026-01-20 17:51:55 +00:00
Hugo van Kemenade
a126893fa8
gh-143960: Add support for OpenSSL 3.6, drop EOL 3.2 (#143961)
Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
2026-01-20 17:25:31 +02:00
Neil Schemenauer
795d5c5b44
gh-144054: shutdown fix for deferred ref counting (GH-144055)
When shutting down, disable deferred refcounting for all GC objects. It
is important to do this also for untracked objects, which before this
change were getting missed.

Small code cleanup:
We can remove the shutdown case disable_deferred_refcounting() call
inside scan_heap_visitor() if we are careful about it.  The key is
that frame_disable_deferred_refcounting() might fail if the object
is untracked.
2026-01-20 14:45:12 +01:00
Sergey B Kirpichev
31c81ab0a2
Update struct.__doc__: _Bool available unconditionally (#143716)
This amends commit a9296e7f3be4d6c22271b25c86467ff867c63bbb.
2026-01-20 13:09:20 +01:00
VanshAgarwal24036
fe629262c0
gh-144050: Fix stat.filemode pure Python file type detection (GH-144059) 2026-01-20 14:05:42 +02:00
Yongtao Huang
76b484b9d1
gh-143999: Fix: handle suspended state on types.coroutine wrappers (GH-144000) 2026-01-20 10:20:06 +00:00
AZero13
fa44efa0ef
gh-144023: Prevent follow_symlinks from being allowed with an fd of 0 (GH-144022)
The check was (fd > 0), should be (fd >= 0).
2026-01-20 11:50:51 +02:00
Terry Jan Reedy
fa3abf5a51
gh-143774 - Improve IDLE Format Paragraph doc (#143775)
Add a reminder to not rewrap code line to the Menu => Format => Reformat Paragraph entry.
In Editing and Nagivagion, add a new 'Format block' subsection that defines 'paragraph'
to better match what is dependably handled as more or less expected.
In particular, specify equal indents and that the resulting indent equals original indent.
Also mention that selections are expanded to complete lines and how to modify max length.

(Also fix a couple case errors in cross references.)
2026-01-20 04:07:29 -05:00
Serhiy Storchaka
71cbffde61
gh-143754: Add Tkinter methods pack_content(), place_content() and grid_content() (GH-143845)
They use Tk commands with new name like "pack content instead of old
"pack slaves".
2026-01-20 09:16:36 +02:00
Bodhi Silberling
05eab96435
Fix typo: 'exept' -> 'except' in Lib/dbm/dumb.py (GH-144060) 2026-01-20 08:50:34 +02:00
Yongtao Huang
e66597d6c8
gh-143874: Use self.message instead of raw print in _exec_in_closure() (#143875) 2026-01-19 16:11:07 -08:00
Bartosz Sławecki
f84ea11071
gh-102809: No longer mention Misc/gdbinit in the code (#143980)
Fix misleading comment on `PyObject_Dump`.
2026-01-19 18:36:55 +01:00
Adorilson Bezerra
3c9c3d33cb
gh-106318: Add examples for str.rpartition() method (#143891) 2026-01-19 15:15:55 +01:00
Yongtao Huang
375e372c66
gh-143689: Fix BufferedReader.read1 leaving object in reentrant state on error (#143690)
BufferedReader.read1() could leave the buffered object in a
reentrant (locked) state when an exception was raised while
allocating the output buffer.

This change ensures the internal buffered lock is always released
on error, keeping the object in a consistent state after failures.

Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
Co-authored-by: sobolevn <mail@sobolevn.me>
2026-01-19 15:09:30 +01:00
Aniket
72bacb0cd0
gh-142461: Move misplaced NEWS entries to an appropriate section (GH-143392) 2026-01-19 14:59:26 +02:00
Seth Michael Larson
17d1490aa9
gh-143935: Email preserve parens when folding comments (#143936)
Fix a bug in the folding of comments when flattening an email message
using a modern email policy. Comments consisting of a very long sequence of
non-foldable characters could trigger a forced line wrap that omitted the
required leading space on the continuation line, causing the remainder of
the comment to be interpreted as a new header field. This enabled header
injection with carefully crafted inputs.

Co-authored-by: Denis Ledoux <dle@odoo.com>
2026-01-19 12:38:22 +00:00
Serhiy Storchaka
bb2b9ba49d
gh-143897: Remove the isxidstart() and isxidcontinue() methods of unicodedata.ucd_3_2_0 (GH-143898)
They are now only exposed as the unicodedata function.
2026-01-19 12:37:41 +00:00
AZero13
7dca4e3af1
gh-142440: Fix _decimal builds configured with EXTRA_FUNCTIONALITY (GH-142441)
There was a typo in _decimal.Context.apply().
2026-01-19 14:19:20 +02:00
Serhiy Storchaka
813fc7a291
gh-143754: Modernize Tkinter docs (GH-143841)
Use more relevant terminology instead of "master"/"slave" widgets where possible.
2026-01-19 14:01:29 +02:00
b9788213
cb6a662bb0
gh-143866: Verify return value of pathlib.write_{bytes,text} methods in tests (#143870)
Co-authored-by: sobolevn <mail@sobolevn.me>
2026-01-19 14:24:20 +03:00
Sergey B Kirpichev
d8ab1c79b0
gh-75459: versionadded for PyObject_CallFinalizer*() API (GH-143982) 2026-01-19 11:22:16 +01:00
Bartosz Sławecki
59d3594ca1
gh-143831: Compare cells by identity in forward references (#143848) 2026-01-18 21:29:11 -08:00
🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко)
78b1370de9
Notify Sviat of GHA changes through codeowners (#143945)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2026-01-19 00:28:38 +02:00
Gregory P. Smith
c879b2a7a5
gh-141860: Add on_error= keyword arg to multiprocessing.set_forkserver_preload (GH-141859)
Add a keyword-only `on_error` parameter to `multiprocessing.set_forkserver_preload()`. This allows the user to have exceptions during optional `forkserver` start method module preloading cause the forkserver subprocess to warn (generally to stderr) or exit with an error (preventing use of the forkserver) instead of being silently ignored.

This _also_ fixes an oversight, errors when preloading a `__main__` module are now treated the similarly. Those would always raise unlike other modules in preload, but that had gone unnoticed as up until bug fix PR GH-135295 in 3.14.1 and 3.13.8, the `__main__` module was never actually preloaded.

Based on original work by Nick Neumann @aggieNick02 in GH-99515.
2026-01-18 14:04:18 -08:00
AN Long
54bedcf714
gh-144012: Check null binary op extend (#144014) 2026-01-19 02:38:37 +08:00
Bartosz Sławecki
63cc1257db
gh-143952: Fix asyncio tools to work with the new remote debugging API (#143954) 2026-01-17 19:16:12 -05:00
Shamil
7ca9e7ad05
Fix UBSan error in _testcapi: correct create_attr_from_spec signature (#143097)
Co-authored-by: sobolevn <mail@sobolevn.me>
2026-01-17 18:05:09 +00:00
Seth Michael Larson
f7fceed79c
gh-143916: Reject control characters in wsgiref.headers.Headers (GH-143917)
* Add 'test.support' fixture for C0 control characters
* gh-143916: Reject control characters in wsgiref.headers.Headers
2026-01-17 09:46:21 -08:00
Hai Zhu
b4b73245d8
gh-143421: Use new buffer to save optimized uops (GH-143682) 2026-01-17 15:52:16 +00:00
Peter Bierma
d51c01a271
gh-141004: Document PyException_HEAD and PyDescr_COMMON (GH-143896) 2026-01-17 10:39:29 -05:00
Hai Zhu
61ec66acd5
gh-143946: Show JitOptSymbol on abstract stack when set PYTHON_OPT_DEBUG > 4 (GH-143957) 2026-01-17 15:20:35 +00:00
Ken Jin
7e28ae550f
gh-142913: Export JIT functions for _testinternalcapi (#143958)
* Export JIT functions for _testinternalcapi

* Add testinternalcapi to paths to run JIT CI on
2026-01-17 13:31:38 +00:00
Hai Zhu
7d151e552c
gh-143943: add missing header file for pycore_optimizer_types.h (GH-143947) 2026-01-17 12:26:19 +00:00
Neil Schemenauer
bb25f7280a
gh-132657: Add maybe_enable_deferred_ref_count() (gh-142843)
If we are specializing to `LOAD_GLOBAL_MODULE` or `LOAD_ATTR_MODULE`, try
to enable deferred reference counting for the value, if the object is owned by
a different thread.  This applies to the free-threaded build only and should
improve scaling of multi-threaded programs.
2026-01-17 01:54:43 +00:00
Ken Jin
949b5ec8e6
gh-142913: Remove JIT interpreter for interpreter generator tests (GH-143944) 2026-01-17 00:34:39 +00:00
Hugo van Kemenade
3199cfcf76
gh-138122: Default highlight to sh (#143782) 2026-01-16 21:40:28 +02:00
Dino Viehland
5996636ab5
gh-142913: Update generated code that conflicted with other landed change (#143932)
Update test cases
2026-01-16 11:10:15 -08:00
Dino Viehland
4d5a676aa0
gh-142913: Add test case for interpreter generator w/ overridden opcodes (#142911)
Add test case for interpreter generator w/ overridden opcodes
2026-01-16 10:33:29 -08:00
Muneeb Ullah
3e93225798
gh-143674: Document F/D complex format characters in struct module (#143675)
Add documentation for the F (complex from two floats) and D (complex
from two doubles) format characters in the struct module docstring.
These format characters were implemented but not documented.

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
2026-01-16 18:09:56 +01:00
Serhiy Storchaka
21ed1e2a94
gh-65784: Add support for parametrized resource wantobjects in regrtests (GH-143570)
This allows to run Tkinter tests with the specified value of
tkinter.wantobjects, for example "-u wantobjects=0".
2026-01-16 18:38:38 +02:00
Seth Michael Larson
edeebe22cb
gh-143572: Run 'python3-libraries' fuzzer in CI using CIFuzz (#143749)
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <578543+webknjaz@users.noreply.github.com>
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2026-01-16 18:09:39 +02:00
Peter Bierma
19e64afddf
gh-141070: Rename PyUnstable_Object_Dump to PyObject_Dump (GH-142848) 2026-01-16 09:19:43 -05:00
Serhiy Storchaka
780e9692fe
gh-143672: Finish conversion of the struct module to Argument Clinic (GH-143857)
Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
2026-01-16 13:35:23 +02:00
Serhiy Storchaka
66680f1230
gh-143672: Add more tests for struct.pack_into() (GH-143901)
Add tests for negative offset, out of bound offset, invalid type of offset,
non-writeable buffer, non-continuous buffer, invalid type of buffer.

Repeat all tests for struct.Struct.pack_into().
2026-01-16 11:24:43 +00:00
Mark Shannon
ae53da5758
GH-143493: Conform to spec for generator expressions while supporting virtual iterators (GH-143569)
* Moves the `GET_ITER` instruction into the generator function preamble.
  This means the the iterable is converted into an iterator during generator
  creation, as documented, but keeps it in the same code object allowing
  optimization.
2026-01-16 09:11:58 +00:00
zhong
c461aa99e2
gh-143602: Fix duplicate buffer exports in io.BytesIO.write (#143629)
Fix an inconsistency issue in io.BytesIO.write() where the buffer was exported
twice, which could lead to unexpected data overwrites and position drift when
the buffer changes between exports.
2026-01-15 16:08:55 +01:00
Victor Stinner
3514ba2175
gh-142434: Use ppoll() if available in select.poll (#143529) 2026-01-15 13:49:46 +01:00
Petr Viktorin
a009e78b79
gh-141004: Mark up docs of old PyMem macros (GH-143783)
These had a docs-only deprecation notice since the first version of the docs in this repo. Nowadays we call things “soft deprecated” if there's just a note in the docs.

The deprecated directive needs a version, I went with the first one that had the notice (2.0; it's not in 1.6):
- https://docs.python.org/release/2.0/api/memoryInterface.html
- https://docs.python.org/release/1.6/api/memoryInterface.html

Since PEP 445, they are now direct aliases; there are no (additional) binary compatibility concerns over the preferred names.
2026-01-15 12:37:30 +01:00
Victor Stinner
1597d00d96
gh-80620: Fix test_time.test_gmtime() for 32-bit time_t (#143861) 2026-01-15 11:54:30 +01:00
Peter Gessler
f5685a266b
gh-80620: Support negative timestamps on windows in time.gmtime, time.localtime, and datetime module (#143463)
Previously, negative timestamps (representing dates before 1970-01-01) were
not supported on Windows due to platform limitations. The changes introduce a
fallback implementation using the Windows FILETIME API, allowing negative
timestamps to be correctly handled in both UTC and local time conversions.
Additionally, related test code is updated to remove Windows-specific skips
and error handling, ensuring consistent behavior across platforms.

Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-15 10:51:11 +01:00
Serhiy Storchaka
565685f6e8
Add regression test for add() after remove() with hash collision in set (GH-143781) 2026-01-15 10:41:08 +02:00
Muneeb Ullah
421bd1770a
gh-143834: Fix PyLong_AsNativeBytes docs for negative number padding (GH-143840) 2026-01-15 09:22:01 +01:00
sobolevn
bdba5f0db2
gh-143635: Fix crash in ga_repr_items_list (#143670) 2026-01-15 06:50:47 +03:00
Donghee Na
794f758cd8
gh-141504: Refactor policy object into a single opt_config (gh-143644) 2026-01-15 09:53:00 +09:00
Donghee Na
a73ba4d46e
gh-143632: Skip unittest for mmap.set_name at musl environment (gh-143839) 2026-01-15 09:52:29 +09:00
Serhiy Storchaka
b8e925b4f8
gh-141805: Fix crash after concurrent addition objects with the same hash to set (GH-143815)
This happens when the set contained several elements with the same hash,
and then some of them were removed.
2026-01-14 23:29:17 +02:00
Raymond Hettinger
f4de184980
gh-143825: Micro-optimizations to _make_key. (gh-143844) 2026-01-14 15:20:53 -06:00
Mohammad Miadh Angkad
499706b843
GH-143842: Make optimizer color table static (GH-143846) 2026-01-14 20:37:13 +00:00
Savannah Ostrowski
313d5cde5d
Add Savannah to CODEOWNERS for WASI (#143847) 2026-01-14 12:08:32 -08:00
Serhiy Storchaka
85013d7a55
Fix refleaks in new unicodedata classes added in gh-74902 (GH-143843) 2026-01-14 19:55:11 +00:00
Priyanshu Singh
14f96a8d8f
gh-143632: Disable PR_SET_VMA_ANON_NAME on musl libc (gh-143730) 2026-01-14 17:47:55 +00:00
Hugo van Kemenade
2b33b52219 Post 3.15.0a5 2026-01-14 19:30:22 +02:00
Hugo van Kemenade
d51cc01c19 Python 3.15.0a5 2026-01-14 16:41:32 +02:00
Serhiy Storchaka
bab1d7a561
gh-74902: Add Unicode Grapheme Cluster Break algorithm (GH-143076)
Add the unicodedata.iter_graphemes() function to iterate over grapheme
clusters according to rules defined in Unicode Standard Annex #29.

Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break()
and unicodedata.extended_pictographic() functions to get the properties
of the character which are related to the above algorithm.

Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>
2026-01-14 14:37:57 +00:00
Peter Bierma
0e0d51cdce
gh-141004: Document several soft-deprecated C APIs (GH-141634)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-14 08:20:33 -05:00
Serhiy Storchaka
a471a32f4b
gh-143214: Add the wrapcol parameter in binascii.b2a_base64() and base64.b64encode() (GH-143216) 2026-01-14 14:44:53 +02:00
Ken Jin
e370c8db52
gh-143123: Protect against recursive tracer calls/finalization (GH-143126)
* Stronger check for recursive traces

* Add a stop_tracing field

* Stop early when tracing exceptions
2026-01-14 12:23:14 +00:00
Mark Shannon
6db952eae9
GH-143613: Add colours and some more edges to executor visualization graph (GH-143809) 2026-01-14 11:34:58 +00:00
Duane Griffin
ce8f5f98c6
gh-106287: Do not write objects after an unmarshalling error (GH-132715)
Writing out an object may involve a slot lookup, which is not safe to do with
an exception raised. In debug mode an assertion failure will occur if this
happens.
2026-01-14 11:25:46 +00:00
Hai Zhu
94dbce1397
gh-138050: Use cold flag instead of warm flag in MAKE_WARM (GH-143827) 2026-01-14 10:27:33 +00:00
Petr Viktorin
7e8a1b5061
gh-141004: Document old provisional names for vectorcall/code API (GH-143568)
gh-141004: Document old provisional names for vectorcall & code API

For vectorcall API, the underscore-prefixed provisional names
continue to be available per PEP 590:
https://peps.python.org/pep-0590/#finalizing-the-api
Use `versionadded`, `versionchanged`, and `:no-typesetting:`
to mark this up.

For PyCode API, use `:no-typesetting:` rather than `index::` so that
these are semantically documented as C functions.
2026-01-14 10:54:45 +01:00
Russell Keith-Magee
1857a40807
gh-143460: Correct unlimited stack size skip for Emscripten (#143765)
Modifies the test skip for unlimited stack size to add Emscripten to the
exclusion list.
2026-01-14 06:03:04 +08:00
Ken Jin
2873c31edf
gh-131798: JIT optimizer: Support custom binary op and property frames (GH-143735) 2026-01-13 19:11:53 +00:00
Hugo van Kemenade
dfc66e5c8d Merge branch 'main' of https://github.com/python/cpython 2026-01-13 20:44:19 +02:00
Hugo van Kemenade
da0e8c39dc Post 3.15.0a4 2026-01-13 20:39:00 +02:00
Alper
fca7fec88c
gh-116738: Make lzma module thread-safe (#142947) 2026-01-13 14:02:27 +00:00
Bartosz Sławecki
865eb12e07
gh-143728: Keep TypedDict and NamedTuple in class role in docs (#143702) 2026-01-13 13:52:29 +00:00
Hugo van Kemenade
80e9eaf071
Bump GitHub Actions (#143757) 2026-01-13 15:43:26 +02:00
Yongtao Huang
f53a801e91
remove duplicate error constants in _winapi (#143684) 2026-01-13 19:11:38 +05:30
Lakshya Upadhyaya
95a17b4a85
gh-141045: Document that shutil.Error is a subclass of OSError (#141152) 2026-01-13 19:08:26 +05:30
Yongtao Huang
103a384bfd
gh-143249: Fix buffer leak when overlapped operation fails to start on windows (#143250) 2026-01-13 19:01:24 +05:30
Stan Ulbrych
1176facbf2
gh-141004: Don't trigger run-tests when Tools/check-c-api-docs/ignored_c_api.txt is changed (GH-143583) 2026-01-13 08:28:09 -05:00
Petr Viktorin
f0a0467c17
gh-143054: Disallow non-top-level Cut for now (GH-143622)
The behaviour of Cut in nested parentheses, Repeat, Opt, and similar
is somewhat chaotic. Apparently even the academic papers on PEG aren't
as clear as they could be.

And it doesn't really matter. Python only uses top-level cuts.
When that changes, we can clarify as much as necessary (and even
change the implementation to make sense for what we'll need).

Document that this is deliberately unspecified, and add a test to
make sure any decision is deliberate, tested and documented.
2026-01-13 13:21:59 +01:00
Yashraj
a7ba3b124f
gh-141004: Document symbol visibility macros (PyAPI_DATA, Py_EXPORTED_SYMBOL, Py_LOCAL_SYMBOL,Py_IMPORTED_SYMBOL) (GH-143508)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-13 13:15:02 +01:00
Hugo van Kemenade
43cd277ae9 Python 3.15.0a4 2026-01-13 12:26:56 +02:00
Cody Maloney
510ab7d6e1
gh-132108: Add Buffer Protocol support to int.from_bytes to improve performance (#132109)
Speed up conversion from `bytes-like` objects like `bytearray` while
keeping conversion from `bytes` stable.

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-13 10:33:26 +01:00
Stan Ulbrych
e5b5a15804
gh-141004: GHA: Run check-c-api-docs check on docs-only PRs (GH-143573)
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
2026-01-13 10:18:48 +01:00
Hugo van Kemenade
cbf9b8cc08
gh-143658: importlib.metadata: Use str.translate to improve performance of importlib.metadata.Prepared.normalized (#143660)
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com>
2026-01-13 08:54:15 +02:00
Raymond Hettinger
a6bc60da02
Update random combinatoric recipes and add tests (gh-143762) 2026-01-12 17:55:02 -06:00
Gregory P. Smith
298d5440eb
gh-143706: Fix sys.argv not set during multiprocessing forkserver __main__ preload (#143717)
The forkserver was not passing sys.argv to its main() function, causing
sys.argv to be empty during `__main__` module import in child processes. This
fixes a non-obvious regression inadvertently introduced by the gh-126631 main
preloading fix.
2026-01-12 15:47:21 -08:00
Łukasz Langa
d5882c5b70
Bump automatic certificate update on Windows during builds (GH-143741)
Without this, OpenSSL that we use to download external dependencies might use a stale certificate store and be unable to connect to servers. We need to use a Windows-specific HTTP client that uses CryptoAPI directly to trigger certificate updates.

We only do it on failure to avoid hitting servers twice. And we only do it once per each URL.
2026-01-13 00:01:01 +01:00
Savannah Ostrowski
0bee481576
Add Savannah to optimizer CODEOWNERS (#143761) 2026-01-12 14:56:05 -08:00
Ken Jin
5a45279320
gh-134584: Add more contributors for JIT refcounting work (GH-143760) 2026-01-12 22:14:52 +00:00
reiden
c556786b8b
gh-134584: Eliminate redundant refcounting in TO_BOOL_INT, TO_BOOL_LIST (GH-143759) 2026-01-12 21:56:26 +00:00
Raymond Hettinger
66e1399311
Minor readability/usability improvement to the recipes section (gh-143753) 2026-01-12 14:12:23 -06:00
Savannah Ostrowski
4766237d19
GH-43374: Revert "GH-43374: Fix urlretrieve reporthook to report actual bytes r… (#143711)
Revert "GH-43374: Fix urlretrieve reporthook to report actual bytes read (#142653)"

This reverts commit 68a01f901f446f71aac88431e01bb18b9fa35bd0.
2026-01-12 19:03:42 +00:00
Serhiy Storchaka
c559135c93
gh-143196: Fix crash in non-standard use of internal JSON encoder object (GH-143618)
The internal encoder object returned by undocumented function
json.encoder.c_make_encoder() (aka _json.make_encoder()) crashed
when it was called with non-zero second argument.
2026-01-12 18:23:05 +00:00
Aniket
3d44f0ab65
gh-142461: Move misplaced NEWS entries to an appropriate section (GH-143411) 2026-01-12 20:13:44 +02:00
Terry Jan Reedy
ec254e2b40
gh-139707: Copy-strip change to idle.rst into idlelib (#143718)
On Windows, running `python -c "from idlelib.help import copy_strip;
copy_strip()"` after idle.html is built results in
`f:\dev\3x\Doc\build\html\library\idle.html
copied to f:\dev\3x\Lib\idlelib\help.html`
This PR commits the html change to idlelib/help.html.
2026-01-12 12:55:06 -05:00
Sam Gross
7d155d7915
gh-140795: Remove 'exc' field in SSLObject (gh-143491)
The 'exc' field was used by our debug SSL callbacks. Keep the exception
in the normal per-thread state to avoid shared mutable state between
threads.

This also avoids a reference count leak if the Python callback raised an
exception because it can be called multiple times per SSL operation.
2026-01-12 17:14:13 +00:00
Lysandros Nikolaou
1de46715ec
gh-142518: Document thread-safety guarantees of list operations (#142519)
* Add everything to code blocks
* Improve wording around atomicity; specify exact types
* Better explain lock-free and atomicity
2026-01-12 17:43:05 +01:00
Nadeshiko Manju
e535bdb0a2
gh-134584: Eliminate redundant refcounting from _CONTAINS_{OP|OP_SET|OP_DICT} (GH-143731)
Signed-off-by: Manjusaka <me@manjusaka.me>
2026-01-12 15:47:31 +00:00
VanshAgarwal24036
c315748060
gh-143544: Fix possible use-after-free in the JSON decoder when JSONDecodeError disappears during raising it (#143561)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2026-01-12 16:39:32 +01:00
Pablo Galindo Salgado
fe78c1e749
gh-143253: Add libabigail suppression file for internal types (#143254)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2026-01-12 17:37:43 +02:00
Kuang Yu Heng
971f387bbb
gh-137113 docs: note readline no longer supported in REPL after 3.13 (GH-137142)
Add a note to the readline module documentation stating that
Python 3.13 and later no longer supports readline in the default REPL,
as per gh-118840. Includes workaround using PYTHON_BASIC_REPL.

Update tutorial to remove the reference, and use a different key to
test things out.

Signed-off-by: Kuang Yu Heng <yuheng3107@gmail.com>
2026-01-12 15:30:35 +00:00
Cajetan Rodrigues
054a565c64
gh-134584: JIT: Remove redundant refcount for _BINARY_OP_SUBSCR_DICT (GH-143724) 2026-01-12 14:13:55 +00:00
Yashraj
42f7c2dfba
gh-141004: Document PyUnicode_IS_COMPACT and PyUnicode_IS_COMPACT_ASCII macros (GH-143494) 2026-01-12 13:59:59 +01:00
Guo Ci
7f50a5febd
gh-140806: add docs for enum.bin function (#140807)
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2026-01-12 13:34:18 +02:00
Inada Naoki
43c76587c1
gh-143189: fix insertdict() for non-Unicode key (#143285) 2026-01-12 13:21:23 +02:00
Petr Viktorin
bd83a57463
gh-143578: Restore note about patchlevel.h (#143596)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2026-01-12 11:21:14 +00:00
Serhiy Storchaka
f3759d21dd
gh-142306: Improve errors for Element.remove() (GH-142308)
* Raise TypeError for non-element argument in the Python implementation
  too.
* Include the repr of the elements in the ValueError error message.
2026-01-12 11:49:18 +02:00
Serhiy Storchaka
dbd10a6c29
gh-142881: Fix concurrent and reentrant call of atexit.unregister() (GH-142901) 2026-01-12 10:45:10 +02:00
Serhiy Storchaka
5f28aa2f37
gh-143346: Fix calculation of the line width for wrapped Base64 in plistlib (GH-143347)
It was incorrect in case of mixed tabs and spaces in indentation.
2026-01-12 10:40:06 +02:00
Ken Jin
548526bbbe
gh-134584: JIT: Remove redundant refcounting for UNARY_{INVERT|NEGATIVE} (GH-143704) 2026-01-11 20:42:55 +00:00
Bartosz Sławecki
9d13ca97c1
gh-142972: Document arbitrary ordering in Path.glob and Path.rglob (GH-143025) 2026-01-11 14:36:47 -05:00
James Hilton-Balfe
265381b7e8
gh-128335: Make slice generic at runtime (#128336)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2026-01-11 11:27:24 -08:00
Bartosz Sławecki
620a5b9269
gh-127750: Fix annotations in singledispatchmethod signature tests (GH-143571)
These tests relied on a bug -- gh-84644, which is that singledispatch
doesn't verify the annotation is on the "first" parameter.
2026-01-11 21:24:19 +02:00
Ken Jin
9633f95b95
gh-139038: Update JIT perf figures for 3.14a4 (GH-143705) 2026-01-11 18:55:05 +00:00
Sergey B Kirpichev
23b93770f6
gh-143420: Clarify sequence behavior for slice indexes (#143422) 2026-01-11 19:17:47 +02:00
AN Long
e22b68568a
gh-86139: Correct NamedTuple and TypedDict's type in typing.rst (#143692)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2026-01-11 15:53:25 +00:00
Sergey B Kirpichev
515ae4078d
gh-78724: Raise RuntimeError's when calling methods on non-ready Struct()'s (GH-143643) 2026-01-11 15:52:01 +00:00
Serhiy Storchaka
d1282efb2b
gh-143638: Forbid cuncurrent use of the Pickler and Unpickler objects in C implementation (GH-143664)
Previously, this could cause crash or data corruption, now concurrent calls
of methods of the same object raise RuntimeError.
2026-01-11 14:01:13 +02:00
Manuel Jacob
03e6457096
Remove outdated comment about excess_args warning in object.__init__ (#143669)
The code emitting a warning was removed in 96384b93aae1d1e45dda21c4024d7d083c91626d.
2026-01-11 10:27:11 +03:00
Raymond Hettinger
75d73c3674
Sync the batched() example with the grouper() example. (gh-143676) 2026-01-10 22:00:13 -06:00
Raymond Hettinger
718c15fa95
Add derangements() recipe (gh-143671) 2026-01-10 19:47:27 -06:00
emmanuel
aa8578dc54
Doc: remove duplicate GitHub issue reference in "What's New in Python 3.13" (#143654) 2026-01-10 14:34:30 +01:00
Bénédikt Tran
ce6bae92da
gh-143377: fix crashes in _interpreters.capture_exception (#143418) 2026-01-10 12:37:54 +01:00
Nadeshiko Manju
e2f0160026
gh-143604: Hold strong reference to executor during JIT tracing (GH-143646)
Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2026-01-10 11:15:48 +00:00
Jeong, YunWon
e7f5ffa0de
gh-143641: Make ready_to_import always remove tempdir from sys.path (#143642)
Make ready_to_import always remove tempdir from sys.path
2026-01-10 12:51:42 +03:00
Heikki Toivonen
78e868fa28
gh-143586: Add Counter comparison tests to enforce missing element semantics (gh-143587) 2026-01-10 00:17:57 -06:00
Ken Jin
95259116ec
gh-142764: Restore REPLACE_OPCODE_IF_EVALUATES_PURE optimization for some ops (GH-143335) 2026-01-09 23:18:22 +00:00
Ken Jin
e0fb278064
gh-143421: Allocate all JIT state in one go (GH-143626) 2026-01-09 19:00:49 +00:00
Petr Viktorin
1932127ec7
gh-141004: Document remaining constants from patchlevel.h (GH-143598)
Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-09 17:40:07 +00:00
Semyon Moroz
499d3a8d50
gh-136438: Make sure test_interpreters pass with all optimization levels (GH-136499) 2026-01-09 18:06:54 +01:00
Serhiy Storchaka
66bca383bd
gh-143006: Fix and optimize mixed comparison of float and int (GH-143084)
When comparing negative non-integer float and int with the same number
of bits in the integer part, __neg__() in the int subclass returning
not an int caused an assertion error.

Now the integer is no longer negated. Also, reduced the number of
temporary created Python objects.
2026-01-09 19:06:45 +02:00
Ken Jin
b852236b26
gh-143421: Lazily allocate tracer code and opt buffers (GH-143597) 2026-01-09 16:56:19 +00:00
Jan André Reuter
61e036691c
gh-143460: Skip infinite recusion tests for infinite stack size (#143606)
Avoid tests being killed due to OOM on Linux if a system is configured with
'ulimit -s unlimited' by skipping tests relying on infinite recursion.

While unclear if Python should support 'ulimit -s unlimited', we should at
least try to avoid failing a PGO build running tests due to an unlimited
stack size being set.

Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
2026-01-09 16:11:37 +00:00
Adorilson Bezerra
c696f33d9e
gh-106318: Add examples for str.rfind() method (#143582) 2026-01-09 16:24:31 +01:00
Adorilson Bezerra
6d6c7ed737
gh-106318: Improve str.removeprefix() and str.removesuffix() docs (#143580)
Add doctest role and 'See also'.
2026-01-09 16:22:50 +01:00
Victor Stinner
ba10100c39
gh-143191: Use _PyOS_MIN_STACK_SIZE in _thread.stack_size() (#143601)
The stack size must be at least _PyOS_MIN_STACK_SIZE+SYSTEM_PAGE_SIZE
bytes.
2026-01-09 16:08:40 +01:00
A.Ibrahim
a4086d7f89
gh-142829: Fix use-after-free in Context.__eq__ via re-entrant ContextVar.set (#142905) 2026-01-09 17:57:34 +05:30
AN Long
a9ca49d9c6
gh-72239: Add note distinguishing concurrent.futures.Future from asyncio.Future (#143288) 2026-01-09 17:48:32 +05:30
Victor Stinner
39a2bcf949
gh-143547: Fix PyErr_FormatUnraisable() fallback (#143557)
Hold a strong reference to 'hook' while calling the default
unraisable took to log hook failure.

Fix test_sys.UnraisableHookTest: use the right decorator function to
disable colors. Previously, tests were always skipped.
2026-01-09 13:16:22 +01:00
zhong
6d54b6ac7d
gh-143378: Fix use-after-free when BytesIO is concurrently mutated during write operations (GH-143408)
PyObject_GetBuffer() can execute user code (e.g. via __buffer__), which may
close or otherwise mutate a BytesIO object while write() or writelines()
is in progress. This could invalidate the internal buffer and lead to a
use-after-free.

Ensure that PyObject_GetBuffer() is called before validation checks.
2026-01-09 13:50:56 +02:00
Henry Chen
dcdb23f9db
gh-143429: Use compile-time NaN encoding detection for test_struct (#143432) 2026-01-09 10:46:48 +00:00
Petr Viktorin
234a15dc4e
gh-141004: Improve make check-c-api-docs (GH-143564)
- Gather all documented names into a set in a single pass.
  This makes the check much faster.

- Do not match substrings (e.g. documenting `PyErr_WarnEx`
  doesn't mean that `PyErr_Warn` is documented)

- Consider `PY`-prefixed names (a lot of old macros use this)
2026-01-09 11:16:50 +01:00
Yongtao Huang
fd6d41b292
Test: remove unintended unittest.main() in live collector tests (#143590) 2026-01-09 08:49:54 +00:00
Adorilson Bezerra
af9f783a7e
gh-106318: Add examples for str.replace() method (GH-143581) 2026-01-09 09:01:00 +01:00
Savannah Ostrowski
68a01f901f
GH-43374: Fix urlretrieve reporthook to report actual bytes read (#142653) 2026-01-09 00:32:05 +00:00
Brett Cannon
dfeefbe8ea
GH-139686: Revert "gh-139686: Make reloading a lazy module no-op (GH-139857)" (#143584)
This reverts commits 57db12514ac686f0a752ec8fe1c08b6daa0c6219 and 0a97941245f1dda6d838f9aaf0512104e5253929.
2026-01-08 22:08:03 +00:00
Russell Keith-Magee
b54a1d272e
Add Emscripten test skips for recently added tests. (#143551)
Some recent changes introduced tests that can't run on Emscripten.
This change adds test skips for those tests.
2026-01-09 05:47:38 +08:00
Sam Gross
e2f15aec16
gh-120321: Make gen.gi_frame.clear() thread-safe (gh-143112) 2026-01-08 14:45:54 -05:00
Hai Zhu
aeb3403563
gh-143421: Move JitOptContext from stack allocation to per-thread heap allocation (GH-143536)
* move JitOptContext to _PyThreadStateImpl
* make _PyUOpInstruction buffer a part of _PyThreadStateImpl

Co-authored-by: Kumar Aditya <kumaraditya@python.org>
2026-01-08 19:38:21 +00:00
Heikki Toivonen
cea2d2475d
gh-143445: Optimize deepcopy for 1.04x speedup (#143449)
Gains according to pyperformance:

```
deepcopy:
Mean +- std dev: 411 us +- 2 us -> 396 us +- 3 us: 1.04x faster
Significant (t=28.94)

deepcopy_reduce:
Mean +- std dev: 4.38 us +- 0.05 us -> 4.23 us +- 0.04 us: 1.04x faster
Significant (t=20.05)
```

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2026-01-08 16:28:02 +01:00
Sam Gross
8cf5c4d89a
gh-142908: Don't use DK_IS_UNICODE in interpreter (gh-142909)
`DK_IS_UNICODE()` includes split keys and we don't want to specialize on
those accesses..
2026-01-08 09:32:20 -05:00
Sam Gross
49c3b0a67a
gh-142095: Use thread local frame info in py-bt and py-bt-full when available (gh-143371)
In optimized and `-Og` builds, arguments and local variables are frequently
unavailable in gdb. This makes `py-bt` fail to print anything useful. Use the
`PyThreadState*` pointers `_Py_tss_gilstate` and `Py_tss_tstate` to find the
interpreter frame if we can't get the frame from the
`_PyEval_EvalFrameDefault` call.

Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-08 09:18:24 -05:00
Alex Willmer
5462002bbe
gh-143513: Clarify changed argument name of ResourceReader.is_resource (GH-143523) 2026-01-08 08:40:17 -05:00
Victor Stinner
efaa56f73c
gh-143528: Fix test_time.test_thread_time() (#143558)
Tolerate 100 ms instead of 20 ms to support slow CIs.
2026-01-08 14:05:57 +01:00
Aniket
f3e069a7ab
gh-140025: Fix queue.SimpleQueue.__sizeof__() to return correct size (#143137) 2026-01-08 12:40:25 +00:00
Serhiy Storchaka
c07e5ec0a9
gh-143553: Add support for parametrized resources in regrtests (GH-143554)
For example, "-u xpickle=2.7" will run test_xpickle only against Python 2.7.
2026-01-08 13:51:38 +02:00
Bartosz Sławecki
6c9f7b4406
gh-143517: Fix an edge case in rewriting stringified starred annotations (#143518) 2026-01-07 19:41:27 -08:00
VanshAgarwal24036
67d3d0344f
gh-143089: Fix ParamSpec default examples to use list instead of tuple (#143179) 2026-01-07 19:32:44 -08:00
Sam Gross
f11f5ebfe6
gh-129824: Fix data race on runtime->gilstate.check_enabled (gh-143530) 2026-01-07 17:56:14 -05:00
Tomas R.
228d95582e
Fix a typo in STORE_SLICE docs (#143500) 2026-01-07 23:36:22 +01:00
Dino Viehland
bfc3d8d77f
gh-143531: Use macro to check if PEP 523 is hooked (#143532)
Use macro to check if PEP 523 is hooked
2026-01-07 13:26:19 -08:00
Savannah Ostrowski
9a3263ff8f
GH-142950: Process format specifiers before colourization in argparse help (#142960) 2026-01-07 17:39:47 +00:00
Sam Gross
d04394929b
gh-117657: Remove remaining TSan suppressions (gh-143499) 2026-01-07 17:34:05 +00:00
Adorilson Bezerra
51a56a3a7b
gh-106318: Add examples for str.casefold() and str.lower() methods (#142154)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
2026-01-07 16:16:25 +00:00
T. Wouters
b2827de18f
gh-141536: Fix a crash when running test_capi *after* test_code in the same process (#141537) 2026-01-07 16:40:33 +01:00
Łukasz Langa
b3e4a3462f
gh-138568: Make help mode in PyREPL not exit on empty line input (#143512)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: yihong0618 <zouzou0208@gmail.com>
2026-01-07 16:30:58 +01:00
Giampaolo Rodola
a1eedaee98
gh-94172: delete mention of ssl_version from ftplib doc (#143515)
`FTP_TLS.ssl_version` class attribute was removed in https://github.com/python/cpython/commit/ef0e72b31d2.
2026-01-07 15:11:50 +01:00
Petr Viktorin
8565ddd288
gh-141671: PyMODINIT_FUNC: apply __declspec(dllexport) on Windows (GH-141672) 2026-01-07 13:09:48 +01:00
Serhiy Storchaka
b866a1c73f
gh-75572: Fix test_xpickle and disable it by default (GH-143503) 2026-01-06 23:51:12 +00:00
Adorilson Bezerra
4fb6a31bce
gh-106318: Add example for str.index() (#137023) 2026-01-06 23:42:11 +02:00
Nadeshiko Manju
0a5c04a5ce
gh-134584: Eliminate redundant refcounting from TO_BOOL_STR (GH-143417)
Signed-off-by: Manjusaka <me@manjusaka.me>
2026-01-06 21:11:53 +00:00
Sam Gross
98e55d70bc
gh-132070: Fix PyObject_Realloc thread-safety in free threaded Python (gh-143441)
The PyObject header reference count fields must be initialized using
atomic operations because they may be concurrently read by another
thread (e.g., from `_Py_TryIncref`).
2026-01-06 15:55:37 -05:00
Ken Jin
df355348f0
gh-142982: Specialize CALL_FUNCTION_EX (GH-143391) 2026-01-06 20:34:08 +00:00
Andrii Hrimov
ff7d1cec41
gh-133315: Document that compat32 should not be used with MIMEPart/EmailMessage (#143307) 2026-01-06 15:18:37 -05:00
Xiao Yuan
51227b6b1a
Doc: Fix missing closing parenthesis in argparse example (#143488) 2026-01-06 09:53:43 -08:00
Kumar Aditya
74bb3ca1f8
gh-142863: optimize list and set calls with generator expressions (#142864) 2026-01-06 23:23:28 +05:30
ADITYA RAI
e79c9b7031
gh-142302: Fix mkstemp() documentation: clarify file descriptor inheritance behavior (#142338)
The documentation incorrectly stated that the file descriptor is not
inherited by child processes. In reality, the close-on-exec flag (when
available) only prevents inheritance across exec() calls, not fork().

Co-authored-by: Victor Stinner <vstinner@python.org>
2026-01-06 17:08:25 +00:00
Ken Jin
8735daf3e8
gh-75572: Forward-port test_xpickle from Python 2 to 3 (GH-22452)
Move data classes used in tests to separate file test_picklecommon.py,
so it can be imported in old Python versions.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2026-01-06 19:03:03 +02:00
Adorilson Bezerra
7b0a372b20
gh-106318: Add examples for str.isprintable() (#140043)
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
2026-01-06 17:43:44 +01:00
Ken Jin
90c44bc803
gh-131798: Support generator frames in the JIT optimizer (GH-143340) 2026-01-06 16:39:57 +00:00
Kumar Aditya
faa3dc7c64
gh-143469: enable LOAD_ATTR_MODULE specialization even if __getattr__ is defined (#143470) 2026-01-06 22:09:18 +05:30
Gleb Smirnoff
05406b221d
gh-142991: socketmodule: fixed getsockaddrlen() for PF_DIVERT socket (#142993) 2026-01-06 17:25:29 +01:00
reiden
54f1ed0299
gh-134584: Eliminate redundant refcounting in TO_BOOL_ALWAYS_TRUE (GH-143427)
Co-authored-by: Ken Jin <kenjin@python.org>
2026-01-06 16:11:56 +00:00
Ken Jin
d9c1235db4
gh-139038: Add macOS performance numbers for the JIT (GH-143479) 2026-01-06 15:19:35 +00:00
Sam Gross
841b7482dd
gh-143424: Fix assertion in _PyMutex_LockTimed (gh-143439)
The assertion doesn't necessarily hold for `threading.Lock`, so allow
the lock to be unlocked if `_PY_LOCK_PYTHONLOCK` is set on the flags.
2026-01-06 09:40:14 -05:00
Clay Dugo
d745b60ef2
gh-143474: Add os.RWF_ATOMIC constant (GH-143475) 2026-01-06 09:35:51 -05:00
Sam Gross
71119a164c
gh-129824: Fix data race in _PyBuiltins_AddExceptions with subinterpreters (gh-143446) 2026-01-06 09:15:55 -05:00
Semyon Moroz
dd750b3485
gh-143331: Schedule to remove format "N" for Decimal (#143372) 2026-01-06 15:12:05 +02:00
Hugo van Kemenade
efb4e6c733
Allow Android to fail CI (GH-143471) 2026-01-06 12:31:21 +00:00
Łukasz Langa
7dae1077cd
gh-143394: On macOS, run main PyREPL tests as "Apple Terminal" as well (GH-143461) 2026-01-06 11:10:18 +01:00
Serhiy Storchaka
4d21297d28
gh-41779: Allow defining any __slots__ for a class derived from tuple (GH-141763) 2026-01-06 11:36:00 +02:00
Kumar Aditya
d6f77e6a3f
gh-116738: make entering of contextvars.Context thread safe (#143074) 2026-01-06 12:24:02 +05:30
Emma Smith
12d363bb66
gh-134740: Document distribution discovery in importlib.metadata (#134751)
Document the following items listed in `__all__` but missing from
documentation:

- `distributions()`: mentioned in doc strings as well
- `DistributionFinder`: mentioned but didn't have it's own :class:
  entry
- `DistributionFinder.Context`: mentioned but didn't have it's own :class:
  entry)
- `Distribution.discover()`: mentioned in doc strings

Closes #134740
Partially addresses #110937

---------

Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
2026-01-06 01:25:30 +00:00
Chris Eibl
04ace41fe2
gh-143394: filter for turning off automatic margins in test_no_newline (GH-143433) 2026-01-05 22:46:08 +01:00
Sam Gross
7a572d9f21
gh-143108: Don't instrument some faulthandler related functions for TSan (#143450) 2026-01-05 22:13:29 +01:00
Ritam Pal
240a6c3262
gh-143434: Fix typo in zstd documentation (#143437) 2026-01-05 11:55:11 -08:00
Ori Avtalion
1569275117
gh-138205: explicit mention to mmap.mmap.resize in "Porting to Python 3.15" notes (#143440) 2026-01-05 20:27:45 +01:00
Łukasz Langa
4f9a8d075e
gh-69605: In test_pyrepl.test_already_imported*, invalidate FS import caches (#143436)
This makes the test robust against file systems with low timestamp resolution, which otherwise would fail to re-import in the tight loop of the test.
2026-01-05 18:00:48 +01:00
Loïc Simon
bfac54d861
gh-69605: Check for already imported modules in PyREPL module completion (GH-139461)
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
2026-01-05 16:18:54 +01:00
Denizhan Dakılır
68fcb958eb
gh-143082: Make readline work when sys.stdin is explicitly passed to pdb (#143083) 2026-01-04 20:39:11 -08:00
Bénédikt Tran
c99f766743
gh-143309: fix test_execve_env_concurrent_mutation_with_fspath_posix buildbot failure (#143415) 2026-01-04 21:44:47 +01:00
Chris Eibl
e6bfe4d886
gh-139757: Add BINARY_OP_SUBSCR_USTR_INT (GH-143389) 2026-01-04 14:14:27 +00:00
Ken Jin
6116d70bbd
gh-143403: Fix a UAF in _BINARY_OP_INPLACE_ADD_UNICODE (GH-143404)
Fix a UAF in `_BINARY_OP_INPLACE_ADD_UNICODE`
2026-01-04 13:10:39 +00:00
Bénédikt Tran
9609574e7f
gh-143309: fix UAF in os.execve when the environment is concurrently mutated (#143314) 2026-01-03 23:05:57 +01:00
Aaron Wieczorek
6c53af18f6
gh-143308: fix UAF when PickleBuffer is concurrently mutated in a callback (#143312)
Co-authored-by: Aaron Wieczorek <woz@Aarons-MacBook-Pro.local>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2026-01-03 23:01:59 +01:00
László Kiss Kollár
ef3b8829e4
gh-142927: Clarify pstats file output in docs and CLI (#143388)
When running the `profiling.sampling` module in pstats mode, the output
can be emitted in two different ways: text to stdout or a binary file
when the `--output` argument is set.

The current documentation and help text is confusing as it does not
distinguish between these two output formats so it may be surprising to
the user to get different formats depending whether `--output` is set or not.
2026-01-03 21:16:29 +00:00
Donghee Na
12283f6373
gh-141504: Factor out tracing and optimization heuristics into a single object (gh-143381) 2026-01-03 15:22:14 +00:00
Jeong, YunWon
3c56f9e2cc
gh-143001: Add @cpython_only to test_threading.ThreadTests.test_PyThreadState_SetAsyncExc (gh-143002) 2026-01-04 00:01:49 +09:00
Pablo Galindo Salgado
27434c68f8
gh-138122: Skip threads on EPERM in blocking mode profiler (GH-143368)
When using blocking mode in the remote debugging profiler, ptrace calls
to seize threads can fail with EPERM if the thread has exited between
listing and attaching, is in a special kernel state, or is already being
traced. Previously this raised a RuntimeError that was caught by the
Python sampling loop,and retried indefinitely since EPERM is
a persistent condition that will not resolve on its own.

Treat EPERM the same as ESRCH by returning 1 (skip this thread) instead
of -1 (fatal error). This allows profiling to continue with the threads
that can be traced rather than entering an endless retry loop printing
the same error message repeatedly.
2026-01-03 15:37:16 +01:00
Olga Matoula
2c39b9d2f2
gh-136924: Suspend REPL colorizing when in a REPL interactive command (GH-136926)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2026-01-03 14:35:34 +01:00
Bartosz Sławecki
6d05e55de0
Fix dunder name typo in compiler code comment (#143374) 2026-01-03 10:08:42 +00:00
Bartosz Sławecki
e7c542de5f
gh-140648: Make asyncio REPL respect the -I flag (isolated mode) (#143045) 2026-01-03 13:33:04 +05:30
Kumar Aditya
abdbe0b807
gh-142615: disallow multiple initializations of asyncio.Task and asyncio.Future (#142616) 2026-01-03 13:27:02 +05:30
Aaron Wieczorek
b538c2832d
gh-143361: Pass PY_VECTORCALL_ARGUMENTS_OFFSET in _Py_CallBuiltinClass_StackRefSteal (GH-143367)
Co-authored-by: Aaron Wieczorek <woz@Aarons-MacBook-Pro.local>
2026-01-02 20:43:00 +00:00
Tomas R.
136f6d8355
gh-76187: Document the c typecode for multiprocessing.Array. (#132504)
* Document the `c` typecode for `multiprocessing.Array`.

* Add quotes

* Mention that 'w' is not supported
2026-01-02 20:18:32 +01:00
Ken Jin
f7a03bb944
gh-134584: JIT: Remove redundant refcount from STORE_FAST (GH-143336) 2026-01-02 18:22:21 +00:00
Donghee Na
98258326a9
no-issue: Fix optimizer_generator.py to print abstract_uop_name properly (gh-143366) 2026-01-02 17:18:15 +00:00
Stan Ulbrych
0417dabe3f
gh-138897: Use _pyrepl.pager for _sitebuiltins._Printer (GH-138898)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2026-01-02 16:24:48 +01:00
Chris Eibl
ef6f92a2a6
gh-139262: Prevent swallowing REPL input on Windows (GH-139263)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2026-01-02 16:17:45 +01:00
Johann Christensen
864c5985ea
gh-143148: Replace pre-commit with prek in CI (#143149)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2026-01-02 16:58:23 +02:00
Łukasz Langa
08a17ed061
gh-128067: In test_pyrepl, discover escape sequences from terminfo instead of using hard-coded values (#143356) 2026-01-02 15:42:04 +01:00
Donghee Na
61f2ad9a3a
gh-139757: Fix unintended bytecode specialization for non-ascii string (gh-143352) 2026-01-02 14:05:08 +00:00
Jan-Eric Nitschke
8a2deea1fc
gh-128067: Fix pyrepl overriding printed output without newlines (#138732)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2026-01-02 14:04:37 +01:00
Pablo Galindo Salgado
09ce592499
gh-142927: Hide _sync_coordinator frames from profiler output (#143337) 2026-01-02 12:09:36 +00:00
Donghee Na
315f474d11
gh-139757: Add _GUARD_NOS_COMPACT_ASCII for compact ascii optimization (gh-143332) 2026-01-02 20:58:14 +09:00
Alex Waygood
18f3c59e57
Update docstrings of typing.Hashable, typing.List, etc., to reflect their deprecation (#143295) 2026-01-02 11:31:41 +00:00
Bénédikt Tran
9712dc1d9e
gh-143310: fix crash in Tcl object conversion with concurrent mutations (#143321) 2026-01-02 10:51:05 +01:00
Gregory P. Smith
61fc72a4a4
gh-124951: Optimize base64 encode & decode for an easy 2-3x speedup [no SIMD] (GH-143262)
Optimize base64 encoding/decoding by eliminating loop-carried dependencies. Key changes:
- Add `base64_encode_trio()` and `base64_decode_quad()` helper functions that process complete groups independently
- Add `base64_encode_fast()` and `base64_decode_fast()` wrappers
- Update `b2a_base64` and `a2b_base64` to use fast path for complete groups

Performance gains (encode/decode speedup vs main, PGO builds):
```
             64 bytes    64K        1M
  Zen2:      1.2x/1.8x   1.7x/2.8x  1.5x/2.8x
  Zen4:      1.2x/1.7x   1.6x/3.0x  1.5x/3.0x  [old data, likely faster]
  M4:        1.3x/1.9x   2.3x/2.8x  2.4x/2.9x  [old data, likely faster]
  RPi5-32:   1.2x/1.2x   2.4x/2.4x  2.0x/2.1x
```

Based on my exploratory work done in https://github.com/python/cpython/compare/main...gpshead:cpython:claude/vectorize-base64-c-S7Hku

See PR and issue for further thoughts on sometimes MUCH faster SIMD vectorized versions of this.
2026-01-01 22:03:05 -08:00
Pablo Galindo Salgado
6b9a6c6ec3
gh-138122: Move local imports to module level in sampling profiler (#143257) 2026-01-02 02:31:39 +00:00
Pablo Galindo Salgado
e5ad7b7694
gh-138122: Integrate live profiler TUI with _colorize theming system (#142360)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2026-01-01 21:10:52 +00:00
Pablo Galindo Salgado
513ae175bb
gh-142927: Fix heatmap caller navigation for interior lines (#143180) 2026-01-01 19:05:59 +00:00
ivonastojanovic
5d133351c6
gh-142927: Auto-open HTML output in browser after generation (#143178) 2026-01-01 19:05:45 +00:00
Ken Jin
faa26044ce
gh-134584: Fix _CALL_BUILTIN_O test to reflect real-world usage (GH-143333)
Fix test to reflect real-world usage
2026-01-01 18:54:49 +00:00
Nadeshiko Manju
d00d39f58e
gh-134584: Eliminate redundant refcounting from _LOAD_ATTR_SLOT (GH-143320)
Signed-off-by: Manjusaka <me@manjusaka.me>
Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2026-01-01 17:27:02 +00:00
Donghee Na
1fb8e0eb51
gh-134584: Eliminate redundant refcounting from _CALL{_BUILTIN_O, _METHOD_DESCRIPTOR_O} (GH-143330)
Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2026-01-01 17:25:38 +00:00
Bénédikt Tran
422ca074bc
Amend NEWS entries for PRs GH-139553 and GH-142790 (#143329) 2026-01-01 14:16:26 +00:00
Bartosz Sławecki
2d9f4e357a
gh-143048: Remove outdated mention to curses in the "Interactive Mode" docs (#143049) 2026-01-01 12:52:21 +01:00
Bénédikt Tran
7f6c16a956
gh-142830: prevent some crashes when mutating sqlite3 callbacks (#143245) 2026-01-01 11:55:05 +01:00
Lakshya Upadhyaya
c5215978eb
gh-140920: remove incorrect mentions to concurrent.futures.interpreter.ExecutionFailed (#141723)
Remove documentation for inexistant `concurrent.futures.interpreter.ExecutionFailed`
and replace its occurrences by `concurrent.interpreters.ExecutionFailed` since this
is the documented exception.
2025-12-31 21:45:41 +01:00
AN Long
3c4429f65a
gh-135852: Remove out of tree pywin32 dependency for NTEventLogHandler (GH-137860)
Add RegisterEventSource(), DeregisterEventSource(), ReportEvent()
and a number of EVENTLOG_* constants to _winapi.
2025-12-31 12:50:50 +02:00
Sam Gross
96ab379dca
gh-140795: Keep 'err' in local variable in _ssl.c (gh-143275)
The error return code doesn't need to be mutable state on the SSLSocket.
This simplifes thread-safety and avoids potential reentrancy issues.
2025-12-30 19:45:44 -05:00
Sam Gross
469fe33edd
gh-143121: Avoid thread leak in configure (gh-143122)
If you are building with `--with-thread-sanitizer` and don't use the
suppression file, then running configure will report a thread leak.

Call `pthread_join()` to avoid the thread leak.
2025-12-30 19:45:23 -05:00
Gregory P. Smith
04899b8539
gh-115634: document ProcessPoolExecutor max_tasks_per_child bug (GH-140897) 2025-12-30 15:24:32 -08:00
Serhiy Storchaka
aa8a43d179
gh-143237: Fix support of named pipes in the rotating logging handlers (GH-143259)
This fixes regression introduced in GH-105887.
2025-12-30 14:56:29 +00:00
Adam Turner
7e3a5a7e79
gh-130167: Add a What's New entry for changes to `textwrap.{de,in}dent` (#131924)
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-30 14:57:28 +02:00
Hugo van Kemenade
0aedf2f9cf
gh-143284: Temporarily install Sphinx<9 to fix Chinese search (#143286) 2025-12-30 14:31:41 +02:00
dgpb
23ad9c5d01
gh-142939: difflib.get_close_matches performance (#142940) 2025-12-30 07:15:59 +00:00
Sebastian Rittau
ef834dee89
gh-128546: Document that getaddrinfo() can return raw data (#128547)
Document that getaddrinfo() can return raw data

This is the case for IPv6 addresses if Python was compiled with
--disable-ipv6.
2025-12-29 21:23:30 -08:00
Thanos
79c03ac001
gh-69686: Remove untrue part of __import__ replacement docs (#143261)
Remove untrue part of `__import__` replacement docs

The original statement effectively says that replacing `__import__` at global scope affects import statements, and not only that, but only import statements within the rest of the executing module. None of that has been true since at least Python 2.7, I think.

This was likely missed in python/cpython#69686.
2025-12-29 21:16:54 -08:00
Bénédikt Tran
b6b0e14b3d
gh-143200: fix UAFs in Element.__{set,get}item__ when the element is concurrently mutated (#143226) 2025-12-29 18:30:51 +01:00
Ken Jin
6cb245d260
gh-143183: Link trace to side exits, rather than stop (GH-143268) 2025-12-29 15:10:42 +00:00
Samuel
f37f57dfe6
gh-131421: Fix ASDL kw_defaults being expr* instead of expr?* (GH-133773)
Also fix docs ASDL highlighting.
2025-12-29 13:43:09 +02:00
Ken Jin
daa9aa4c0a
gh-143183: Rewind stop tracing to previous target (GH-143187)
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
2025-12-28 22:12:31 +00:00
Tomas R.
713684de53
gh-131798: Remove bounds check when indexing into tuples with a constant index (#137607)
* Remove bounds check when indexing into tuples with a constant index

* Add news entry

* fixup after rebase
2025-12-28 22:06:06 +01:00
Loïc Simon
c3febba73b
gh-140870: Full coverage for _pyrepl._module_completer (#143244)
Full coverage for _pyrepl._module_completer

Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
2025-12-28 20:06:06 +01:00
wangxiaolei
0efbad60e1
gh-142994, gh-142996: document missing async generator and coroutine field entries in inspect (#142997) 2025-12-28 19:03:30 +01:00
Prithviraj Chaudhuri
fa9a4254e8
gh-142195: Fixed Popen.communicate indefinite loops (GH-143203)
Changed condition to evaluate if timeout is less than or equals to 0. This is needed for simulated time environments such as Shadow where the time will match exactly on the boundary.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2025-12-28 16:57:44 +00:00
AN Long
c3bfe5d5aa
gh-63016: fix failing mmap.flush tests on FreeBSD (#143230)
Fix `mmap.flush` tests introduced in 1af21ea32043ad5bd4eaacd48a1718d4e0bef945
where some flag combinations are not supported on FreeBSD.
2025-12-28 16:36:52 +01:00
Fatih Çelik
3ca1f2a370
gh-143241: Fix infinite loop in zoneinfo._common.load_data (#143243)
Correctly reject truncated TZif files in `ZoneInfo.from_file`.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2025-12-28 15:48:43 +01:00
Pablo Galindo Salgado
3ccc76f036
gh-143228: Fix UAF in perf trampoline during finalization (#143233) 2025-12-28 13:50:23 +00:00
Pablo Galindo Salgado
836b2810d5
gh-136186: Fix more flaky tests in test_external_inspection (#143235) 2025-12-28 12:52:32 +00:00
Serhiy Storchaka
522563549a
gh-143003: Fix possible shared buffer overflow in bytearray.extend() (GH-143086)
When __length_hint__() returns 0 for non-empty iterator, the data can be
written past the shared 0-terminated buffer, corrupting it.
2025-12-28 12:30:36 +00:00
Ken Jin
23abbf1f2b
gh-139922: Link to results in MSVC tail calling in What's New 3.15 (GH-143242)
Link to results in MSVC tail calling for whats new in 3.15
2025-12-28 12:15:24 +00:00
Bénédikt Tran
61ee04834b
gh-142557: fix UAF in bytearray.__mod__ when object is mutated while formatting %-style arguments (#143213) 2025-12-27 14:57:13 +00:00
Bénédikt Tran
84fcdbd86e
gh-142664: fix PyObject_Hash invokation post GH-143217 (#143223) 2025-12-27 14:30:09 +00:00
Pablo Galindo Salgado
3a728e5f93
gh-131591: Do not free page caches that weren't allocated (#143205) 2025-12-27 13:38:11 +00:00
Bénédikt Tran
00e24b80e0
gh-142664: fix UAF in memoryview.__hash__ via re-entrant data's __hash__ (#143217) 2025-12-27 13:12:03 +00:00
Bénédikt Tran
7726119651
gh-138122: fix AC warnings in Modules/_remote_debugging/module.c (#143218) 2025-12-27 12:57:03 +00:00
Bénédikt Tran
9976c2b634
gh-143195: fix UAF in {bytearray,memoryview}.hex(sep) via re-entrant sep.__len__ (#143209) 2025-12-27 13:32:52 +01:00
Rafael Fontenelle
f5e11facf2
no-issue: Fix override value in os.rst (gh-123522) 2025-12-27 19:48:01 +09:00
AN Long
1af21ea320
gh-63016: Add flags parameter on mmap.flush (#139553)
Co-authored-by: Victor Stinner <vstinner@python.org>
2025-12-27 10:33:56 +00:00
SYan212
57d569942c
Fix typos in docs (#143193) 2025-12-27 09:52:28 +02:00
Duane Hilton
5d1e78f7b5
gh-143181: Fix 'overriden' -> 'overridden' in c-api/module.rst (#143182)
* Doc: Fix typo 'overriden' -> 'overridden' in c-api/module.rst

* Fix 'overriden' -> 'overridden' in tests
2025-12-27 02:23:57 -05:00
Pablo Galindo Salgado
54362898f3
gh-140739: Fix missing exception on allocation failure in BinaryWriter (#143204) 2025-12-27 01:39:21 +00:00
Marta Gómez Macías
9d92ac1225
gh-143040: Exit taychon live mode gracefully and display profiled script errors (#143101) 2025-12-27 00:36:15 +00:00
Hai Zhu
a1c6308346
gh-134584: Eliminate redundant refcounting from IS_OP (GH-143171)
Eliminate redundant refcounting from IS_OP
2025-12-26 20:30:02 +00:00
Hai Zhu
b3f2d80569
gh-134584: Eliminate redundant refcounting from _COMPARE_OP_X (GH-143186) 2025-12-26 16:12:28 +00:00
Pablo Galindo Salgado
d3d4cf9432
gh-140739: Fix crashes from corrupted remote memory (#143190) 2025-12-26 16:06:48 +00:00
Yongtao Huang
de22e718bb
Remove redundant pycore_optimizer.h includes (#143184)
`pycore_optimizer.h` was included redundantly in
Objects/frameobject.c and Python/instrumentation.c.
Both includes are unnecessary and can be safely removed.
No functional change.

Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
2025-12-26 11:11:11 +00:00
László Kiss Kollár
888d101445
gh-138122: Remove default duration for statistical profiling (#143174)
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
2025-12-25 19:21:16 +00:00
Hugo van Kemenade
ea3fd785cb
gh-142927: Tachyon: Fix contrast ratio in top panel (#142936) 2025-12-25 18:28:13 +00:00
ivonastojanovic
59ede34c8c
gh-138122: Convert GIL/GC/exception stats from tiles to progress bars (#143177) 2025-12-25 18:22:54 +00:00
Yongtao Huang
b9a4806430
gh-143164: Fix incorrect error message for ctypes bitfield overflow (GH-143165)
Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
2025-12-25 19:08:43 +02:00
Peter Bierma
8611f74e08
gh-142975: During GC, mark frozen objects with a merged zero refcount for destruction (GH-143156) 2025-12-25 16:31:41 +00:00
Yongtao Huang
579c5b496b
gh-143145: Fix possible reference leak in ctypes _build_result() (GH-143131)
The result tuple was leaked if __ctypes_from_outparam__() failed for any item.

Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
2025-12-25 14:24:25 +00:00
Hauke D
8d46f961c3
gh-143103: Added pad parameter to base64.z85encode() (GH-143106)
This makes it analogous to a85encode() and b85encode() and allows the
user to more easily meet the Z85 specification, which requires input
lengths to be a multiple of 4.
2025-12-25 13:34:44 +02:00
kaushal trivedi
86d904588e
gh-143004: Fix possible use-after-free in collections.Counter.update() (GH-143044)
This happened when the Counter was mutated when incrementing
the value for an existing key.
2025-12-25 09:13:39 +00:00
Ken Jin
cf6758ff9e
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124)
These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-24 22:03:00 +00:00
Sam Gross
594a4631c3
gh-120321: Fix TSan reported races on gi_frame_state (gh-143128) 2025-12-24 16:10:43 -05:00
Cody Maloney
305aff0a66
Move News for gh-142560 to Core and Builtins (GH-143154) 2025-12-24 21:03:10 +02:00
R. David Murray
7342890ed7
gh-142517: Fix typo in news item. (#143150) 2025-12-24 13:14:23 -05:00
Stan Ulbrych
3509fa5a12
gh-143135: Fix sys.flags.inspect when PYTHONINSPECT=0 (GH-143136) 2025-12-24 19:56:59 +02:00
Tom Kuson
84b7e6970f
gh-140717: Add exc_text to LogRecord attributes table (GH-140718) 2025-12-24 17:30:20 +00:00
Pablo Galindo Salgado
7c44f37170
gh-138122: Extend binary profiling format with full source location and opcode (#143088)
Co-authored-by: Stan Ulbrych <stan@ulbrych.org>
2025-12-24 16:15:11 +00:00
R. David Murray
1e17ccd030
Correctly fold unknown-8bit originating from encoded words. (#142517)
The unknown-8bit trick was designed to deal with unknown bytes in an
ASCII message, and it works fine for that.  However, I also tried to
extend it to handle bytes that can't be decoded using the charset
specified in an encoded word, and there it fails because there can be
other non-ASCII characters that were *successfully* decoded.  The fix is
simple: do the unknown-8bit encoding using the utf-8 codec.  This is
especially appropriate since anyone trying to do recovery on an unknown
byte string will probably attempt utf-8 first.
2025-12-24 09:14:39 -05:00
László Kiss Kollár
d4dc3dd9aa
gh-138122: Replace --interval with --sampling-rate (#143085) 2025-12-24 13:46:33 +00:00
Sam Gross
e8e044eda3
gh-143100: Fix memcpy data race in setobject.c (gh-143127) 2025-12-24 08:02:19 -05:00
Sam Gross
4ee6929d60
gh-143121: Skip test that leak threads under TSan (gh-143125) 2025-12-24 08:02:02 -05:00
Sam Gross
57937a8e5e
gh-142145: Avoid timing measurements in quadratic behavior test (gh-143105)
Count the number of Element attribute accesses as a proxy for work done.
With double the amount of work, a ratio of 2.0 indicates linear scaling
and 4.0 quadratic scaling. Use 3.2 as an intermediate threshold.
2025-12-24 08:01:45 -05:00
Pablo Galindo Salgado
9af7a20cae
gh-136186: Fix flaky tests in test_external_inspection (#143110) 2025-12-24 12:38:17 +00:00
Diego Russo
fc2f0fea6b
JIT: Move executor to a register (#143072) 2025-12-24 09:44:16 +00:00
Sam Gross
50ecd6b880
gh-143108: Don't instrument faulthandler.c for TSan (#143109)
The dumping of tracebacks has data races and that's okay (it's best
effort).
2025-12-24 02:12:55 +01:00
Hai Zhu
cc48bf0fde
gh-134584: Eliminate redundant refcounting from _BINARY_OP_SUBSCR_TUPLE_INT (GH-143094) 2025-12-23 21:47:12 +00:00
Sam Gross
cbe0cb779a
gh-143100: Add temporary suppression for set_swap_bodies (gh-143114) 2025-12-23 15:52:59 -05:00
AN Long
c2202a7e66
gh-109263: Start process from spawn context in multiprocessing no longer have side effect (GH-135813) 2025-12-23 19:28:32 +00:00
Diego Russo
450e836aef
JIT: don't leak shim memory when shutting down the interpreter (#142984) 2025-12-23 17:50:00 +00:00
Mark Shannon
20aeb3a463
GH-143026: Fix assertion error in executor management. (GH-143104) 2025-12-23 17:19:34 +00:00
Tomas R.
25c294b6ea
gh-134584: Eliminate redundant refcounting from _CALL_TYPE_1 (GH-135818) 2025-12-23 17:01:10 +00:00
Ken Jin
c8b80f5e23
gh-134584: Add another contributor to whats new 3.15 (GH-143107)
Add another contributor to whats new 3.15
2025-12-23 16:47:46 +00:00
Hugo van Kemenade
f783cc37eb
Update pre-commit with zizmor and Ruff fixes (#143095) 2025-12-23 17:51:02 +02:00
Victor Stinner
6536fab194
gh-130796: Undeprecate locale.getdefaultlocale() (#143069) 2025-12-23 16:31:10 +01:00
Ken Jin
c4ab024530
gh-142448: Disable JIT tracing when monitoring is enabled (GH-142842) 2025-12-23 11:27:23 +00:00
Pablo Galindo Salgado
28da1fb7c0
gh-142368: Fix transient error handling in inspection tests (#143093) 2025-12-23 11:01:32 +00:00
Pablo Galindo Salgado
81c8eb85e1
gh-138122: Add blocking mode for accurate stack traces in Tachyon (#142998) 2025-12-23 10:49:47 +00:00
Stan Ulbrych
f9704f1d84
gh-84232: Fix pydoc docs.python.org link generation (#139995)
Co-authored-by: Éric <merwok@netwok.org>
2025-12-23 10:29:58 +02:00
Zheng Yu
a273bc99d2
gh-122431: Correct the non-negative error message in readline.append_history_file (GH-143075)
"positive" -> "non-negative", since zero is included.
2025-12-22 23:35:23 -05:00
Hai Zhu
5b5ee3c4bf
gh-134584: Eliminate redundant refcounting from _LOAD_ATTR_WITH_HINT (GH-143062)
Eliminate redundant refcounting from _LOAD_ATTR_WITH_HINT
2025-12-23 00:28:08 +00:00
Pablo Galindo Salgado
9e51301234
gh-138122: Allow tachyon to write and read binary output (#142730) 2025-12-22 23:57:20 +00:00
Ken Jin
714037ba84
gh-139922: Add tail call for MSVC for whats new in 3.15 (GH-143087) 2025-12-22 23:56:26 +00:00
Chris Eibl
be3c131640
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068)
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Co-authored-by: Brandt Bucher <brandt@python.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-22 23:01:34 +00:00
Ken Jin
665d2807a0
gh-139109: Add terminator to JIT code when halting due to invalid dependencies (#143033)
* Add terminator to JIT code when  halting due to invalid dependencies

* 📜🤖 Added by blurb_it.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2025-12-22 20:54:47 +00:00
Stan Ulbrych
3c0888b25b
gh-89152: Note truth testing exception in stdtypes.rst (#137640)
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2025-12-22 11:36:44 -08:00
AZero13
a88d1b8dab
gh-143010: Prevent a TOCTOU issue by only calling open once (#143011)
* gh-143010: Prevent a TOCTOU issue by gh-143010: Prevent a TOCTOU issue by only calling open once

RDM: per  AZero13's research the 'x' option did not exist when this code was written,  This
modernization can thus drop the fd trick in _create_carefully and just use open with 'x' to achieve the same goal more securely.

Co-authored-by: sobolevn <mail@sobolevn.me>
2025-12-22 12:48:11 -05:00
Chris Eibl
700e9fad70
GH-142513: fix missing return in executor_clear (GH-143073)
fix missing return in executor_clear
2025-12-22 17:10:52 +00:00
Kumar Aditya
487e91c120
gh-129069: fix more thread safety issues in list (#143019) 2025-12-22 21:45:28 +05:30
Kumar Aditya
e728b006de
gh-143057: avoid locking in tracemalloc C-APIs when it is not enabled (#143065) 2025-12-22 21:08:07 +05:30
Ken Jin
9ded3dd4e9
gh-142476: Fix Windows crashing with JIT (GH-143021)
Fix Windows crashing with JIT
2025-12-22 14:57:13 +00:00
Hugo van Kemenade
ff7f62eb23
gh-142927: Tachyon: Comma separate thousands and fix singular/plurals (#142934) 2025-12-22 14:15:57 +00:00
Yongtao Huang
3960878143
Remove unreachable code in mmapmodule error path on Windows (GH-143063)
mmapmodule: remove unreachable code in Windows error path

Remove an unreachable `return NULL` after `PyErr_SetFromWindowsErr()` in
the Windows mmap resize error path.

Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
2025-12-22 11:16:50 +01:00
Bartosz Sławecki
6213a512bf
gh-143046: Make asyncio REPL respect the -q flag (quiet mode) (#143047) 2025-12-22 07:05:15 +00:00
saucoide
09044dd42b
gh-80744: do not read .pdbrc twice when cwd == $home (#136816) 2025-12-21 08:58:07 -08:00
Gregory P. Smith
b8d3fddba6
gh-70647: Better promote how to safely parse yearless dates in datetime. (GH-116179)
* gh-70647: Better promote how to safely parse yearless dates in datetime.

Every four years people encounter this because it just isn't obvious.
This moves the footnote up to a note with a code example.

We'd love to change the default year value for datetime but doing
that could have other consequences for existing code.  This documented
workaround *always* works.

* doctest code within note is bad, dedent.

* Update to match the error message.

* remove no longer referenced footnote

* ignore the warning in the doctest

* use Petr's suggestion for the docs to hide the warning processing

* cover date.strptime (3.14) as well
2025-12-20 22:47:40 -08:00
Gregory P. Smith
8d2d7bb2e7
gh-142145: relax the no-longer-quadratic test timing (#143030)
* gh-142145: relax the no-longer-quadratic test timing

* require cpu resource
2025-12-20 23:42:06 +00:00
Sam Gross
2b4feee648
gh-122581: Use parser mutex in default build for subinterpreters (gh-142959) 2025-12-20 15:37:31 -05:00
Sam Gross
7607712b61
gh-120321: Avoid -Wunreachable-code warning on Clang (gh-143022) 2025-12-20 14:42:12 -05:00
Hai Zhu
3cc57505e5
gh-142834: pdb commands command should use last available breakpoint (#142835) 2025-12-20 09:27:34 -08:00
AZero13
5989095dfd
gh-143012: use Py_ssize_t cast for PyBytes_FromStringAndSize (#143013) 2025-12-20 10:37:10 +03:00
Hugo van Kemenade
5b5263648f
gh-142927: Tachyon: Start with user's default light/dark theme (#142987) 2025-12-20 02:36:09 +01:00
Sam Gross
e46f28c6af
gh-129069: Fix listobject.c data races due to memmove (gh-142957)
The use of memmove and _Py_memory_repeat were not thread-safe in the
free threading build in some cases. In theory, memmove and
_Py_memory_repeat can copy byte-by-byte instead of pointer-by-pointer,
so concurrent readers could see uninitialized data or tearing.

Additionally, we should be using "release" (or stronger) ordering to be
compliant with the C11 memory model when copying objects within a list.
2025-12-19 18:06:47 -05:00
Sam Gross
4ea3c1a047
gh-120321: Fix TSan reported race in gen_clear_frame (gh-142995)
TSan treats compare-exchanges that fail as if they are writes
so there is a false positive with the read of gi_frame_state in
gen_close.
2025-12-19 17:33:49 -05:00
Sam Gross
08bc03ff2a
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599)
This makes generator frame state transitions atomic in the free
threading build, which avoids segfaults when trying to execute
a generator from multiple threads concurrently.

There are still a few operations that aren't thread-safe and may crash
if performed concurrently on the same generator/coroutine:

 * Accessing gi_yieldfrom/cr_await/ag_await
 * Accessing gi_frame/cr_frame/ag_frame
 * Async generator operations
2025-12-19 19:10:37 +00:00
Shamil
e2a7db7175
gh-142476: fix memory leak when creating JIT executors (GH-142492) 2025-12-19 19:07:11 +00:00
Ken Jin
6b4bc6e6a2
gh-134584: JIT: Borrow references for immortal promoted globals (GH-142921)
JIT: Borrow references for immortal promoted globals
2025-12-19 19:06:34 +00:00
stratakis
6a4f10325d
gh-142776: Ensure fp file descriptor is closed on all code paths in import.c (GH-142777) 2025-12-19 10:14:52 -08:00
Ken Jin
786f464c74
gh-142961: Fix constant folding len(tuple) in JIT (GH-142963) 2025-12-19 17:43:36 +00:00
Petr Viktorin
049c2526bf
gh-134160: Start "Extending and embedding" with a Diataxis-style tutorial (GH-142314)
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Éric <merwok@netwok.org>
Co-authored-by: Daniele Nicolodi <daniele@grinta.net>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
2025-12-19 17:48:34 +01:00
Diego Russo
685272eb8a
JIT: Rename trampoline.c to shim.c (#142974) 2025-12-19 14:39:41 +00:00
Rogdham
4aef138325
gh-136282: Configparser: create unnamed sections via mapping protocol access (GH-136313) 2025-12-19 13:44:03 +01:00
Andrej
610aabfef2
gh-142527: Docs: Clarify that random.seed() discards the sign of an integer input (#142483)
If *a* is an integer, the sign of *a* is discarded in the C source code. Clarify this behavior to prevent foot guns, where a common use case might naively assume that flipping the sign will produce different sequences (e.g. for a train/test split of a synthetic data generator in machine learning).

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2025-12-19 00:29:35 -08:00
wangxiaolei
220f0b1077
gh-142560: prevent use-after-free in search-like methods by exporting buffer in bytearray (#142938) 2025-12-19 08:02:23 +00:00
Savannah Ostrowski
1391ee664c
GH-134584: Remove redundant refcount for BINARY_OP_SUBSCR_STR_INT (#142844) 2025-12-18 21:29:54 +00:00
Ethan Furman
e79c39101a
gh-118342: [Enum] update docs (GH-137290)
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
2025-12-18 18:31:37 +00:00
LloydZ
33d94abafd
gh-134584: Eliminate redundant refcounting from _BINARY_OP_SUBSCR_LIST_INT (GH-142926) 2025-12-18 18:25:36 +00:00
Sam Gross
f54d44d333
gh-129068: Make range iterators thread-safe (gh-142886)
Now that we specialize range iteration in the interpreter for the common
case where the iterator has only one reference, there's not a
significant performance cost to making the iteration thread-safe.
2025-12-18 13:11:51 -05:00
Kumar Aditya
e22c49522b
gh-142890: remove unnecessary interp parameter from dict functions and _PyDict_NotifyEvent (#142923) 2025-12-18 22:48:56 +05:30
Serhiy Storchaka
4a8ecbad80
gh-142681: Move NormalizationTest-3.2.0.txt to more safe place. (GH-142935) 2025-12-18 18:04:28 +01:00
Mark Shannon
e4058d7cb1
GH-142513: Reimplement executor management (GH-142931)
* Invalidating an executor does not cause arbitrary code to run
* Executors are only freed at safe points
2025-12-18 16:43:44 +00:00
Donghee Na
14f0b5191a
gh-142419: Add mmap.set_name method for user custom annotation (gh-142480) 2025-12-18 23:33:49 +09:00
Hugo van Kemenade
d2abd5733b
gh-76007: Deprecate VERSION in xml.etree.ElementTree & version in xml.sax.expatreader & xml.sax.handler (#142898) 2025-12-18 14:22:23 +00:00
Max R
0f01530bd5
Fix typo in format_string docstring (GH-136742) 2025-12-18 14:43:19 +01:00
Bartosz Sławecki
ddfc155d3a
gh-142784: make the asyncio REPL call loop.close() at exit (#142785) 2025-12-18 13:00:12 +00:00
MonadChains
1c544acaa5
gh-124098: Fix incorrect inclusion of handler methods without protocol prefix in OpenerDirector (GH-136873) 2025-12-18 13:50:05 +01:00
Donghee Na
71a7cb8887
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825) 2025-12-18 21:33:18 +09:00
James
fc80096a07
gh-137063: Document that ast node types replaced by Constant are no longer available (#137064) 2025-12-18 13:17:42 +01:00
ivonastojanovic
cbc0851ada
gh-138122: Improve bytecode panel (#142910)
The bytecode panel appears when a user generates a heatmap with
--opcodes and clicks the button to unfold a line and display the
bytecode instructions. Currently, an empty space appears on the
left where the line number, self, and total columns are displayed.
This area should instead extend those columns, rather than leaving
a gap.
2025-12-18 11:43:39 +00:00
Ken Jin
8b64dd853d
gh-139757: Treat call specially in JIT assembly backend optimizer on x86-64 and AArch64 (GH-142907)
Co-authored-by: Savannah Ostrowski <savannah@python.org>
2025-12-17 22:47:47 +00:00
Sam Gross
6e625f87d2
gh-129748: Remove TSan suppression for mi_block_set_nextx (gh-142887)
The function was already changed to use a relaxed atomic store in gh-134238.
2025-12-17 15:27:04 -05:00
Savannah Ostrowski
92243dc62c
GH-100964: Fix reference cycle in exhausted generator frames (#141112) 2025-12-17 19:21:45 +00:00
Sam Gross
25397f9541
gh-142766: Clear frame when generator.close() is called (gh-142838) 2025-12-17 13:06:32 -05:00
Pablo Galindo Salgado
77b56eafde
gh-91048: Refactor common data into context object in Modules/_remote_debugging (#142879) 2025-12-17 17:43:52 +00:00
Ken Jin
fba4584ffc
gh-142849: Fix segfault in executor_to_gv (GH-142885)
Fix segfault in `executor_to_gv`
2025-12-17 17:05:21 +00:00
sobolevn
e61a447d0e
gh-142873: Do not check for PyContextVar_CheckExact twice in PyContextVar_Set (#142874) 2025-12-17 19:41:36 +03:00
Damian Birchler
77c8e6a2b8
gh-142876: remove reference to thread in documentation of asyncio.Queue.shutdown (#142888) 2025-12-17 16:40:03 +00:00
Petr Viktorin
7d81eab923
gh-142225: Add PyABIInfo_VAR to to _testcapimodule & _testinternalcapi (GH-142833) 2025-12-17 16:33:09 +01:00
Pablo Galindo Salgado
568a819f67
gh-138122: Validate base frame before caching in remote debugging frame cache (#142852) 2025-12-17 15:12:28 +00:00
Benjamin Johnson
2b466c47c3
gh-112127: Fix possible use-after-free in atexit.unregister() (GH-114092)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2025-12-17 17:09:57 +02:00
Ken Jin
49627dc991
Use other name for JIT contributor (#142877) 2025-12-17 14:21:02 +00:00
Keming
d4095f25e8
gh-142654: show the clear error message when sampling on an unknown PID in tachyon (#142655)
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
2025-12-17 14:15:22 +00:00
Savannah Ostrowski
1fc3039d71
gh-139038: Add JIT What's New for 3.15 (#142845)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-17 12:14:44 +00:00
Mark Shannon
c7dcb26520
GH-142621: JIT: Avoid memory load for symbols within 4GB on AArch64 (GH-142820) 2025-12-17 12:07:07 +00:00
sobolevn
248eb3efb3
gh-142859: Add Tools/check-c-api-docs to mypy check (#142860) 2025-12-17 13:23:30 +02:00
Lysandros Nikolaou
1e9a0ee682
gh-140374: Add glossary entries related to multithreading (#140375)
---------

Co-authored-by: Daniele Parmeggiani <8658291+dpdani@users.noreply.github.com>
2025-12-17 12:09:51 +01:00
yihong
454485e564
gh-139743: Avoid print twice verbose version for sqlite tests (GH-142850)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-12-17 09:44:47 +00:00
wangxiaolei
8307a14d0e
gh-142783: Fix possible use after free in zoneinfo module (GH-142790) 2025-12-17 08:35:08 +00:00
Kumar Aditya
4fd006e712
gh-142752: add more thread safety tests for mock (#142791) 2025-12-17 13:17:12 +05:30
Jason R. Coombs
c35b812e77
gh-142836: Avoid /proc fd pipes on Solaris (#142853) 2025-12-17 07:38:00 +00:00
Ken Jin
89729f2ef7
gh-142543: Mark tracer functions as Py_NO_INLINE (GH-142846) 2025-12-17 00:12:32 +00:00
Nadeshiko Manju
4345253981
gh-134584: Eliminate redundant refcounting from _STORE_ATTR_WITH_HINT (GH-142767)
Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 22:21:04 +00:00
ivonastojanovic
8c87bcd7f2
gh-138122: Update Tachyon dark theme colors (#142841) 2025-12-16 20:58:56 +00:00
Nadeshiko Manju
6ee51a36b3
gh-134584: Eliminate redundant refcounting from _LOAD_ATTR_INSTANCE_VALUE (GH-142769)
Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 20:39:20 +00:00
Mark Shannon
92d4aeafd5
GH-142629: JIT: Fix out of bounds memory read in lltrace (GH-142821)
JIT: Fix out of bounds memory read in lltrace
2025-12-16 19:57:15 +00:00
Gabriele N. Tornetta
16a305f152
Make RESUME monitoring more readable and robust (GH-142136) 2025-12-16 16:23:27 +00:00
Edward Xu
a043407510
gh-142495: Make defaultdict keep existed value when racing with __missing__ (GH-142668) 2025-12-16 17:04:20 +02:00
Hugo van Kemenade
47ec96f133 Post 3.15.0a3 2025-12-16 16:31:52 +02:00
689 changed files with 57445 additions and 15029 deletions

6
.gitattributes vendored
View File

@ -83,8 +83,10 @@ Include/opcode.h generated
Include/opcode_ids.h generated
Include/token.h generated
Lib/_opcode_metadata.py generated
Lib/keyword.py generated
Lib/idlelib/help.html generated
Lib/keyword.py generated
Lib/pydoc_data/topics.py generated
Lib/pydoc_data/module_docs.py generated
Lib/test/certdata/*.pem generated
Lib/test/certdata/*.0 generated
Lib/test/levenshtein_examples.json generated
@ -92,6 +94,8 @@ Lib/test/test_stable_abi_ctypes.py generated
Lib/test/test_zoneinfo/data/*.json generated
Lib/token.py generated
Misc/sbom.spdx.json generated
Modules/_testinternalcapi/test_cases.c.h generated
Modules/_testinternalcapi/test_targets.h generated
Objects/typeslots.inc generated
PC/python3dll.c generated
Parser/parser.c generated

20
.github/CODEOWNERS vendored
View File

@ -63,8 +63,8 @@
.azure-pipelines/ @AA-Turner
# GitHub & related scripts
.github/ @ezio-melotti @hugovk @AA-Turner
Tools/build/compute-changes.py @AA-Turner
.github/ @ezio-melotti @hugovk @AA-Turner @webknjaz
Tools/build/compute-changes.py @AA-Turner @hugovk @webknjaz
Tools/build/verify_ensurepip_wheels.py @AA-Turner @pfmoore @pradyunsg
# Pre-commit
@ -143,6 +143,9 @@ Misc/externals.spdx.json @sethmlarson
Misc/sbom.spdx.json @sethmlarson
Tools/build/generate_sbom.py @sethmlarson
# ABI check
Misc/libabigail.abignore @encukou
# ----------------------------------------------------------------------------
# Platform Support
@ -173,9 +176,10 @@ Tools/wasm/config.site-wasm32-emscripten @freakboy3742 @emmatyping
Tools/wasm/emscripten @freakboy3742 @emmatyping
# WebAssembly (WASI)
Tools/wasm/wasi-env @brettcannon @emmatyping
Tools/wasm/wasi.py @brettcannon @emmatyping
Tools/wasm/wasi @brettcannon @emmatyping
Platforms/WASI @brettcannon @emmatyping @savannahostrowski
Tools/wasm/wasi-env @brettcannon @emmatyping @savannahostrowski
Tools/wasm/wasi.py @brettcannon @emmatyping @savannahostrowski
Tools/wasm/wasi @brettcannon @emmatyping @savannahostrowski
# Windows
PC/ @python/windows-team
@ -290,9 +294,9 @@ InternalDocs/jit.md @brandtbucher @savannahostrowski @diegorusso @AA-T
# Micro-op / μop / Tier 2 Optimiser
Python/optimizer.c @markshannon @Fidget-Spinner
Python/optimizer_analysis.c @markshannon @tomasr8 @Fidget-Spinner
Python/optimizer_bytecodes.c @markshannon @tomasr8 @Fidget-Spinner
Python/optimizer_symbols.c @markshannon @tomasr8 @Fidget-Spinner
Python/optimizer_analysis.c @markshannon @tomasr8 @Fidget-Spinner @savannahostrowski
Python/optimizer_bytecodes.c @markshannon @tomasr8 @Fidget-Spinner @savannahostrowski
Python/optimizer_symbols.c @markshannon @tomasr8 @Fidget-Spinner @savannahostrowski
# Parser, Lexer, and Grammar
Grammar/python.gram @pablogsal @lysnikolaou

View File

@ -20,7 +20,7 @@ jobs:
issues: write
timeout-minutes: 5
steps:
- uses: actions/github-script@v7
- uses: actions/github-script@v8
with:
# language=JavaScript
script: |

View File

@ -64,7 +64,7 @@ jobs:
run: |
apt update && apt install git -yq
git config --global --add safe.directory "$GITHUB_WORKSPACE"
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 1
persist-credentials: false
@ -101,10 +101,10 @@ jobs:
needs: build-context
if: needs.build-context.outputs.run-tests == 'true'
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3.x'
- name: Runner image version
@ -142,9 +142,14 @@ jobs:
- name: Check for unsupported C global variables
if: github.event_name == 'pull_request' # $GITHUB_EVENT_NAME
run: make check-c-globals
- name: Check for undocumented C APIs
run: make check-c-api-docs
check-c-api-docs:
name: C API Docs
needs: build-context
if: >-
needs.build-context.outputs.run-tests == 'true'
|| needs.build-context.outputs.run-docs == 'true'
uses: ./.github/workflows/reusable-check-c-api-docs.yml
build-windows:
name: >-
@ -256,7 +261,7 @@ jobs:
# Keep 1.1.1w in our list despite it being upstream EOL and otherwise
# unsupported as it most resembles other 1.1.1-work-a-like ssl APIs
# supported by important vendors such as AWS-LC.
openssl_ver: [1.1.1w, 3.0.18, 3.2.6, 3.3.5, 3.4.3, 3.5.4]
openssl_ver: [1.1.1w, 3.0.18, 3.3.5, 3.4.3, 3.5.4, 3.6.0]
# See Tools/ssl/make_ssl_data.py for notes on adding a new version
env:
OPENSSL_VER: ${{ matrix.openssl_ver }}
@ -264,7 +269,7 @@ jobs:
OPENSSL_DIR: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }}
LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }}/lib
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Runner image version
@ -280,7 +285,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ matrix.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
@ -316,7 +321,7 @@ jobs:
OPENSSL_DIR: ${{ github.workspace }}/multissl/aws-lc/${{ matrix.awslc_ver }}
LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/aws-lc/${{ matrix.awslc_ver }}/lib
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Runner image version
@ -332,7 +337,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/aws-lc/${AWSLC_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore AWS-LC build'
id: cache-aws-lc
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ./multissl/aws-lc/${{ matrix.awslc_ver }}
key: ${{ matrix.os }}-multissl-aws-lc-${{ matrix.awslc_ver }}
@ -381,7 +386,7 @@ jobs:
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Build and test
@ -394,7 +399,7 @@ jobs:
timeout-minutes: 60
runs-on: macos-14
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
@ -426,7 +431,7 @@ jobs:
OPENSSL_VER: 3.0.18
PYTHONSTRICTEXTENSIONBUILD: 1
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Register gcc problem matcher
@ -440,7 +445,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
@ -490,7 +495,7 @@ jobs:
./python -m venv "$VENV_LOC" && "$VENV_PYTHON" -m pip install -r "${GITHUB_WORKSPACE}/Tools/requirements-hypothesis.txt"
- name: 'Restore Hypothesis database'
id: cache-hypothesis-database
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ${{ env.CPYTHON_BUILDDIR }}/.hypothesis/
key: hypothesis-database-${{ github.head_ref || github.run_id }}
@ -517,7 +522,7 @@ jobs:
-x test_subprocess \
-x test_signal \
-x test_sysconfig
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@v6
if: always()
with:
name: hypothesis-example-db
@ -538,7 +543,7 @@ jobs:
PYTHONSTRICTEXTENSIONBUILD: 1
ASAN_OPTIONS: detect_leaks=0:allocator_may_return_null=1:handle_segv=0
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Runner image version
@ -548,7 +553,7 @@ jobs:
- name: Install dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Set up GCC-10 for ASAN
uses: egor-tensin/setup-gcc@v1
uses: egor-tensin/setup-gcc@v2
with:
version: 10
- name: Configure OpenSSL env vars
@ -558,7 +563,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ matrix.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
@ -608,7 +613,7 @@ jobs:
needs: build-context
if: needs.build-context.outputs.run-ubuntu == 'true'
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Runner image version
@ -636,45 +641,45 @@ jobs:
run: |
"$BUILD_DIR/cross-python/bin/python3" -m test test_sysconfig test_site test_embed
# CIFuzz job based on https://google.github.io/oss-fuzz/getting-started/continuous-integration/
cifuzz:
name: CIFuzz
runs-on: ubuntu-latest
timeout-minutes: 60
# ${{ '' } is a hack to nest jobs under the same sidebar category.
name: CIFuzz${{ '' }} # zizmor: ignore[obfuscation]
needs: build-context
if: needs.build-context.outputs.run-ci-fuzz == 'true'
if: >-
needs.build-context.outputs.run-ci-fuzz == 'true'
|| needs.build-context.outputs.run-ci-fuzz-stdlib == 'true'
permissions:
security-events: write
strategy:
fail-fast: false
matrix:
sanitizer: [address, undefined, memory]
steps:
- name: Build fuzzers (${{ matrix.sanitizer }})
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: cpython3
sanitizer: ${{ matrix.sanitizer }}
- name: Run fuzzers (${{ matrix.sanitizer }})
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
fuzz-seconds: 600
oss-fuzz-project-name: cpython3
output-sarif: true
sanitizer: ${{ matrix.sanitizer }}
- name: Upload crash
if: failure() && steps.build.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.sanitizer }}-artifacts
path: ./out/artifacts
- name: Upload SARIF
if: always() && steps.build.outcome == 'success'
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: cifuzz-sarif/results.sarif
checkout_path: cifuzz-sarif
sanitizer:
- address
- undefined
- memory
oss-fuzz-project-name:
- cpython3
- python3-libraries
exclude:
# Note that the 'no-exclude' sentinel below is to prevent
# an empty string value from excluding all jobs and causing
# GHA to create a 'default' matrix entry with all empty values.
- oss-fuzz-project-name: >-
${{
needs.build-context.outputs.run-ci-fuzz == 'true'
&& 'no-exclude'
|| 'cpython3'
}}
- oss-fuzz-project-name: >-
${{
needs.build-context.outputs.run-ci-fuzz-stdlib == 'true'
&& 'no-exclude'
|| 'python3-libraries'
}}
uses: ./.github/workflows/reusable-cifuzz.yml
with:
oss-fuzz-project-name: ${{ matrix.oss-fuzz-project-name }}
sanitizer: ${{ matrix.sanitizer }}
all-required-green: # This job does nothing and is only used for the branch protection
name: All required checks pass
@ -685,13 +690,13 @@ jobs:
- check-docs
- check-autoconf-regen
- check-generated-files
- check-c-api-docs
- build-windows
- build-windows-msi
- build-macos
- build-ubuntu
- build-ubuntu-ssltests-awslc
- build-ubuntu-ssltests-openssl
- build-android
- build-ios
- build-wasi
- test-hypothesis
@ -706,6 +711,7 @@ jobs:
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe
with:
allowed-failures: >-
build-android,
build-windows-msi,
build-ubuntu-ssltests-awslc,
build-ubuntu-ssltests-openssl,
@ -721,8 +727,19 @@ jobs:
'
|| ''
}}
${{
!fromJSON(needs.build-context.outputs.run-tests)
&& !fromJSON(needs.build-context.outputs.run-docs)
&& 'check-c-api-docs,'
|| ''
}}
${{ !fromJSON(needs.build-context.outputs.run-windows-tests) && 'build-windows,' || '' }}
${{ !fromJSON(needs.build-context.outputs.run-ci-fuzz) && 'cifuzz,' || '' }}
${{
!fromJSON(needs.build-context.outputs.run-ci-fuzz)
&& !fromJSON(needs.build-context.outputs.run-ci-fuzz-stdlib)
&& 'cifuzz,' ||
''
}}
${{ !fromJSON(needs.build-context.outputs.run-macos) && 'build-macos,' || '' }}
${{
!fromJSON(needs.build-context.outputs.run-ubuntu)

View File

@ -7,6 +7,7 @@ on:
- 'Python/optimizer*.c'
- 'Python/executor_cases.c.h'
- 'Python/optimizer_cases.c.h'
- '**_testinternalcapi**'
- '!Python/perf_jit_trampoline.c'
- '!**/*.md'
- '!**/*.ini'
@ -17,6 +18,7 @@ on:
- 'Python/optimizer*.c'
- 'Python/executor_cases.c.h'
- 'Python/optimizer_cases.c.h'
- '**_testinternalcapi**'
- '!Python/perf_jit_trampoline.c'
- '!**/*.md'
- '!**/*.ini'
@ -38,7 +40,7 @@ jobs:
runs-on: ubuntu-24.04
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Build tier two interpreter
@ -92,10 +94,10 @@ jobs:
architecture: aarch64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3.11'
@ -140,10 +142,10 @@ jobs:
llvm:
- 21
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Build with JIT enabled and GIL disabled
@ -168,10 +170,10 @@ jobs:
llvm:
- 21
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Build with JIT
@ -195,10 +197,10 @@ jobs:
llvm:
- 21
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Build with JIT and tailcall

View File

@ -19,10 +19,7 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- uses: pre-commit/action@v3.0.1
- uses: j178/prek-action@v1

View File

@ -26,6 +26,7 @@ on:
- "Tools/build/update_file.py"
- "Tools/build/verify_ensurepip_wheels.py"
- "Tools/cases_generator/**"
- "Tools/check-c-api-docs/**"
- "Tools/clinic/**"
- "Tools/jit/**"
- "Tools/peg_generator/**"
@ -58,15 +59,16 @@ jobs:
"Lib/tomllib",
"Tools/build",
"Tools/cases_generator",
"Tools/check-c-api-docs",
"Tools/clinic",
"Tools/jit",
"Tools/peg_generator",
]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: "3.13"
cache: pip

View File

@ -13,12 +13,12 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/setup-node@v4
- uses: actions/setup-node@v6
with:
node-version: 20
- run: npm install mailgun.js form-data
- name: Send notification
uses: actions/github-script@v7
uses: actions/github-script@v8
env:
MAILGUN_API_KEY: ${{ secrets.MAILGUN_PYTHON_ORG_MAILGUN_KEY }}
with:

View File

@ -0,0 +1,25 @@
name: Reusable C API Docs Check
on:
workflow_call:
permissions:
contents: read
env:
FORCE_COLOR: 1
jobs:
check-c-api-docs:
name: 'Check if all C APIs are documented'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Check for undocumented C APIs
run: python Tools/check-c-api-docs/main.py

46
.github/workflows/reusable-cifuzz.yml vendored Normal file
View File

@ -0,0 +1,46 @@
# CIFuzz job based on https://google.github.io/oss-fuzz/getting-started/continuous-integration/
name: Reusable CIFuzz
on:
workflow_call:
inputs:
oss-fuzz-project-name:
description: OSS-Fuzz project name
required: true
type: string
sanitizer:
description: OSS-Fuzz sanitizer
required: true
type: string
jobs:
cifuzz:
name: ${{ inputs.oss-fuzz-project-name }} (${{ inputs.sanitizer }})
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Build fuzzers (${{ inputs.sanitizer }})
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: ${{ inputs.oss-fuzz-project-name }}
sanitizer: ${{ inputs.sanitizer }}
- name: Run fuzzers (${{ inputs.sanitizer }})
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
fuzz-seconds: 600
oss-fuzz-project-name: ${{ inputs.oss-fuzz-project-name }}
output-sarif: true
sanitizer: ${{ inputs.sanitizer }}
- name: Upload crash
if: failure() && steps.build.outcome == 'success'
uses: actions/upload-artifact@v6
with:
name: ${{ inputs.sanitizer }}-artifacts
path: ./out/artifacts
- name: Upload SARIF
if: always() && steps.build.outcome == 'success'
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: cifuzz-sarif/results.sarif
checkout_path: cifuzz-sarif

View File

@ -21,8 +21,11 @@ on: # yamllint disable-line rule:truthy
description: Whether to run the Android tests
value: ${{ jobs.compute-changes.outputs.run-android }} # bool
run-ci-fuzz:
description: Whether to run the CIFuzz job
description: Whether to run the CIFuzz job for 'cpython' fuzzer
value: ${{ jobs.compute-changes.outputs.run-ci-fuzz }} # bool
run-ci-fuzz-stdlib:
description: Whether to run the CIFuzz job for 'python3-libraries' fuzzer
value: ${{ jobs.compute-changes.outputs.run-ci-fuzz-stdlib }} # bool
run-docs:
description: Whether to build the docs
value: ${{ jobs.compute-changes.outputs.run-docs }} # bool
@ -56,6 +59,7 @@ jobs:
outputs:
run-android: ${{ steps.changes.outputs.run-android }}
run-ci-fuzz: ${{ steps.changes.outputs.run-ci-fuzz }}
run-ci-fuzz-stdlib: ${{ steps.changes.outputs.run-ci-fuzz-stdlib }}
run-docs: ${{ steps.changes.outputs.run-docs }}
run-ios: ${{ steps.changes.outputs.run-ios }}
run-macos: ${{ steps.changes.outputs.run-macos }}
@ -66,14 +70,14 @@ jobs:
run-windows-tests: ${{ steps.changes.outputs.run-windows-tests }}
steps:
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: "3"
- run: >-
echo '${{ github.event_name }}'
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
ref: >-

View File

@ -27,7 +27,7 @@ jobs:
refspec_pr: '+${{ github.event.pull_request.head.sha }}:remotes/origin/${{ github.event.pull_request.head.ref }}'
steps:
- name: 'Check out latest PR branch commit'
uses: actions/checkout@v4
uses: actions/checkout@v6
with:
persist-credentials: false
ref: >-
@ -52,7 +52,7 @@ jobs:
git fetch origin "${refspec_base}" --shallow-since="${DATE}" \
--no-tags --prune --no-recurse-submodules
- name: 'Set up Python'
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: '3'
cache: 'pip'
@ -82,10 +82,10 @@ jobs:
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/cache@v4
- uses: actions/cache@v5
with:
path: ~/.cache/pip
key: ubuntu-doc-${{ hashFiles('Doc/requirements.txt') }}
@ -108,11 +108,11 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: 'Set up Python'
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: '3'
cache: 'pip'

View File

@ -28,7 +28,7 @@ jobs:
PYTHONSTRICTEXTENSIONBUILD: 1
TERM: linux
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Runner image version

View File

@ -26,7 +26,7 @@ jobs:
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Runner image version
@ -99,7 +99,7 @@ jobs:
run: find "${GITHUB_WORKSPACE}" -name 'san_log.*' | xargs head -n 1000
- name: Archive logs
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
with:
name: >-
${{ inputs.sanitizer }}-logs-${{

View File

@ -31,7 +31,7 @@ jobs:
PYTHONSTRICTEXTENSIONBUILD: 1
TERM: linux
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Register gcc problem matcher
@ -51,7 +51,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ inputs.os }}-multissl-openssl-${{ env.OPENSSL_VER }}

View File

@ -18,7 +18,7 @@ jobs:
CROSS_BUILD_PYTHON: cross-build/build
CROSS_BUILD_WASI: cross-build/wasm32-wasip1
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
# No problem resolver registered as one doesn't currently exist for Clang.
@ -28,7 +28,7 @@ jobs:
version: ${{ env.WASMTIME_VERSION }}
- name: "Restore WASI SDK"
id: cache-wasi-sdk
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: ${{ env.WASI_SDK_PATH }}
key: ${{ runner.os }}-wasi-sdk-${{ env.WASI_SDK_VERSION }}
@ -41,20 +41,20 @@ jobs:
- name: "Add ccache to PATH"
run: echo "PATH=/usr/lib/ccache:$PATH" >> "$GITHUB_ENV"
- name: "Install Python"
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: '3.x'
- name: "Runner image version"
run: echo "IMAGE_OS_VERSION=${ImageOS}-${ImageVersion}" >> "$GITHUB_ENV"
- name: "Configure build Python"
run: python3 Tools/wasm/wasi configure-build-python -- --config-cache --with-pydebug
run: python3 Platforms/WASI configure-build-python -- --config-cache --with-pydebug
- name: "Make build Python"
run: python3 Tools/wasm/wasi make-build-python
run: python3 Platforms/WASI make-build-python
- name: "Configure host"
# `--with-pydebug` inferred from configure-build-python
run: python3 Tools/wasm/wasi configure-host -- --config-cache
run: python3 Platforms/WASI configure-host -- --config-cache
- name: "Make host"
run: python3 Tools/wasm/wasi make-host
run: python3 Platforms/WASI make-host
- name: "Display build info"
run: make --directory "${CROSS_BUILD_WASI}" pythoninfo
- name: "Test"

View File

@ -23,7 +23,7 @@ jobs:
ARCH: ${{ inputs.arch }}
IncludeFreethreaded: true
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Build CPython installer

View File

@ -26,7 +26,7 @@ jobs:
env:
ARCH: ${{ inputs.arch }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Register MSVC problem matcher

View File

@ -72,33 +72,32 @@ jobs:
architecture: x86_64
runner: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Native Windows (debug)
- name: Native Windows MSVC (release)
if: runner.os == 'Windows' && matrix.architecture != 'ARM64'
shell: cmd
shell: pwsh
run: |
choco install llvm --allow-downgrade --no-progress --version ${{ matrix.llvm }}.1.0
set PlatformToolset=clangcl
set LLVMToolsVersion=${{ matrix.llvm }}.1.0
set LLVMInstallDir=C:\Program Files\LLVM
call ./PCbuild/build.bat --tail-call-interp -d -p ${{ matrix.architecture }}
call ./PCbuild/rt.bat -d -p ${{ matrix.architecture }} -q --multiprocess 0 --timeout 4500 --verbose2 --verbose3
choco install visualstudio2026buildtools --no-progress -y --force --params "--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --locale en-US --passive"
$env:PATH = "C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\MSBuild\Current\bin;$env:PATH"
$env:PlatformToolset = "v145"
./PCbuild/build.bat --tail-call-interp -c Release -p ${{ matrix.architecture }}
./PCbuild/rt.bat -p ${{ matrix.architecture }} -q --multiprocess 0 --timeout 4500 --verbose2 --verbose3
# No tests (yet):
- name: Emulated Windows (release)
- name: Emulated Windows Clang (release)
if: runner.os == 'Windows' && matrix.architecture == 'ARM64'
shell: cmd
shell: pwsh
run: |
choco install llvm --allow-downgrade --no-progress --version ${{ matrix.llvm }}.1.0
set PlatformToolset=clangcl
set LLVMToolsVersion=${{ matrix.llvm }}.1.0
set LLVMInstallDir=C:\Program Files\LLVM
$env:PlatformToolset = "clangcl"
$env:LLVMToolsVersion = "${{ matrix.llvm }}.1.0"
$env:LLVMInstallDir = "C:\Program Files\LLVM"
./PCbuild/build.bat --tail-call-interp -p ${{ matrix.architecture }}
- name: Native macOS (release)

View File

@ -25,10 +25,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: '3'
- name: Compare checksum of bundled wheels to the ones published on PyPI

2
.github/zizmor.yml vendored
View File

@ -1,4 +1,4 @@
# Configuration for the zizmor static analysis tool, run via pre-commit in CI
# Configuration for the zizmor static analysis tool, run via prek in CI
# https://woodruffw.github.io/zizmor/configuration/
rules:
dangerous-triggers:

View File

@ -1,6 +1,6 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.13.2
rev: v0.14.10
hooks:
- id: ruff-check
name: Run Ruff (lint) on Apple/
@ -40,19 +40,19 @@ repos:
files: ^Apple
- id: ruff-format
name: Run Ruff (format) on Doc/
args: [--check]
args: [--exit-non-zero-on-fix]
files: ^Doc/
- id: ruff-format
name: Run Ruff (format) on Tools/build/check_warnings.py
args: [--check, --config=Tools/build/.ruff.toml]
args: [--exit-non-zero-on-fix, --config=Tools/build/.ruff.toml]
files: ^Tools/build/check_warnings.py
- id: ruff-format
name: Run Ruff (format) on Tools/wasm/
args: [--check, --config=Tools/wasm/.ruff.toml]
args: [--exit-non-zero-on-fix, --config=Tools/wasm/.ruff.toml]
files: ^Tools/wasm/
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 25.9.0
rev: 25.12.0
hooks:
- id: black
name: Run Black on Tools/jit/
@ -83,24 +83,24 @@ repos:
files: '^\.github/CODEOWNERS|\.(gram)$'
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.34.0
rev: 0.36.0
hooks:
- id: check-dependabot
- id: check-github-workflows
- id: check-readthedocs
- repo: https://github.com/rhysd/actionlint
rev: v1.7.7
rev: v1.7.9
hooks:
- id: actionlint
- repo: https://github.com/woodruffw/zizmor-pre-commit
rev: v1.14.1
rev: v1.19.0
hooks:
- id: zizmor
- repo: https://github.com/sphinx-contrib/sphinx-lint
rev: v1.0.0
rev: v1.0.2
hooks:
- id: sphinx-lint
args: [--enable=default-role]

View File

@ -140,7 +140,8 @@ doctest:
pydoc-topics: BUILDER = pydoc-topics
pydoc-topics: build
@echo "Building finished; now run this:" \
"cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py"
"cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py" \
"&& cp build/pydoc-topics/module_docs.py ../Lib/pydoc_data/module_docs.py"
.PHONY: gettext
gettext: BUILDER = gettext

View File

@ -34,6 +34,23 @@ See :ref:`stable` for a discussion of API and ABI stability across versions.
This can be ``0xA`` for alpha, ``0xB`` for beta, ``0xC`` for release
candidate or ``0xF`` for final.
.. c:namespace:: NULL
.. c:macro:: PY_RELEASE_LEVEL_ALPHA
:no-typesetting:
.. c:macro:: PY_RELEASE_LEVEL_BETA
:no-typesetting:
.. c:macro:: PY_RELEASE_LEVEL_GAMMA
:no-typesetting:
.. c:macro:: PY_RELEASE_LEVEL_FINAL
:no-typesetting:
For completeness, the values are available as macros:
:c:macro:`!PY_RELEASE_LEVEL_ALPHA` (``0xA``),
:c:macro:`!PY_RELEASE_LEVEL_BETA` (``0xB``),
:c:macro:`!PY_RELEASE_LEVEL_GAMMA` (``0xC``), and
:c:macro:`!PY_RELEASE_LEVEL_FINAL` (``0xF``).
.. c:macro:: PY_RELEASE_SERIAL
The ``2`` in ``3.4.1a2``. Zero for final releases.
@ -46,6 +63,12 @@ See :ref:`stable` for a discussion of API and ABI stability across versions.
Use this for numeric comparisons, for example,
``#if PY_VERSION_HEX >= ...``.
.. c:macro:: PY_VERSION
The Python version as a string, for example, ``"3.4.1a2"``.
These macros are defined in :source:`Include/patchlevel.h`.
Run-time version
----------------

View File

@ -347,6 +347,8 @@ please see individual documentation for details.
.. versionadded:: 3.9
.. c:function:: PyObject* _PyObject_Vectorcall(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames)
:no-typesetting:
.. c:function:: PyObject* PyObject_Vectorcall(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames)
@ -358,7 +360,12 @@ please see individual documentation for details.
Return the result of the call on success, or raise an exception and return
*NULL* on failure.
.. versionadded:: 3.9
.. versionadded:: 3.8 as ``_PyObject_Vectorcall``
.. versionchanged:: 3.9
Renamed to the current name, without the leading underscore.
The old provisional name is :term:`soft deprecated`.
.. c:function:: PyObject* PyObject_VectorcallDict(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwdict)

View File

@ -69,13 +69,14 @@ bound into a function.
The old name is deprecated, but will remain available until the
signature changes again.
.. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(...)
:no-typesetting:
.. c:function:: PyCodeObject* PyUnstable_Code_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, PyObject *qualname, int firstlineno, PyObject *linetable, PyObject *exceptiontable)
Similar to :c:func:`PyUnstable_Code_New`, but with an extra "posonlyargcount" for positional-only arguments.
The same caveats that apply to ``PyUnstable_Code_New`` also apply to this function.
.. index:: single: PyCode_NewWithPosOnlyArgs (C function)
.. versionadded:: 3.8 as ``PyCode_NewWithPosOnlyArgs``
.. versionchanged:: 3.11
@ -298,6 +299,9 @@ These functions are part of the unstable C API tier:
this functionality is a CPython implementation detail, and the API
may change without deprecation warnings.
.. c:function:: Py_ssize_t _PyEval_RequestCodeExtraIndex(freefunc free)
:no-typesetting:
.. c:function:: Py_ssize_t PyUnstable_Eval_RequestCodeExtraIndex(freefunc free)
Return a new opaque index value used to adding data to code objects.
@ -310,8 +314,6 @@ may change without deprecation warnings.
*free* will be called on non-``NULL`` data stored under the new index.
Use :c:func:`Py_DecRef` when storing :c:type:`PyObject`.
.. index:: single: _PyEval_RequestCodeExtraIndex (C function)
.. versionadded:: 3.6 as ``_PyEval_RequestCodeExtraIndex``
.. versionchanged:: 3.12
@ -320,6 +322,9 @@ may change without deprecation warnings.
The old private name is deprecated, but will be available until the API
changes.
.. c:function:: int _PyCode_GetExtra(PyObject *code, Py_ssize_t index, void **extra)
:no-typesetting:
.. c:function:: int PyUnstable_Code_GetExtra(PyObject *code, Py_ssize_t index, void **extra)
Set *extra* to the extra data stored under the given index.
@ -328,8 +333,6 @@ may change without deprecation warnings.
If no data was set under the index, set *extra* to ``NULL`` and return
0 without setting an exception.
.. index:: single: _PyCode_GetExtra (C function)
.. versionadded:: 3.6 as ``_PyCode_GetExtra``
.. versionchanged:: 3.12
@ -338,13 +341,14 @@ may change without deprecation warnings.
The old private name is deprecated, but will be available until the API
changes.
.. c:function:: int _PyCode_SetExtra(PyObject *code, Py_ssize_t index, void *extra)
:no-typesetting:
.. c:function:: int PyUnstable_Code_SetExtra(PyObject *code, Py_ssize_t index, void *extra)
Set the extra data stored under the given index to *extra*.
Return 0 on success. Set an exception and return -1 on failure.
.. index:: single: _PyCode_SetExtra (C function)
.. versionadded:: 3.6 as ``_PyCode_SetExtra``
.. versionchanged:: 3.12

View File

@ -130,6 +130,8 @@ The following functions provide locale-independent string to number conversions.
*flags* can be zero or more of the following values or-ed together:
.. c:namespace:: NULL
.. c:macro:: Py_DTSF_SIGN
Always precede the returned string with a sign
@ -151,9 +153,21 @@ The following functions provide locale-independent string to number conversions.
.. versionadded:: 3.11
If *ptype* is non-``NULL``, then the value it points to will be set to one of
``Py_DTST_FINITE``, ``Py_DTST_INFINITE``, or ``Py_DTST_NAN``, signifying that
*val* is a finite number, an infinite number, or not a number, respectively.
If *ptype* is non-``NULL``, then the value it points to will be set to one
of the following constants depending on the type of *val*:
.. list-table::
:header-rows: 1
:align: left
* - *\*ptype*
- type of *val*
* - .. c:macro:: Py_DTST_FINITE
- finite number
* - .. c:macro:: Py_DTST_INFINITE
- infinite number
* - .. c:macro:: Py_DTST_NAN
- not a number
The return value is a pointer to *buffer* with the converted string or
``NULL`` if the conversion failed. The caller is responsible for freeing the

View File

@ -10,11 +10,6 @@ found in the dictionary of type objects.
.. XXX document these!
.. c:var:: PyTypeObject PyProperty_Type
The type object for the built-in descriptor types.
.. c:function:: PyObject* PyDescr_NewGetSet(PyTypeObject *type, struct PyGetSetDef *getset)
@ -74,9 +69,26 @@ found in the dictionary of type objects.
.. c:function:: PyObject* PyWrapper_New(PyObject *, PyObject *)
.. c:macro:: PyDescr_COMMON
This is a :term:`soft deprecated` macro including the common fields for a
descriptor object.
This was included in Python's C API by mistake; do not use it in extensions.
For creating custom descriptor objects, create a class implementing the
descriptor protocol (:c:member:`~PyTypeObject.tp_descr_get` and
:c:member:`~PyTypeObject.tp_descr_set`).
Built-in descriptors
^^^^^^^^^^^^^^^^^^^^
.. c:var:: PyTypeObject PyProperty_Type
The type object for property objects. This is the same object as
:class:`property` in the Python layer.
.. c:var:: PyTypeObject PySuper_Type
The type object for super objects. This is the same object as

View File

@ -793,6 +793,17 @@ Exception Classes
Return :c:member:`~PyTypeObject.tp_name` of the exception class *ob*.
.. c:macro:: PyException_HEAD
This is a :term:`soft deprecated` macro including the base fields for an
exception object.
This was included in Python's C API by mistake and is not designed for use
in extensions. For creating custom exception objects, use
:c:func:`PyErr_NewException` or otherwise create a class inheriting from
:c:data:`PyExc_BaseException`.
Exception Objects
=================

View File

@ -131,3 +131,22 @@ the :mod:`io` APIs instead.
Write string *s* to file object *p*. Return ``0`` on success or ``-1`` on
failure; the appropriate exception will be set.
Deprecated API
^^^^^^^^^^^^^^
These are :term:`soft deprecated` APIs that were included in Python's C API
by mistake. They are documented solely for completeness; use other
``PyFile*`` APIs instead.
.. c:function:: PyObject *PyFile_NewStdPrinter(int fd)
Use :c:func:`PyFile_FromFd` with defaults (``fd, NULL, "w", -1, NULL, NULL, NULL, 0``) instead.
.. c:var:: PyTypeObject PyStdPrinter_Type
Type of file-like objects used internally at Python startup when :py:mod:`io` is
not yet available.
Use Python :py:func:`open` or :c:func:`PyFile_FromFd` to create file objects instead.

View File

@ -45,6 +45,7 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
A reference to *frame* is stolen by this function. The *frame* argument
must not be ``NULL``.
.. c:function:: PyCodeObject* PyGen_GetCode(PyGenObject *gen)
Return a new :term:`strong reference` to the code object wrapped by *gen*.
@ -82,3 +83,14 @@ Asynchronous Generator Objects
This function always succeeds.
.. versionadded:: 3.6
Deprecated API
^^^^^^^^^^^^^^
.. c:macro:: PyAsyncGenASend_CheckExact(op)
This is a :term:`soft deprecated` API that was included in Python's C API
by mistake.
It is solely here for completeness; do not use this API.

View File

@ -107,6 +107,46 @@ header files properly declare the entry points to be ``extern "C"``. As a result
there is no need to do anything special to use the API from C++.
.. _capi-system-includes:
System includes
---------------
:file:`Python.h` includes several standard header files.
C extensions should include the standard headers that they use,
and should not rely on these implicit includes.
The implicit includes are:
* ``<assert.h>``
* ``<intrin.h>`` (on Windows)
* ``<inttypes.h>``
* ``<limits.h>``
* ``<math.h>``
* ``<stdarg.h>``
* ``<wchar.h>``
* ``<sys/types.h>`` (if present)
The following are included for backwards compatibility, unless using
:ref:`Limited API <limited-c-api>` 3.13 or newer:
* ``<ctype.h>``
* ``<unistd.h>`` (on POSIX)
The following are included for backwards compatibility, unless using
:ref:`Limited API <limited-c-api>` 3.11 or newer:
* ``<errno.h>``
* ``<stdio.h>``
* ``<stdlib.h>``
* ``<string.h>``
.. note::
Since Python may define some pre-processor definitions which affect the standard
headers on some systems, you *must* include :file:`Python.h` before any standard
headers are included.
Useful macros
=============
@ -182,6 +222,14 @@ complete listing.
Equivalent to :c:macro:`Py_LOCAL` but additionally requests the function
be inlined.
.. c:macro:: Py_LOCAL_SYMBOL
Macro used to declare a symbol as local to the shared library (hidden).
On supported platforms, it ensures the symbol is not exported.
On compatible versions of GCC/Clang, it
expands to ``__attribute__((visibility("hidden")))``.
.. c:macro:: Py_MAX(x, y)
Return the maximum value between ``x`` and ``y``.
@ -336,6 +384,38 @@ complete listing.
sizeof(array) / sizeof((array)[0])
.. c:macro:: Py_EXPORTED_SYMBOL
Macro used to declare a symbol (function or data) as exported.
On Windows, this expands to ``__declspec(dllexport)``.
On compatible versions of GCC/Clang, it
expands to ``__attribute__((visibility("default")))``.
This macro is for defining the C API itself; extension modules should not use it.
.. c:macro:: Py_IMPORTED_SYMBOL
Macro used to declare a symbol as imported.
On Windows, this expands to ``__declspec(dllimport)``.
This macro is for defining the C API itself; extension modules should not use it.
.. c:macro:: PyAPI_FUNC(type)
Macro used by CPython to declare a function as part of the C API.
Its expansion depends on the platform and build configuration.
This macro is intended for defining CPython's C API itself;
extension modules should not use it for their own symbols.
.. c:macro:: PyAPI_DATA(type)
Macro used by CPython to declare a public global variable as part of the C API.
Its expansion depends on the platform and build configuration.
This macro is intended for defining CPython's C API itself;
extension modules should not use it for their own symbols.
.. _api-objects:
Objects, Types and Reference Counts

View File

@ -256,6 +256,8 @@ To allocate and free memory, see :ref:`allocating-objects`.
collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set); this may
change in the future.
.. versionadded:: 3.4
.. c:function:: int PyObject_CallFinalizerFromDealloc(PyObject *op)
@ -266,6 +268,8 @@ To allocate and free memory, see :ref:`allocating-objects`.
should happen. Otherwise, this function returns 0 and destruction can
continue normally.
.. versionadded:: 3.4
.. seealso::
:c:member:`~PyTypeObject.tp_dealloc` for example code.

View File

@ -453,8 +453,8 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate.
Otherwise, returns the number of bytes required to store the value.
If this is equal to or less than *n_bytes*, the entire value was copied.
All *n_bytes* of the buffer are written: large buffers are padded with
zeroes.
All *n_bytes* of the buffer are written: remaining bytes filled by
copies of the sign bit.
If the returned value is greater than *n_bytes*, the value was
truncated: as many of the lowest bits of the value as could fit are written,
@ -687,7 +687,7 @@ Export API
.. versionadded:: 3.14
.. c:struct:: PyLongLayout
.. c:type:: PyLongLayout
Layout of an array of "digits" ("limbs" in the GMP terminology), used to
represent absolute value for arbitrary precision integers.
@ -727,7 +727,7 @@ Export API
Get the native layout of Python :class:`int` objects.
See the :c:struct:`PyLongLayout` structure.
See the :c:type:`PyLongLayout` structure.
The function must not be called before Python initialization nor after
Python finalization. The returned layout is valid until Python is
@ -735,7 +735,7 @@ Export API
in a process, and so it can be cached.
.. c:struct:: PyLongExport
.. c:type:: PyLongExport
Export of a Python :class:`int` object.
@ -769,7 +769,7 @@ Export API
Export a Python :class:`int` object.
*export_long* must point to a :c:struct:`PyLongExport` structure allocated
*export_long* must point to a :c:type:`PyLongExport` structure allocated
by the caller. It must not be ``NULL``.
On success, fill in *\*export_long* and return ``0``.
@ -799,7 +799,7 @@ The :c:type:`PyLongWriter` API can be used to import an integer.
.. versionadded:: 3.14
.. c:struct:: PyLongWriter
.. c:type:: PyLongWriter
A Python :class:`int` writer instance.
@ -827,7 +827,7 @@ The :c:type:`PyLongWriter` API can be used to import an integer.
The layout of *digits* is described by :c:func:`PyLong_GetNativeLayout`.
Digits must be in the range [``0``; ``(1 << bits_per_digit) - 1``]
(where the :c:struct:`~PyLongLayout.bits_per_digit` is the number of bits
(where the :c:type:`~PyLongLayout.bits_per_digit` is the number of bits
per digit).
Any unused most significant digits must be set to ``0``.
@ -855,3 +855,31 @@ The :c:type:`PyLongWriter` API can be used to import an integer.
If *writer* is ``NULL``, no operation is performed.
The writer instance and the *digits* array are invalid after the call.
Deprecated API
^^^^^^^^^^^^^^
These macros are :term:`soft deprecated`. They describe parameters
of the internal representation of :c:type:`PyLongObject` instances.
Use :c:func:`PyLong_GetNativeLayout` instead, along with :c:func:`PyLong_Export`
to read integer data or :c:type:`PyLongWriter` to write it.
These currently use the same layout, but are designed to continue working correctly
even if CPython's internal integer representation changes.
.. c:macro:: PyLong_SHIFT
This is equivalent to :c:member:`~PyLongLayout.bits_per_digit` in
the output of :c:func:`PyLong_GetNativeLayout`.
.. c:macro:: PyLong_BASE
This is currently equivalent to :c:expr:`1 << PyLong_SHIFT`.
.. c:macro:: PyLong_MASK
This is currently equivalent to :c:expr:`(1 << PyLong_SHIFT) - 1`

View File

@ -293,17 +293,39 @@ The following type-oriented macros are provided for convenience. Note that
Same as :c:func:`PyMem_Free`.
In addition, the following macro sets are provided for calling the Python memory
allocator directly, without involving the C API functions listed above. However,
note that their use does not preserve binary compatibility across Python
versions and is therefore deprecated in extension modules.
* ``PyMem_MALLOC(size)``
* ``PyMem_NEW(type, size)``
* ``PyMem_REALLOC(ptr, size)``
* ``PyMem_RESIZE(ptr, type, size)``
* ``PyMem_FREE(ptr)``
* ``PyMem_DEL(ptr)``
Deprecated aliases
------------------
These are :term:`soft deprecated` aliases to existing functions and macros.
They exist solely for backwards compatibility.
.. list-table::
:widths: auto
:header-rows: 1
* * Deprecated alias
* Corresponding function or macro
* * .. c:macro:: PyMem_MALLOC(size)
* :c:func:`PyMem_Malloc`
* * .. c:macro:: PyMem_NEW(type, size)
* :c:macro:`PyMem_New`
* * .. c:macro:: PyMem_REALLOC(ptr, size)
* :c:func:`PyMem_Realloc`
* * .. c:macro:: PyMem_RESIZE(ptr, type, size)
* :c:macro:`PyMem_Resize`
* * .. c:macro:: PyMem_FREE(ptr)
* :c:func:`PyMem_Free`
* * .. c:macro:: PyMem_DEL(ptr)
* :c:func:`PyMem_Free`
.. versionchanged:: 3.4
The macros are now aliases of the corresponding functions and macros.
Previously, their behavior was the same, but their use did not necessarily
preserve binary compatibility across Python versions.
.. deprecated:: 2.0
.. _objectinterface:

View File

@ -426,10 +426,10 @@ To retrieve the state from a given module, use the following functions:
module state.
.. c:function:: int PyModule_GetStateSize(PyObject *, Py_ssize_t *result)
.. c:function:: int PyModule_GetStateSize(PyObject *module, Py_ssize_t *result)
Set *\*result* to the size of the module's state, as specified using
:c:macro:`Py_mod_state_size` (or :c:member:`PyModuleDef.m_size`),
Set *\*result* to the size of *module*'s state, as specified
using :c:macro:`Py_mod_state_size` (or :c:member:`PyModuleDef.m_size`),
and return 0.
On error, set *\*result* to -1, and return -1 with an exception set.
@ -571,7 +571,7 @@ A module's token -- and the *your_token* value to use in the above code -- is:
of that slot;
- For modules created from an ``PyModExport_*``
:ref:`export hook <extension-export-hook>`: the slots array that the export
hook returned (unless overriden with :c:macro:`Py_mod_token`).
hook returned (unless overridden with :c:macro:`Py_mod_token`).
.. c:macro:: Py_mod_token
@ -597,7 +597,7 @@ A module's token -- and the *your_token* value to use in the above code -- is:
.. c:function:: int PyModule_GetToken(PyObject *module, void** result)
Set *\*result* to the module's token and return 0.
Set *\*result* to the module token for *module* and return 0.
On error, set *\*result* to NULL, and return -1 with an exception set.
@ -645,7 +645,7 @@ rather than from an extension's :ref:`export hook <extension-export-hook>`.
.. c:function:: int PyModule_Exec(PyObject *module)
Execute the :c:data:`Py_mod_exec` slot(s) of the given *module*.
Execute the :c:data:`Py_mod_exec` slot(s) of *module*.
On success, return 0.
On error, return -1 with an exception set.
@ -820,15 +820,18 @@ struct:
.. versionadded:: 3.5
.. c:macro:: PYTHON_API_VERSION
PYTHON_API_STRING
The C API version. Defined for backwards compatibility.
The C API version, as an integer (``1013``) and string (``"1013"``), respectively.
Defined for backwards compatibility.
Currently, this constant is not updated in new Python versions, and is not
useful for versioning. This may change in the future.
.. c:macro:: PYTHON_ABI_VERSION
PYTHON_ABI_STRING
Defined as ``3`` for backwards compatibility.
Defined as ``3`` and ``"3"``, respectively, for backwards compatibility.
Currently, this constant is not updated in new Python versions, and is not
useful for versioning. This may change in the future.
@ -1018,6 +1021,9 @@ or code that creates modules dynamically.
``PyModuleDef`` (such as when using :ref:`multi-phase-initialization`,
``PyModule_Create``, or ``PyModule_FromDefAndSpec``).
Return ``0`` on success.
Return ``-1`` with an exception set on error.
.. versionadded:: 3.5
.. c:function:: int PyUnstable_Module_SetGIL(PyObject *module, void *gil)

View File

@ -85,7 +85,7 @@ Object Protocol
instead of the :func:`repr`.
.. c:function:: void PyUnstable_Object_Dump(PyObject *op)
.. c:function:: void PyObject_Dump(PyObject *op)
Dump an object *op* to ``stderr``. This should only be used for debugging.

View File

@ -166,3 +166,20 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
Empty an existing set of all elements. Return ``0`` on
success. Return ``-1`` and raise :exc:`SystemError` if *set* is not an instance of
:class:`set` or its subtype.
Deprecated API
^^^^^^^^^^^^^^
.. c:macro:: PySet_MINSIZE
A :term:`soft deprecated` constant representing the size of an internal
preallocated table inside :c:type:`PySetObject` instances.
This is documented solely for completeness, as there are no guarantees
that a given version of CPython uses preallocated tables with a fixed
size.
In code that does not deal with unstable set internals,
:c:macro:`!PySet_MINSIZE` can be replaced with a small constant like ``8``.
If looking for the size of a set, use :c:func:`PySet_Size` instead.

View File

@ -1373,6 +1373,9 @@ and :c:data:`PyType_Type` effectively act as defaults.)
type structure.
.. c:macro:: _Py_TPFLAGS_HAVE_VECTORCALL
:no-typesetting:
.. c:macro:: Py_TPFLAGS_HAVE_VECTORCALL
This bit is set when the class implements
@ -1384,7 +1387,12 @@ and :c:data:`PyType_Type` effectively act as defaults.)
This bit is inherited if :c:member:`~PyTypeObject.tp_call` is also
inherited.
.. versionadded:: 3.9
.. versionadded:: 3.8 as ``_Py_TPFLAGS_HAVE_VECTORCALL``
.. versionchanged:: 3.9
Renamed to the current name, without the leading underscore.
The old provisional name is :term:`soft deprecated`.
.. versionchanged:: 3.12

View File

@ -65,6 +65,27 @@ Python:
.. versionadded:: 3.3
The structure of a particular object can be determined using the following
macros.
The macros cannot fail; their behavior is undefined if their argument
is not a Python Unicode object.
.. c:namespace:: NULL
.. c:macro:: PyUnicode_IS_COMPACT(o)
True if *o* uses the :c:struct:`PyCompactUnicodeObject` structure.
.. versionadded:: 3.3
.. c:macro:: PyUnicode_IS_COMPACT_ASCII(o)
True if *o* uses the :c:struct:`PyASCIIObject` structure.
.. versionadded:: 3.3
The following APIs are C macros and static inlined functions for fast checks and
access to internal read-only data of Unicode objects:

View File

@ -389,8 +389,14 @@ func,PyList_SetSlice,3.2,,
func,PyList_Size,3.2,,
func,PyList_Sort,3.2,,
data,PyList_Type,3.2,,
type,PyLongExport,3.15,,full-abi
type,PyLongLayout,3.15,,full-abi
type,PyLongObject,3.2,,opaque
data,PyLongRangeIter_Type,3.2,,
type,PyLongWriter,3.15,,opaque
func,PyLongWriter_Create,3.15,,
func,PyLongWriter_Discard,3.15,,
func,PyLongWriter_Finish,3.15,,
func,PyLong_AsDouble,3.2,,
func,PyLong_AsInt,3.13,,
func,PyLong_AsInt32,3.14,,
@ -409,6 +415,8 @@ func,PyLong_AsUnsignedLongLong,3.2,,
func,PyLong_AsUnsignedLongLongMask,3.2,,
func,PyLong_AsUnsignedLongMask,3.2,,
func,PyLong_AsVoidPtr,3.2,,
func,PyLong_Export,3.15,,
func,PyLong_FreeExport,3.15,,
func,PyLong_FromDouble,3.2,,
func,PyLong_FromInt32,3.14,,
func,PyLong_FromInt64,3.14,,
@ -425,6 +433,7 @@ func,PyLong_FromUnsignedLongLong,3.2,,
func,PyLong_FromUnsignedNativeBytes,3.14,,
func,PyLong_FromVoidPtr,3.2,,
func,PyLong_GetInfo,3.2,,
func,PyLong_GetNativeLayout,3.15,,
data,PyLong_Type,3.2,,
macro,PyMODEXPORT_FUNC,3.15,,
data,PyMap_Type,3.2,,

View File

@ -7,6 +7,8 @@ Deprecations
.. include:: pending-removal-in-3.17.rst
.. include:: pending-removal-in-3.18.rst
.. include:: pending-removal-in-3.19.rst
.. include:: pending-removal-in-3.20.rst

View File

@ -33,16 +33,6 @@ Pending removal in Python 3.15
* ``load_module()`` method: use ``exec_module()`` instead.
* :class:`locale`:
* The :func:`~locale.getdefaultlocale` function
has been deprecated since Python 3.11.
Its removal was originally planned for Python 3.13 (:gh:`90817`),
but has been postponed to Python 3.15.
Use :func:`~locale.getlocale`, :func:`~locale.setlocale`,
and :func:`~locale.getencoding` instead.
(Contributed by Hugo van Kemenade in :gh:`111187`.)
* :mod:`pathlib`:
* :meth:`!.PurePath.is_reserved`

View File

@ -0,0 +1,9 @@
Pending removal in Python 3.18
------------------------------
* :mod:`decimal`:
* The non-standard and undocumented :class:`~decimal.Decimal` format
specifier ``'N'``, which is only supported in the :mod:`!decimal` module's
C implementation, has been deprecated since Python 3.13.
(Contributed by Serhiy Storchaka in :gh:`89902`.)

View File

@ -1,9 +1,9 @@
Pending removal in Python 3.20
------------------------------
* The ``__version__`` attribute has been deprecated in these standard library
modules and will be removed in Python 3.20.
Use :py:data:`sys.version_info` instead.
* The ``__version__``, ``version`` and ``VERSION`` attributes have been
deprecated in these standard library modules and will be removed in
Python 3.20. Use :py:data:`sys.version_info` instead.
- :mod:`argparse`
- :mod:`csv`
@ -24,6 +24,9 @@ Pending removal in Python 3.20
- :mod:`tkinter.font`
- :mod:`tkinter.ttk`
- :mod:`wsgiref.simple_server`
- :mod:`xml.etree.ElementTree`
- :mod:`!xml.sax.expatreader`
- :mod:`xml.sax.handler`
- :mod:`zlib`
(Contributed by Hugo van Kemenade and Stan Ulbrych in :gh:`76007`.)

View File

@ -3,154 +3,20 @@
.. _extending-intro:
******************************
Extending Python with C or C++
******************************
********************************
Using the C API: Assorted topics
********************************
It is quite easy to add new built-in modules to Python, if you know how to
program in C. Such :dfn:`extension modules` can do two things that can't be
done directly in Python: they can implement new built-in object types, and they
can call C library functions and system calls.
To support extensions, the Python API (Application Programmers Interface)
defines a set of functions, macros and variables that provide access to most
aspects of the Python run-time system. The Python API is incorporated in a C
source file by including the header ``"Python.h"``.
The compilation of an extension module depends on its intended use as well as on
your system setup; details are given in later chapters.
.. note::
The C extension interface is specific to CPython, and extension modules do
not work on other Python implementations. In many cases, it is possible to
avoid writing C extensions and preserve portability to other implementations.
For example, if your use case is calling C library functions or system calls,
you should consider using the :mod:`ctypes` module or the `cffi
<https://cffi.readthedocs.io/>`_ library rather than writing
custom C code.
These modules let you write Python code to interface with C code and are more
portable between implementations of Python than writing and compiling a C
extension module.
.. _extending-simpleexample:
A Simple Example
================
Let's create an extension module called ``spam`` (the favorite food of Monty
Python fans...) and let's say we want to create a Python interface to the C
library function :c:func:`system` [#]_. This function takes a null-terminated
character string as argument and returns an integer. We want this function to
be callable from Python as follows:
.. code-block:: pycon
>>> import spam
>>> status = spam.system("ls -l")
Begin by creating a file :file:`spammodule.c`. (Historically, if a module is
called ``spam``, the C file containing its implementation is called
:file:`spammodule.c`; if the module name is very long, like ``spammify``, the
module name can be just :file:`spammify.c`.)
The first two lines of our file can be::
#define PY_SSIZE_T_CLEAN
#include <Python.h>
which pulls in the Python API (you can add a comment describing the purpose of
the module and a copyright notice if you like).
.. note::
Since Python may define some pre-processor definitions which affect the standard
headers on some systems, you *must* include :file:`Python.h` before any standard
headers are included.
``#define PY_SSIZE_T_CLEAN`` was used to indicate that ``Py_ssize_t`` should be
used in some APIs instead of ``int``.
It is not necessary since Python 3.13, but we keep it here for backward compatibility.
See :ref:`arg-parsing-string-and-buffers` for a description of this macro.
All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or
``PY``, except those defined in standard header files.
.. tip::
For backward compatibility, :file:`Python.h` includes several standard header files.
C extensions should include the standard headers that they use,
and should not rely on these implicit includes.
If using the limited C API version 3.13 or newer, the implicit includes are:
* ``<assert.h>``
* ``<intrin.h>`` (on Windows)
* ``<inttypes.h>``
* ``<limits.h>``
* ``<math.h>``
* ``<stdarg.h>``
* ``<wchar.h>``
* ``<sys/types.h>`` (if present)
If :c:macro:`Py_LIMITED_API` is not defined, or is set to version 3.12 or older,
the headers below are also included:
* ``<ctype.h>``
* ``<unistd.h>`` (on POSIX)
If :c:macro:`Py_LIMITED_API` is not defined, or is set to version 3.10 or older,
the headers below are also included:
* ``<errno.h>``
* ``<stdio.h>``
* ``<stdlib.h>``
* ``<string.h>``
The next thing we add to our module file is the C function that will be called
when the Python expression ``spam.system(string)`` is evaluated (we'll see
shortly how it ends up being called)::
static PyObject *
spam_system(PyObject *self, PyObject *args)
{
const char *command;
int sts;
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
sts = system(command);
return PyLong_FromLong(sts);
}
There is a straightforward translation from the argument list in Python (for
example, the single expression ``"ls -l"``) to the arguments passed to the C
function. The C function always has two arguments, conventionally named *self*
and *args*.
The *self* argument points to the module object for module-level functions;
for a method it would point to the object instance.
The *args* argument will be a pointer to a Python tuple object containing the
arguments. Each item of the tuple corresponds to an argument in the call's
argument list. The arguments are Python objects --- in order to do anything
with them in our C function we have to convert them to C values. The function
:c:func:`PyArg_ParseTuple` in the Python API checks the argument types and
converts them to C values. It uses a template string to determine the required
types of the arguments as well as the types of the C variables into which to
store the converted values. More about this later.
:c:func:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right
type and its components have been stored in the variables whose addresses are
passed. It returns false (zero) if an invalid argument list was passed. In the
latter case it also raises an appropriate exception so the calling function can
return ``NULL`` immediately (as we saw in the example).
The :ref:`tutorial <first-extension-module>` walked you through
creating a C API extension module, but left many areas unexplained.
This document looks at several concepts that you'll need to learn
in order to write more complex extensions.
.. _extending-errors:
Intermezzo: Errors and Exceptions
=================================
Errors and Exceptions
=====================
An important convention throughout the Python interpreter is the following: when
a function fails, it should set an exception condition and return an error value
@ -321,194 +187,14 @@ call to :c:func:`PyErr_SetString` as shown below::
}
.. _backtoexample:
Back to the Example
===================
Going back to our example function, you should now be able to understand this
statement::
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
It returns ``NULL`` (the error indicator for functions returning object pointers)
if an error is detected in the argument list, relying on the exception set by
:c:func:`PyArg_ParseTuple`. Otherwise the string value of the argument has been
copied to the local variable :c:data:`!command`. This is a pointer assignment and
you are not supposed to modify the string to which it points (so in Standard C,
the variable :c:data:`!command` should properly be declared as ``const char
*command``).
The next statement is a call to the Unix function :c:func:`system`, passing it
the string we just got from :c:func:`PyArg_ParseTuple`::
sts = system(command);
Our :func:`!spam.system` function must return the value of :c:data:`!sts` as a
Python object. This is done using the function :c:func:`PyLong_FromLong`. ::
return PyLong_FromLong(sts);
In this case, it will return an integer object. (Yes, even integers are objects
on the heap in Python!)
If you have a C function that returns no useful argument (a function returning
:c:expr:`void`), the corresponding Python function must return ``None``. You
need this idiom to do so (which is implemented by the :c:macro:`Py_RETURN_NONE`
macro)::
Py_INCREF(Py_None);
return Py_None;
:c:data:`Py_None` is the C name for the special Python object ``None``. It is a
genuine Python object rather than a ``NULL`` pointer, which means "error" in most
contexts, as we have seen.
.. _methodtable:
The Module's Method Table and Initialization Function
=====================================================
I promised to show how :c:func:`!spam_system` is called from Python programs.
First, we need to list its name and address in a "method table"::
static PyMethodDef spam_methods[] = {
...
{"system", spam_system, METH_VARARGS,
"Execute a shell command."},
...
{NULL, NULL, 0, NULL} /* Sentinel */
};
Note the third entry (``METH_VARARGS``). This is a flag telling the interpreter
the calling convention to be used for the C function. It should normally always
be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means
that an obsolete variant of :c:func:`PyArg_ParseTuple` is used.
When using only ``METH_VARARGS``, the function should expect the Python-level
parameters to be passed in as a tuple acceptable for parsing via
:c:func:`PyArg_ParseTuple`; more information on this function is provided below.
The :c:macro:`METH_KEYWORDS` bit may be set in the third field if keyword
arguments should be passed to the function. In this case, the C function should
accept a third ``PyObject *`` parameter which will be a dictionary of keywords.
Use :c:func:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a
function.
The method table must be referenced in the module definition structure::
static struct PyModuleDef spam_module = {
...
.m_methods = spam_methods,
...
};
This structure, in turn, must be passed to the interpreter in the module's
initialization function. The initialization function must be named
:c:func:`!PyInit_name`, where *name* is the name of the module, and should be the
only non-\ ``static`` item defined in the module file::
PyMODINIT_FUNC
PyInit_spam(void)
{
return PyModuleDef_Init(&spam_module);
}
Note that :c:macro:`PyMODINIT_FUNC` declares the function as ``PyObject *`` return type,
declares any special linkage declarations required by the platform, and for C++
declares the function as ``extern "C"``.
:c:func:`!PyInit_spam` is called when each interpreter imports its module
:mod:`!spam` for the first time. (See below for comments about embedding Python.)
A pointer to the module definition must be returned via :c:func:`PyModuleDef_Init`,
so that the import machinery can create the module and store it in ``sys.modules``.
When embedding Python, the :c:func:`!PyInit_spam` function is not called
automatically unless there's an entry in the :c:data:`PyImport_Inittab` table.
To add the module to the initialization table, use :c:func:`PyImport_AppendInittab`,
optionally followed by an import of the module::
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int
main(int argc, char *argv[])
{
PyStatus status;
PyConfig config;
PyConfig_InitPythonConfig(&config);
/* Add a built-in module, before Py_Initialize */
if (PyImport_AppendInittab("spam", PyInit_spam) == -1) {
fprintf(stderr, "Error: could not extend in-built modules table\n");
exit(1);
}
/* Pass argv[0] to the Python interpreter */
status = PyConfig_SetBytesString(&config, &config.program_name, argv[0]);
if (PyStatus_Exception(status)) {
goto exception;
}
/* Initialize the Python interpreter. Required.
If this step fails, it will be a fatal error. */
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto exception;
}
PyConfig_Clear(&config);
/* Optionally import the module; alternatively,
import can be deferred until the embedded script
imports it. */
PyObject *pmodule = PyImport_ImportModule("spam");
if (!pmodule) {
PyErr_Print();
fprintf(stderr, "Error: could not import module 'spam'\n");
}
// ... use Python C API here ...
return 0;
exception:
PyConfig_Clear(&config);
Py_ExitStatusException(status);
}
.. note::
If you declare a global variable or a local static one, the module may
experience unintended side-effects on re-initialisation, for example when
removing entries from ``sys.modules`` or importing compiled modules into
multiple interpreters within a process
(or following a :c:func:`fork` without an intervening :c:func:`exec`).
If module state is not yet fully :ref:`isolated <isolating-extensions-howto>`,
authors should consider marking the module as having no support for subinterpreters
(via :c:macro:`Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED`).
A more substantial example module is included in the Python source distribution
as :file:`Modules/xxlimited.c`. This file may be used as a template or simply
read as an example.
.. _compilation:
Compilation and Linkage
=======================
Embedding an extension
======================
There are two more things to do before you can use your new extension: compiling
and linking it with the Python system. If you use dynamic loading, the details
may depend on the style of dynamic loading your system uses; see the chapters
about building extension modules (chapter :ref:`building`) and additional
information that pertains only to building on Windows (chapter
:ref:`building-on-windows`) for more information about this.
If you can't use dynamic loading, or if you want to make your module a permanent
If you want to make your module a permanent
part of the Python interpreter, you will have to change the configuration setup
and rebuild the interpreter. Luckily, this is very simple on Unix: just place
and rebuild the interpreter. On Unix, place
your file (:file:`spammodule.c` for example) in the :file:`Modules/` directory
of an unpacked source distribution, add a line to the file
:file:`Modules/Setup.local` describing your file:
@ -536,7 +222,7 @@ on the line in the configuration file as well, for instance:
Calling Python Functions from C
===============================
So far we have concentrated on making C functions callable from Python. The
The tutorial concentrated on making C functions callable from Python. The
reverse is also useful: calling Python functions from C. This is especially the
case for libraries that support so-called "callback" functions. If a C
interface makes use of callbacks, the equivalent Python often needs to provide a
@ -581,7 +267,7 @@ be part of a module definition::
}
This function must be registered with the interpreter using the
:c:macro:`METH_VARARGS` flag; this is described in section :ref:`methodtable`. The
:c:macro:`METH_VARARGS` flag in :c:type:`PyMethodDef.ml_flags`. The
:c:func:`PyArg_ParseTuple` function and its arguments are documented in section
:ref:`parsetuple`.
@ -676,14 +362,21 @@ the above example, we use :c:func:`Py_BuildValue` to construct the dictionary. :
Py_DECREF(result);
.. index:: single: PyArg_ParseTuple (C function)
.. _parsetuple:
Extracting Parameters in Extension Functions
============================================
.. index:: single: PyArg_ParseTuple (C function)
The :ref:`tutorial <first-extension-module>` uses a ":c:data:`METH_O`"
function, which is limited to a single Python argument.
If you want more, you can use :c:data:`METH_VARARGS` instead.
With this flag, the C function will receive a :py:class:`tuple` of arguments
instead of a single object.
The :c:func:`PyArg_ParseTuple` function is declared as follows::
For unpacking the tuple, CPython provides the :c:func:`PyArg_ParseTuple`
function, declared as follows::
int PyArg_ParseTuple(PyObject *arg, const char *format, ...);
@ -693,6 +386,19 @@ whose syntax is explained in :ref:`arg-parsing` in the Python/C API Reference
Manual. The remaining arguments must be addresses of variables whose type is
determined by the format string.
For example, to receive a single Python :py:class:`str` object and turn it
into a C buffer, you would use ``"s"`` as the format string::
const char *command;
if (!PyArg_ParseTuple(args, "s", &command)) {
return NULL;
}
If an error is detected in the argument list, :c:func:`!PyArg_ParseTuple`
returns ``NULL`` (the error indicator for functions returning object pointers);
your function may return ``NULL``, relying on the exception set by
:c:func:`PyArg_ParseTuple`.
Note that while :c:func:`PyArg_ParseTuple` checks that the Python arguments have
the required types, it cannot check the validity of the addresses of C variables
passed to the call: if you make mistakes there, your code will probably crash or
@ -703,7 +409,6 @@ Note that any Python object references which are provided to the caller are
Some example calls::
#define PY_SSIZE_T_CLEAN
#include <Python.h>
::
@ -773,6 +478,17 @@ Some example calls::
Keyword Parameters for Extension Functions
==========================================
If you also want your function to accept
:term:`keyword arguments <keyword argument>`, use the :c:data:`METH_KEYWORDS`
flag in combination with :c:data:`METH_VARARGS`.
(:c:data:`!METH_KEYWORDS` can also be used with other flags; see its
documentation for the allowed combinations.)
In this case, the C function should accept a third ``PyObject *`` parameter
which will be a dictionary of keywords.
Use :c:func:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a
function.
.. index:: single: PyArg_ParseTupleAndKeywords (C function)
The :c:func:`PyArg_ParseTupleAndKeywords` function is declared as follows::
@ -833,19 +549,6 @@ Philbrick (philbrick@hks.com)::
{NULL, NULL, 0, NULL} /* sentinel */
};
static struct PyModuleDef keywdarg_module = {
.m_base = PyModuleDef_HEAD_INIT,
.m_name = "keywdarg",
.m_size = 0,
.m_methods = keywdarg_methods,
};
PyMODINIT_FUNC
PyInit_keywdarg(void)
{
return PyModuleDef_Init(&keywdarg_module);
}
.. _buildvalue:
@ -986,11 +689,11 @@ needed. Ownership of a reference can be transferred. There are three ways to
dispose of an owned reference: pass it on, store it, or call :c:func:`Py_DECREF`.
Forgetting to dispose of an owned reference creates a memory leak.
It is also possible to :dfn:`borrow` [#]_ a reference to an object. The
It is also possible to :dfn:`borrow` [#borrow]_ a reference to an object. The
borrower of a reference should not call :c:func:`Py_DECREF`. The borrower must
not hold on to the object longer than the owner from which it was borrowed.
Using a borrowed reference after the owner has disposed of it risks using freed
memory and should be avoided completely [#]_.
memory and should be avoided completely [#dont-check-refcount]_.
The advantage of borrowing over owning a reference is that you don't need to
take care of disposing of the reference on all possible paths through the code
@ -1169,7 +872,7 @@ checking.
The C function calling mechanism guarantees that the argument list passed to C
functions (``args`` in the examples) is never ``NULL`` --- in fact it guarantees
that it is always a tuple [#]_.
that it is always a tuple [#old-calling-convention]_.
It is a severe error to ever let a ``NULL`` pointer "escape" to the Python user.
@ -1226,8 +929,8 @@ the module whose functions one wishes to call might not have been loaded yet!
Portability therefore requires not to make any assumptions about symbol
visibility. This means that all symbols in extension modules should be declared
``static``, except for the module's initialization function, in order to
avoid name clashes with other extension modules (as discussed in section
:ref:`methodtable`). And it means that symbols that *should* be accessible from
avoid name clashes with other extension modules. And it means that symbols
that *should* be accessible from
other extension modules must be exported in a different way.
Python provides a special mechanism to pass C-level information (pointers) from
@ -1269,8 +972,9 @@ file corresponding to the module provides a macro that takes care of importing
the module and retrieving its C API pointers; client modules only have to call
this macro before accessing the C API.
The exporting module is a modification of the :mod:`!spam` module from section
:ref:`extending-simpleexample`. The function :func:`!spam.system` does not call
The exporting module is a modification of the :mod:`!spam` module from the
:ref:`tutorial <first-extension-module>`.
The function :func:`!spam.system` does not call
the C library function :c:func:`system` directly, but a function
:c:func:`!PySpam_System`, which would of course do something more complicated in
reality (such as adding "spam" to every command). This function
@ -1412,15 +1116,14 @@ code distribution).
.. rubric:: Footnotes
.. [#] An interface for this function already exists in the standard module :mod:`os`
--- it was chosen as a simple and straightforward example.
.. [#borrow] The metaphor of "borrowing" a reference is not completely correct:
the owner still has a copy of the reference.
.. [#] The metaphor of "borrowing" a reference is not completely correct: the owner
still has a copy of the reference.
.. [#] Checking that the reference count is at least 1 **does not work** --- the
.. [#dont-check-refcount] Checking that the reference count is at least 1
**does not work** --- the
reference count itself could be in freed memory and may thus be reused for
another object!
.. [#] These guarantees don't hold when you use the "old" style calling convention ---
.. [#old-calling-convention] These guarantees don't hold when you use the
"old" style calling convention ---
this is still found in much existing code.

View File

@ -0,0 +1,667 @@
.. highlight:: c
.. _extending-simpleexample:
.. _first-extension-module:
*********************************
Your first C API extension module
*********************************
This tutorial will take you through creating a simple
Python extension module written in C or C++.
We will use the low-level Python C API directly.
For easier ways to create extension modules, see
the :ref:`recommended third party tools <c-api-tools>`.
The tutorial assumes basic knowledge about Python: you should be able to
define functions in Python code before starting to write them in C.
See :ref:`tutorial-index` for an introduction to Python itself.
The tutorial should be approachable for anyone who can write a basic C library.
While we will mention several concepts that a C beginner would not be expected
to know, like ``static`` functions or linkage declarations, understanding these
is not necessary for success.
We will focus on giving you a "feel" of what Python's C API is like.
It will not teach you important concepts, like error handling
and reference counting, which are covered in later chapters.
We will assume that you use a Unix-like system (including macOS and
Linux), or Windows.
On other systems, you might need to adjust some details -- for example,
a system command name.
You need to have a suitable C compiler and Python development headers installed.
On Linux, headers are often in a package like ``python3-dev``
or ``python3-devel``.
You need to be able to install Python packages.
This tutorial uses `pip <https://pip.pypa.io/>`__ (``pip install``), but you
can substitute any tool that can build and install ``pyproject.toml``-based
projects, like `uv <https://docs.astral.sh/uv/>`_ (``uv pip install``).
Preferably, have a :ref:`virtual environment <venv-def>` activated.
.. note::
This tutorial uses APIs that were added in CPython 3.15.
To create an extension that's compatible with earlier versions of CPython,
please follow an earlier version of this documentation.
This tutorial uses C syntax added in C11 and C++20.
If your extension needs to be compatible with earlier standards,
please follow tutorials in documentation for Python 3.14 or below.
What we'll do
=============
Let's create an extension module called ``spam`` [#why-spam]_,
which will include a Python interface to the C
standard library function :c:func:`system`.
This function is defined in ``stdlib.h``.
It takes a C string as argument, runs the argument as a system
command, and returns a result value as an integer.
A manual page for :c:func:`system` might summarize it this way::
#include <stdlib.h>
int system(const char *command);
Note that like many functions in the C standard library,
this function is already exposed in Python.
In production, use :py:func:`os.system` or :py:func:`subprocess.run`
rather than the module you'll write here.
We want this function to be callable from Python as follows:
.. code-block:: pycon
>>> import spam
>>> status = spam.system("whoami")
User Name
>>> status
0
.. note::
The system command ``whoami`` prints out your username.
It's useful in tutorials like this one because it has the same name on
both Unix and Windows.
Start with the headers
======================
Begin by creating a directory for this tutorial, and switching to it
on the command line.
Then, create a file named :file:`spammodule.c` in your directory.
[#why-spammodule]_
In this file, we'll include two headers: :file:`Python.h` to pull in
all declarations of the Python C API, and :file:`stdlib.h` for the
:c:func:`system` function. [#stdlib-h]_
Add the following lines to :file:`spammodule.c`:
.. literalinclude:: ../includes/capi-extension/spammodule-01.c
:start-at: <Python.h>
:end-at: <stdlib.h>
Be sure to put :file:`stdlib.h`, and any other standard library includes,
*after* :file:`Python.h`.
On some systems, Python may define some pre-processor definitions
that affect the standard headers.
Running your build tool
=======================
With only the includes in place, your extension won't do anything.
Still, it's a good time to compile it and try to import it.
This will ensure that your build tool works, so that you can make
and test incremental changes as you follow the rest of the text.
CPython itself does not come with a tool to build extension modules;
it is recommended to use a third-party project for this.
In this tutorial, we'll use `meson-python`_.
(If you want to use another one, see :ref:`first-extension-other-tools`.)
.. at the time of writing, meson-python has the least overhead for a
simple extension using PyModExport.
Change this if another tool makes things easier.
``meson-python`` requires defining a "project" using two extra files.
First, add ``pyproject.toml`` with these contents:
.. code-block:: toml
[build-system]
build-backend = 'mesonpy'
requires = ['meson-python']
[project]
# Placeholder project information
# (change this before distributing the module)
name = 'sampleproject'
version = '0'
Then, create ``meson.build`` containing the following:
.. code-block:: meson
project('sampleproject', 'c')
py = import('python').find_installation(pure: false)
py.extension_module(
'spam', # name of the importable Python module
'spammodule.c', # the C source file
install: true,
)
.. note::
See `meson-python documentation <meson-python>`_ for details on
configuration.
Now, build install the *project in the current directory* (``.``) via ``pip``:
.. code-block:: sh
python -m pip install .
.. tip::
If you don't have ``pip`` installed, run ``python -m ensurepip``,
preferably in a :ref:`virtual environment <venv-def>`.
(Or, if you prefer another tool that can build and install
``pyproject.toml``-based projects, use that.)
.. _meson-python: https://mesonbuild.com/meson-python/
.. _virtual environment: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments
Note that you will need to run this command again every time you change your
extension.
Unlike Python, C has an explicit compilation step.
When your extension is compiled and installed, start Python and try to
import it.
This should fail with the following exception:
.. code-block:: pycon
>>> import spam
Traceback (most recent call last):
...
ImportError: dynamic module does not define module export function (PyModExport_spam or PyInit_spam)
Module export hook
==================
The exception you got when you tried to import the module told you that Python
is looking for a "module export function", also known as a
:ref:`module export hook <extension-export-hook>`.
Let's define one.
First, add a prototype below the ``#include`` lines:
.. literalinclude:: ../includes/capi-extension/spammodule-01.c
:start-after: /// Export hook prototype
:end-before: ///
.. tip::
The prototype is not strictly necessary, but some modern compilers emit
warnings without it.
It's generally better to add the prototype than to disable the warning.
The :c:macro:`PyMODEXPORT_FUNC` macro declares the function's
return type, and adds any special linkage declarations needed
to make the function visible and usable when CPython loads it.
After the prototype, add the function itself.
For now, make it return ``NULL``:
.. code-block:: c
PyMODEXPORT_FUNC
PyModExport_spam(void)
{
return NULL;
}
Compile and load the module again.
You should get a different error this time.
.. code-block:: pycon
>>> import spam
Traceback (most recent call last):
...
SystemError: module export hook for module 'spam' failed without setting an exception
Simply returning ``NULL`` is *not* correct behavior for an export hook,
and CPython complains about it.
That's good -- it means that CPython found the function!
Let's now make it do something useful.
The slot table
==============
Rather than ``NULL``, the export hook should return the information needed to
create a module.
Let's start with the basics: the name and docstring.
The information should be defined in a ``static`` array of
:c:type:`PyModuleDef_Slot` entries, which are essentially key-value pairs.
Define this array just before your export hook:
.. code-block:: c
static PyModuleDef_Slot spam_slots[] = {
{Py_mod_name, "spam"},
{Py_mod_doc, "A wonderful module with an example function"},
{0, NULL}
};
For both :c:data:`Py_mod_name` and :c:data:`Py_mod_doc`, the values are C
strings -- that is, NUL-terminated, UTF-8 encoded byte arrays.
Note the zero-filled sentinel entry at the end.
If you forget it, you'll trigger undefined behavior.
The array is defined as ``static`` -- that is, not visible outside this ``.c`` file.
This will be a common theme.
CPython only needs to access the export hook; all global variables
and all other functions should generally be ``static``, so that they don't
clash with other extensions.
Return this array from your export hook instead of ``NULL``:
.. code-block:: c
:emphasize-lines: 4
PyMODEXPORT_FUNC
PyModExport_spam(void)
{
return spam_slots;
}
Now, recompile and try it out:
.. code-block:: pycon
>>> import spam
>>> print(spam)
<module 'spam' from '/home/encukou/dev/cpython/spam.so'>
You have an extension module!
Try ``help(spam)`` to see the docstring.
The next step will be adding a function.
.. _backtoexample:
Exposing a function
===================
To expose the :c:func:`system` C function directly to Python,
we'll need to write a layer of glue code to convert arguments from Python
objects to C values, and the C return value back to Python.
One of the simplest ways to write glue code is a ":c:data:`METH_O`" function,
which takes two Python objects and returns one.
All Python objects -- regardless of the Python type -- are represented in C
as pointers to the :c:type:`PyObject` structure.
Add such a function above the slots array::
static PyObject *
spam_system(PyObject *self, PyObject *arg)
{
Py_RETURN_NONE;
}
For now, we ignore the arguments, and use the :c:macro:`Py_RETURN_NONE`
macro, which expands to a ``return`` statement that properly returns
a Python :py:data:`None` object.
Recompile your extension to make sure you don't have syntax errors.
We haven't yet added ``spam_system`` to the module, so you might get a
warning that ``spam_system`` is unused.
.. _methodtable:
Method definitions
------------------
To expose the C function to Python, you will need to provide several pieces of
information in a structure called
:c:type:`PyMethodDef` [#why-pymethoddef]_:
* ``ml_name``: the name of the Python function;
* ``ml_doc``: a docstring;
* ``ml_meth``: the C function to be called; and
* ``ml_flags``: a set of flags describing details like how Python arguments are
passed to the C function.
We'll use :c:data:`METH_O` here -- the flag that matches our
``spam_system`` function's signature.
Because modules typically create several functions, these definitions
need to be collected in an array, with a zero-filled sentinel at the end.
Add this array just below the ``spam_system`` function:
.. literalinclude:: ../includes/capi-extension/spammodule-01.c
:start-after: /// Module method table
:end-before: ///
As with module slots, a zero-filled sentinel marks the end of the array.
Next, we'll add the method to the module.
Add a :c:data:`Py_mod_methods` slot to your :c:type:`PyMethodDef` array:
.. literalinclude:: ../includes/capi-extension/spammodule-01.c
:start-after: /// Module slot table
:end-before: ///
:emphasize-lines: 5
Recompile your extension again, and test it.
Be sure to restart the Python interpreter, so that ``import spam`` picks
up the new version of the module.
You should now be able to call the function:
.. code-block:: pycon
>>> import spam
>>> print(spam.system)
<built-in function system>
>>> print(spam.system('whoami'))
None
Note that our ``spam.system`` does not yet run the ``whoami`` command;
it only returns ``None``.
Check that the function accepts exactly one argument, as specified by
the :c:data:`METH_O` flag:
.. code-block:: pycon
>>> print(spam.system('too', 'many', 'arguments'))
Traceback (most recent call last):
...
TypeError: spam.system() takes exactly one argument (3 given)
Returning an integer
====================
Now, let's take a look at the return value.
Instead of ``None``, we'll want ``spam.system`` to return a number -- that is,
a Python :py:type:`int` object.
Eventually this will be the exit code of a system command,
but let's start with a fixed value, say, ``3``.
The Python C API provides a function to create a Python :py:type:`int` object
from a C ``int`` value: :c:func:`PyLong_FromLong`. [#why-pylongfromlong]_
To call it, replace the ``Py_RETURN_NONE`` with the following 3 lines:
.. this could be a one-liner, but we want to show the data types here
.. code-block:: c
:emphasize-lines: 4-6
static PyObject *
spam_system(PyObject *self, PyObject *arg)
{
int status = 3;
PyObject *result = PyLong_FromLong(status);
return result;
}
Recompile, restart the Python interpreter again,
and check that the function now returns 3:
.. code-block:: pycon
>>> import spam
>>> spam.system('whoami')
3
Accepting a string
==================
Finally, let's handle the function argument.
Our C function, :c:func:`!spam_system`, takes two arguments.
The first one, ``PyObject *self``, will be set to the ``spam`` module
object.
This isn't useful in our case, so we'll ignore it.
The other one, ``PyObject *arg``, will be set to the object that the user
passed from Python.
We expect that it should be a Python string.
In order to use the information in it, we will need
to convert it to a C value -- in this case, a C string (``const char *``).
There's a slight type mismatch here: Python's :py:class:`str` objects store
Unicode text, but C strings are arrays of bytes.
So, we'll need to *encode* the data, and we'll use the UTF-8 encoding for it.
(UTF-8 might not always be correct for system commands, but it's what
:py:meth:`str.encode` uses by default,
and the C API has special support for it.)
The function to encode a Python string into a UTF-8 buffer is named
:c:func:`PyUnicode_AsUTF8` [#why-pyunicodeasutf8]_.
Call it like this:
.. code-block:: c
:emphasize-lines: 4
static PyObject *
spam_system(PyObject *self, PyObject *arg)
{
const char *command = PyUnicode_AsUTF8(arg);
int status = 3;
PyObject *result = PyLong_FromLong(status);
return result;
}
If :c:func:`PyUnicode_AsUTF8` is successful, *command* will point to the
resulting array of bytes.
This buffer is managed by the *arg* object, which means we don't need to free
it, but we must follow some rules:
* We should only use the buffer inside the ``spam_system`` function.
When ``spam_system`` returns, *arg* and the buffer it manages might be
garbage-collected.
* We must not modify it. This is why we use ``const``.
If :c:func:`PyUnicode_AsUTF8` was *not* successful, it returns a ``NULL``
pointer.
When calling *any* Python C API, we always need to handle such error cases.
The way to do this in general is left for later chapters of this documentation.
For now, be assured that we are already handling errors from
:c:func:`PyLong_FromLong` correctly.
For the :c:func:`PyUnicode_AsUTF8` call, the correct way to handle errors is
returning ``NULL`` from ``spam_system``.
Add an ``if`` block for this:
.. code-block:: c
:emphasize-lines: 5-7
static PyObject *
spam_system(PyObject *self, PyObject *arg)
{
const char *command = PyUnicode_AsUTF8(arg);
if (command == NULL) {
return NULL;
}
int status = 3;
PyObject *result = PyLong_FromLong(status);
return result;
}
That's it for the setup.
Now, all that is left is calling the C library function :c:func:`system` with
the ``char *`` buffer, and using its result instead of the ``3``:
.. code-block:: c
:emphasize-lines: 8
static PyObject *
spam_system(PyObject *self, PyObject *arg)
{
const char *command = PyUnicode_AsUTF8(arg);
if (command == NULL) {
return NULL;
}
int status = system(command);
PyObject *result = PyLong_FromLong(status);
return result;
}
Compile your module, restart Python, and test.
This time, you should see your username -- the output of the ``whoami``
system command:
.. code-block:: pycon
>>> import spam
>>> result = spam.system('whoami')
User Name
>>> result
0
You might also want to test error cases:
.. code-block:: pycon
>>> import spam
>>> result = spam.system('nonexistent-command')
sh: line 1: nonexistent-command: command not found
>>> result
32512
>>> spam.system(3)
Traceback (most recent call last):
...
TypeError: bad argument type for built-in operation
The result
==========
Congratulations!
You have written a complete Python C API extension module,
and completed this tutorial!
Here is the entire source file, for your convenience:
.. _extending-spammodule-source:
.. literalinclude:: ../includes/capi-extension/spammodule-01.c
:start-at: ///
.. _first-extension-other-tools:
Appendix: Other build tools
===========================
You should be able to follow this tutorial -- except the
*Running your build tool* section itself -- with a build tool other
than ``meson-python``.
The Python Packaging User Guide has a `list of recommended tools <https://packaging.python.org/en/latest/guides/tool-recommendations/#build-backends-for-extension-modules>`_;
be sure to choose one for the C language.
Workaround for missing PyInit function
--------------------------------------
If your build tool output complains about missing ``PyInit_spam``,
add the following function to your module for now:
.. code-block:: c
// A workaround
void *PyInit_spam(void) { return NULL; }
This is a shim for an old-style :ref:`initialization function <extension-export-hook>`,
which was required in extension modules for CPython 3.14 and below.
Current CPython does not need it, but some build tools may still assume that
all extension modules need to define it.
If you use this workaround, you will get the exception
``SystemError: initialization of spam failed without raising an exception``
instead of
``ImportError: dynamic module does not define module export function``.
Compiling directly
------------------
Using a third-party build tool is heavily recommended,
as it will take care of various details of your platform and Python
installation, of naming the resulting extension, and, later, of distributing
your work.
If you are building an extension for as *specific* system, or for yourself
only, you might instead want to run your compiler directly.
The way to do this is system-specific; be prepared for issues you will need
to solve yourself.
Linux
^^^^^
On Linux, the Python development package may include a ``python3-config``
command that prints out the required compiler flags.
If you use it, check that it corresponds to the CPython interpreter you'll use
to load the module.
Then, start with the following command:
.. code-block:: sh
gcc --shared $(python3-config --cflags --ldflags) spammodule.c -o spam.so
This should generate a ``spam.so`` file that you need to put in a directory
on :py:attr:`sys.path`.
.. rubric:: Footnotes
.. [#why-spam] ``spam`` is the favorite food of Monty Python fans...
.. [#why-spammodule] The source file name is entirely up to you,
though some tools can be picky about the ``.c`` extension.
This tutorial uses the traditional ``*module.c`` suffix.
Some people would just use :file:`spam.c` to implement a module
named ``spam``,
projects where Python isn't the primary language might use ``py_spam.c``,
and so on.
.. [#stdlib-h] Including :file:`stdlib.h` is technically not necessary,
since :file:`Python.h` includes it and
:ref:`several other standard headers <capi-system-includes>` for its own use
or for backwards compatibility.
However, it is good practice to explicitly include what you need.
.. [#why-pymethoddef] The :c:type:`!PyMethodDef` structure is also used
to create methods of classes, so there's no separate
":c:type:`!PyFunctionDef`".
.. [#why-pylongfromlong] The name :c:func:`PyLong_FromLong`
might not seem obvious.
``PyLong`` refers to a the Python :py:class:`int`, which was originally
called ``long``; the ``FromLong`` refers to the C ``long`` (or ``long int``)
type.
.. [#why-pyunicodeasutf8] Here, ``PyUnicode`` refers to the original name of
the Python :py:class:`str` class: ``unicode``.

View File

@ -5,15 +5,17 @@
##################################################
This document describes how to write modules in C or C++ to extend the Python
interpreter with new modules. Those modules can not only define new functions
but also new object types and their methods. The document also describes how
interpreter with new modules. Those modules can do what Python code does --
define functions, object types and methods -- but also interact with native
libraries or achieve better performance by avoiding the overhead of an
interpreter. The document also describes how
to embed the Python interpreter in another application, for use as an extension
language. Finally, it shows how to compile and link extension modules so that
they can be loaded dynamically (at run time) into the interpreter, if the
underlying operating system supports this feature.
This document assumes basic knowledge about Python. For an informal
introduction to the language, see :ref:`tutorial-index`. :ref:`reference-index`
This document assumes basic knowledge about C and Python. For an informal
introduction to Python, see :ref:`tutorial-index`. :ref:`reference-index`
gives a more formal definition of the language. :ref:`library-index` documents
the existing object types, functions and modules (both built-in and written in
Python) that give the language its wide application range.
@ -21,37 +23,75 @@ Python) that give the language its wide application range.
For a detailed description of the whole Python/C API, see the separate
:ref:`c-api-index`.
To support extensions, Python's C API (Application Programmers Interface)
defines a set of functions, macros and variables that provide access to most
aspects of the Python run-time system. The Python API is incorporated in a C
source file by including the header ``"Python.h"``.
.. note::
The C extension interface is specific to CPython, and extension modules do
not work on other Python implementations. In many cases, it is possible to
avoid writing C extensions and preserve portability to other implementations.
For example, if your use case is calling C library functions or system calls,
you should consider using the :mod:`ctypes` module or the `cffi
<https://cffi.readthedocs.io/>`_ library rather than writing
custom C code.
These modules let you write Python code to interface with C code and are more
portable between implementations of Python than writing and compiling a C
extension module.
.. toctree::
:hidden:
first-extension-module.rst
extending.rst
newtypes_tutorial.rst
newtypes.rst
building.rst
windows.rst
embedding.rst
Recommended third party tools
=============================
This guide only covers the basic tools for creating extensions provided
This document only covers the basic tools for creating extensions provided
as part of this version of CPython. Some :ref:`third party tools
<c-api-tools>` offer both simpler and more sophisticated approaches to creating
C and C++ extensions for Python.
While this document is aimed at extension authors, it should also be helpful to
the authors of such tools.
For example, the tutorial module can serve as a simple test case for a build
tool or sample expected output of a code generator.
Creating extensions without third party tools
=============================================
C API Tutorial
==============
This tutorial describes how to write a simple module in C or C++,
using the Python C API -- that is, using the basic tools provided
as part of this version of CPython.
#. :ref:`first-extension-module`
Guides for intermediate topics
==============================
This section of the guide covers creating C and C++ extensions without
assistance from third party tools. It is intended primarily for creators
of those tools, rather than being a recommended way to create your own
C extensions.
.. seealso::
:pep:`489` -- Multi-phase extension module initialization
.. toctree::
:maxdepth: 2
:numbered:
extending.rst
newtypes_tutorial.rst
newtypes.rst
building.rst
windows.rst
* :ref:`extending-intro`
* :ref:`defining-new-types`
* :ref:`new-types-topics`
* :ref:`building`
* :ref:`building-on-windows`
Embedding the CPython runtime in a larger application
=====================================================
@ -61,8 +101,4 @@ interpreter as the main application, it is desirable to instead embed
the CPython runtime inside a larger application. This section covers
some of the details involved in doing that successfully.
.. toctree::
:maxdepth: 2
:numbered:
embedding.rst
* :ref:`embedding`

View File

@ -1226,13 +1226,13 @@ This converts the list into a set, thereby removing duplicates, and then back
into a list.
How do you remove multiple items from a list
--------------------------------------------
How do you remove multiple items from a list?
---------------------------------------------
As with removing duplicates, explicitly iterating in reverse with a
delete condition is one possibility. However, it is easier and faster
to use slice replacement with an implicit or explicit forward iteration.
Here are three variations.::
Here are three variations::
mylist[:] = filter(keep_function, mylist)
mylist[:] = (x for x in mylist if keep_condition)

View File

@ -134,6 +134,14 @@ Glossary
iterator's :meth:`~object.__anext__` method until it raises a
:exc:`StopAsyncIteration` exception. Introduced by :pep:`492`.
atomic operation
An operation that appears to execute as a single, indivisible step: no
other thread can observe it half-done, and its effects become visible all
at once. Python does not guarantee that high-level statements are atomic
(for example, ``x += 1`` performs multiple bytecode operations and is not
atomic). Atomicity is only guaranteed where explicitly documented. See
also :term:`race condition` and :term:`data race`.
attached thread state
A :term:`thread state` that is active for the current OS thread.
@ -289,6 +297,22 @@ Glossary
advanced mathematical feature. If you're not aware of a need for them,
it's almost certain you can safely ignore them.
concurrency
The ability of a computer program to perform multiple tasks at the same
time. Python provides libraries for writing programs that make use of
different forms of concurrency. :mod:`asyncio` is a library for dealing
with asynchronous tasks and coroutines. :mod:`threading` provides
access to operating system threads and :mod:`multiprocessing` to
operating system processes. Multi-core processors can execute threads and
processes on different CPU cores at the same time (see
:term:`parallelism`).
concurrent modification
When multiple threads modify shared data at the same time. Concurrent
modification without proper synchronization can cause
:term:`race conditions <race condition>`, and might also trigger a
:term:`data race <data race>`, data corruption, or both.
context
This term has different meanings depending on where and how it is used.
Some common meanings:
@ -363,6 +387,28 @@ Glossary
the :term:`cyclic garbage collector <garbage collection>` is to identify these groups and break the reference
cycles so that the memory can be reclaimed.
data race
A situation where multiple threads access the same memory location
concurrently, at least one of the accesses is a write, and the threads
do not use any synchronization to control their access. Data races
lead to :term:`non-deterministic` behavior and can cause data corruption.
Proper use of :term:`locks <lock>` and other :term:`synchronization primitives
<synchronization primitive>` prevents data races. Note that data races
can only happen in native code, but that :term:`native code` might be
exposed in a Python API. See also :term:`race condition` and
:term:`thread-safe`.
deadlock
A situation in which two or more tasks (threads, processes, or coroutines)
wait indefinitely for each other to release resources or complete actions,
preventing any from making progress. For example, if thread A holds lock
1 and waits for lock 2, while thread B holds lock 2 and waits for lock 1,
both threads will wait indefinitely. In Python this often arises from
acquiring multiple locks in conflicting orders or from circular
join/await dependencies. Deadlocks can be avoided by always acquiring
multiple :term:`locks <lock>` in a consistent order. See also
:term:`lock` and :term:`reentrant`.
decorator
A function returning another function, usually applied as a function
transformation using the ``@wrapper`` syntax. Common examples for
@ -662,6 +708,14 @@ Glossary
requires the GIL to be held in order to use it. This refers to having an
:term:`attached thread state`.
global state
Data that is accessible throughout a program, such as module-level
variables, class variables, or C static variables in :term:`extension modules
<extension module>`. In multi-threaded programs, global state shared
between threads typically requires synchronization to avoid
:term:`race conditions <race condition>` and
:term:`data races <data race>`.
hash-based pyc
A bytecode cache file that uses the hash rather than the last-modified
time of the corresponding source file to determine its validity. See
@ -706,7 +760,9 @@ Glossary
tuples. Such an object cannot be altered. A new object has to
be created if a different value has to be stored. They play an important
role in places where a constant hash value is needed, for example as a key
in a dictionary.
in a dictionary. Immutable objects are inherently :term:`thread-safe`
because their state cannot be modified after creation, eliminating concerns
about improperly synchronized :term:`concurrent modification`.
import path
A list of locations (or :term:`path entries <path entry>`) that are
@ -796,8 +852,9 @@ Glossary
CPython does not consistently apply the requirement that an iterator
define :meth:`~iterator.__iter__`.
And also please note that the free-threading CPython does not guarantee
the thread-safety of iterator operations.
And also please note that :term:`free-threaded <free threading>`
CPython does not guarantee :term:`thread-safe` behavior of iterator
operations.
key function
@ -835,10 +892,11 @@ Glossary
:keyword:`if` statements.
In a multi-threaded environment, the LBYL approach can risk introducing a
race condition between "the looking" and "the leaping". For example, the
code, ``if key in mapping: return mapping[key]`` can fail if another
:term:`race condition` between "the looking" and "the leaping". For example,
the code, ``if key in mapping: return mapping[key]`` can fail if another
thread removes *key* from *mapping* after the test, but before the lookup.
This issue can be solved with locks or by using the EAFP approach.
This issue can be solved with :term:`locks <lock>` or by using the
:term:`EAFP` approach. See also :term:`thread-safe`.
lexical analyzer
@ -857,6 +915,19 @@ Glossary
clause is optional. If omitted, all elements in ``range(256)`` are
processed.
lock
A :term:`synchronization primitive` that allows only one thread at a
time to access a shared resource. A thread must acquire a lock before
accessing the protected resource and release it afterward. If a thread
attempts to acquire a lock that is already held by another thread, it
will block until the lock becomes available. Python's :mod:`threading`
module provides :class:`~threading.Lock` (a basic lock) and
:class:`~threading.RLock` (a :term:`reentrant` lock). Locks are used
to prevent :term:`race conditions <race condition>` and ensure
:term:`thread-safe` access to shared data. Alternative design patterns
to locks exist such as queues, producer/consumer patterns, and
thread-local state. See also :term:`deadlock`, and :term:`reentrant`.
loader
An object that loads a module.
It must define the :meth:`!exec_module` and :meth:`!create_module` methods
@ -942,8 +1013,11 @@ Glossary
See :term:`method resolution order`.
mutable
Mutable objects can change their value but keep their :func:`id`. See
also :term:`immutable`.
An :term:`object` with state that is allowed to change during the course
of the program. In multi-threaded programs, mutable objects that are
shared between threads require careful synchronization to avoid
:term:`race conditions <race condition>`. See also :term:`immutable`,
:term:`thread-safe`, and :term:`concurrent modification`.
named tuple
The term "named tuple" applies to any type or class that inherits from
@ -995,6 +1069,13 @@ Glossary
See also :term:`module`.
native code
Code that is compiled to machine instructions and runs directly on the
processor, as opposed to code that is interpreted or runs in a virtual
machine. In the context of Python, native code typically refers to
C, C++, Rust or Fortran code in :term:`extension modules <extension module>`
that can be called from Python. See also :term:`extension module`.
nested scope
The ability to refer to a variable in an enclosing definition. For
instance, a function defined inside another function can refer to
@ -1011,6 +1092,15 @@ Glossary
properties, :meth:`~object.__getattribute__`, class methods, and static
methods.
non-deterministic
Behavior where the outcome of a program can vary between executions with
the same inputs. In multi-threaded programs, non-deterministic behavior
often results from :term:`race conditions <race condition>` where the
relative timing or interleaving of threads affects the result.
Proper synchronization using :term:`locks <lock>` and other
:term:`synchronization primitives <synchronization primitive>` helps
ensure deterministic behavior.
object
Any data with state (attributes or value) and defined behavior
(methods). Also the ultimate base class of any :term:`new-style
@ -1041,6 +1131,16 @@ Glossary
See also :term:`regular package` and :term:`namespace package`.
parallelism
Executing multiple operations at the same time (e.g. on multiple CPU
cores). In Python builds with the
:term:`global interpreter lock (GIL) <global interpreter lock>`, only one
thread runs Python bytecode at a time, so taking advantage of multiple
CPU cores typically involves multiple processes
(e.g. :mod:`multiprocessing`) or native extensions that release the GIL.
In :term:`free-threaded <free threading>` Python, multiple Python threads
can run Python code simultaneously on different cores.
parameter
A named entity in a :term:`function` (or method) definition that
specifies an :term:`argument` (or in some cases, arguments) that the
@ -1215,6 +1315,18 @@ Glossary
>>> email.mime.text.__name__
'email.mime.text'
race condition
A condition of a program where the its behavior
depends on the relative timing or ordering of events, particularly in
multi-threaded programs. Race conditions can lead to
:term:`non-deterministic` behavior and bugs that are difficult to
reproduce. A :term:`data race` is a specific type of race condition
involving unsynchronized access to shared memory. The :term:`LBYL`
coding style is particularly susceptible to race conditions in
multi-threaded code. Using :term:`locks <lock>` and other
:term:`synchronization primitives <synchronization primitive>`
helps prevent race conditions.
reference count
The number of references to an object. When the reference count of an
object drops to zero, it is deallocated. Some objects are
@ -1236,6 +1348,25 @@ Glossary
See also :term:`namespace package`.
reentrant
A property of a function or :term:`lock` that allows it to be called or
acquired multiple times by the same thread without causing errors or a
:term:`deadlock`.
For functions, reentrancy means the function can be safely called again
before a previous invocation has completed, which is important when
functions may be called recursively or from signal handlers. Thread-unsafe
functions may be :term:`non-deterministic` if they're called reentrantly in a
multithreaded program.
For locks, Python's :class:`threading.RLock` (reentrant lock) is
reentrant, meaning a thread that already holds the lock can acquire it
again without blocking. In contrast, :class:`threading.Lock` is not
reentrant - attempting to acquire it twice from the same thread will cause
a deadlock.
See also :term:`lock` and :term:`deadlock`.
REPL
An acronym for the "readevalprint loop", another name for the
:term:`interactive` interpreter shell.
@ -1340,6 +1471,18 @@ Glossary
See also :term:`borrowed reference`.
synchronization primitive
A basic building block for coordinating (synchronizing) the execution of
multiple threads to ensure :term:`thread-safe` access to shared resources.
Python's :mod:`threading` module provides several synchronization primitives
including :class:`~threading.Lock`, :class:`~threading.RLock`,
:class:`~threading.Semaphore`, :class:`~threading.Condition`,
:class:`~threading.Event`, and :class:`~threading.Barrier`. Additionally,
the :mod:`queue` module provides multi-producer, multi-consumer queues
that are especially useful in multithreaded programs. These
primitives help prevent :term:`race conditions <race condition>` and
coordinate thread execution. See also :term:`lock`.
t-string
t-strings
String literals prefixed with ``t`` or ``T`` are commonly called
@ -1392,6 +1535,19 @@ Glossary
See :ref:`Thread State and the Global Interpreter Lock <threads>` for more
information.
thread-safe
A module, function, or class that behaves correctly when used by multiple
threads concurrently. Thread-safe code uses appropriate
:term:`synchronization primitives <synchronization primitive>` like
:term:`locks <lock>` to protect shared mutable state, or is designed
to avoid shared mutable state entirely. In the
:term:`free-threaded <free threading>` build, built-in types like
:class:`dict`, :class:`list`, and :class:`set` use internal locking
to make many operations thread-safe, although thread safety is not
necessarily guaranteed. Code that is not thread-safe may experience
:term:`race conditions <race condition>` and :term:`data races <data race>`
when used in multi-threaded programs.
token
A small unit of source code, generated by the

View File

@ -8,6 +8,16 @@ execute Python code remotely.
Most platforms require elevated privileges to attach to another Python process.
Disabling remote debugging
--------------------------
To disable remote debugging support, use any of the following:
* Set the :envvar:`PYTHON_DISABLE_REMOTE_DEBUG` environment variable to ``1`` before
starting the interpreter.
* Use the :option:`-X disable_remote_debug` command-line option.
* Compile Python with the :option:`--without-remote-debug` build flag.
.. _permission-requirements:
Permission requirements
@ -614,4 +624,3 @@ To inject and execute a Python script in a remote process:
6. Set ``_PY_EVAL_PLEASE_STOP_BIT`` in the ``eval_breaker`` field.
7. Resume the process (if suspended). The script will execute at the next safe
evaluation point.

View File

@ -0,0 +1,55 @@
/* This file needs to be kept in sync with the tutorial
* at Doc/extending/first-extension-module.rst
*/
/// Includes
#include <Python.h>
#include <stdlib.h> // for system()
/// Implementation of spam.system
static PyObject *
spam_system(PyObject *self, PyObject *arg)
{
const char *command = PyUnicode_AsUTF8(arg);
if (command == NULL) {
return NULL;
}
int status = system(command);
PyObject *result = PyLong_FromLong(status);
return result;
}
/// Module method table
static PyMethodDef spam_methods[] = {
{
.ml_name="system",
.ml_meth=spam_system,
.ml_flags=METH_O,
.ml_doc="Execute a shell command.",
},
{NULL, NULL, 0, NULL} /* Sentinel */
};
/// Module slot table
static PyModuleDef_Slot spam_slots[] = {
{Py_mod_name, "spam"},
{Py_mod_doc, "A wonderful module with an example function"},
{Py_mod_methods, spam_methods},
{0, NULL}
};
/// Export hook prototype
PyMODEXPORT_FUNC PyModExport_spam(void);
/// Module export hook
PyMODEXPORT_FUNC
PyModExport_spam(void)
{
return spam_slots;
}

View File

@ -753,7 +753,7 @@ upper-cased name. For example::
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo-bar')
>>> parser.parse_args(['--foo-bar', 'FOO-BAR']
>>> parser.parse_args(['--foo-bar', 'FOO-BAR'])
Namespace(foo_bar='FOO-BAR')
>>> parser.print_help()
usage: [-h] [--foo-bar FOO-BAR]

View File

@ -9,7 +9,7 @@
--------------
This module defines an object type which can compactly represent an array of
basic values: characters, integers, floating-point numbers. Arrays are sequence
basic values: characters, integers, floating-point numbers. Arrays are mutable :term:`sequence`
types and behave very much like lists, except that the type of objects stored in
them is constrained. The type is specified at object creation time by using a
:dfn:`type code`, which is a single character. The following type codes are
@ -93,7 +93,7 @@ The module defines the following type:
otherwise, the initializer's iterator is passed to the :meth:`extend` method
to add initial items to the array.
Array objects support the ordinary sequence operations of indexing, slicing,
Array objects support the ordinary :ref:`mutable <typesseq-mutable>` :term:`sequence` operations of indexing, slicing,
concatenation, and multiplication. When using slice assignment, the assigned
value must be an array object with the same type code; in all other cases,
:exc:`TypeError` is raised. Array objects also implement the buffer interface,

View File

@ -139,12 +139,13 @@ Node classes
The :meth:`~object.__repr__` output of :class:`~ast.AST` nodes includes
the values of the node fields.
.. deprecated:: 3.8
.. deprecated-removed:: 3.8 3.14
Old classes :class:`!ast.Num`, :class:`!ast.Str`, :class:`!ast.Bytes`,
:class:`!ast.NameConstant` and :class:`!ast.Ellipsis` are still available,
but they will be removed in future Python releases. In the meantime,
instantiating them will return an instance of a different class.
Previous versions of Python provided the AST classes :class:`!ast.Num`,
:class:`!ast.Str`, :class:`!ast.Bytes`, :class:`!ast.NameConstant` and
:class:`!ast.Ellipsis`, which were deprecated in Python 3.8. These classes
were removed in Python 3.14, and their functionality has been replaced with
:class:`ast.Constant`.
.. deprecated:: 3.9
@ -2419,12 +2420,12 @@ and classes for traversing abstract syntax trees:
during traversal. For this a special visitor exists
(:class:`NodeTransformer`) that allows modifications.
.. deprecated:: 3.8
.. deprecated-removed:: 3.8 3.14
Methods :meth:`!visit_Num`, :meth:`!visit_Str`, :meth:`!visit_Bytes`,
:meth:`!visit_NameConstant` and :meth:`!visit_Ellipsis` are deprecated
now and will not be called in future Python versions. Add the
:meth:`visit_Constant` method to handle all constant nodes.
:meth:`!visit_NameConstant` and :meth:`!visit_Ellipsis` will not be called
in Python 3.14+. Add the :meth:`visit_Constant` method instead to handle
all constant nodes.
.. class:: NodeTransformer()

View File

@ -107,7 +107,7 @@ Queue
The queue can no longer grow.
Future calls to :meth:`~Queue.put` raise :exc:`QueueShutDown`.
Currently blocked callers of :meth:`~Queue.put` will be unblocked
and will raise :exc:`QueueShutDown` in the formerly blocked thread.
and will raise :exc:`QueueShutDown` in the formerly awaiting task.
If *immediate* is false (the default), the queue can be wound
down normally with :meth:`~Queue.get` calls to extract tasks

View File

@ -51,7 +51,7 @@ The :rfc:`4648` encodings are suitable for encoding binary data so that it can b
safely sent by email, used as parts of URLs, or included as part of an HTTP
POST request.
.. function:: b64encode(s, altchars=None)
.. function:: b64encode(s, altchars=None, *, wrapcol=0)
Encode the :term:`bytes-like object` *s* using Base64 and return the encoded
:class:`bytes`.
@ -61,11 +61,19 @@ POST request.
This allows an application to e.g. generate URL or filesystem safe Base64
strings. The default is ``None``, for which the standard Base64 alphabet is used.
If *wrapcol* is non-zero, insert a newline (``b'\n'``) character
after at most every *wrapcol* characters.
If *wrapcol* is zero (default), do not insert any newlines.
May assert or raise a :exc:`ValueError` if the length of *altchars* is not 2. Raises a
:exc:`TypeError` if *altchars* is not a :term:`bytes-like object`.
.. versionchanged:: 3.15
Added the *wrapcol* parameter.
.. function:: b64decode(s, altchars=None, validate=False)
b64decode(s, altchars=None, validate=True, *, ignorechars)
Decode the Base64 encoded :term:`bytes-like object` or ASCII string
*s* and return the decoded :class:`bytes`.
@ -77,15 +85,30 @@ POST request.
A :exc:`binascii.Error` exception is raised
if *s* is incorrectly padded.
If *validate* is ``False`` (the default), characters that are neither
If *ignorechars* is specified, it should be a :term:`bytes-like object`
containing characters to ignore from the input when *validate* is true.
The default value of *validate* is ``True`` if *ignorechars* is specified,
``False`` otherwise.
If *validate* is false, characters that are neither
in the normal base-64 alphabet nor the alternative alphabet are
discarded prior to the padding check. If *validate* is ``True``,
these non-alphabet characters in the input result in a
:exc:`binascii.Error`.
discarded prior to the padding check, but the ``+`` and ``/`` characters
keep their meaning if they are not in *altchars* (they will be discarded
in future Python versions).
If *validate* is true, these non-alphabet characters in the input
result in a :exc:`binascii.Error`.
For more information about the strict base64 check, see :func:`binascii.a2b_base64`
May assert or raise a :exc:`ValueError` if the length of *altchars* is not 2.
.. deprecated:: next
Accepting the ``+`` and ``/`` characters with an alternative alphabet
is now deprecated.
.. versionchanged:: next
Added the *ignorechars* parameter.
.. function:: standard_b64encode(s)
@ -116,6 +139,9 @@ POST request.
``/`` in the standard Base64 alphabet, and return the decoded
:class:`bytes`.
.. deprecated:: next
Accepting the ``+`` and ``/`` characters is now deprecated.
.. function:: b32encode(s)
@ -214,9 +240,9 @@ Refer to the documentation of the individual functions for more information.
instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This
feature is not supported by the "standard" Ascii85 encoding.
*wrapcol* controls whether the output should have newline (``b'\n'``)
characters added to it. If this is non-zero, each output line will be
at most this many characters long, excluding the trailing newline.
If *wrapcol* is non-zero, insert a newline (``b'\n'``) character
after at most every *wrapcol* characters.
If *wrapcol* is zero (default), do not insert any newlines.
*pad* controls whether the input is padded to a multiple of 4
before encoding. Note that the ``btoa`` implementation always pads.
@ -239,8 +265,7 @@ Refer to the documentation of the individual functions for more information.
*adobe* controls whether the input sequence is in Adobe Ascii85 format
(i.e. is framed with <~ and ~>).
*ignorechars* should be a :term:`bytes-like object` or ASCII string
containing characters to ignore
*ignorechars* should be a byte string containing characters to ignore
from the input. This should only contain whitespace characters, and by
default contains all whitespace characters in ASCII.
@ -267,14 +292,20 @@ Refer to the documentation of the individual functions for more information.
.. versionadded:: 3.4
.. function:: z85encode(s)
.. function:: z85encode(s, pad=False)
Encode the :term:`bytes-like object` *s* using Z85 (as used in ZeroMQ)
and return the encoded :class:`bytes`. See `Z85 specification
<https://rfc.zeromq.org/spec/32/>`_ for more information.
If *pad* is true, the input is padded with ``b'\0'`` so its length is a
multiple of 4 bytes before encoding.
.. versionadded:: 3.13
.. versionchanged:: 3.15
The *pad* parameter was added.
.. function:: z85decode(s)

View File

@ -49,16 +49,22 @@ The :mod:`binascii` module defines the following functions:
.. function:: a2b_base64(string, /, *, strict_mode=False)
a2b_base64(string, /, *, strict_mode=True, ignorechars)
Convert a block of base64 data back to binary and return the binary data. More
than one line may be passed at a time.
If *ignorechars* is specified, it should be a :term:`bytes-like object`
containing characters to ignore from the input when *strict_mode* is true.
The default value of *strict_mode* is ``True`` if *ignorechars* is specified,
``False`` otherwise.
If *strict_mode* is true, only valid base64 data will be converted. Invalid base64
data will raise :exc:`binascii.Error`.
Valid base64:
* Conforms to :rfc:`3548`.
* Conforms to :rfc:`4648`.
* Contains only characters from the base64 alphabet.
* Contains no excess data after padding (including excess padding, newlines, etc.).
* Does not start with a padding.
@ -66,16 +72,28 @@ The :mod:`binascii` module defines the following functions:
.. versionchanged:: 3.11
Added the *strict_mode* parameter.
.. versionchanged:: next
Added the *ignorechars* parameter.
.. function:: b2a_base64(data, *, newline=True)
Convert binary data to a line of ASCII characters in base64 coding. The return
value is the converted line, including a newline char if *newline* is
true. The output of this function conforms to :rfc:`3548`.
.. function:: b2a_base64(data, *, wrapcol=0, newline=True)
Convert binary data to a line(s) of ASCII characters in base64 coding,
as specified in :rfc:`4648`.
If *wrapcol* is non-zero, insert a newline (``b'\n'``) character
after at most every *wrapcol* characters.
If *wrapcol* is zero (default), do not insert any newlines.
If *newline* is true (default), a newline character will be added
at the end of the output.
.. versionchanged:: 3.6
Added the *newline* parameter.
.. versionchanged:: 3.15
Added the *wrapcol* parameter.
.. function:: a2b_qp(data, header=False)

View File

@ -73,7 +73,7 @@ Reading and writing compressed files
argument is not None, a :exc:`!TypeError` will be raised.
When writing, the *options* argument can be a dictionary
providing advanced decompression parameters; see
providing advanced compression parameters; see
:class:`CompressionParameter` for detailed information about supported
parameters. The *level* argument is the compression level to use when
writing compressed data. Only one of *level* or *options* may be non-None.
@ -117,7 +117,7 @@ Reading and writing compressed files
argument is not None, a :exc:`!TypeError` will be raised.
When writing, the *options* argument can be a dictionary
providing advanced decompression parameters; see
providing advanced compression parameters; see
:class:`CompressionParameter` for detailed information about supported
parameters. The *level* argument is the compression level to use when
writing compressed data. Only one of *level* or *options* may be passed. The

View File

@ -21,6 +21,11 @@ or separate processes, using :class:`ProcessPoolExecutor`.
Each implements the same interface, which is defined
by the abstract :class:`Executor` class.
:class:`concurrent.futures.Future` must not be confused with
:class:`asyncio.Future`, which is designed for use with :mod:`asyncio`
tasks and coroutines. See the :doc:`asyncio's Future <asyncio-future>`
documentation for a detailed comparison of the two.
.. include:: ../includes/wasm-notavail.rst
Executor Objects
@ -308,7 +313,7 @@ the bytes over a shared :mod:`socket <socket>` or
.. note::
The executor may replace uncaught exceptions from *initializer*
with :class:`~concurrent.futures.interpreter.ExecutionFailed`.
with :class:`~concurrent.interpreters.ExecutionFailed`.
Other caveats from parent :class:`ThreadPoolExecutor` apply here.
@ -320,11 +325,11 @@ likewise serializes the return value when sending it back.
When a worker's current task raises an uncaught exception, the worker
always tries to preserve the exception as-is. If that is successful
then it also sets the ``__cause__`` to a corresponding
:class:`~concurrent.futures.interpreter.ExecutionFailed`
:class:`~concurrent.interpreters.ExecutionFailed`
instance, which contains a summary of the original exception.
In the uncommon case that the worker is not able to preserve the
original as-is then it directly preserves the corresponding
:class:`~concurrent.futures.interpreter.ExecutionFailed`
:class:`~concurrent.interpreters.ExecutionFailed`
instance instead.
@ -379,6 +384,11 @@ in a REPL or a lambda should not be expected to work.
default in absence of a *mp_context* parameter. This feature is incompatible
with the "fork" start method.
.. note::
Bugs have been reported when using the *max_tasks_per_child* feature that
can result in the :class:`ProcessPoolExecutor` hanging in some
circumstances. Follow its eventual resolution in :gh:`115634`.
.. versionchanged:: 3.3
When one of the worker processes terminates abruptly, a
:exc:`~concurrent.futures.process.BrokenProcessPool` error is now raised.
@ -715,15 +725,6 @@ Exception classes
.. versionadded:: 3.14
.. exception:: ExecutionFailed
Raised from :class:`~concurrent.futures.InterpreterPoolExecutor` when
the given initializer fails or from
:meth:`~concurrent.futures.Executor.submit` when there's an uncaught
exception from the submitted task.
.. versionadded:: 3.14
.. currentmodule:: concurrent.futures.process
.. exception:: BrokenProcessPool

View File

@ -77,6 +77,32 @@ Context Variables
to restore the variable to its previous value via the
:meth:`ContextVar.reset` method.
For convenience, the token object can be used as a context manager
to avoid calling :meth:`ContextVar.reset` manually::
var = ContextVar('var', default='default value')
with var.set('new value'):
assert var.get() == 'new value'
assert var.get() == 'default value'
It is a shorthand for::
var = ContextVar('var', default='default value')
token = var.set('new value')
try:
assert var.get() == 'new value'
finally:
var.reset(token)
assert var.get() == 'default value'
.. versionadded:: 3.14
Added support for using tokens as context managers.
.. method:: reset(token)
Reset the context variable to the value it had before the
@ -93,24 +119,18 @@ Context Variables
# After the reset call the var has no value again, so
# var.get() would raise a LookupError.
The same *token* cannot be used twice.
.. class:: Token
*Token* objects are returned by the :meth:`ContextVar.set` method.
They can be passed to the :meth:`ContextVar.reset` method to revert
the value of the variable to what it was before the corresponding
*set*.
*set*. A single token cannot reset a context variable more than once.
The token supports :ref:`context manager protocol <context-managers>`
to restore the corresponding context variable value at the exit from
:keyword:`with` block::
var = ContextVar('var', default='default value')
with var.set('new value'):
assert var.get() == 'new value'
assert var.get() == 'default value'
Tokens support the :ref:`context manager protocol <context-managers>`
to automatically reset context variables. See :meth:`ContextVar.set`.
.. versionadded:: 3.14

View File

@ -2651,9 +2651,42 @@ Broadly speaking, ``d.strftime(fmt)`` acts like the :mod:`time` module's
``time.strftime(fmt, d.timetuple())`` although not all objects support a
:meth:`~date.timetuple` method.
For the :meth:`.datetime.strptime` class method, the default value is
``1900-01-01T00:00:00.000``: any components not specified in the format string
will be pulled from the default value. [#]_
For the :meth:`.datetime.strptime` and :meth:`.date.strptime` class methods,
the default value is ``1900-01-01T00:00:00.000``: any components not specified
in the format string will be pulled from the default value.
.. note::
When used to parse partial dates lacking a year, :meth:`.datetime.strptime`
and :meth:`.date.strptime` will raise when encountering February 29 because
the default year of 1900 is *not* a leap year. Always add a default leap
year to partial date strings before parsing.
.. testsetup::
# doctest seems to turn the warning into an error which makes it
# show up and require matching and prevents the actual interesting
# exception from being raised.
# Manually apply the catch_warnings context manager
import warnings
catch_warnings = warnings.catch_warnings()
catch_warnings.__enter__()
warnings.simplefilter("ignore")
.. testcleanup::
catch_warnings.__exit__()
.. doctest::
>>> from datetime import datetime
>>> value = "2/29"
>>> datetime.strptime(value, "%m/%d")
Traceback (most recent call last):
...
ValueError: day 29 must be in range 1..28 for month 2 in year 1900
>>> datetime.strptime(f"1904 {value}", "%Y %m/%d")
datetime.datetime(1904, 2, 29, 0, 0)
Using ``datetime.strptime(date_string, format)`` is equivalent to::
@ -2790,7 +2823,7 @@ Notes:
include a year in the format. If the value you need to parse lacks a year,
append an explicit dummy leap year. Otherwise your code will raise an
exception when it encounters leap day because the default year used by the
parser is not a leap year. Users run into this bug every four years...
parser (1900) is not a leap year. Users run into that bug every leap year.
.. doctest::
@ -2817,5 +2850,3 @@ Notes:
.. [#] See R. H. van Gent's `guide to the mathematics of the ISO 8601 calendar
<https://web.archive.org/web/20220531051136/https://webspace.science.uu.nl/~gent0113/calendar/isocalendar.htm>`_
for a good explanation.
.. [#] Passing ``datetime.strptime('Feb 29', '%b %d')`` will fail since 1900 is not a leap year.

View File

@ -768,7 +768,7 @@ not have to be) the original ``STACK[-2]``.
end = STACK.pop()
start = STACK.pop()
container = STACK.pop()
values = STACK.pop()
value = STACK.pop()
container[start:end] = value
.. versionadded:: 3.12

View File

@ -57,7 +57,7 @@ message objects.
:class:`~email.policy.default` policy, which follows the rules of the email
RFCs except for line endings (instead of the RFC mandated ``\r\n``, it uses
the Python standard ``\n`` line endings). For more information see the
:mod:`~email.policy` documentation.
:mod:`~email.policy` documentation. [2]_
.. method:: as_string(unixfrom=False, maxheaderlen=None, policy=None)
@ -749,3 +749,9 @@ message objects.
.. [1] Originally added in 3.4 as a :term:`provisional module <provisional
package>`. Docs for legacy message class moved to
:ref:`compat32_message`.
.. [2] The :class:`EmailMessage` class requires a policy that provides a
``content_manager`` attribute for content management methods like
``set_content()`` and ``get_content()`` to work. The legacy
:const:`~email.policy.compat32` policy does not support these methods
and should not be used with :class:`EmailMessage`.

View File

@ -662,6 +662,13 @@ The header objects and their attributes are described in
An instance of :class:`Compat32`, providing backward compatibility with the
behavior of the email package in Python 3.2.
.. note::
The :const:`compat32` policy should not be used as a policy for
:class:`~email.message.EmailMessage` objects, and should only be used
to serialize messages that were created using the :const:`compat32`
policy.
.. rubric:: Footnotes

View File

@ -153,6 +153,12 @@ Module Contents
Return a list of all power-of-two integers contained in a flag.
:func:`enum.bin`
Like built-in :func:`bin`, except negative values are represented in
two's complement, and the leading bit always indicates sign
(``0`` implies positive, ``1`` implies negative).
.. versionadded:: 3.6 ``Flag``, ``IntFlag``, ``auto``
.. versionadded:: 3.11 ``StrEnum``, ``EnumCheck``, ``ReprEnum``, ``FlagBoundary``, ``property``, ``member``, ``nonmember``, ``global_enum``, ``show_flag_values``
@ -947,12 +953,13 @@ Utilities and Decorators
the member's name. Care must be taken if mixing *auto()* with manually
specified values.
*auto* instances are only resolved when at the top level of an assignment:
*auto* instances are only resolved when at the top level of an assignment, either by
itself or as part of a tuple:
* ``FIRST = auto()`` will work (auto() is replaced with ``1``);
* ``SECOND = auto(), -2`` will work (auto is replaced with ``2``, so ``2, -2`` is
used to create the ``SECOND`` enum member;
* ``THREE = [auto(), -3]`` will *not* work (``<auto instance>, -3`` is used to
* ``THREE = [auto(), -3]`` will *not* work (``[<auto instance>, -3]`` is used to
create the ``THREE`` enum member)
.. versionchanged:: 3.11.1
@ -1034,6 +1041,20 @@ Utilities and Decorators
.. versionadded:: 3.11
.. function:: bin(num, max_bits=None)
Like built-in :func:`bin`, except negative values are represented in
two's complement, and the leading bit always indicates sign
(``0`` implies positive, ``1`` implies negative).
>>> import enum
>>> enum.bin(10)
'0b0 1010'
>>> enum.bin(~10) # ~10 is -11
'0b1 0101'
.. versionadded:: 3.10
---------------
Notes

View File

@ -524,14 +524,9 @@ FTP_TLS objects
:class:`!FTP_TLS` class inherits from :class:`FTP`,
defining these additional methods and attributes:
.. attribute:: FTP_TLS.ssl_version
The SSL version to use (defaults to :data:`ssl.PROTOCOL_SSLv23`).
.. method:: FTP_TLS.auth()
Set up a secure control connection by using TLS or SSL, depending on what
is specified in the :attr:`ssl_version` attribute.
Set up a secure control connection by using TLS.
.. versionchanged:: 3.4
The method now supports hostname check with
@ -548,7 +543,7 @@ FTP_TLS objects
.. method:: FTP_TLS.prot_p()
Set up secure data connection.
Set up secure data connection by using TLS.
.. method:: FTP_TLS.prot_c()

View File

@ -138,6 +138,8 @@ are always available. They are listed here in alphabetical order.
>>> f'{14:#b}', f'{14:b}'
('0b1110', '1110')
See also :func:`enum.bin` to represent negative values as twos-complement.
See also :func:`format` for more information.

View File

@ -294,9 +294,9 @@ The following example demonstrates how to use the :mod:`http.cookies` module.
Set-Cookie: chips=ahoy
Set-Cookie: vienna=finger
>>> C = cookies.SimpleCookie()
>>> C.load('keebler="E=everybody; L=\\"Loves\\"; fudge=\\012;";')
>>> C.load('keebler="E=everybody; L=\\"Loves\\"; fudge=;";')
>>> print(C)
Set-Cookie: keebler="E=everybody; L=\"Loves\"; fudge=\012;"
Set-Cookie: keebler="E=everybody; L=\"Loves\"; fudge=;"
>>> C = cookies.SimpleCookie()
>>> C["oreo"] = "doublestuff"
>>> C["oreo"]["path"] = "/"

View File

@ -158,7 +158,7 @@ Go to Line
Show Completions
Open a scrollable list allowing selection of existing names. See
:ref:`Completions <completions>` in the Editing and navigation section below.
:ref:`Completions <completions>` in the Editing and Navigation section below.
Expand Word
Expand a prefix you have typed to match a full word in the same window;
@ -167,7 +167,7 @@ Expand Word
Show Call Tip
After an unclosed parenthesis for a function, open a small window with
function parameter hints. See :ref:`Calltips <calltips>` in the
Editing and navigation section below.
Editing and Navigation section below.
Show Surrounding Parens
Highlight the surrounding parenthesis.
@ -178,9 +178,9 @@ Format menu (Editor window only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Format Paragraph
Reformat the current blank-line-delimited paragraph in comment block or
multiline string or selected line in a string. All lines in the
paragraph will be formatted to less than N columns, where N defaults to 72.
Rewrap the text block containing the text insert cursor.
Avoid code lines. See :ref:`Format block<format-block>` in the
Editing and Navigation section below.
Indent Region
Shift selected lines right by the indent width (default 4 spaces).
@ -566,6 +566,20 @@ In an editor, import statements have no effect until one runs the file.
One might want to run a file after writing import statements, after
adding function definitions, or after opening an existing file.
.. _format-block:
Format block
^^^^^^^^^^^^
Reformat Paragraph rewraps a block ('paragraph') of contiguous equally
indented non-blank comments, a similar block of text within a multiline
string, or a selected subset of either.
If needed, add a blank line to separate string from code.
Partial lines in a selection expand to complete lines.
The resulting lines have the same indent as before
but have maximum total length of N columns (characters).
Change the default N of 72 on the Window tab of IDLE Settings.
.. _code-context:
Code Context

View File

@ -418,6 +418,16 @@ Distributions
equal, even if they relate to the same installed distribution and
accordingly have the same attributes.
.. method:: discover(cls, *, context=None, **kwargs)
Returns an iterable of :class:`Distribution` instances for all packages.
The optional argument *context* is a :class:`DistributionFinder.Context`
instance, used to modify the search for distributions. Alternatively,
*kwargs* may contain keyword arguments for constructing a new
:class:`!DistributionFinder.Context`.
While the module level API described above is the most common and convenient usage,
you can get all of that information from the :class:`!Distribution` class.
:class:`!Distribution` is an abstract object that represents the metadata for
@ -466,6 +476,61 @@ This metadata finder search defaults to ``sys.path``, but varies slightly in how
- ``importlib.metadata`` does not honor :class:`bytes` objects on ``sys.path``.
- ``importlib.metadata`` will incidentally honor :py:class:`pathlib.Path` objects on ``sys.path`` even though such values will be ignored for imports.
.. class:: DistributionFinder
A :class:`~importlib.abc.MetaPathFinder` subclass capable of discovering
installed distributions.
Custom providers should implement this interface in order to
supply metadata.
.. class:: Context(**kwargs)
A :class:`!Context` gives a custom provider a means to
solicit additional details from the callers of distribution discovery
functions like :func:`distributions` or :meth:`Distribution.discover`
beyond :attr:`!.name` and :attr:`!.path` when searching
for distributions.
For example, a provider could expose suites of packages in either a
"public" or "private" ``realm``. A caller of distribution discovery
functions may wish to query only for distributions in a particular realm
and could call ``distributions(realm="private")`` to signal to the
custom provider to only include distributions from that
realm.
Each :class:`!DistributionFinder` must expect any parameters and should
attempt to honor the canonical parameters defined below when
appropriate.
See the section on :ref:`implementing-custom-providers` for more details.
.. attribute:: name
Specific name for which a distribution finder should match.
A :attr:`!.name` of ``None`` matches all distributions.
.. attribute:: path
A property providing the sequence of directory paths that a
distribution finder should search.
Typically refers to Python installed package paths such as
"site-packages" directories and defaults to :attr:`sys.path`.
.. function:: distributions(**kwargs)
Returns an iterable of :class:`Distribution` instances for all packages.
The *kwargs* argument may contain either a keyword argument ``context``, a
:class:`DistributionFinder.Context` instance, or pass keyword arguments for
constructing a new :class:`!DistributionFinder.Context`. The
:class:`!DistributionFinder.Context` is used to modify the search for
distributions.
.. _implementing-custom-providers:
Implementing Custom Providers
=============================
@ -493,7 +558,7 @@ interface expected of finders by Python's import system.
``importlib.metadata`` extends this protocol by looking for an optional
``find_distributions`` callable on the finders from
:data:`sys.meta_path` and presents this extended interface as the
``DistributionFinder`` abstract base class, which defines this abstract
:class:`DistributionFinder` abstract base class, which defines this abstract
method::
@abc.abstractmethod
@ -502,9 +567,11 @@ method::
loading the metadata for packages for the indicated ``context``.
"""
The ``DistributionFinder.Context`` object provides ``.path`` and ``.name``
properties indicating the path to search and name to match and may
supply other relevant context sought by the consumer.
The :class:`DistributionFinder.Context` object provides
:attr:`~DistributionFinder.Context.path` and
:attr:`~DistributionFinder.Context.name` properties indicating the path to
search and name to match and may supply other relevant context sought by the
consumer.
In practice, to support finding distribution package
metadata in locations other than the file system, subclass
@ -529,7 +596,7 @@ Imagine a custom finder that loads Python modules from a database::
That importer now presumably provides importable modules from a
database, but it provides no metadata or entry points. For this
custom importer to provide metadata, it would also need to implement
``DistributionFinder``::
:class:`DistributionFinder`::
from importlib.metadata import DistributionFinder

View File

@ -63,11 +63,14 @@
If the resource does not concretely exist on the file system,
raise :exc:`FileNotFoundError`.
.. method:: is_resource(name)
.. method:: is_resource(path)
:abstractmethod:
Returns ``True`` if the named *name* is considered a resource.
:exc:`FileNotFoundError` is raised if *name* does not exist.
Returns ``True`` if the named *path* is considered a resource.
:exc:`FileNotFoundError` is raised if *path* does not exist.
.. versionchanged:: 3.10
The argument *name* was renamed to *path*.
.. method:: contents()
:abstractmethod:

View File

@ -210,12 +210,6 @@ Functions
:exc:`ModuleNotFoundError` is raised when the module being reloaded lacks
a :class:`~importlib.machinery.ModuleSpec`.
.. versionchanged:: 3.15
If *module* is a lazy module that has not yet been materialized (i.e.,
loaded via :class:`importlib.util.LazyLoader` and not yet accessed),
calling :func:`reload` is a no-op and returns the module unchanged.
This prevents the reload from unintentionally triggering the lazy load.
.. warning::
This function is not thread-safe. Calling it from multiple threads can result
in unexpected behavior. It's recommended to use the :class:`threading.Lock`
@ -602,172 +596,6 @@ ABC hierarchy::
itself does not end in ``__init__``.
.. class:: ResourceReader
*Superseded by TraversableResources*
An :term:`abstract base class` to provide the ability to read
*resources*.
From the perspective of this ABC, a *resource* is a binary
artifact that is shipped within a package. Typically this is
something like a data file that lives next to the ``__init__.py``
file of the package. The purpose of this class is to help abstract
out the accessing of such data files so that it does not matter if
the package and its data file(s) are stored e.g. in a zip file
versus on the file system.
For any of methods of this class, a *resource* argument is
expected to be a :term:`path-like object` which represents
conceptually just a file name. This means that no subdirectory
paths should be included in the *resource* argument. This is
because the location of the package the reader is for, acts as the
"directory". Hence the metaphor for directories and file
names is packages and resources, respectively. This is also why
instances of this class are expected to directly correlate to
a specific package (instead of potentially representing multiple
packages or a module).
Loaders that wish to support resource reading are expected to
provide a method called ``get_resource_reader(fullname)`` which
returns an object implementing this ABC's interface. If the module
specified by fullname is not a package, this method should return
:const:`None`. An object compatible with this ABC should only be
returned when the specified module is a package.
.. versionadded:: 3.7
.. deprecated-removed:: 3.12 3.14
Use :class:`importlib.resources.abc.TraversableResources` instead.
.. method:: open_resource(resource)
:abstractmethod:
Returns an opened, :term:`file-like object` for binary reading
of the *resource*.
If the resource cannot be found, :exc:`FileNotFoundError` is
raised.
.. method:: resource_path(resource)
:abstractmethod:
Returns the file system path to the *resource*.
If the resource does not concretely exist on the file system,
raise :exc:`FileNotFoundError`.
.. method:: is_resource(name)
:abstractmethod:
Returns ``True`` if the named *name* is considered a resource.
:exc:`FileNotFoundError` is raised if *name* does not exist.
.. method:: contents()
:abstractmethod:
Returns an :term:`iterable` of strings over the contents of
the package. Do note that it is not required that all names
returned by the iterator be actual resources, e.g. it is
acceptable to return names for which :meth:`is_resource` would
be false.
Allowing non-resource names to be returned is to allow for
situations where how a package and its resources are stored
are known a priori and the non-resource names would be useful.
For instance, returning subdirectory names is allowed so that
when it is known that the package and resources are stored on
the file system then those subdirectory names can be used
directly.
The abstract method returns an iterable of no items.
.. class:: Traversable
An object with a subset of :class:`pathlib.Path` methods suitable for
traversing directories and opening files.
For a representation of the object on the file-system, use
:meth:`importlib.resources.as_file`.
.. versionadded:: 3.9
.. deprecated-removed:: 3.12 3.14
Use :class:`importlib.resources.abc.Traversable` instead.
.. attribute:: name
Abstract. The base name of this object without any parent references.
.. method:: iterdir()
:abstractmethod:
Yield ``Traversable`` objects in ``self``.
.. method:: is_dir()
:abstractmethod:
Return ``True`` if ``self`` is a directory.
.. method:: is_file()
:abstractmethod:
Return ``True`` if ``self`` is a file.
.. method:: joinpath(child)
:abstractmethod:
Return Traversable child in ``self``.
.. method:: __truediv__(child)
:abstractmethod:
Return ``Traversable`` child in ``self``.
.. method:: open(mode='r', *args, **kwargs)
:abstractmethod:
*mode* may be 'r' or 'rb' to open as text or binary. Return a handle
suitable for reading (same as :attr:`pathlib.Path.open`).
When opening as text, accepts encoding parameters such as those
accepted by :class:`io.TextIOWrapper`.
.. method:: read_bytes()
Read contents of ``self`` as bytes.
.. method:: read_text(encoding=None)
Read contents of ``self`` as text.
.. class:: TraversableResources
An abstract base class for resource readers capable of serving
the :meth:`importlib.resources.files` interface. Subclasses
:class:`importlib.resources.abc.ResourceReader` and provides
concrete implementations of the :class:`importlib.resources.abc.ResourceReader`'s
abstract methods. Therefore, any loader supplying
:class:`importlib.abc.TraversableResources` also supplies ResourceReader.
Loaders that wish to support resource reading are expected to
implement this interface.
.. versionadded:: 3.9
.. deprecated-removed:: 3.12 3.14
Use :class:`importlib.resources.abc.TraversableResources` instead.
.. method:: files()
:abstractmethod:
Returns a :class:`importlib.resources.abc.Traversable` object for the loaded
package.
:mod:`importlib.machinery` -- Importers and path hooks
------------------------------------------------------

View File

@ -273,6 +273,9 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
+-----------------+-------------------+---------------------------+
| | ag_running | is the generator running? |
+-----------------+-------------------+---------------------------+
| | ag_suspended | is the generator |
| | | suspended? |
+-----------------+-------------------+---------------------------+
| | ag_code | code |
+-----------------+-------------------+---------------------------+
| coroutine | __name__ | name |
@ -286,6 +289,9 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
+-----------------+-------------------+---------------------------+
| | cr_running | is the coroutine running? |
+-----------------+-------------------+---------------------------+
| | cr_suspended | is the coroutine |
| | | suspended? |
+-----------------+-------------------+---------------------------+
| | cr_code | code |
+-----------------+-------------------+---------------------------+
| | cr_origin | where coroutine was |
@ -319,6 +325,18 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
Add ``__builtins__`` attribute to functions.
.. versionchanged:: 3.11
Add ``gi_suspended`` attribute to generators.
.. versionchanged:: 3.11
Add ``cr_suspended`` attribute to coroutines.
.. versionchanged:: 3.12
Add ``ag_suspended`` attribute to async generators.
.. versionchanged:: 3.14
Add ``f_generator`` attribute to frames.
@ -506,7 +524,7 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
.. versionchanged:: 3.13
Functions wrapped in :func:`functools.partialmethod` now return ``True``
if the wrapped function is a :term:`coroutine function`.
if the wrapped function is a :term:`asynchronous generator` function.
.. function:: isasyncgen(object)

View File

@ -47,7 +47,7 @@ Iterator Arguments Results
Iterator Arguments Results Example
============================ ============================ ================================================= =============================================================
:func:`accumulate` p [,func] p0, p0+p1, p0+p1+p2, ... ``accumulate([1,2,3,4,5]) → 1 3 6 10 15``
:func:`batched` p, n (p0, p1, ..., p_n-1), ... ``batched('ABCDEFG', n=2) → AB CD EF G``
:func:`batched` p, n (p0, p1, ..., p_n-1), ... ``batched('ABCDEFG', n=3) → ABC DEF G``
:func:`chain` p, q, ... p0, p1, ... plast, q0, q1, ... ``chain('ABC', 'DEF') → A B C D E F``
:func:`chain.from_iterable` iterable p0, p1, ... plast, q0, q1, ... ``chain.from_iterable(['ABC', 'DEF']) → A B C D E F``
:func:`compress` data, selectors (d[0] if s[0]), (d[1] if s[1]), ... ``compress('ABCDEF', [1,0,1,0,1,1]) → A C E F``
@ -181,7 +181,7 @@ loops that truncate the stream.
Roughly equivalent to::
def batched(iterable, n, *, strict=False):
# batched('ABCDEFG', 2) → AB CD EF G
# batched('ABCDEFG', 3) → ABC DEF G
if n < 1:
raise ValueError('n must be at least one')
iterator = iter(iterable)
@ -819,7 +819,7 @@ well as with the built-in itertools such as ``map()``, ``filter()``,
A secondary purpose of the recipes is to serve as an incubator. The
``accumulate()``, ``compress()``, and ``pairwise()`` itertools started out as
recipes. Currently, the ``sliding_window()``, ``iter_index()``, and ``sieve()``
recipes. Currently, the ``sliding_window()``, ``derangements()``, and ``sieve()``
recipes are being tested to see whether they prove their worth.
Substantially all of these recipes and many, many others can be installed from
@ -838,11 +838,16 @@ and :term:`generators <generator>` which incur interpreter overhead.
.. testcode::
from itertools import (accumulate, batched, chain, combinations, compress,
count, cycle, filterfalse, groupby, islice, permutations, product,
repeat, starmap, tee, zip_longest)
from collections import Counter, deque
from contextlib import suppress
from functools import reduce
from math import comb, prod, sumprod, isqrt
from operator import itemgetter, getitem, mul, neg
from math import comb, isqrt, prod, sumprod
from operator import getitem, is_not, itemgetter, mul, neg
# ==== Basic one liners ====
def take(n, iterable):
"Return first n items of the iterable as a list."
@ -899,8 +904,8 @@ and :term:`generators <generator>` which incur interpreter overhead.
def first_true(iterable, default=False, predicate=None):
"Returns the first true value or the *default* if there is no true value."
# first_true([a,b,c], x) → a or b or c or x
# first_true([a,b], x, f) → a if f(a) else b if f(b) else x
# first_true([a, b, c], x) → a or b or c or x
# first_true([a, b], x, f) → a if f(a) else b if f(b) else x
return next(filter(predicate, iterable), default)
def all_equal(iterable, key=None):
@ -908,6 +913,8 @@ and :term:`generators <generator>` which incur interpreter overhead.
# all_equal('4٤௪౪໔', key=int) → True
return len(take(2, groupby(iterable, key))) <= 1
# ==== Data pipelines ====
def unique_justseen(iterable, key=None):
"Yield unique elements, preserving order. Remember only the element just seen."
# unique_justseen('AAAABBBCCDAABBB') → A B C D A B
@ -940,7 +947,7 @@ and :term:`generators <generator>` which incur interpreter overhead.
def sliding_window(iterable, n):
"Collect data into overlapping fixed-length chunks or blocks."
# sliding_window('ABCDEFG', 4) → ABCD BCDE CDEF DEFG
# sliding_window('ABCDEFG', 3) → ABC BCD CDE DEF EFG
iterator = iter(iterable)
window = deque(islice(iterator, n - 1), maxlen=n)
for x in iterator:
@ -949,7 +956,7 @@ and :term:`generators <generator>` which incur interpreter overhead.
def grouper(iterable, n, *, incomplete='fill', fillvalue=None):
"Collect data into non-overlapping fixed-length chunks or blocks."
# grouper('ABCDEFG', 3, fillvalue='x') → ABC DEF Gxx
# grouper('ABCDEFG', 3, fillvalue='x') → ABC DEF Gxx
# grouper('ABCDEFG', 3, incomplete='strict') → ABC DEF ValueError
# grouper('ABCDEFG', 3, incomplete='ignore') → ABC DEF
iterators = [iter(iterable)] * n
@ -978,6 +985,16 @@ and :term:`generators <generator>` which incur interpreter overhead.
slices = starmap(slice, combinations(range(len(seq) + 1), 2))
return map(getitem, repeat(seq), slices)
def derangements(iterable, r=None):
"Produce r length permutations without fixed points."
# derangements('ABCD') → BADC BCDA BDAC CADB CDAB CDBA DABC DCAB DCBA
# Algorithm credited to Stefan Pochmann
seq = tuple(iterable)
pos = tuple(range(len(seq)))
have_moved = map(map, repeat(is_not), repeat(pos), permutations(pos, r=r))
valid_derangements = map(all, have_moved)
return compress(permutations(seq, r=r), valid_derangements)
def iter_index(iterable, value, start=0, stop=None):
"Return indices where a value occurs in a sequence or iterable."
# iter_index('AABCADEAF', 'A') → 0 1 4 7
@ -1004,10 +1021,7 @@ and :term:`generators <generator>` which incur interpreter overhead.
while True:
yield function()
The following recipes have a more mathematical flavor:
.. testcode::
# ==== Mathematical operations ====
def multinomial(*counts):
"Number of distinct arrangements of a multiset."
@ -1026,9 +1040,11 @@ The following recipes have a more mathematical flavor:
# sum_of_squares([10, 20, 30]) → 1400
return sumprod(*tee(iterable))
# ==== Matrix operations ====
def reshape(matrix, columns):
"Reshape a 2-D matrix to have a given number of columns."
# reshape([(0, 1), (2, 3), (4, 5)], 3) → (0, 1, 2), (3, 4, 5)
# reshape([(0, 1), (2, 3), (4, 5)], 3) → (0, 1, 2) (3, 4, 5)
return batched(chain.from_iterable(matrix), columns, strict=True)
def transpose(matrix):
@ -1038,10 +1054,12 @@ The following recipes have a more mathematical flavor:
def matmul(m1, m2):
"Multiply two matrices."
# matmul([(7, 5), (3, 5)], [(2, 5), (7, 9)]) → (49, 80), (41, 60)
# matmul([(7, 5), (3, 5)], [(2, 5), (7, 9)]) → (49, 80) (41, 60)
n = len(m2[0])
return batched(starmap(sumprod, product(m1, transpose(m2))), n)
# ==== Polynomial arithmetic ====
def convolve(signal, kernel):
"""Discrete linear convolution of two iterables.
Equivalent to polynomial multiplication.
@ -1096,6 +1114,8 @@ The following recipes have a more mathematical flavor:
powers = reversed(range(1, n))
return list(map(mul, coefficients, powers))
# ==== Number theory ====
def sieve(n):
"Primes less than n."
# sieve(30) → 2 3 5 7 11 13 17 19 23 29
@ -1663,6 +1683,36 @@ The following recipes have a more mathematical flavor:
['A', 'AB', 'ABC', 'ABCD', 'B', 'BC', 'BCD', 'C', 'CD', 'D']
>>> ' '.join(map(''.join, derangements('ABCD')))
'BADC BCDA BDAC CADB CDAB CDBA DABC DCAB DCBA'
>>> ' '.join(map(''.join, derangements('ABCD', 3)))
'BAD BCA BCD BDA CAB CAD CDA CDB DAB DCA DCB'
>>> ' '.join(map(''.join, derangements('ABCD', 2)))
'BA BC BD CA CD DA DC'
>>> ' '.join(map(''.join, derangements('ABCD', 1)))
'B C D'
>>> ' '.join(map(''.join, derangements('ABCD', 0)))
''
>>> # Compare number of derangements to https://oeis.org/A000166
>>> [len(list(derangements(range(n)))) for n in range(10)]
[1, 0, 1, 2, 9, 44, 265, 1854, 14833, 133496]
>>> # Verify that identical objects are treated as unique by position
>>> identical = 'X'
>>> distinct = 'x'
>>> seq1 = ('A', identical, 'B', identical)
>>> result1 = ' '.join(map(''.join, derangements(seq1)))
>>> result1
'XAXB XBXA XXAB BAXX BXAX BXXA XAXB XBAX XBXA'
>>> seq2 = ('A', identical, 'B', distinct)
>>> result2 = ' '.join(map(''.join, derangements(seq2)))
>>> result2
'XAxB XBxA XxAB BAxX BxAX BxXA xAXB xBAX xBXA'
>>> result1 == result2
False
>>> result1.casefold() == result2.casefold()
True
>>> list(powerset([1,2,3]))
[(), (1,), (2,), (3,), (1, 2), (1, 3), (2, 3), (1, 2, 3)]
>>> all(len(list(powerset(range(n)))) == 2**n for n in range(18))

View File

@ -31,7 +31,7 @@ The :mod:`linecache` module defines the following functions:
.. index:: triple: module; search; path
If *filename* indicates a frozen module (starting with ``'<frozen '``), the function
will attepmt to get the real file name from ``module_globals['__file__']`` if
will attempt to get the real file name from ``module_globals['__file__']`` if
*module_globals* is not ``None``.
If a file named *filename* is not found, the function first checks

View File

@ -370,8 +370,6 @@ The :mod:`locale` module defines the following exception and functions:
determined.
The "C" locale is represented as ``(None, None)``.
.. deprecated-removed:: 3.11 3.15
.. function:: getlocale(category=LC_CTYPE)

View File

@ -1011,6 +1011,11 @@ the options available to you.
| exc_info | You shouldn't need to | Exception tuple (à la ``sys.exc_info``) or, |
| | format this yourself. | if no exception has occurred, ``None``. |
+----------------+-------------------------+-----------------------------------------------+
| exc_text | You shouldn't need to | Exception information formatted as a string. |
| | format this yourself. | This is set when :meth:`Formatter.format` is |
| | | invoked, or ``None`` if no exception has |
| | | occurred. |
+----------------+-------------------------+-----------------------------------------------+
| filename | ``%(filename)s`` | Filename portion of ``pathname``. |
+----------------+-------------------------+-----------------------------------------------+
| funcName | ``%(funcName)s`` | Name of function containing the logging call. |

View File

@ -212,7 +212,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
Writable :term:`bytes-like object` is now accepted.
.. method:: flush([offset[, size]])
.. method:: flush([offset[, size]], *, flags=MS_SYNC)
Flushes changes made to the in-memory copy of a file back to disk. Without
use of this call there is no guarantee that changes are written back before
@ -221,6 +221,12 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
whole extent of the mapping is flushed. *offset* must be a multiple of the
:const:`PAGESIZE` or :const:`ALLOCATIONGRANULARITY`.
The *flags* parameter specifies the synchronization behavior.
*flags* must be one of the :ref:`MS_* constants <ms-constants>` available
on the system.
On Windows, the *flags* parameter is ignored.
``None`` is returned to indicate success. An exception is raised when the
call failed.
@ -235,6 +241,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
specified alone, and the flush operation will extend from *offset*
to the end of the mmap.
.. versionchanged:: 3.15
Added *flags* parameter to control synchronization behavior.
.. method:: madvise(option[, start[, length]])
@ -328,6 +337,17 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
.. versionadded:: 3.13
.. method:: set_name(name, /)
Annotate the memory mapping with the given *name* for easier identification
in ``/proc/<pid>/maps`` if the kernel supports the feature and :option:`-X dev <-X>` is passed
to Python or if Python is built in :ref:`debug mode <debug-build>`.
The length of *name* must not exceed 67 bytes including the ``'\0'`` terminator.
.. availability:: Linux >= 5.17 (kernel built with ``CONFIG_ANON_VMA_NAME`` option)
.. versionadded:: 3.15
.. method:: size()
Return the length of the file, which can be larger than the size of the
@ -450,3 +470,22 @@ MAP_* Constants
:data:`MAP_TPRO`, :data:`MAP_TRANSLATED_ALLOW_EXECUTE`, and
:data:`MAP_UNIX03` constants.
.. _ms-constants:
MS_* Constants
++++++++++++++
.. data:: MS_SYNC
MS_ASYNC
MS_INVALIDATE
These flags control the synchronization behavior for :meth:`mmap.flush`:
* :data:`MS_SYNC` - Synchronous flush: writes are scheduled and the call
blocks until they are physically written to storage.
* :data:`MS_ASYNC` - Asynchronous flush: writes are scheduled but the call
returns immediately without waiting for completion.
* :data:`MS_INVALIDATE` - Invalidate cached data: invalidates other mappings
of the same file so they can see the changes.
.. versionadded:: 3.15

View File

@ -7,6 +7,8 @@
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
**Source code:** :source:`PC/msvcrtmodule.c`
--------------
These functions provide access to some useful capabilities on Windows platforms.

View File

@ -1234,22 +1234,32 @@ Miscellaneous
.. versionchanged:: 3.11
Accepts a :term:`path-like object`.
.. function:: set_forkserver_preload(module_names)
.. function:: set_forkserver_preload(module_names, *, on_error='ignore')
Set a list of module names for the forkserver main process to attempt to
import so that their already imported state is inherited by forked
processes. Any :exc:`ImportError` when doing so is silently ignored.
This can be used as a performance enhancement to avoid repeated work
in every process.
processes. This can be used as a performance enhancement to avoid repeated
work in every process.
For this to work, it must be called before the forkserver process has been
launched (before creating a :class:`Pool` or starting a :class:`Process`).
The *on_error* parameter controls how :exc:`ImportError` exceptions during
module preloading are handled: ``"ignore"`` (default) silently ignores
failures, ``"warn"`` causes the forkserver subprocess to emit an
:exc:`ImportWarning` to stderr, and ``"fail"`` causes the forkserver
subprocess to exit with the exception traceback on stderr, making
subsequent process creation fail with :exc:`EOFError` or
:exc:`ConnectionError`.
Only meaningful when using the ``'forkserver'`` start method.
See :ref:`multiprocessing-start-methods`.
.. versionadded:: 3.4
.. versionchanged:: next
Added the *on_error* parameter.
.. function:: set_start_method(method, force=False)
Set the method which should be used to start child processes.
@ -1693,11 +1703,14 @@ inherited by child processes.
value is actually a synchronized wrapper for the array.
*typecode_or_type* determines the type of the elements of the returned array:
it is either a ctypes type or a one character typecode of the kind used by
the :mod:`array` module. If *size_or_initializer* is an integer, then it
determines the length of the array, and the array will be initially zeroed.
Otherwise, *size_or_initializer* is a sequence which is used to initialize
the array and whose length determines the length of the array.
it is either a :ref:`ctypes type <ctypes-fundamental-data-types>` or a one
character typecode of the kind used by the :mod:`array` module with the
exception of ``'w'``, which is not supported. In addition, the ``'c'``
typecode is an alias for :class:`ctypes.c_char`. If *size_or_initializer*
is an integer, then it determines the length of the array, and the array
will be initially zeroed. Otherwise, *size_or_initializer* is a sequence
which is used to initialize the array and whose length determines the length
of the array.
If *lock* is ``True`` (the default) then a new lock object is created to
synchronize access to the value. If *lock* is a :class:`Lock` or

View File

@ -1556,6 +1556,15 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo
.. versionadded:: 3.15
.. data:: RWF_ATOMIC
Write data atomically. Requires alignment to the device's atomic write unit.
.. availability:: Linux >= 6.11
.. versionadded:: 3.15
.. function:: ptsname(fd, /)
Return the name of the slave pseudo-terminal device associated with the
@ -1598,6 +1607,7 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo
- :data:`RWF_SYNC`
- :data:`RWF_APPEND`
- :data:`RWF_DONTCACHE`
- :data:`RWF_ATOMIC`
Return the total number of bytes actually written.
@ -1969,7 +1979,8 @@ can be inherited by child processes. Since Python 3.4, file descriptors
created by Python are non-inheritable by default.
On UNIX, non-inheritable file descriptors are closed in child processes at the
execution of a new program, other file descriptors are inherited.
execution of a new program, other file descriptors are inherited. Note that
non-inheritable file descriptors are still *inherited* by child processes on :func:`os.fork`.
On Windows, non-inheritable handles and file descriptors are closed in child
processes, except for standard streams (file descriptors 0, 1 and 2: stdin, stdout
@ -4251,7 +4262,7 @@ features:
import os
# semaphore with start value '1'
fd = os.eventfd(1, os.EFD_SEMAPHORE | os.EFC_CLOEXEC)
fd = os.eventfd(1, os.EFD_SEMAPHORE | os.EFD_CLOEXEC)
try:
# acquire semaphore
v = os.eventfd_read(fd)
@ -5993,7 +6004,7 @@ Miscellaneous System Information
.. versionchanged:: 3.13
If :option:`-X cpu_count <-X>` is given or :envvar:`PYTHON_CPU_COUNT` is set,
:func:`cpu_count` returns the overridden value *n*.
:func:`cpu_count` returns the override value *n*.
.. function:: getloadavg()
@ -6015,7 +6026,7 @@ Miscellaneous System Information
in the **system**.
If :option:`-X cpu_count <-X>` is given or :envvar:`PYTHON_CPU_COUNT` is set,
:func:`process_cpu_count` returns the overridden value *n*.
:func:`process_cpu_count` returns the override value *n*.
See also the :func:`sched_getaffinity` function.

View File

@ -1331,6 +1331,10 @@ Reading directories
PosixPath('setup.py'),
PosixPath('test_pathlib.py')]
.. note::
The paths are returned in no particular order.
If you need a specific order, sort the results.
.. seealso::
:ref:`pathlib-pattern-language` documentation.
@ -1365,6 +1369,10 @@ Reading directories
Glob the given relative *pattern* recursively. This is like calling
:func:`Path.glob` with "``**/``" added in front of the *pattern*.
.. note::
The paths are returned in no particular order.
If you need a specific order, sort the results.
.. seealso::
:ref:`pathlib-pattern-language` and :meth:`Path.glob` documentation.

View File

@ -520,7 +520,8 @@ can be overridden by the local file.
To remove all commands from a breakpoint, type ``commands`` and follow it
immediately with ``end``; that is, give no commands.
With no *bpnumber* argument, ``commands`` refers to the last breakpoint set.
With no *bpnumber* argument, ``commands`` refers to the most recently set
breakpoint that still exists.
You can use breakpoint commands to start your program up again. Simply use
the :pdbcmd:`continue` command, or :pdbcmd:`step`,

View File

@ -56,19 +56,6 @@ files.
The :mod:`pickle` module differs from :mod:`marshal` in several significant ways:
* The :mod:`pickle` module keeps track of the objects it has already serialized,
so that later references to the same object won't be serialized again.
:mod:`marshal` doesn't do this.
This has implications both for recursive objects and object sharing. Recursive
objects are objects that contain references to themselves. These are not
handled by marshal, and in fact, attempting to marshal recursive objects will
crash your Python interpreter. Object sharing happens when there are multiple
references to the same object in different places in the object hierarchy being
serialized. :mod:`pickle` stores such objects only once, and ensures that all
other references point to the master copy. Shared objects remain shared, which
can be very important for mutable objects.
* :mod:`marshal` cannot be used to serialize user-defined classes and their
instances. :mod:`pickle` can save and restore class instances transparently,
however the class definition must be importable and live in the same module as

View File

@ -1,4 +1,4 @@
.. highlight:: shell-session
.. highlight:: sh
.. _profiling-module:

View File

@ -1,4 +1,4 @@
.. highlight:: shell-session
.. highlight:: sh
.. _profiling-sampling:
@ -53,7 +53,7 @@ counts**, not direct measurements. Tachyon counts how many times each function
appears in the collected samples, then multiplies by the sampling interval to
estimate time.
For example, with a 100 microsecond sampling interval over a 10-second profile,
For example, with a 10 kHz sampling rate over a 10-second profile,
Tachyon collects approximately 100,000 samples. If a function appears in 5,000
samples (5% of total), Tachyon estimates it consumed 5% of the 10-second
duration, or about 500 milliseconds. This is a statistical estimate, not a
@ -142,7 +142,7 @@ Use live mode for real-time monitoring (press ``q`` to quit)::
Profile for 60 seconds with a faster sampling rate::
python -m profiling.sampling run -d 60 -i 50 script.py
python -m profiling.sampling run -d 60 -r 20khz script.py
Generate a line-by-line heatmap::
@ -200,6 +200,36 @@ On most systems, attaching to another process requires appropriate permissions.
See :ref:`profiling-permissions` for platform-specific requirements.
.. _replay-command:
The ``replay`` command
----------------------
The ``replay`` command converts binary profile files to other output formats::
python -m profiling.sampling replay profile.bin
python -m profiling.sampling replay --flamegraph -o profile.html profile.bin
This command is useful when you have captured profiling data in binary format
and want to analyze it later or convert it to a visualization format. Binary
profiles can be replayed multiple times to different formats without
re-profiling.
::
# Convert binary to pstats (default, prints to stdout)
python -m profiling.sampling replay profile.bin
# Convert binary to flame graph
python -m profiling.sampling replay --flamegraph -o output.html profile.bin
# Convert binary to gecko format for Firefox Profiler
python -m profiling.sampling replay --gecko -o profile.json profile.bin
# Convert binary to heatmap
python -m profiling.sampling replay --heatmap -o my_heatmap profile.bin
Profiling in production
-----------------------
@ -211,8 +241,8 @@ is unaware it is being profiled.
When profiling production systems, keep these guidelines in mind:
Start with shorter durations (10-30 seconds) to get quick results, then extend
if you need more statistical accuracy. The default 10-second duration is usually
sufficient to identify major hotspots.
if you need more statistical accuracy. By default, profiling runs until the
target process completes, which is usually sufficient to identify major hotspots.
If possible, profile during representative load rather than peak traffic.
Profiles collected during normal operation are easier to interpret than those
@ -296,10 +326,10 @@ The default configuration works well for most use cases:
* - Option
- Default
* - Default for ``--interval`` / ``-i``
- 100 µs between samples (~10,000 samples/sec)
* - Default for ``--sampling-rate`` / ``-r``
- 1 kHz
* - Default for ``--duration`` / ``-d``
- 10 seconds
- Run to completion
* - Default for ``--all-threads`` / ``-a``
- Main thread only
* - Default for ``--native``
@ -312,35 +342,35 @@ The default configuration works well for most use cases:
- Disabled
* - Default for ``--subprocesses``
- Disabled
* - Default for ``--blocking``
- Disabled (non-blocking sampling)
Sampling interval and duration
------------------------------
Sampling rate and duration
--------------------------
The two most fundamental parameters are the sampling interval and duration.
The two most fundamental parameters are the sampling rate and duration.
Together, these determine how many samples will be collected during a profiling
session.
The :option:`--interval` option (:option:`-i`) sets the time between samples in
microseconds. The default is 100 microseconds, which produces approximately
10,000 samples per second::
The :option:`--sampling-rate` option (:option:`-r`) sets how frequently samples
are collected. The default is 1 kHz (10,000 samples per second)::
python -m profiling.sampling run -i 50 script.py
python -m profiling.sampling run -r 20khz script.py
Lower intervals capture more samples and provide finer-grained data at the
cost of slightly higher profiler CPU usage. Higher intervals reduce profiler
Higher rates capture more samples and provide finer-grained data at the
cost of slightly higher profiler CPU usage. Lower rates reduce profiler
overhead but may miss short-lived functions. For most applications, the
default interval provides a good balance between accuracy and overhead.
default rate provides a good balance between accuracy and overhead.
The :option:`--duration` option (:option:`-d`) sets how long to profile in seconds. The
default is 10 seconds::
The :option:`--duration` option (:option:`-d`) sets how long to profile in seconds. By
default, profiling continues until the target process exits or is interrupted::
python -m profiling.sampling run -d 60 script.py
Longer durations collect more samples and produce more statistically reliable
results, especially for code paths that execute infrequently. When profiling
a program that runs for a fixed time, you may want to set the duration to
match or exceed the expected runtime.
Specifying a duration is useful when attaching to long-running processes or when
you want to limit profiling to a specific time window. When profiling a script,
the default behavior of running to completion is usually what you want.
Thread selection
@ -362,6 +392,50 @@ This option is particularly useful when investigating concurrency issues or
when work is distributed across a thread pool.
.. _blocking-mode:
Blocking mode
-------------
By default, Tachyon reads the target process's memory without stopping it.
This non-blocking approach is ideal for most profiling scenarios because it
imposes virtually zero overhead on the target application: the profiled
program runs at full speed and is unaware it is being observed.
However, non-blocking sampling can occasionally produce incomplete or
inconsistent stack traces in applications with many generators or coroutines
that rapidly switch between yield points, or in programs with very fast-changing
call stacks where functions enter and exit between the start and end of a single
stack read, resulting in reconstructed stacks that mix frames from different
execution states or that never actually existed.
For these cases, the :option:`--blocking` option stops the target process during
each sample::
python -m profiling.sampling run --blocking script.py
python -m profiling.sampling attach --blocking 12345
When blocking mode is enabled, the profiler suspends the target process,
reads its stack, then resumes it. This guarantees that each captured stack
represents a real, consistent snapshot of what the process was doing at that
instant. The trade-off is that the target process runs slower because it is
repeatedly paused.
.. warning::
Do not use very high sample rates (low ``--interval`` values) with blocking
mode. Suspending and resuming a process takes time, and if the sampling
interval is too short, the target will spend more time stopped than running.
For blocking mode, intervals of 1000 microseconds (1 millisecond) or higher
are recommended. The default 100 microsecond interval may cause noticeable
slowdown in the target application.
Use blocking mode only when you observe inconsistent stacks in your profiles,
particularly with generator-heavy or coroutine-heavy code. For most
applications, the default non-blocking mode provides accurate results with
zero impact on the target process.
Special frames
--------------
@ -497,9 +571,9 @@ appended:
- For pstats format (which defaults to stdout), subprocesses produce files like
``profile_12345.pstats``
The subprocess profilers inherit most sampling options from the parent (interval,
duration, thread selection, native frames, GC frames, async-aware mode, and
output format). All Python descendant processes are profiled recursively,
The subprocess profilers inherit most sampling options from the parent (sampling
rate, duration, thread selection, native frames, GC frames, async-aware mode,
and output format). All Python descendant processes are profiled recursively,
including grandchildren and further descendants.
Subprocess detection works by periodically scanning for new descendants of
@ -804,9 +878,9 @@ interesting functions that highlights:
Use :option:`--no-summary` to suppress both the legend and summary sections.
To save pstats output to a file instead of stdout::
To save pstats output to a binary file instead of stdout::
python -m profiling.sampling run -o profile.txt script.py
python -m profiling.sampling run -o profile.pstats script.py
The pstats format supports several options for controlling the display.
The :option:`--sort` option determines the column used for ordering results::
@ -1041,6 +1115,59 @@ intuitive view that shows exactly where time is spent without requiring
interpretation of hierarchical visualizations.
Binary format
-------------
Binary format (:option:`--binary`) produces a compact binary file for efficient
storage of profiling data::
python -m profiling.sampling run --binary -o profile.bin script.py
python -m profiling.sampling attach --binary -o profile.bin 12345
The :option:`--compression` option controls data compression:
- ``auto`` (default): Use zstd compression if available, otherwise no
compression
- ``zstd``: Force zstd compression (requires :mod:`compression.zstd` support)
- ``none``: Disable compression
::
python -m profiling.sampling run --binary --compression=zstd -o profile.bin script.py
To analyze binary profiles, use the :ref:`replay-command` to convert them to
other formats like flame graphs or pstats output.
Record and replay workflow
==========================
The binary format combined with the replay command enables a record-and-replay
workflow that separates data capture from analysis. Rather than generating
visualizations during profiling, you capture raw data to a compact binary file
and convert it to different formats later.
This approach has three main benefits:
- Sampling runs faster because the work of building data structures for
visualization is deferred until replay.
- A single binary capture can be converted to multiple output formats
without re-profiling: pstats for a quick overview, flame graph for visual
exploration, heatmap for line-level detail.
- Binary files are compact and easy to share with colleagues who can convert
them to their preferred format.
A typical workflow::
# Capture profile in production or during tests
python -m profiling.sampling attach --binary -o profile.bin 12345
# Later, analyze with different formats
python -m profiling.sampling replay profile.bin
python -m profiling.sampling replay --flamegraph -o profile.html profile.bin
python -m profiling.sampling replay --heatmap -o heatmap profile.bin
Live mode
=========
@ -1252,17 +1379,21 @@ Global options
Attach to and profile a running process by PID.
.. option:: replay
Convert a binary profile file to another output format.
Sampling options
----------------
.. option:: -i <microseconds>, --interval <microseconds>
.. option:: -r <rate>, --sampling-rate <rate>
Sampling interval in microseconds. Default: 100.
Sampling rate (for example, ``10000``, ``10khz``, ``10k``). Default: ``1khz``.
.. option:: -d <seconds>, --duration <seconds>
Profiling duration in seconds. Default: 10.
Profiling duration in seconds. Default: run to completion.
.. option:: -a, --all-threads
@ -1296,6 +1427,13 @@ Sampling options
Also profile subprocesses. Each subprocess gets its own profiler
instance and output file. Incompatible with ``--live``.
.. option:: --blocking
Pause the target process during each sample. This ensures consistent
stack traces at the cost of slowing down the target. Use with longer
intervals (1000 µs or higher) to minimize impact. See :ref:`blocking-mode`
for details.
Mode options
------------
@ -1317,7 +1455,9 @@ Output options
.. option:: --pstats
Generate text statistics output. This is the default.
Generate pstats statistics. This is the default.
When written to stdout, the output is a text table; with :option:`-o`,
it is a binary pstats file.
.. option:: --collapsed
@ -1335,12 +1475,30 @@ Output options
Generate HTML heatmap with line-level sample counts.
.. option:: --binary
Generate high-performance binary format for later conversion with the
``replay`` command.
.. option:: --compression <type>
Compression for binary format: ``auto`` (use zstd if available, default),
``zstd``, or ``none``.
.. option:: -o <path>, --output <path>
Output file or directory path. Default behavior varies by format:
``--pstats`` writes to stdout, ``--flamegraph`` and ``--gecko`` generate
files like ``flamegraph.PID.html``, and ``--heatmap`` creates a directory
named ``heatmap_PID``.
:option:`--pstats` prints a text table to stdout, while ``-o`` writes a
binary pstats file. Other formats generate a file named
``<format>_<PID>.<ext>`` (for example, ``flamegraph_12345.html``).
:option:`--heatmap` creates a directory named ``heatmap_<PID>``.
.. option:: --browser
Automatically open HTML output (:option:`--flamegraph` and
:option:`--heatmap`) in your default web browser after generation.
When profiling with :option:`--subprocesses`, only the main process
opens the browser; subprocess outputs are never auto-opened.
pstats display options

View File

@ -78,7 +78,7 @@ Bookkeeping functions
instead of the system time (see the :func:`os.urandom` function for details
on availability).
If *a* is an int, it is used directly.
If *a* is an int, its absolute value is used directly.
With version 2 (the default), a :class:`str`, :class:`bytes`, or :class:`bytearray`
object gets converted to an :class:`int` and all of its bits are used.
@ -634,11 +634,12 @@ from the combinatoric iterators in the :mod:`itertools` module
or the :pypi:`more-itertools` project:
.. testcode::
import random
def random_product(*args, repeat=1):
"Random selection from itertools.product(*args, **kwds)"
pools = [tuple(pool) for pool in args] * repeat
def random_product(*iterables, repeat=1):
"Random selection from itertools.product(*iterables, repeat=repeat)"
pools = tuple(map(tuple, iterables)) * repeat
return tuple(map(random.choice, pools))
def random_permutation(iterable, r=None):
@ -663,15 +664,89 @@ or the :pypi:`more-itertools` project:
return tuple(pool[i] for i in indices)
def random_derangement(iterable):
"Choose a permutation where no element is in its original position."
"Choose a permutation where no element stays in its original position."
seq = tuple(iterable)
if len(seq) < 2:
raise ValueError('derangements require at least two values')
perm = list(seq)
if not seq:
return ()
raise IndexError('No derangments to choose from')
perm = list(range(len(seq)))
start = tuple(perm)
while True:
random.shuffle(perm)
if all(p != q for p, q in zip(seq, perm)):
return tuple(perm)
if all(p != q for p, q in zip(start, perm)):
return tuple([seq[i] for i in perm])
.. doctest::
:hide:
>>> import random
>>> random.seed(8675309)
>>> random_product('ABCDEFG', repeat=5)
('D', 'B', 'E', 'F', 'E')
>>> random.seed(8675309)
>>> random_permutation('ABCDEFG')
('D', 'B', 'E', 'C', 'G', 'A', 'F')
>>> random_permutation('ABCDEFG', 5)
('A', 'G', 'D', 'C', 'B')
>>> random.seed(8675309)
>>> random_combination('ABCDEFG', 7)
('A', 'B', 'C', 'D', 'E', 'F', 'G')
>>> random_combination('ABCDEFG', 6)
('A', 'B', 'C', 'D', 'F', 'G')
>>> random_combination('ABCDEFG', 5)
('A', 'B', 'C', 'E', 'F')
>>> random_combination('ABCDEFG', 4)
('B', 'C', 'D', 'G')
>>> random_combination('ABCDEFG', 3)
('B', 'E', 'G')
>>> random_combination('ABCDEFG', 2)
('E', 'G')
>>> random_combination('ABCDEFG', 1)
('C',)
>>> random_combination('ABCDEFG', 0)
()
>>> random.seed(8675309)
>>> random_combination_with_replacement('ABCDEFG', 7)
('B', 'C', 'D', 'E', 'E', 'E', 'G')
>>> random_combination_with_replacement('ABCDEFG', 3)
('A', 'B', 'E')
>>> random_combination_with_replacement('ABCDEFG', 2)
('A', 'G')
>>> random_combination_with_replacement('ABCDEFG', 1)
('E',)
>>> random_combination_with_replacement('ABCDEFG', 0)
()
>>> random.seed(8675309)
>>> random_derangement('')
()
>>> random_derangement('A')
Traceback (most recent call last):
...
IndexError: No derangments to choose from
>>> random_derangement('AB')
('B', 'A')
>>> random_derangement('ABC')
('C', 'A', 'B')
>>> random_derangement('ABCD')
('B', 'A', 'D', 'C')
>>> random_derangement('ABCDE')
('B', 'C', 'A', 'E', 'D')
>>> # Identical inputs treated as distinct
>>> identical = 20
>>> random_derangement((10, identical, 30, identical))
(20, 30, 10, 20)
The default :func:`.random` returns multiples of 2⁻⁵³ in the range
*0.0 ≤ x < 1.0*. All such numbers are evenly spaced and are exactly

View File

@ -403,3 +403,9 @@ support history save/restore. ::
def save_history(self, histfile):
readline.set_history_length(1000)
readline.write_history_file(histfile)
.. note::
The new :term:`REPL` introduced in version 3.13 doesn't support readline.
However, readline can still be used by setting the :envvar:`PYTHON_BASIC_REPL`
environment variable.

View File

@ -478,6 +478,8 @@ linearly scanned again. :c:func:`!select` is *O*\ (*highest file descriptor*), w
.. versionchanged:: 3.15
Accepts any real number as *timeout*, not only integer or float.
If ``ppoll()`` function is available, *timeout* has a resolution
of ``1`` ns (``1e-6`` ms) instead of ``1`` ms.
.. _kqueue-objects:

View File

@ -515,7 +515,7 @@ Directory and files operations
.. exception:: Error
This exception collects exceptions that are raised during a multi-file
Subclass of :exc:`OSError` collecting exceptions raised during a multi-file
operation. For :func:`copytree`, the exception argument is a list of 3-tuples
(*srcname*, *dstname*, *exception*).

View File

@ -1072,10 +1072,16 @@ The :mod:`socket` module also offers various network-related services:
a string representing the canonical name of the *host* if
:const:`AI_CANONNAME` is part of the *flags* argument; else *canonname*
will be empty. *sockaddr* is a tuple describing a socket address, whose
format depends on the returned *family* (a ``(address, port)`` 2-tuple for
:const:`AF_INET`, a ``(address, port, flowinfo, scope_id)`` 4-tuple for
:const:`AF_INET6`), and is meant to be passed to the :meth:`socket.connect`
method.
format depends on the returned *family* and flags Python was compiled with,
and is meant to be passed to the :meth:`socket.connect` method.
*sockaddr* can be one of the following:
* a ``(address, port)`` 2-tuple for :const:`AF_INET`
* a ``(address, port, flowinfo, scope_id)`` 4-tuple for :const:`AF_INET6` if
Python was compiled with ``--enable-ipv6`` (the default)
* a 2-tuple containing raw data for :const:`AF_INET6` if Python was
compiled with ``--disable-ipv6``
.. note::

View File

@ -46,8 +46,10 @@ Any object can be tested for truth value, for use in an :keyword:`if` or
By default, an object is considered true unless its class defines either a
:meth:`~object.__bool__` method that returns ``False`` or a
:meth:`~object.__len__` method that
returns zero, when called with the object. [1]_ Here are most of the built-in
objects considered false:
returns zero, when called with the object. [1]_ If one of the methods raises an
exception when called, the exception is propagated and the object does
not have a truth value (for example, :data:`NotImplemented`).
Here are most of the built-in objects considered false:
.. index::
single: None (Built-in object)
@ -1091,11 +1093,14 @@ Notes:
still ``0``.
(4)
The slice of *s* from *i* to *j* is defined as the sequence of items with index
*k* such that ``i <= k < j``. If *i* or *j* is greater than ``len(s)``, use
``len(s)``. If *i* is omitted or ``None``, use ``0``. If *j* is omitted or
``None``, use ``len(s)``. If *i* is greater than or equal to *j*, the slice is
empty.
The slice of *s* from *i* to *j* is defined as the sequence of items with
index *k* such that ``i <= k < j``.
* If *i* is omitted or ``None``, use ``0``.
* If *j* is omitted or ``None``, use ``len(s)``.
* If *i* or *j* is less than ``-len(s)``, use ``0``.
* If *i* or *j* is greater than ``len(s)``, use ``len(s)``.
* If *i* is greater than or equal to *j*, the slice is empty.
(5)
The slice of *s* from *i* to *j* with step *k* is defined as the sequence of
@ -1436,6 +1441,109 @@ application).
list appear empty for the duration, and raises :exc:`ValueError` if it can
detect that the list has been mutated during a sort.
.. admonition:: Thread safety
Reading a single element from a :class:`list` is
:term:`atomic <atomic operation>`:
.. code-block::
:class: green
lst[i] # list.__getitem__
The following methods traverse the list and use :term:`atomic <atomic operation>`
reads of each item to perform their function. That means that they may
return results affected by concurrent modifications:
.. code-block::
:class: maybe
item in lst
lst.index(item)
lst.count(item)
All of the above methods/operations are also lock-free. They do not block
concurrent modifications. Other operations that hold a lock will not block
these from observing intermediate states.
All other operations from here on block using the per-object lock.
Writing a single item via ``lst[i] = x`` is safe to call from multiple
threads and will not corrupt the list.
The following operations return new objects and appear
:term:`atomic <atomic operation>` to other threads:
.. code-block::
:class: good
lst1 + lst2 # concatenates two lists into a new list
x * lst # repeats lst x times into a new list
lst.copy() # returns a shallow copy of the list
Methods that only operate on a single elements with no shifting required are
:term:`atomic <atomic operation>`:
.. code-block::
:class: good
lst.append(x) # append to the end of the list, no shifting required
lst.pop() # pop element from the end of the list, no shifting required
The :meth:`~list.clear` method is also :term:`atomic <atomic operation>`.
Other threads cannot observe elements being removed.
The :meth:`~list.sort` method is not :term:`atomic <atomic operation>`.
Other threads cannot observe intermediate states during sorting, but the
list appears empty for the duration of the sort.
The following operations may allow lock-free operations to observe
intermediate states since they modify multiple elements in place:
.. code-block::
:class: maybe
lst.insert(idx, item) # shifts elements
lst.pop(idx) # idx not at the end of the list, shifts elements
lst *= x # copies elements in place
The :meth:`~list.remove` method may allow concurrent modifications since
element comparison may execute arbitrary Python code (via
:meth:`~object.__eq__`).
:meth:`~list.extend` is safe to call from multiple threads. However, its
guarantees depend on the iterable passed to it. If it is a :class:`list`, a
:class:`tuple`, a :class:`set`, a :class:`frozenset`, a :class:`dict` or a
:ref:`dictionary view object <dict-views>` (but not their subclasses), the
``extend`` operation is safe from concurrent modifications to the iterable.
Otherwise, an iterator is created which can be concurrently modified by
another thread. The same applies to inplace concatenation of a list with
other iterables when using ``lst += iterable``.
Similarly, assigning to a list slice with ``lst[i:j] = iterable`` is safe
to call from multiple threads, but ``iterable`` is only locked when it is
also a :class:`list` (but not its subclasses).
Operations that involve multiple accesses, as well as iteration, are never
atomic. For example:
.. code-block::
:class: bad
# NOT atomic: read-modify-write
lst[i] = lst[i] + 1
# NOT atomic: check-then-act
if lst:
item = lst.pop()
# NOT thread-safe: iteration while modifying
for item in lst:
process(item) # another thread may modify lst
Consider external synchronization when sharing :class:`list` instances
across threads. See :ref:`freethreading-python-howto` for more information.
.. _typesseq-tuple:
@ -1842,6 +1950,14 @@ expression support in the :mod:`re` module).
lowercase letter ``'ß'`` is equivalent to ``"ss"``. Since it is already
lowercase, :meth:`lower` would do nothing to ``'ß'``; :meth:`casefold`
converts it to ``"ss"``.
For example:
.. doctest::
>>> 'straße'.lower()
'straße'
>>> 'straße'.casefold()
'strasse'
The casefolding algorithm is `described in section 3.13.3 'Default Case
Folding' of the Unicode Standard
@ -2043,7 +2159,18 @@ expression support in the :mod:`re` module).
.. method:: str.index(sub[, start[, end]])
Like :meth:`~str.find`, but raise :exc:`ValueError` when the substring is
not found.
not found. For example:
.. doctest::
>>> 'spam, spam, spam'.index('eggs')
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
'spam, spam, spam'.index('eggs')
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
ValueError: substring not found
See also :meth:`rindex`.
.. method:: str.isalnum()
@ -2188,6 +2315,15 @@ expression support in the :mod:`re` module).
Nonprintable characters are those in group Separator or Other (Z or C),
except the ASCII space.
For example:
.. doctest::
>>> ''.isprintable(), ' '.isprintable()
(True, True)
>>> '\t'.isprintable(), '\n'.isprintable()
(False, False)
.. method:: str.isspace()
@ -2278,7 +2414,12 @@ expression support in the :mod:`re` module).
.. method:: str.lower()
Return a copy of the string with all the cased characters [4]_ converted to
lowercase.
lowercase. For example:
.. doctest::
>>> 'Lower Method Example'.lower()
'lower method example'
The lowercasing algorithm used is `described in section 3.13.2 'Default Case
Conversion' of the Unicode Standard
@ -2334,7 +2475,9 @@ expression support in the :mod:`re` module).
If the string starts with the *prefix* string, return
``string[len(prefix):]``. Otherwise, return a copy of the original
string::
string:
.. doctest::
>>> 'TestHook'.removeprefix('Test')
'Hook'
@ -2343,12 +2486,16 @@ expression support in the :mod:`re` module).
.. versionadded:: 3.9
See also :meth:`removesuffix` and :meth:`startswith`.
.. method:: str.removesuffix(suffix, /)
If the string ends with the *suffix* string and that *suffix* is not empty,
return ``string[:-len(suffix)]``. Otherwise, return a copy of the
original string::
original string:
.. doctest::
>>> 'MiscTests'.removesuffix('Tests')
'Misc'
@ -2357,12 +2504,22 @@ expression support in the :mod:`re` module).
.. versionadded:: 3.9
See also :meth:`removeprefix` and :meth:`endswith`.
.. method:: str.replace(old, new, /, count=-1)
Return a copy of the string with all occurrences of substring *old* replaced by
*new*. If *count* is given, only the first *count* occurrences are replaced.
If *count* is not specified or ``-1``, then all occurrences are replaced.
For example:
.. doctest::
>>> 'spam, spam, spam'.replace('spam', 'eggs')
'eggs, eggs, eggs'
>>> 'spam, spam, spam'.replace('spam', 'eggs', 1)
'eggs, spam, spam'
.. versionchanged:: 3.13
*count* is now supported as a keyword argument.
@ -2373,6 +2530,16 @@ expression support in the :mod:`re` module).
Return the highest index in the string where substring *sub* is found, such
that *sub* is contained within ``s[start:end]``. Optional arguments *start*
and *end* are interpreted as in slice notation. Return ``-1`` on failure.
For example:
.. doctest::
>>> 'spam, spam, spam'.rfind('sp')
12
>>> 'spam, spam, spam'.rfind('sp', 0, 10)
6
See also :meth:`find` and :meth:`rindex`.
.. method:: str.rindex(sub[, start[, end]])
@ -2395,6 +2562,19 @@ expression support in the :mod:`re` module).
after the separator. If the separator is not found, return a 3-tuple containing
two empty strings, followed by the string itself.
For example:
.. doctest::
>>> 'Monty Python'.rpartition(' ')
('Monty', ' ', 'Python')
>>> "Monty Python's Flying Circus".rpartition(' ')
("Monty Python's Flying", ' ', 'Circus')
>>> 'Monty Python'.rpartition('-')
('', '', 'Monty Python')
See also :meth:`partition`.
.. method:: str.rsplit(sep=None, maxsplit=-1)

View File

@ -180,6 +180,12 @@ Examining Symbol Tables
Return a tuple containing names of :term:`free (closure) variables <closure variable>`
in this function.
.. method:: get_cells()
Return a tuple containing names of :term:`cell variables <closure variable>` in this table.
.. versionadded:: next
.. class:: Class
@ -291,6 +297,12 @@ Examining Symbol Tables
Return ``True`` if the symbol is referenced in its block, but not assigned
to.
.. method:: is_cell()
Return ``True`` if the symbol is referenced but not assigned in a nested block.
.. versionadded:: next
.. method:: is_free_class()
Return *True* if a class-scoped symbol is free from

View File

@ -1997,6 +1997,9 @@ always available. Unless explicitly noted otherwise, all variables are read-only
interpreter is pre-release (alpha, beta, or release candidate) then the
local and remote interpreters must be the same exact version.
See :ref:`remote-debugging` for more information about the remote debugging
mechanism.
.. audit-event:: sys.remote_exec pid script_path
When the code is executed in the remote process, an
@ -2015,6 +2018,7 @@ always available. Unless explicitly noted otherwise, all variables are read-only
.. availability:: Unix, Windows.
.. versionadded:: 3.14
See :pep:`768` for more details.
.. function:: _enablelegacywindowsfsencoding()

View File

@ -225,8 +225,9 @@ The module defines the following user-callable items:
properly implements the :const:`os.O_EXCL` flag for :func:`os.open`. The
file is readable and writable only by the creating user ID. If the
platform uses permission bits to indicate whether a file is executable,
the file is executable by no one. The file descriptor is not inherited
by child processes.
the file is executable by no one.
The file descriptor is :ref:`not inherited by child processes <fd_inheritance>`.
Unlike :func:`TemporaryFile`, the user of :func:`mkstemp` is responsible
for deleting the temporary file when done with it.

View File

@ -492,6 +492,12 @@ The :mod:`test.support` module defines the following functions:
tests.
.. function:: get_resource_value(resource)
Return the value specified for *resource* (as :samp:`-u {resource}={value}`).
Return ``None`` if *resource* is disabled or no value is specified.
.. function:: python_is_optimized()
Return ``True`` if Python was not built with ``-O0`` or ``-Og``.

View File

@ -102,6 +102,10 @@ functions should be good enough; otherwise, you should use an instance of
print(repr(s)) # prints ' hello\n world\n '
print(repr(dedent(s))) # prints 'hello\n world\n'
.. versionchanged:: 3.14
The :func:`!dedent` function now correctly normalizes blank lines containing
only whitespace characters. Previously, the implementation only normalized
blank lines containing tabs and spaces.
.. function:: indent(text, prefix, predicate=None)

View File

@ -177,12 +177,12 @@ the modern themed widget set and API::
.. attribute:: master
The widget object that contains this widget. For :class:`Tk`, the
*master* is :const:`None` because it is the main window. The terms
:attr:`!master` is :const:`None` because it is the main window. The terms
*master* and *parent* are similar and sometimes used interchangeably
as argument names; however, calling :meth:`winfo_parent` returns a
string of the widget name whereas :attr:`master` returns the object.
string of the widget name whereas :attr:`!master` returns the object.
*parent*/*child* reflects the tree-like relationship while
*master*/*slave* reflects the container structure.
*master* (or *container*)/*content* reflects the container structure.
.. attribute:: children
@ -638,15 +638,15 @@ The Packer
.. index:: single: packing (widgets)
The packer is one of Tk's geometry-management mechanisms. Geometry managers
are used to specify the relative positioning of widgets within their container -
their mutual *master*. In contrast to the more cumbersome *placer* (which is
are used to specify the relative positioning of widgets within their container.
In contrast to the more cumbersome *placer* (which is
used less commonly, and we do not cover here), the packer takes qualitative
relationship specification - *above*, *to the left of*, *filling*, etc - and
works everything out to determine the exact placement coordinates for you.
The size of any *master* widget is determined by the size of the "slave widgets"
inside. The packer is used to control where slave widgets appear inside the
master into which they are packed. You can pack widgets into frames, and frames
The size of any container widget is determined by the size of the "content widgets"
inside. The packer is used to control where content widgets appear inside the
container into which they are packed. You can pack widgets into frames, and frames
into other frames, in order to achieve the kind of layout you desire.
Additionally, the arrangement is dynamically adjusted to accommodate incremental
changes to the configuration, once it is packed.
@ -673,7 +673,7 @@ For more extensive information on the packer and the options that it can take,
see the man pages and page 183 of John Ousterhout's book.
anchor
Anchor type. Denotes where the packer is to place each slave in its parcel.
Anchor type. Denotes where the packer is to place each content in its parcel.
expand
Boolean, ``0`` or ``1``.
@ -682,10 +682,10 @@ fill
Legal values: ``'x'``, ``'y'``, ``'both'``, ``'none'``.
ipadx and ipady
A distance - designating internal padding on each side of the slave widget.
A distance - designating internal padding on each side of the content.
padx and pady
A distance - designating external padding on each side of the slave widget.
A distance - designating external padding on each side of the content.
side
Legal values are: ``'left'``, ``'right'``, ``'top'``, ``'bottom'``.
@ -758,8 +758,8 @@ subclassed from the :class:`Wm` class, and so can call the :class:`Wm` methods
directly.
To get at the toplevel window that contains a given widget, you can often just
refer to the widget's master. Of course if the widget has been packed inside of
a frame, the master won't represent a toplevel window. To get at the toplevel
refer to the widget's :attr:`master`. Of course if the widget has been packed inside of
a frame, the :attr:`!master` won't represent a toplevel window. To get at the toplevel
window that contains an arbitrary widget, you can call the :meth:`_root` method.
This method begins with an underscore to denote the fact that this function is
part of the implementation, and not an interface to Tk functionality.

View File

@ -2442,6 +2442,10 @@ types.
Removed the ``_field_types`` attribute in favor of the more
standard ``__annotations__`` attribute which has the same information.
.. versionchanged:: 3.9
``NamedTuple`` is now a function rather than a class.
It can still be used as a class base, as described above.
.. versionchanged:: 3.11
Added support for generic namedtuples.
@ -2588,7 +2592,7 @@ types.
.. class:: TypedDict(dict)
Special construct to add type hints to a dictionary.
At runtime it is a plain :class:`dict`.
At runtime ":class:`!TypedDict` instances" are simply :class:`dicts <dict>`.
``TypedDict`` declares a dictionary type that expects all of its
instances to have a certain set of keys, where each key is
@ -2811,6 +2815,10 @@ types.
.. versionadded:: 3.8
.. versionchanged:: 3.9
``TypedDict`` is now a function rather than a class.
It can still be used as a class base, as described above.
.. versionchanged:: 3.11
Added support for marking individual keys as :data:`Required` or :data:`NotRequired`.
See :pep:`655`.

Some files were not shown because too many files have changed in this diff Show More