If the error handler is used, a new bytes object is created to set as
the object attribute of UnicodeDecodeError, and that bytes object then
replaces the original data. A pointer to the decoded data will became invalid
after destroying that temporary bytes object. So we need other way to return
the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().
_PyBytes_DecodeEscape() does not have such issue, because it does not
use the error handlers registry, but it should be changed for compatibility
with _PyUnicode_DecodeUnicodeEscapeInternal().
(cherry picked from commit 9f69a58623bd01349a18ba0c7a9cb1dad6a51e8e)
(cherry picked from commit 6279eb8c076d89d3739a6edb393e43c7929b429d)
(cherry picked from commit a75953b347716fff694aa59a7c7c2489fa50d1f5)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:
- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`
Based on the "flush" idea from #115138 (comment) .
- Please treat as a security fix related to CVE-2023-52425.
(cherry picked from commit 6a95676)
(cherry picked from commit 73807eb)
(cherry picked from commit eda2963)
---------
Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
(cherry picked from commit 48c49739f5502fc7aa82f247ab2e4d7b55bdca62)
(cherry picked from commit d58a5f453f59f44ccf09b1a9b11a0b879ac6f35b)
Co-authored-by: Yilei Yang <yileiyang@google.com>
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
gh-108987: Fix _thread.start_new_thread() race condition (#109135)
Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.
thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.
Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().
(cherry picked from commit 517cd82ea7d01b344804413ef05610934a43a241)
gh-104690: thread_run() checks for tstate dangling pointer (#109056)
thread_run() of _threadmodule.c now calls
_PyThreadState_CheckConsistency() to check if tstate is a dangling
pointer when Python is built in debug mode.
Rename ceval_gil.c is_tstate_valid() to
_PyThreadState_CheckConsistency() to reuse it in _threadmodule.c.
(cherry picked from commit f63d37877ad166041489a968233b57540f8456e8)
gh-104018: remove unused format "z" handling in string formatfloat() (GH-104107)
This is a cleanup overlooked in PR GH-104033.
(cherry picked from commit 69621d1b09c996e43a1e13d2fa4c317d3dd4d738)
Co-authored-by: John Belmonte <john@neggie.net>
Several platforms don't define the static_assert macro despite having
compiler support for the _Static_assert keyword. The macro needs to be
defined since it is used unconditionally in the Python code. So it
should always be safe to define it if undefined and not in C++11 (or
later) mode.
Hence, remove the checks for particular platforms or libc versions,
and just define static_assert anytime it needs to be defined but isn't.
That way, all platforms that need the fix will get it, regardless of
whether someone specifically thought of them.
Also document that certain macOS versions are among the platforms that
need this.
The C2x draft (currently expected to become C23) makes static_assert
a keyword to match C++. So only define the macro for up to C17.
(cherry picked from commit 96e1901a59ed3bb6188743d60395666969a3ba42)
Co-authored-by: Joshua Root <jmr@macports.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-101037: Fix potential memory underallocation for zeros of int subtypes (GH-101038)
This PR fixes object allocation in long_subtype_new to ensure that there's at least one digit in all cases, and makes sure that the value of that digit is copied over from the source long.
Needs backport to 3.11, but not any further: the change to require at least one digit was only introduced for Python 3.11.
Fixes GH-101037.
(cherry picked from commit 401fdf9c851eb61229250ebffa942adca99b36d1)
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
The Py_CLEAR(), Py_SETREF() and Py_XSETREF() macros now only evaluate
their argument once. If an argument has side effects, these side
effects are no longer duplicated.
Add test_py_clear() and test_py_setref() unit tests to _testcapi.
(cherry picked from commit c03e05c2e72f3ea5e797389e7d1042eef85ad37a)
When merging the v3.11.0 tag into 3.11, some files were incorrectly updated and some others were not properly deleted.
Automerge-Triggered-By: GH:pablogsal
