* [3.9] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944)
If the error handler is used, a new bytes object is created to set as
the object attribute of UnicodeDecodeError, and that bytes object then
replaces the original data. A pointer to the decoded data will became invalid
after destroying that temporary bytes object. So we need other way to return
the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().
_PyBytes_DecodeEscape() does not have such issue, because it does not
use the error handlers registry, but it should be changed for compatibility
with _PyUnicode_DecodeUnicodeEscapeInternal().
(cherry picked from commit 9f69a58623bd01349a18ba0c7a9cb1dad6a51e8e)
(cherry picked from commit 6279eb8c076d89d3739a6edb393e43c7929b429d)
(cherry picked from commit a75953b347716fff694aa59a7c7c2489fa50d1f5)
(cherry picked from commit 0c33e5baedf18ebcb04bc41dff7cfc614d5ea5fe)
(cherry picked from commit 8b528cacbbde60504f6ac62784d04889d285f18b)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Previously *consumed was not set in this case.
(cherry picked from commit f08e52ccb027f6f703302b8c1a82db9fd3934270).
(cherry picked from commit b8b3e6afc0a48c3cbb7c36d2f73e332edcd6058c)
When ValueError is raised if an integer is larger than the limit,
mention sys.set_int_max_str_digits() in the error message.
(cherry picked from commit e841ffc915e82e5ea6e3b473205417d63494808d)
Co-authored-by: Ned Deily <nad@python.org>
gh-97616: list_resize() checks for integer overflow (GH-97617)
Fix multiplying a list by an integer (list *= int): detect the
integer overflow when the new allocated length is close to the
maximum size. Issue reported by Jordan Limor.
list_resize() now checks for integer overflow before multiplying the
new allocated length by the list item size (sizeof(PyObject*)).
(cherry picked from commit a5f092f3c469b674b8d9ccbd4e4377230c9ac7cf)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Correctly pre-check for int-to-str conversion (#96537)
Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)
The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.
The justification for the current check. The C code check is:
```c
max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10
```
In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$
From this it follows that
$$\frac{M}{3L} < \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$
So
$$2^{L(s-1)} > 10^M.$$
But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.
<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
If the error handler returns position less or equal than the starting
position of non-encodable characters, most of built-in encoders didn't
properly re-size the output buffer. This led to out-of-bounds writes,
and segfaults.
(cherry picked from commit 18b07d773e09a2719e69aeaa925d5abb7ba0c068)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
The left-hand side expression of the if-check can be converted to a
constant by the compiler, but the addition on the right-hand side is
performed during runtime.
Move the addition from the right-hand side to the left-hand side by
turning it into a subtraction there. Since the values are known to
be large enough to not turn negative, this is a safe operation.
Prevents a very unlikely integer overflow on 32 bit systems.
Fixes GH-91421.
(cherry picked from commit 0859368335d470b9ff33fc53ed9a85ec2654b278)
Co-authored-by: Tobias Stoeckmann <stoeckmann@users.noreply.github.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
(cherry picked from commit 8be7c2bc5ad5e295f0f855bb31db412eef2c7c92)
Co-authored-by: Dave Goncalves <davegoncalves@gmail.com>
Rename the private undocumented float.__set_format__() method to
float.__setformat__() to fix a typo introduced in Python 3.7. The
method is only used by test_float.
The change enables again test_float tests on the float format which
were previously skipped because of the typo.
The typo was introduced in Python 3.7 by bpo-20185
in commit b5c51d3dd95bbfde533655fb86ac0f96f771ba7b.
(cherry picked from commit 7d03c8be5af2f1559dbc35b775b3116dfd63cfb6)
Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called.
(cherry picked from commit 4a66615ba736f84eadf9456bfd5d32a94cccf117)
Fix a race condition on setting a type __bases__ attribute: the
internal function add_subclass() now gets the
PyTypeObject.tp_subclasses member after calling PyWeakref_NewRef()
which can trigger a garbage collection which can indirectly modify
PyTypeObject.tp_subclasses.
(cherry picked from commit f1c6ae3270913e095d24ae13ecf96f5a32c8c503)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
The docstrings for MappingProxyType's keys(), values(), and items()
methods were never updated to reflect the changes that Python 3 brought
to these APIs, namely returning views rather than lists.
(cherry picked from commit 2d10fa9bc4cf83c5e5dd73decc9a138d6d247374)
Co-authored-by: Joshua Bronson <jabronson@gmail.com>
* Use Py_EnterRecursiveCall() in issubclass()
Reviewed-by: Gregory P. Smith <greg@krypto.org> [Google]
(cherry picked from commit 423fa1c1817abfa8c3d1bc308ddbbd8f28b69d68)
Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.raw_unicode_escape_decode().
It is True by default to match the former behavior.
(cherry picked from commit 39aa98346d5dd8ac591a7cafb467af21c53f1e5d)
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
(cherry picked from commit c96d1546b11b4c282a7e21737cb1f5d16349656d)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
It happened with fast range iterator when the calculated stop = start + step * len
was out of the C long range.
(cherry picked from commit 936f6a16b9ef85bd56b18a247b962801e954c30e)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* bpo-45083: Include the exception class qualname when formatting an exception (GH-28119)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
(cherry picked from commit b4b6342848ec0459182a992151099252434cc619)
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.
Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.
_PyMem_IsPtrFreed() argument is now constant.
(cherry picked from commit 615069eb08494d089bf24e43547fbc482ed699b8)
Co-authored-by: Victor Stinner <vstinner@python.org>
The non-GC-type branch of subtype_dealloc is using the type of an object after freeing in the same unsafe way as GH-26274 fixes. (I believe the old news entry covers this change well enough.)
https://bugs.python.org/issue44184
(cherry picked from commit 074e7659f208051b6b973f7fdb654dd22b93aaa2)
Co-authored-by: T. Wouters <thomas@python.org>
(cherry picked from commit d33943a6c368c2184e238019c63ac7a267da5594)
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
(cherry picked from commit 59242431991794064824cf2ab70886367613f29e)
Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
These are passed and called as PyCFunction, however they are defined here without the (ignored) args parameter.
This works fine in some C compilers, but fails in webassembly or anything else that has strict function pointer call type checking.
(cherry picked from commit ab383eb6f03896b0ef6634ee3d776344fcb9e5b8)
Co-authored-by: Joe Marshall <joe.marshall@nottingham.ac.uk>
Co-authored-by: Joe Marshall <joe.marshall@nottingham.ac.uk>
