If the error handler is used, a new bytes object is created to set as
the object attribute of UnicodeDecodeError, and that bytes object then
replaces the original data. A pointer to the decoded data will became invalid
after destroying that temporary bytes object. So we need other way to return
the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().
_PyBytes_DecodeEscape() does not have such issue, because it does not
use the error handlers registry, but it should be changed for compatibility
with _PyUnicode_DecodeUnicodeEscapeInternal().
(cherry picked from commit 9f69a58623bd01349a18ba0c7a9cb1dad6a51e8e)
(cherry picked from commit 6279eb8c076d89d3739a6edb393e43c7929b429d)
(cherry picked from commit a75953b347716fff694aa59a7c7c2489fa50d1f5)
(cherry picked from commit 0c33e5baedf18ebcb04bc41dff7cfc614d5ea5fe)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Previously *consumed was not set in this case.
(cherry picked from commit b8b3e6afc0a48c3cbb7c36d2f73e332edcd6058c)
(cherry picked from commit f08e52ccb027f6f703302b8c1a82db9fd3934270)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Cast size_t to Py_ssize_t, rather than casting it to long. On 64-bit
Windows, long is 32-bit whereas Py_ssize_t is 64-bit.
(cherry picked from commit 9707bf228e008485a3fbb63aa7ee28cf88014f91)
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix subscription of types.GenericAlias instances containing bare generic types:
for example tuple[A, T][int], where A is a generic type, and T is a type
variable.
gh-97943: PyFunction_GetAnnotations should return a borrowed reference. (GH-97949)
(cherry picked from commit 6bfb0be80486c614cd60dce44c9fe7b3e6c76e3b)
Co-authored-by: larryhastings <larry@hastings.org>
Fix multiplying a list by an integer (list *= int): detect the
integer overflow when the new allocated length is close to the
maximum size. Issue reported by Jordan Limor.
list_resize() now checks for integer overflow before multiplying the
new allocated length by the list item size (sizeof(PyObject*)).
(cherry picked from commit a5f092f3c469b674b8d9ccbd4e4377230c9ac7cf)
Co-authored-by: Victor Stinner <vstinner@python.org>
When ValueError is raised if an integer is larger than the limit,
mention sys.set_int_max_str_digits() in the error message.
(cherry picked from commit e841ffc915e82e5ea6e3b473205417d63494808d)
Co-authored-by: Victor Stinner <vstinner@python.org>
Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)
The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.
The justification for the current check. The C code check is:
```c
max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10
```
In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$
From this it follows that
$$\frac{M}{3L} < \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$
So
$$2^{L(s-1)} > 10^M.$$
But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.
<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
(cherry picked from commit b126196838bbaf5f4d35120e0e6bcde435b0b480)
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.
This PR comes fresh from a pile of work done in our private PSRT security response team repo.
This backports https://github.com/python/cpython/pull/96499 aka 511ca9452033ef95bc7d7fc404b8161068226002
Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).
<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->
I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#).
This PR fixes the error message from float(s) in the case where s contains only whitespace.
(cherry picked from commit 97e9cfa75a80b54a0630b7371f35e368a12749d1)
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Because of the way wrap_descr_get is written, the second argument
to __get__ methods implemented through the wrapper is always
optional.
(cherry picked from commit 4e08fbcfdfa57ea94091aabdd09413708e3fb2bf)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Classes ReferenceType, ProxyType and CallableProxyType have now correct
atrtributes __module__, __name__ and __qualname__.
It makes them (types, not instances) pickleable.
(cherry picked from commit 8352e322e87ba39c71e578b65ad8ae156ca3e0c7)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
(cherry picked from commit e12f34b6d8200508bd50cdc9c6c5637732ff56e7)
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
merge_freemem(): set keys to NULL do it's harmless to call this again.
(cherry picked from commit 9652900969df77b1ac245595419431df19296af9)
Co-authored-by: Tim Peters <tim.peters@gmail.com>
If the error handler returns position less or equal than the starting
position of non-encodable characters, most of built-in encoders didn't
properly re-size the output buffer. This led to out-of-bounds writes,
and segfaults.
(cherry picked from commit 18b07d773e09a2719e69aeaa925d5abb7ba0c068)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
The left-hand side expression of the if-check can be converted to a
constant by the compiler, but the addition on the right-hand side is
performed during runtime.
Move the addition from the right-hand side to the left-hand side by
turning it into a subtraction there. Since the values are known to
be large enough to not turn negative, this is a safe operation.
Prevents a very unlikely integer overflow on 32 bit systems.
Fixes GH-91421.
(cherry picked from commit 0859368335d470b9ff33fc53ed9a85ec2654b278)
Co-authored-by: Tobias Stoeckmann <stoeckmann@users.noreply.github.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
(cherry picked from commit 8be7c2bc5ad5e295f0f855bb31db412eef2c7c92)
Co-authored-by: Dave Goncalves <davegoncalves@gmail.com>
When an exception is created in a nested call to PyObject_GetAttr, any
external calls will override the context information of the
AttributeError that we have already placed in the most internal call.
This will cause the suggestions we create to nor work properly as the
attribute name and object that we will be using are the incorrect ones.
To avoid this, we need to check first if these attributes are already
set and bail out if that's the case..
(cherry picked from commit 3b3be05a164da43f201e35b6dafbc840993a4d18)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Rename the private undocumented float.__set_format__() method to
float.__setformat__() to fix a typo introduced in Python 3.7. The
method is only used by test_float.
The change enables again test_float tests on the float format which
were previously skipped because of the typo.
The typo was introduced in Python 3.7 by bpo-20185
in commit b5c51d3dd95bbfde533655fb86ac0f96f771ba7b.
(cherry picked from commit 7d03c8be5af2f1559dbc35b775b3116dfd63cfb6)
Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called.
(cherry picked from commit 4a66615ba736f84eadf9456bfd5d32a94cccf117)
Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>
Fix a race condition on setting a type __bases__ attribute: the
internal function add_subclass() now gets the
PyTypeObject.tp_subclasses member after calling PyWeakref_NewRef()
which can trigger a garbage collection which can indirectly modify
PyTypeObject.tp_subclasses.
(cherry picked from commit f1c6ae3270913e095d24ae13ecf96f5a32c8c503)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
