[3.13] gh-139516: Fix lambda colon start format spec in f-string in tokenizer (GH-139657)
(cherry picked from commit 539461d9ec8e5322ead638f7be733fd196aa6c79)
gh-137314: Fix incorrect treatment of format specs in raw fstrings (GH-137328)
(cherry picked from commit 0153d82a5ab0c6ac16c046bdd4438ea11b58d59d)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
gh-132769: Refactor possible read-out-of-bounds in `lexer.c` (GH-132770)
(cherry picked from commit ea8ec95cfadbf58a11ef8e41341254d982a1a479)
Co-authored-by: sobolevn <mail@sobolevn.me>
gh-129093: Fix f-string debug text sometimes getting cut off when expression contains `!` (GH-129159)
(cherry picked from commit 767cf708449fbf13826d379ecef64af97d779510)
Co-authored-by: Tomas R <tomas.roun8@gmail.com>
gh-123229: Fix valgrind warning by initializing the f-string buffers to 0 in the tokenizer (GH-123263)
(cherry picked from commit adc5190014efcf7b7a4c5dfc9998faa8345527ed)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* The lexer, which include the actual lexeme producing logic, goes into
the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
readline), go into the `tokenizer` directory and include logic for
creating a lexer instance and managing the buffer for different modes.
---------
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
