81557 Commits

Author SHA1 Message Date
John Hawthorn
ad54fbf281 Add missing GVL hooks for M:N threads and ractors
[Bug #20019]

This fixes GVL instrumentation in three locations it was missing:
- Suspending when blocking on a Ractor
- Suspending when doing a coroutine transfer from an M:N thread
- Resuming after an M:N thread starts

Co-authored-by: Matthew Draper <matthew@trebex.net>
2023-12-02 10:06:07 -08:00
Burdette Lamar
715cf9b690
[DOC] More on What's Here (#9099)
* More on What's Here

* More on What's Here
2023-12-02 11:46:57 -06:00
Peter Zhu
d1691617d6 Pin instruction storage
The operands in each instruction needs to be pinned because if
auto-compaction runs in iseq_set_sequence, then the objects could exist
on the generated_iseq buffer, which would not be reference updated which
can lead to T_MOVED (and subsequently T_NONE) objects on the iseq.
2023-12-02 09:06:03 -05:00
Nobuyoshi Nakada
092a17e7bd
[DOC] Markup class name Time as code 2023-12-02 21:48:00 +09:00
Nobuyoshi Nakada
ae3e9f114c
[DOC] Include timezones document in timev.rb
So that it can be referred as a part of the document of `Time`.
2023-12-02 21:46:52 +09:00
Nobuyoshi Nakada
c9c1670f7e [ruby/rdoc] Markup punctuations need to be separated with a space
https://github.com/ruby/rdoc/commit/83f0149fc1
2023-12-02 12:40:12 +00:00
Gary Tou
4ee1f0fb5d [ruby/irb] Implement history command
(https://github.com/ruby/irb/pull/761)

* Implement `history` command

Lists IRB input history with indices. Also aliased as `hist`.

* Add tests for `history` command

* Address feedback: `puts` with multiple arguments instead of `join`ing

* Address feedback: Handle nil from splitting an empty input string

* Refactor line truncation

* Add `-g` grep option to `history` command

* Add `history` command to README

* Remove unused `*args` parameter

* Allow spaces to be included in grep

* Allow `/` to be included in grep regex

* Handle `input` being an empty string

* Exclude "#{index}: " from matching the grep regex

* Add new line after joining

https://github.com/ruby/irb/commit/3f9eacbfa9
2023-12-02 04:32:04 +00:00
yui-knk
9f6c6f88c3 Lrama v0.5.11 2023-12-02 08:58:32 +09:00
eileencodes
818813c2bd Implement paren node for defined?
Implements and adds a test for passing a parentheses node to `defined?`.
2023-12-01 14:56:25 -08:00
Ryan Davis
6ecc1ca9b1
[DOC] Update ARGF.readlines documentation to match/reference IO.readlines. 2023-12-01 22:51:08 +00:00
Peter Zhu
0aed37b973 Make expandarray compaction safe
The expandarray instruction can allocate an array, which can trigger
a GC compaction. However, since it does not increment the sp until the
end of the instruction, the objects it places on the stack are not
marked or reference updated by the GC, which can cause the objects to
move which leaves broken or incorrect objects on the stack.

This commit changes the instruction to be handles_sp so the sp is
incremented inside of the instruction right after the object is written
on the stack.
2023-12-01 17:13:56 -05:00
Kevin Newton
492c82cb41 [ruby/prism] Prism.parse_success?(source)
A lot of tools use Ripper/RubyVM::AbstractSyntaxTree to determine
if a source is valid. These tools both create an AST instead of
providing an API that will return a boolean only.

This new API only creates the C structs, but doesn't bother
reifying them into Ruby/the serialization API. Instead it only
returns true/false, which is significantly more efficient.

https://github.com/ruby/prism/commit/7014740118
2023-12-01 20:53:34 +00:00
Peter Zhu
b77551adee Remove unneeded local variables 2023-12-01 15:21:01 -05:00
Haldun Bayhantopcu
562d949e02 [ruby/prism] Fix parsing heredoc ends
https://github.com/ruby/prism/commit/aa8c702271
2023-12-01 20:10:58 +00:00
eileencodes
39238888bc Implements missing literals for defined?
This PR implements the following literals:

- String
- Symbols
- Integers
- Floats
- Regexs
- Ranges
- Lambdas
- Hashes

and tests for them.
2023-12-01 12:05:22 -08:00
Peter Zhu
ee0eca191f Make String#undump compaction safe 2023-12-01 15:04:31 -05:00
Peter Zhu
80ea7fbad8 Pin embedded shared strings
Embedded shared strings cannot be moved because strings point into the
slot of the shared string. There may be code using the RSTRING_PTR on
the stack, which would pin the string but not pin the shared string,
causing it to move.
2023-12-01 15:04:31 -05:00
Samuel Giddins
0ed55bf097 [rubygems/rubygems] Reduce array allocations when loading definition
The same array was being re-created in a loop (as well as the `generic_local_platform`), which is avoidable by hoisting it to a frozen array created once

https://github.com/rubygems/rubygems/commit/009a3c6d0d
2023-12-01 19:50:03 +00:00
Nobuyoshi Nakada
a607d62d8c [Bug #20033] Dynamic regexp should not assign captures 2023-12-02 03:57:41 +09:00
Kevin Newton
ec83bd7356 [ruby/prism] Provide heredoc? queries
https://github.com/ruby/prism/commit/e148e8fe6a
2023-12-01 18:46:52 +00:00
Jemma Issroff
2a8d9c59ff [PRISM] Account for RescueNodes with no statements
We need a PUTNIL if a RescueNode has no statements.
2023-12-01 13:23:23 -05:00
Jemma Issroff
d6584a0201 [PRISM] Fix behavior of BlockParameters with only one parameter
This commit sets the ambiguous param flag if there is only one
parameter on a block node. It also fixes a small bug with a trailing
comma on params.
2023-12-01 13:07:59 -05:00
hogelog
ef466ac931 [ruby/irb] Scrub past history input before split
(https://github.com/ruby/irb/pull/795)

* Scrub past history input before split

* Don't rewrite ENV["LANG"]

https://github.com/ruby/irb/commit/0f344f66d9
2023-12-01 18:04:03 +00:00
Eric Mueller
73440e1ef2 [rubygems/rubygems] Many major_deprecations supply :removed_message
Generally the removed message is very similar, but often it needs to
specify that the feature has "been removed" instead of "will be
removed", or "been deprecated". And a few chunks of text needed more
substantial updates. And a number of them seemed to have been carefully
crafted to make sense in either context, so I left those alone.

https://github.com/rubygems/rubygems/commit/8d42cf9104
2023-12-01 17:52:38 +00:00
Eric Mueller
079dfa1812 [rubygems/rubygems] major_deprecation accepts :removed_message
If supplied, it uses that in place of the message for the case where the
deprecation version is already past.

https://github.com/rubygems/rubygems/commit/1deb73663c
2023-12-01 17:52:37 +00:00
Nobuyoshi Nakada
e5e1f9813e [Bug #19838] Flush delayed token nonconsecutive with the next token 2023-12-02 02:41:39 +09:00
Nobuyoshi Nakada
add0ab07e8 Extract end_with_newline_p 2023-12-02 02:41:39 +09:00
Nobuyoshi Nakada
6e2b10d456 Prefer proper enum over bare int 2023-12-02 02:41:39 +09:00
Jemma Issroff
9d20909342
[PRISM] Clean up trailing comment 2023-12-01 12:20:16 -05:00
Jemma Issroff
d224618bea [PRISM] Restructure parameters
Prior to this commit, we weren't accounting for hidden variables
on the locals table, so we would have inconsistencies on the stack.
This commit fixes params, and introduces a hidden_variable_count
on the scope, both of which fix parameters.
2023-12-01 12:14:54 -05:00
Kevin Newton
5150ba0dbe [prism] Update to latest numbered parameters 2023-12-01 12:03:09 -05:00
Kevin Newton
cdb74d74af [ruby/prism] Change numbered parameters
Previously numbered parameters were a field on blocks and lambdas
that indicated the maximum number of numbered parameters in either
the block or lambda, respectively. However they also had a
parameters field that would always be nil in these cases.

This changes it so that we introduce a NumberedParametersNode that
goes in place of parameters, which has a single uint8_t maximum
field on it. That field contains the maximum numbered parameter in
either the block or lambda.

As a part of the PR, I'm introducing a new UInt8Field type that
can be used on nodes, which is just to make it a little more
explicit what the maximum values can be (the maximum is actually 9,
since it only goes up to _9). Plus we can do a couple of nice
things in serialization like just read a single byte.

https://github.com/ruby/prism/commit/2d87303903
2023-12-01 12:03:09 -05:00
Matt Valentine-House
90d9c20a0c [PRISM] Compile RescueNode 2023-12-01 16:40:25 +00:00
Kevin Newton
8f3310dc75
Remove unused files in prism 2023-12-01 10:23:07 -05:00
Kevin Newton
64f33aea2b [ruby/prism] Update snapshots
https://github.com/ruby/prism/commit/f4c80c67dc
2023-12-01 15:18:19 +00:00
Martin Emde
cbe57caa24 [ruby/prism] Fix comments after HEREDOCs again.
The problem was deeper than just looking back a single token.
You can push the heredoc_end token way back into the list.
We need to save the last location of a heredoc end to see if
it's the last token in the file.

Fixes https://github.com/ruby/prism/pull/1954

https://github.com/ruby/prism/commit/91dfd4eecd
2023-12-01 15:17:20 +00:00
TSUYUSATO Kitsune
ffeec108cf [ruby/prism] Add missing context names to the switch in debug_context
Now, `PM_DEBUG_LOGGING` can be available.

https://github.com/ruby/prism/commit/569ffacb5f
2023-12-01 15:15:28 +00:00
TSUYUSATO Kitsune
417d700406 [ruby/prism] Improve to handle unterminated strings
Fix https://github.com/ruby/prism/pull/1946

This fixes to set an error position for unterminated strings to the
opening delimiters. Previously, the error position was set to the end
of the delimiter.

The same fix applies to other string-like literals.

Additionally, this fixes https://github.com/ruby/prism/pull/1946; that is, it adds the last part of the
string even though the string literal does not terminate.

https://github.com/ruby/prism/commit/c1240baafd
2023-12-01 15:15:01 +00:00
TSUYUSATO Kitsune
0e599336a2 [ruby/prism] Fix not_provided comment
I don't know when it is changed, but the `not_provided` signature is
changed.

https://github.com/ruby/prism/commit/0255cc307d
2023-12-01 15:13:45 +00:00
Matt Valentine-House
3b21932d14 [PRISM] Use depth_offset not transparent scopes for FOR 2023-12-01 15:04:13 +00:00
Matt Valentine-House
ce5f5ca1d6 [PRISM] Remove transparent scope nodes 2023-12-01 15:04:13 +00:00
Peter Zhu
43ef0da0fb Add assertions for shape cache grandchild nodes 2023-12-01 09:56:32 -05:00
Hiroshi SHIBATA
64c4bbb86c [ruby/prism] Add test/prism/snapshots/heredoc_with_comment.txt
https://github.com/ruby/prism/commit/97b296a0f7
2023-12-01 11:33:26 +00:00
git
e005c51732 Update default gems list at 7d142c08cb5b65e4ba110b116c41a8 [ci skip] 2023-12-01 07:48:10 +00:00
Hiroshi SHIBATA
7d142c08cb
lib/helper only needs on flori/json repo 2023-12-01 16:47:06 +09:00
Hiroshi SHIBATA
86045fca24
Manually merged from flori/json
> https://github.com/flori/json/pull/525
  > Rename escape_slash in script_safe and also escape E+2028 and E+2029

  Co-authored-by: Jean Boussier <jean.boussier@gmail.com>

  > https://github.com/flori/json/pull/454
  > Remove unnecessary initialization of create_id in JSON.parse()

  Co-authored-by: Watson <watson1978@gmail.com>
2023-12-01 16:47:06 +09:00
Jean Boussier
0dfeb17296
Rename escape_slash in script_safe and also escape E+2028 and E+2029
It is rather common to directly interpolate JSON string inside
<script> tags in HTML as to provide configuration or parameters to a
script.

However this may lead to XSS vulnerabilities, to prevent that 3
characters need to be escaped:

  - `/` (forward slash)
  - `U+2028` (LINE SEPARATOR)
  - `U+2029` (PARAGRAPH SEPARATOR)

The forward slash need to be escaped to prevent closing the script
tag early, and the other two are valid JSON but invalid Javascript
and can be used to break JS parsing.

Given that the intent of escaping forward slash is the same than escaping
U+2028 and U+2029, I chos to rename and repurpose the existing `escape_slash`
option.
2023-12-01 16:47:06 +09:00
Jean Boussier
698cb84062
Use ruby_xfree to free buffers
They are allocated with ruby_xmalloc, they should be freed with
ruby_xfree.
2023-12-01 16:47:06 +09:00
Hiroshi SHIBATA
32f289d118
[flori/json] Bump up 2.7.0
https://github.com/flori/json/commit/ca546128f2
2023-12-01 16:47:06 +09:00
John Hawthorn
4b770527c2
[flori/json] Fix "unexpected token" offset for Infinity
Previously in the JSON::Ext parser, when we encountered an "Infinity"
token (and weren't allowing NaN/Infinity) we would try to display the
"unexpected token" at the character before.

https://github.com/flori/json/commit/42ac170712
2023-12-01 16:47:06 +09:00