When debugging the fstring table, I found "UTF-8" to be the most common
interned strings in many benchmarks.
We have a fixed, limited number of these strings, so we might as well permanently
cache their fstrings.
to adopt strict shareable rule.
* (basically) shareable objects only refer shareable objects
* (exception) shareable objects can refere unshareable objects
but should not leak reference to unshareable objects to Ruby world
Previously we were looping over the enc_table, but when I added an
assertion the only thing that loop was doing is the equivalent of
ENC_TO_ENCINDEX(base). However we don't even need the index of base.
Instead we should be able to just use the badirectly.
Previously we would sometimes call enc_register_at several times in
order to update the encoding values after the base encoding may have
been loaded.
This updates enc_register_at to only be used via enc_register, when an
actual new encoding at a new index is being registered.
Other callers (which in all cases found the index by the name matching)
now call enc_load_from_base which is only responsibly for loading the
encoding from the base values.
Not all ractor-related encoding issues were fixed by 1afc07e815051e2f73493f055f2130cb642ba12a.
I found more by running my test-all branch with 3 ractors for each test.
Make sure VM lock is not held when calling `load_transcoder_entry`, as
that causes deadlock inside ractors. `String#encode` now works inside
ractors, among others.
Atomic load the rb_encoding_list
Without this, wbcheck would sometimes hit a missing write barrier.
Co-authored-by: John Hawthorn <john.hawthorn@shopify.com>
Hold VM lock when iterating over global_enc_table.names
This st_table can be inserted into at runtime when autoloading
encodings.
minor optimization when calling Encoding.list
Using `encoding->max_enc_len` as a way to check if the encoding
has been loaded isn't atomic, because it's not atomically set
last.
Intead we can use a dedicated atomic value inside the encoding table.
None of the datastructures involved in the require process are
safe to call on a secondary ractor, however when autoloading
encodings, we do so from the current ractor.
So all sorts of corruption can happen when using an autoloaded
encoding for the first time from a secondary ractor.
This fixes segfaults and errors of the type "Encoding not found" when
using encoding-related methods and internal encoding c functions across
ractors.
Example of a possible segfault in release mode or assertion error in debug mode:
```ruby
rs = []
100.times do
rs << Ractor.new do
"abc".force_encoding(Encoding.list.shuffle.first)
end
end
while rs.any?
r, obj = Ractor.select(*rs)
rs.delete(r)
end
```
While profiling msgpack-ruby I noticed a very substantial amout of time
spent in `rb_enc_associate_index`, called by `rb_utf8_str_new`.
On that benchmark, `rb_utf8_str_new` is 33% of the total runtime,
in big part because it cause GC to trigger often, but even then
`5.3%` of the total runtime is spent in `rb_enc_associate_index`
called by `rb_utf8_str_new`.
After closer inspection, it appears that it's performing a lot of
safety check we can assert we don't need, and other extra useless
operations, because strings are first created and filled as ASCII-8BIT
and then later reassociated to the desired encoding.
By directly allocating the string with the right encoding, it allow
to skip a lot of duplicated and useless operations.
After this change, the time spent in `rb_utf8_str_new` is down
to `28.4%` of total runtime, and most of that is GC.
[Bug #20598]
Just like [Bug #20595], Encoding#name_list and Encoding#aliases can have
their strings corrupted when Encoding.default_internal is set to nil.
Co-authored-by: Matthew Valentine-House <matt@eightbitraptor.com>
[Bug #20595]
enc_set_default_encoding will free the C string if the encoding is nil,
but the C string can be used by the encoding name string. This will cause
the encoding name string to be corrupted.
Consider the following code:
Encoding.default_internal = Encoding::ASCII_8BIT
names = Encoding.default_internal.names
p names
Encoding.default_internal = nil
p names
It outputs:
["ASCII-8BIT", "BINARY", "internal"]
["ASCII-8BIT", "BINARY", "\x00\x00\x00\x00\x00\x00\x00\x00"]
Co-authored-by: Matthew Valentine-House <matt@eightbitraptor.com>
[Feature #18576]
Since outright renaming `ASCII-8BIT` is deemed to backward incompatible,
the next best thing would be to only change its `#inspect`, particularly
in exception messages.
This `st_table` is used to both mark and pin classes
defined from the C API. But `vm->mark_object_ary` already
does both much more efficiently.
Currently a Ruby process starts with 252 rooted classes,
which uses `7224B` in an `st_table` or `2016B` in an `RArray`.
So a baseline of 5kB saved, but since `mark_object_ary` is
preallocated with `1024` slots but only use `405` of them,
it's a net `7kB` save.
`vm->mark_object_ary` is also being refactored.
Prior to this changes, `mark_object_ary` was a regular `RArray`, but
since this allows for references to be moved, it was marked a second
time from `rb_vm_mark()` to pin these objects.
This has the detrimental effect of marking these references on every
minors even though it's a mostly append only list.
But using a custom TypedData we can save from having to mark
all the references on minor GC runs.
Addtionally, immediate values are now ignored and not appended
to `vm->mark_object_ary` as it's just wasted space.
There is a memory leak in Encoding.default_external= and
Encoding.default_internal= because the duplicated name is not freed
when overwriting.
10.times do
1_000_000.times do
Encoding.default_internal = nil
end
puts `ps -o rss= -p #{$$}`
end
Before:
25664
41504
57360
73232
89168
105056
120944
136816
152720
168576
After:
9648
9648
9648
9680
9680
9680
9680
9680
9680
9680
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
Instead of relying on setting an unsetting ruby_verbose, which is
not thread-safe, restructure require_internal and load_lock to
accept a warn argument for whether to warn, and add
rb_require_internal_silent to require without warnings. Use
rb_require_internal_silent when loading encoding.
Note this does not modify ruby_debug and errinfo handling, those
remain thread-unsafe.
Also silent requires when loading transcoders.
rb_enc_from_index(index) doesn't need locking if index specify
ASCII/US_ASCCII/UTF-8.
rb_enc_from_index() is called frequently so it has impact.
user system total real
r_parallel/miniruby 174 0.000209 0.000000 5.559872 ( 1.811501)
r_parallel/master_mini 175 0.000238 0.000000 12.664707 ( 3.523641)
(repeat x1000 `s.split(/,/)` where s = '0,,' * 1000)
