- Fix https://github.com/ruby/rubygems/pull/9186
- ### Problem
Running `bundle pristine` in a Gemfile where there is many git gem
pointing to the same repository will result in a error
"Another git process seems to be running in this repository".
### Context
This error is a regression since https://github.com/ruby/rubygems/commit/a555fd6ccd17
where `bundle pristine` now runs in parallel which could lead
to running simultaneous git operations in the same repository.
### Solution
When Bundler pristine a git gem it does a `git reset --hard` without
specifying a path.
This means the whole repository will be reset. In this case, we can
leverage that by just pristining one gem per unique git sources.
This is also more efficient.
https://github.com/ruby/rubygems/commit/710ba514a8
When running a fresh `bundle install` with gems that contains
executables, Bundler will generate binstubs but soon after will remove
them. This is a regression introduced in https://github.com/ruby/rubygems/commit/573ffad3ea4a.
This results in doing `bundle install && bundle exec foo` to raise an
error saying `foo` couldn't be found.
This issue only appears if `BUNDLE_CLEAN` is set.
At the end of the installation process, when Bundler has installed
gems and generated binstubs, it runs the cleanup.
1. It [detects](4f8aa3b40c/bundler/lib/bundler/runtime.rb (L182)) the executable for the current specs
2. Any existing executables not detected is then removed 4f8aa3b40c/bundler/lib/bundler/runtime.rb (L194).
The issue being that 1. now returns an empty array where as it should
return the executables of the gems from the current bundle.
The problem is in https://github.com/ruby/rubygems/commit/573ffad3ea4a where we
removed the `executables` method from the `EndpointSpecification`.
When Bundler reads the lockfile, it creates a `EndpointSpecification`
object for each spec. At this point, the EndpointSpecification doesn't
know about the `executables` of a gem. Once Bundler fetches the
`gemspec` from the remote, it swaps the the "spec" with the real one
and from here knows what executables the gem has.
Reintroduce the `executables` method and the `bindir` in the
EndpointSpecification class. From what I'm seeing, the removal
of those wasn't needed to resolve the issue where Bundler remembers
CLI flags. This is probably an oversight.
https://github.com/ruby/rubygems/commit/b47f6b0247
Add fallback to `require` when `require_relative` fails to load native
extensions. This addresses an issue on RHEL-based Linux distributions
where Ruby scripts and built native extension shared libraries are
installed in separate directories.
https://github.com/ruby/rubygems/commit/68599bd107
I organized all examples the followings:
```
Total test time: 2468.41 seconds
Total files: 168
Group A: 42 files, 617.08 seconds
Group B: 42 files, 617.05 seconds
Group C: 42 files, 617.14 seconds
Group D: 42 files, 617.14 seconds
```
https://github.com/ruby/rubygems/commit/94d41e6c7c
- ### TL;DR
Bundler is heavily limited by the connection pool which manages a
single connection. By increasing the number of connection, we can
drastiscally speed up the installation process when many gems need
to be downloaded and installed.
### Benchmark
There are various factors that are hard to control such as
compilation time and network speed but after dozens of tests I
can consistently get aroud 70% speed increase when downloading and
installing 472 gems, most having no native extensions (on purpose).
```
# Before
bundle install 28.60s user 12.70s system 179% cpu 23.014 total
# After
bundle install 30.09s user 15.90s system 281% cpu 16.317 total
```
You can find on this gist how this was benchmarked and the Gemfile
used https://gist.github.com/Edouard-chin/c8e39148c0cdf324dae827716fbe24a0
### Context
A while ago in #869, Aaron introduced a connection pool which
greatly improved Bundler speed. It was noted in the PR description
that managing one connection was already good enough and it wasn't
clear whether we needed more connections. Aaron also had the
intuition that we may need to increase the pool for downloading
gems and he was right.
> We need to study how RubyGems uses connections and make a decision
> based on request usage (e.g. only use one connection for many small
> requests like bundler API, and maybe many connections for
> downloading gems)
When bundler downloads and installs gem in parallel 4f85e02fdd/bundler/lib/bundler/installer/parallel_installer.rb (L128)
most threads have to wait for the only connection in the pool to be
available which is not efficient.
### Solution
This commit modifies the pool size for the fetcher that Bundler
uses. RubyGems fetcher will continue to use a single connection.
The bundler fetcher is used in 2 places.
1. When downloading gems 4f85e02fdd/bundler/lib/bundler/source/rubygems.rb (L481-L484)
2. When grabing the index (not the compact index) using the
`bundle install --full-index` flag.
4f85e02fdd/bundler/lib/bundler/fetcher/index.rb (L9)
Having more connections in 2) is not any useful but tweaking the
size based on where the fetcher is used is a bit tricky so I opted
to modify it at the class level.
I fiddle with the pool size and found that 5 seems to be the sweet
spot at least for my environment.
https://github.com/ruby/rubygems/commit/6063fd9963
[Bug #21718]
Otherwise objects that don't define it, but define a fairly liberal
`method_missing` method will run into errors that are hard to understand:
```ruby
class Foo
def method_missing(name, ...)
name
end
end
p Foo.new.inspect
```
```
'Kernel#inspect': wrong argument type Symbol (expected Array) (TypeError)
from ../test.rb:7:in '<main>'
```
This allows for specifying the lockfile to read and write. It mirrors
the --gemfile option, and has higher priority than the lockfile
method in the Gemfile. It also mirrors the bundle lock --lockfile
option.
When the --lockfile option is used, it is applied twice. First, before
the Gemfile is read, to specify the lockfile to operate on, and again
after the Gemfile is read, so that if the Gemfile has a lockfile
method that overrides the defintion's lockfile, the --lockfile option
still has higher precedence.
https://github.com/ruby/rubygems/commit/17acdd4a89
This specifies the lockfile location. This allows for easy support
of different lockfiles per Ruby version or platform.
https://github.com/ruby/rubygems/commit/b54d65bc0a
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
This allows for the same behavior as including `lockfile false`
in the Gemfile. This allows you to get the behavior without
modifying the Gemfile, which is useful if you do not control the
Gemfile.
This is similar to the --no-lock option already supported by
`gem install -g Gemfile`.
https://github.com/ruby/rubygems/commit/6c94623881
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
This allows you to specify the lockfile to use. This is useful if
you want to use different lockfiles for different ruby versions or
platforms. You can also skip writing the lockfile by using a false
value.
https://github.com/ruby/rubygems/commit/2896aa3fc2
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
- With the logger change that is now threadsafe, such code no longer
behaves the same:
```ruby
Bundler.ui.silence do
Bundler.ui.level = 'info'
Bundler.ui.info("foo")
# This used to output something. Now it doesn't.
end
```
IMHO this is the right behaviour since we are in a silence block,
changing the level should have no effect. And fortunately it seems
that we only need to change this spec.
The call to `Bundler.ui.silence` is done in a `around` block
4a13684f07/bundler/spec/spec_helper.rb (L119)https://github.com/ruby/rubygems/commit/e716adb6c9
- The Logger is not thread safe when calling `with_level`.
This now becomes problematic because we are using multiple
threads during the resolution phase in order to fetch git gems.
https://github.com/ruby/rubygems/commit/380653ae74
This is a first pass to improve the way errors are handled and raised in
bundler's tests. The goal is to clean up tests and modernize them -
these were some obvious areas that could be cleaned up.
- Instead of raising "ZOMG" in the load error tests, it now tests for
the actual error and gem raising.
- Improve error messages where applicable.
- All errors raise a specific error class, rather than falling back to a
default and just setting a message.
- Removed arguments and `bundle_dir` option from `TheBundle` class as it wasn't
actually used so therefore we don't need to raise an error for extra
arguments.
- Removed error from `BundlerBuilder`, as it won't work if it's not
`bundler`, also it never uses `name`. The only reaon `name` is passed
in is because of metaprogramming on loading the right builder. I
think that should eventually be refactored.
- Replaced and removed `update_repo3` and `update_repo4` in favor of
just `build_repo3` and `build_repo4`. Rather than tell someone writing
tests to use a different method, automatically use the right method.
https://github.com/ruby/rubygems/commit/68c39c8451
- In 31d67ecc05
we enforced the new behaviour where running `bundle` no longer
installs gems but displays the help.
Users now have a way to configure their preferred default command using
the `BUNDLE_DEFAULT_CLI_COMMAND` flag.
With the preview of Ruby 4.0 now being released, some people will
start to see this new change.
The problem is that the previous behaviour had existed for like an
eternity and we didn't warn users about this change in advance.
I'd like to provide a deprecation/warning cycle because this is
confusing users already and this breaks various CI setup that now
needs to be changed immediately.
https://github.com/ruby/rubygems/commit/e415480ac5
(https://github.com/ruby/rubygems/pull/8183)
* Add new gem templates
* Add `--ext=go` in `bundle gem`
* Add setup-go to .github/workflows/main.yml
* Embed go version in go.mod
* Use go in bundler CI
* Add example method to template
* Install Go in .circleci/config.yml
* Install Go in .gitlab-ci.yml
* Allow hard tabs in go template
* Run `rake update_manifest`
* Fix test
* Move go_gem to gemspec
Respect to 9b0ec80
* nits: ⛳
* includes valid module name in go.mod
* generate header file
* Run `go mod tidy` to create `go.sum`
* Check if `go.sum` is generated only when Go is installed
To avoid test failure in environments where Go is not installed
* Run CI
* Workaround for hung up
c.f. https://github.com/rubygems/rubygems/actions/runs/11639408044/job/32415545422
* Write man for --ext=go
* Re-generate man with `./bin/rake man:build`
* pinning 📌
* Update with `./bin/rake man:build`
* nits: Extract to method
* nits: Use `sys_exec` instead of `system`
* Clean go module cache after test
Workaround following error
```
1) bundle gem gem naming with underscore --ext parameter set with go includes go_gem extension in extconf.rb
Failure/Error: FileUtils.rm_r(dir)
Errno::EACCES:
Permission denied @ apply2files - /home/runner/work/rubygems/rubygems/bundler/tmp/2.2/home/go/pkg/mod/gopkg.in/yaml.v3@v3.0.1/decode_test.go
# ./spec/support/helpers.rb:37:in `block in reset!'
# ./spec/support/helpers.rb:21:in `each'
# ./spec/support/helpers.rb:21:in `reset!'
# ./spec/spec_helper.rb:130:in `block (2 levels) in <top (required)>'
# /home/runner/work/rubygems/rubygems/lib/rubygems.rb:303:in `load'
# /home/runner/work/rubygems/rubygems/lib/rubygems.rb:303:in `activate_and_load_bin_path'
```
Files installed with `go get` have permissions set to 444
ref. https://github.com/golang/go/issues/35615
```
$ ls -l /home/runner/work/rubygems/rubygems/bundler/tmp/2.2/home/go/pkg/mod/gopkg.in/yaml.v3@v3.0.1/decode_test.go
-r--r--r-- 1 runner runner 42320 Nov 15 06:38 /home/runner/work/rubygems/rubygems/bundler/tmp/2.2/home/go/pkg/mod/gopkg.in/yaml.v3@v3.0.1/decode_test.go
```
So they cannot be deleted by `FileUtils.rm_r`.
Therefore, this is necessary to execute `go clean -modcache` separately from `FileUtils.rm_r` to circumvent it.
* Remove needless changes
ref. https://github.com/ruby/rubygems/pull/8183#discussion_r2532902051
* ci: setup-go is needless
* Don't run go command in `bundle gem`
ref. https://github.com/ruby/rubygems/pull/8183#discussion_r2532765470
* Revert unrelated date changes
---------
https://github.com/ruby/rubygems/commit/260d7d60b3
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>
