I stumbled across a bundler bug that had me scratching my head for
awhile, because I hadn't experienced it before.
In some cases when changing the source in a gemfile from a
`Source::Gemspec` to either a `Source::Path` or `Source::Git` only the
parent gem will have it's gem replaced and updated and the child
components will retain the original version. This only happens if the gem
version of the `Source::Gemspec` and `Source::Git` are the same. It also
requires another gem to share a dependency with the one being updated.
For example if I have the following gemfile:
```
gem "rails", "~> 8.1.1"
gem "propshaft"
```
Rails has a component called `actionpack` which `propshaft` depends on.
If I change `rails` to point at a git source (or path source), only the
path for `rails` gets updated:
```
gem "rails", github: "rails/rails", branch: "8-1-stable"
gem "propshaft"
```
Because `actionpack` is a dependency of `propshaft`, it will remain in
the rubygems source in the lock file WHILE the other gems are correctly
pointing to the git source.
Gemfile.lock:
```
GIT
remote: https://github.com/rails/rails.git
revision: https://github.com/ruby/rubygems/commit/9439f463e0ef
branch: 8-1-stable
specs:
actioncable (8.1.1)
...
actionmailbox (8.1.1)
...
actionmailer (8.1.1)
...
actiontext (8.1.1)
...
activejob (8.1.1)
...
activemodel (8.1.1)
...
activerecord (8.1.1)
...
activestorage (8.1.1)
...
rails (8.1.1)
...
railties (8.1.1)
...
GEM
remote: https://rubygems.org/
specs:
action_text-trix (2.1.15)
railties
actionpack (8.1.1) <===== incorrectly left in Rubygems source
...
```
The gemfile will contain `actionpack` in the rubygems source, but will
be missing in the git source so the path will be incorrect. A bundle
show on Rails will point to the correct place:
```
$ bundle show rails
/Users/eileencodes/.gem/ruby/3.4.4/bundler/gems/rails-9439f463e0ef
```
but a bundle show on actionpack will be incorrect:
```
$ bundle show actionpack
/Users/eileencodes/.gem/ruby/3.4.4/gems/actionpack-8.1.1
```
This bug requires the following to reproduce:
1) A gem like Rails that contains components that are released as their
own standalone gem is added to the gemfile pointing to rubygems
2) A second gem is added that depends on one of the gems in the first
gem (like propshaft does on actionpack)
3) The Rails gem is updated to use a git source, pointing to the same
version that is being used by rubygems (ie 8.1.1)
4) `bundle` will only update the path for Rails component gems if no
other gem depends on it.
This incorrectly leaves Rails (or any gem like it) using two different
codepaths / gem source code.
https://github.com/ruby/rubygems/commit/dff76ba4f6
Previously: In #9218 a reproduction is shared where running `bundle clean` using a binstub (`bin/bundle`) results in bundler removing itself. This results in Ruby falling back to its default bundler version. This behavior seems to be present for as long as there has been a default version of bundler (Ruby 2.6+).
Now: Bundler will explicitly add its current version number to the specs to be preserved. This prevents `bundle clean` from removing the current bundler version.
close https://github.com/ruby/rubygems/pull/9218https://github.com/ruby/rubygems/commit/e3f0167ae4
This change updates `write_binary` to use a new class,
`AtomicFileWriter.open` to write the gem's files. This implementation
is borrowed from Active Support's [`atomic_write`](https://github.com/rails/rails/blob/main/activesupport/lib/active_support/core_ext/file/atomic.rb).
Atomic write will write the files to a temporary file and then once
created, sets permissions and renames the file. If the file is corrupted
- ie on failed download, an error occurs, or for some other reason, the
real file will not be created. The changes made here make `verify_gz`
obsolete, we don't need to verify it if we have successfully created the
file atomically. If it exists, it is not corrupt. If it is corrupt, the
file won't exist on disk.
While writing tests for this functionality I replaced the
`RemoteFetcher` stub with `FakeFetcher` except for where we really do
need to overwrite the `RemoteFetcher`. The new test implementation is much
clearer on what it's trying to accomplish versus the prior test
implementation.
https://github.com/ruby/rubygems/commit/0cd4b54291
- ### Problem
Since https://github.com/ruby/rubygems/pull/9131, we are now
compiling make rules simultaneously. The number of jobs
is equal to the number of processors.
This may be problematic for some users as they want to control
this value.
### Solution
The number of jobs passed to `make` will now be equal to the
`BUNDLE_JOBS` value.
### Side note
It's also worth to note that since Bundler installs gems in
parallel, we may end up running multiple `make -j<JOB>` in parallel
which would cause exhaust the number of processors we have.
This problem can be fixed by implementing a GNU jobserver, which I
plan to do. But I felt that this would be too much change in one PR.
https://github.com/ruby/rubygems/commit/d51995deb9
The previous regex didn't properly match quoted strings
it would capture the opening quote as part of the version
if quotes were mismatched.
This change properly parses double-quoted, single-quoted,
and unquoted version strings separately.
https://github.com/ruby/rubygems/commit/81e48c8185
- Fix https://github.com/ruby/rubygems/pull/9186
- ### Problem
Running `bundle pristine` in a Gemfile where there is many git gem
pointing to the same repository will result in a error
"Another git process seems to be running in this repository".
### Context
This error is a regression since https://github.com/ruby/rubygems/commit/a555fd6ccd17
where `bundle pristine` now runs in parallel which could lead
to running simultaneous git operations in the same repository.
### Solution
When Bundler pristine a git gem it does a `git reset --hard` without
specifying a path.
This means the whole repository will be reset. In this case, we can
leverage that by just pristining one gem per unique git sources.
This is also more efficient.
https://github.com/ruby/rubygems/commit/710ba514a8
This change updates `write_binary` to use a new class,
`AtomicFileWriter.open` to write the gem's files. This implementation
is borrowed from Active Support's [`atomic_write`](https://github.com/rails/rails/blob/main/activesupport/lib/active_support/core_ext/file/atomic.rb).
Atomic write will write the files to a temporary file and then once
created, sets permissions and renames the file. If the file is corrupted
- ie on failed download, an error occurs, or for some other reason, the
real file will not be created. The changes made here make `verify_gz`
obsolete, we don't need to verify it if we have successfully created the
file atomically. If it exists, it is not corrupt. If it is corrupt, the
file won't exist on disk.
While writing tests for this functionality I replaced the
`RemoteFetcher` stub with `FakeFetcher` except for where we really do
need to overwrite the `RemoteFetcher`. The new test implementation is much
clearer on what it's trying to accomplish versus the prior test
implementation.
https://github.com/ruby/rubygems/commit/0cd4b54291
When running a fresh `bundle install` with gems that contains
executables, Bundler will generate binstubs but soon after will remove
them. This is a regression introduced in https://github.com/ruby/rubygems/commit/573ffad3ea4a.
This results in doing `bundle install && bundle exec foo` to raise an
error saying `foo` couldn't be found.
This issue only appears if `BUNDLE_CLEAN` is set.
At the end of the installation process, when Bundler has installed
gems and generated binstubs, it runs the cleanup.
1. It [detects](4f8aa3b40c/bundler/lib/bundler/runtime.rb (L182)) the executable for the current specs
2. Any existing executables not detected is then removed 4f8aa3b40c/bundler/lib/bundler/runtime.rb (L194).
The issue being that 1. now returns an empty array where as it should
return the executables of the gems from the current bundle.
The problem is in https://github.com/ruby/rubygems/commit/573ffad3ea4a where we
removed the `executables` method from the `EndpointSpecification`.
When Bundler reads the lockfile, it creates a `EndpointSpecification`
object for each spec. At this point, the EndpointSpecification doesn't
know about the `executables` of a gem. Once Bundler fetches the
`gemspec` from the remote, it swaps the the "spec" with the real one
and from here knows what executables the gem has.
Reintroduce the `executables` method and the `bindir` in the
EndpointSpecification class. From what I'm seeing, the removal
of those wasn't needed to resolve the issue where Bundler remembers
CLI flags. This is probably an oversight.
https://github.com/ruby/rubygems/commit/b47f6b0247
* It looked like:
In a future version of Bundler, running `bundle` without argument will no longer run `bundle install`.
Instead, the `cli_help` command will be displayed. Please use `bundle install` explicitly for scripts like CI/CD.
You can use the future behavior now with `bundle config set default_cli_command cli_help --global`,
or you can continue to use the current behavior with `bundle config set default_cli_command install --global`.
This message will be removed after a default_cli_command value is set.
Bundler version 4.0.0 (2025-12-08 commit https://github.com/ruby/rubygems/commit/9b4819ae18)
* And now looks like:
In a future version of Bundler, running `bundle` without argument will no longer run `bundle install`.
Instead, the `cli_help` command will be displayed. Please use `bundle install` explicitly for scripts like CI/CD.
You can use the future behavior now with `bundle config set default_cli_command cli_help --global`,
or you can continue to use the current behavior with `bundle config set default_cli_command install --global`.
This message will be removed after a default_cli_command value is set.
Bundler version 4.0.0 (2025-12-08 commit https://github.com/ruby/rubygems/commit/9b4819ae18)
https://github.com/ruby/rubygems/commit/979dada8f3
Add fallback to `require` when `require_relative` fails to load native
extensions. This addresses an issue on RHEL-based Linux distributions
where Ruby scripts and built native extension shared libraries are
installed in separate directories.
https://github.com/ruby/rubygems/commit/68599bd107
- ### TL;DR
Bundler is heavily limited by the connection pool which manages a
single connection. By increasing the number of connection, we can
drastiscally speed up the installation process when many gems need
to be downloaded and installed.
### Benchmark
There are various factors that are hard to control such as
compilation time and network speed but after dozens of tests I
can consistently get aroud 70% speed increase when downloading and
installing 472 gems, most having no native extensions (on purpose).
```
# Before
bundle install 28.60s user 12.70s system 179% cpu 23.014 total
# After
bundle install 30.09s user 15.90s system 281% cpu 16.317 total
```
You can find on this gist how this was benchmarked and the Gemfile
used https://gist.github.com/Edouard-chin/c8e39148c0cdf324dae827716fbe24a0
### Context
A while ago in #869, Aaron introduced a connection pool which
greatly improved Bundler speed. It was noted in the PR description
that managing one connection was already good enough and it wasn't
clear whether we needed more connections. Aaron also had the
intuition that we may need to increase the pool for downloading
gems and he was right.
> We need to study how RubyGems uses connections and make a decision
> based on request usage (e.g. only use one connection for many small
> requests like bundler API, and maybe many connections for
> downloading gems)
When bundler downloads and installs gem in parallel 4f85e02fdd/bundler/lib/bundler/installer/parallel_installer.rb (L128)
most threads have to wait for the only connection in the pool to be
available which is not efficient.
### Solution
This commit modifies the pool size for the fetcher that Bundler
uses. RubyGems fetcher will continue to use a single connection.
The bundler fetcher is used in 2 places.
1. When downloading gems 4f85e02fdd/bundler/lib/bundler/source/rubygems.rb (L481-L484)
2. When grabing the index (not the compact index) using the
`bundle install --full-index` flag.
4f85e02fdd/bundler/lib/bundler/fetcher/index.rb (L9)
Having more connections in 2) is not any useful but tweaking the
size based on where the fetcher is used is a bit tricky so I opted
to modify it at the class level.
I fiddle with the pool size and found that 5 seems to be the sweet
spot at least for my environment.
https://github.com/ruby/rubygems/commit/6063fd9963
This allows for specifying the lockfile to read and write. It mirrors
the --gemfile option, and has higher priority than the lockfile
method in the Gemfile. It also mirrors the bundle lock --lockfile
option.
When the --lockfile option is used, it is applied twice. First, before
the Gemfile is read, to specify the lockfile to operate on, and again
after the Gemfile is read, so that if the Gemfile has a lockfile
method that overrides the defintion's lockfile, the --lockfile option
still has higher precedence.
https://github.com/ruby/rubygems/commit/17acdd4a89
(https://github.com/ruby/rubygems/pull/9095)
* Rescue when deleting a non-existent cached gem file
When a gem was in the cache, but another process deletes it first, this
delete command fails.
To work around this, I'm rescuing from Errno::ENOENT and swalling the
error. The file is gone, and we can move on.
* Apply suggestion from @kou
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
---------
https://github.com/ruby/rubygems/commit/b30bcbc648
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
This specifies the lockfile location. This allows for easy support
of different lockfiles per Ruby version or platform.
https://github.com/ruby/rubygems/commit/b54d65bc0a
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
This allows for the same behavior as including `lockfile false`
in the Gemfile. This allows you to get the behavior without
modifying the Gemfile, which is useful if you do not control the
Gemfile.
This is similar to the --no-lock option already supported by
`gem install -g Gemfile`.
https://github.com/ruby/rubygems/commit/6c94623881
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
This allows you to specify the lockfile to use. This is useful if
you want to use different lockfiles for different ruby versions or
platforms. You can also skip writing the lockfile by using a false
value.
https://github.com/ruby/rubygems/commit/2896aa3fc2
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
- The Logger is not thread safe when calling `with_level`.
This now becomes problematic because we are using multiple
threads during the resolution phase in order to fetch git gems.
https://github.com/ruby/rubygems/commit/380653ae74
- ### Problem
When you have a Gemfile that contains git gems, each repository will
be fetched one by one. This is extremelly slow.
A simple Gemfile with 5 git gems (small repositories) can take up
to 10 seconds just to fetch the repos.
We can speed this up by running multiple git process and fetching
repositories silmutaneously.
### Solution
The repositories are fetched in Bundler when `Source::Git#specs` is
called.
The problem is that `source.specs` is called in various places
depending on Gemfile.
I think the issue is that calling `source.specs` feels like that
as a "side effect" we are going to clone repositories. I believe
that fetching repositories should be an explicit call.
For instance:
```ruby
source "https://rubygems.org"
gem "foo", github: "foo/foo"
# The repository foo will be fetched as a side effect to the call to `source.spec_names`
# 6cc7d71dac/bundler/lib/bundler/source_map.rb (L21)
```
```ruby
source "https://rubygems.org"
gem "bar", source: "https://example.org"
gem "foo", github: "foo/foo"
# The repository foo will be fetched on a different codepath
# 6cc7d71dac/bundler/lib/bundler/source/rubygems_aggregate.rb (L35)
# That is because the gem "bar" has a source that doesn't have the `/dependencies` API
# endpoint and therefore Bundler enters a different branch condition.
```
I opted to add a self explanatory call to fetch the git source
repositories just before we start the resolution, and *just*
before any other calls to `source.specs` is performed.
https://github.com/ruby/rubygems/commit/f0ef526f23
- In 31d67ecc05
we enforced the new behaviour where running `bundle` no longer
installs gems but displays the help.
Users now have a way to configure their preferred default command using
the `BUNDLE_DEFAULT_CLI_COMMAND` flag.
With the preview of Ruby 4.0 now being released, some people will
start to see this new change.
The problem is that the previous behaviour had existed for like an
eternity and we didn't warn users about this change in advance.
I'd like to provide a deprecation/warning cycle because this is
confusing users already and this breaks various CI setup that now
needs to be changed immediately.
https://github.com/ruby/rubygems/commit/e415480ac5
This is useful, in case you're using Docker, and an upstream
Dockerfile sets BUNDLER_VERSION to something you don't want.
It's impossible to unset it... only override to be the empty
string.
https://github.com/ruby/rubygems/commit/ffa3eb9ac6
