Bundler | Closer to Code

While officially End-of-Life (EOL), Ruby 2.7 remains critical in many ongoing projects. Despite its EOL status, a significant user base continues to rely on this version for various reasons, ranging from legacy system compatibility to specific feature dependencies. As a developer of Karafka, an open-source software (OSS), I recognize the importance of supporting Ruby 2.7, giving users more time to upgrade than the EOL time. This commitment is reflected in my integration tests that ensure compatibility with Ruby 2.7.

However, a recent update has posed a challenge. On Friday, 15th of December 2023, Karafka integration tests for Ruby 2.7 started failing, citing compatibility issues with the latest Bundler version:

ERROR:  Error installing bundler:
The last version of bundler (>= 0) to support your Ruby & RubyGems was 2.4.22.
Try installing it with `gem install bundler -v 2.4.22`
bundler requires Ruby version >= 3.0.0. The current ruby version is 2.7.8.225.

This message indicates that Bundler no longer supports Ruby versions older than 3.0.0, which can be a significant concern for Continuous Integration (CI) processes that still use Ruby 2.7.

In my setup, I utilize a GitHub Actions Ruby versions matrix. This matrix is configured with custom code that always installs the most recent version of Bundler. The integration tests failed as a direct result of this approach, as Bundler versions newer than 2.4.22 are incompatible with Ruby versions older than 3.0.0.

- name: Install latest Bundler
  run: |
    gem install bundler --no-document
    gem update --system --no-document
    bundle config set without 'tools benchmarks docs'

Aside from upgrading to a newer version of Ruby (which is always recommended for long-term support), the remedy involves enforcing the installation and usage of Bundler version 2.4.22 for projects still running on Ruby 2.7. Below is a script that demonstrates how to implement this solution:

if [[ "$(ruby -v | awk '{print $2}')" == 2.7.8* ]]; then
  gem install bundler -v 2.4.22 --no-document
  gem update --system 3.4.22 --no-document
else
  gem install bundler --no-document
  gem update --system --no-document
fi

In this script, a conditional check is performed to determine the Ruby version. If it's 2.7.8, the script installs Bundler version 2.4.22 and the corresponding compatible version of rubygems-update. For newer Ruby versions, the script defaults to installing the latest Bundler and updates the system gems.

The recent incompatibility between the latest Bundler version and Ruby 2.7 highlights a critical aspect of software development: managing dependencies and ensuring compatibility across different versions. While upgrading to the latest Ruby version is the ideal long-term solution, the provided script offers a viable workaround for maintaining projects on Ruby 2.7, ensuring their stability and functionality in CI environments.

Note: This article is not to deprecate any of the findings and achievements of Alex Birsan. He did great work exploiting specific vulnerabilities and patterns. It is to present the RubyGems side of the story and to reassure you. We actively work to provide a healthy and safe ecosystem for our users.

After reading the Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies I felt, that the Ruby community requires a bit of explanation from people involved in RubyGems security assessment. So here it is.

It's you who is responsible for the security of your software, and bugs do exist

First of all, let me remind you that your system security should never rely solely on other people's OSS work. In the end, it will be your system that will get hacked. Secondly, any software has bugs, whether we're talking about Bundler, RubyGems, Yarn, or any other piece of code.

Looking at this incident as someone heavily involved with making the Ruby ecosystem secure, RubyGems did everything right.

Does RubyGems allow for malicious packages?

Starting from September 2020, Alex has uploaded gems used for his research. Diffend spotted them, and each of the findings was reported to the RubyGems security team.

RubyGems security team assessed each of them, going through the source codes to ensure they were not malicious. And they were not. The reason why they were allowed to stay is that:

A security researcher uploaded them that we knew of.
They included a description and reasoning why they do what they do (see here).
Several people from RubyGems checked them to ensure they were not doing any harm.

RubyGems policy regarding gems is simple:

As long as the gem is not doing any harm or is not misleading in a harmful way, it won't be removed.

Thousands of gems download things upon being installed, run compilers, and send requests over the internet. For some advanced cases, there is no other way.

It is a constant race

We inspect gems and take many countermeasures to ensure the whole ecosystem's safety. While we cannot promise you that we will catch every single attack ever, we are making progress, and we are getting better and better with our detection systems.

Even the day I'm writing this article, we've yanked (removed) 8 gems that were a build-up for a more significant scale attack.

It is not about RubyGems only but also about Bundler

Bundler is a complex piece of code. The resolving engine needs to deal with many corner-cases. Security of resolution is the most important thing, but other factors like correctness and speed are also important.

While we do not know the exact way of installing those gems, our gut feeling is that they might have been resolved "incorrectly". "Incorrectly," however, indicates that they were resolved in an invalid way. At the same time, it may turn out that they were resolved exactly as expected but not as the end-user/programmer wanted. It might be, that it was due to this bug. That is, "dependency of my dependency will be checked in RubyGems."

I use private gems, please help!

If you use private gems with private dependencies, you can do few things at the moment:

Most important: Update Bundler once the patch is released.
Most important: Update Bundler to the 2.2.10 version.
Always use Bundler source blocks to ensure that private gems can only come from private sources. If your private gems depend on other private gems, you may need to declare those gems in the private source block as well.
Setup a tool like Diffend and create rules that will block Bundler from using RubyGems as a source for gems that match your internal gems naming conventions.
Mirror RubyGems and upstream only gems you are interested in + your own.
[workaround] Book the gems names in RubyGems (althought this may also be a potential insight for hackers into internal naming conventions of your company).

Summary

So yes, there was a bug in Bundler (fixed in 2.2.10), but at the same time, if it was not for explicit permission from the RubyGems security team, those gems would have been yanked soon after they were released. That's why in this particular case, I would rather say that those companies got "researched" rather than hacked.

This incident made RubyGems reconsider the "no harm" policy, and it may be subject to change in the near future.

Cover photo by Jan Hrdina on Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0).

Tag: Bundler

The Ruby 2.7 Challenge: Adapting to Bundler’s Latest 2.5+ Update

RubyGems dependency confusion attack side of things