Tag: rubygems

The librdkafka Supply Chain Breakdown: rdkafka-ruby’s Darkest Hour

Opening Note

We all make mistakes, and fundamentally, the havoc caused by this incident was due to a flaw in the design of rdkafka-ruby. While the disappearance of librdkafka from GitHub was unexpected, this article aims to clarify and explain how rdkafka-ruby should have prevented it and what was poorly designed. By examining this incident, I hope to provide insights into better practices for managing dependencies and ensuring more resilient software builds for the Ruby ecosystem.

Incident Summary

On July 10, 2024 15:47 UTC, users of the rdkafka gem faced issues when the librdkafka repository on GitHub unexpectedly went private. This break in the supply chain disrupted installations, causing widespread frustration and, in many cases, completely blocking the ability to deploy rdkafka-based software.

Fetching rdkafka 0.16.0
Installing rdkafka 0.16.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /rdkafka-0.16.0/ext
/usr/local/bin/ruby -rrubygems
/rake-13.2.1/exe/rake
RUBYARCHDIR\=/home/circleci/.rubygems/extensions/x86_64-linux/3.3.0/rdkafka-0.16.0
RUBYLIBDIR\=/home/circleci/.rubygems/extensions/x86_64-linux/3.3.0/rdkafka-0.16.0
2 retrie(s) left for v2.4.0 (404 Not Found)
1 retrie(s) left for v2.4.0 (404 Not Found)
0 retrie(s) left for v2.4.0 (404 Not Found)
404 Not Found
rake aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - ports/archives/v2.4.0
(Errno::ENOENT)
/mini_portile2-2.8.7/lib/mini_portile2/mini_portile.rb:496:in
`verify_file'
/mini_portile2-2.8.7/lib/mini_portile2/mini_portile.rb:133:in
`block in download'
/mini_portile2-2.8.7/lib/mini_portile2/mini_portile.rb:131:in
`each'
/mini_portile2-2.8.7/lib/mini_portile2/mini_portile.rb:131:in
`download'
/mini_portile2-2.8.7/lib/mini_portile2/mini_portile.rb:232:in
`cook'
/rdkafka-0.16.0/ext/Rakefile:38:in `block
in <top (required)>'
/rake-13.2.1/exe/rake:27:in `<main>'
Tasks: TOP => default
(See full trace by running task with --trace)

Detailed Explanation

The rdkafka gem used to rely on downloading librdkafka from the Confluent GitHub repository during the installation process. As a huge proponent of immutable builds that do not depend on external resources, I planned to change this model for a long time. Several months ago, I created a GitHub issue to address this transition. However, the change was delayed due to other priorities within the karafka ecosystem. Unfortunately, this delay resulted in the recent outage.

# Just the relevant code here

recipe.files << {
  :url => "https://codeload.github.com/edenhill/librdkafka/tar.gz/v#{Rdkafka::LIBRDKAFKA_VERSION}",
  :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
}

recipe.configure_options = ["--host=#{recipe.host}"]
recipe.cook

This setup meant that during the bundle install process, the required librdkafka source was fetched and compiled on the fly, which inherently relied on the availability of the external GitHub repository.

Upon discovery, it took me 59 minutes to release the first patched version and approximately four hours to prepare fixes and backport them to all relevant versions of the rdkafka gem, including older ones. Luckily, I was in front of my computer when the incident occurred, allowing me to quickly create and release needed fixes.

Future Steps

Going forward, all future releases will depend only on RubyGems, ensuring no reliance on external build sources like GitHub. I decided to ship the librdkafka releases inside the gem itself, enhancing its reliability and stability of the ecosystem.

releases = File.expand_path(File.join(File.dirname(__FILE__), '../dist'))

recipe.files << {
  :url => "file://#{releases}/librdkafka_#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
  :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
}
recipe.configure_options = ["--host=#{recipe.host}"]
recipe.cook

Fragility of the OSS Supply Chain

This incident highlights our dependence on other OSS projects and repositories. It's essential to remember that mistakes can happen, and we must be prepared. This wasn't the first issue with GitHub downloads. In 2023, a change in GitHub's tar layout broke a lot of software, including ours, that relied on checksums for artifacts verification. To be honest, if we had migrated the building process of rdkafka at that time, this article would not have to be written.

Here are my main takeaways from this incident:

  1. Design Flaws Can Amplify Issues: The incident highlighted how design flaws in dependency management can lead to significant disruptions.
  2. Dependency on External Repositories: Relying on external data sources during the build process can pose risks, mainly when unexpected changes occur.
  3. Importance of Immutable Builds: Adopting immutable builds without external resources can enhance reliability and stability.

RubyGems dependency confusion attack side of things

Note: This article is not to deprecate any of the findings and achievements of Alex Birsan. He did great work exploiting specific vulnerabilities and patterns. It is to present the RubyGems side of the story and to reassure you. We actively work to provide a healthy and safe ecosystem for our users.

After reading the Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies I felt, that the Ruby community requires a bit of explanation from people involved in RubyGems security assessment. So here it is.

It's you who is responsible for the security of your software, and bugs do exist

First of all, let me remind you that your system security should never rely solely on other people's OSS work. In the end, it will be your system that will get hacked. Secondly, any software has bugs, whether we're talking about Bundler, RubyGems, Yarn, or any other piece of code.

Looking at this incident as someone heavily involved with making the Ruby ecosystem secure, RubyGems did everything right.

Does RubyGems allow for malicious packages?

Starting from September 2020, Alex has uploaded gems used for his research. Diffend spotted them, and each of the findings was reported to the RubyGems security team.

RubyGems security team assessed each of them, going through the source codes to ensure they were not malicious. And they were not. The reason why they were allowed to stay is that:

  1. A security researcher uploaded them that we knew of.
  2. They included a description and reasoning why they do what they do (see here).
  3. Several people from RubyGems checked them to ensure they were not doing any harm.

RubyGems policy regarding gems is simple:

As long as the gem is not doing any harm or is not misleading in a harmful way, it won't be removed.

Thousands of gems download things upon being installed, run compilers, and send requests over the internet. For some advanced cases, there is no other way.

It is a constant race

We inspect gems and take many countermeasures to ensure the whole ecosystem's safety. While we cannot promise you that we will catch every single attack ever, we are making progress, and we are getting better and better with our detection systems.

Even the day I'm writing this article, we've yanked (removed) 8 gems that were a build-up for a more significant scale attack.

It is not about RubyGems only but also about Bundler

Bundler is a complex piece of code. The resolving engine needs to deal with many corner-cases. Security of resolution is the most important thing, but other factors like correctness and speed are also important.

While we do not know the exact way of installing those gems, our gut feeling is that they might have been resolved "incorrectly". "Incorrectly," however, indicates that they were resolved in an invalid way. At the same time, it may turn out that they were resolved exactly as expected but not as the end-user/programmer wanted. It might be, that it was due to this bug. That is, "dependency of my dependency will be checked in RubyGems."

I use private gems, please help!

If you use private gems with private dependencies, you can do few things at the moment:

  1. Most important: Update Bundler once the patch is released.
  2. Most important: Update Bundler to the 2.2.10 version.
  3. Always use Bundler source blocks to ensure that private gems can only come from private sources. If your private gems depend on other private gems, you may need to declare those gems in the private source block as well.
  4. Setup a tool like Diffend and create rules that will block Bundler from using RubyGems as a source for gems that match your internal gems naming conventions.
  5. Mirror RubyGems and upstream only gems you are interested in + your own.
  6. [workaround] Book the gems names in RubyGems (althought this may also be a potential insight for hackers into internal naming conventions of your company).

Summary

So yes, there was a bug in Bundler (fixed in 2.2.10), but at the same time, if it was not for explicit permission from the RubyGems security team, those gems would have been yanked soon after they were released. That's why in this particular case, I would rather say that those companies got "researched" rather than hacked.

This incident made RubyGems reconsider the "no harm" policy, and it may be subject to change in the near future.


Cover photo by Jan Hrdina on Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0).

Copyright © 2024 Closer to Code

Theme by Anders NorenUp ↑