Category: Software

RubyGems dependency confusion attack side of things

Note: This article is not to deprecate any of the findings and achievements of Alex Birsan. He did great work exploiting specific vulnerabilities and patterns. It is to present the RubyGems side of the story and to reassure you. We actively work to provide a healthy and safe ecosystem for our users.

After reading the Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies I felt, that the Ruby community requires a bit of explanation from people involved in RubyGems security assessment. So here it is.

It’s you who is responsible for the security of your software, and bugs do exist

First of all, let me remind you that your system security should never rely solely on other people’s OSS work. In the end, it will be your system that will get hacked. Secondly, any software has bugs, whether we’re talking about Bundler, RubyGems, Yarn, or any other piece of code.

Looking at this incident as someone heavily involved with making the Ruby ecosystem secure, RubyGems did everything right.

Does RubyGems allow for malicious packages?

Starting from September 2020, Alex has uploaded gems used for his research. Diffend spotted them, and each of the findings was reported to the RubyGems security team.

RubyGems security team assessed each of them, going through the source codes to ensure they were not malicious. And they were not. The reason why they were allowed to stay is that:

  1. A security researcher uploaded them that we knew of.
  2. They included a description and reasoning why they do what they do (see here).
  3. Several people from RubyGems checked them to ensure they were not doing any harm.

RubyGems policy regarding gems is simple:

As long as the gem is not doing any harm or is not misleading in a harmful way, it won’t be removed.

Thousands of gems download things upon being installed, run compilers, and send requests over the internet. For some advanced cases, there is no other way.

It is a constant race

We inspect gems and take many countermeasures to ensure the whole ecosystem’s safety. While we cannot promise you that we will catch every single attack ever, we are making progress, and we are getting better and better with our detection systems.

Even the day I’m writing this article, we’ve yanked (removed) 8 gems that were a build-up for a more significant scale attack.

It is not about RubyGems only but also about Bundler

Bundler is a complex piece of code. The resolving engine needs to deal with many corner-cases. Security of resolution is the most important thing, but other factors like correctness and speed are also important.

While we do not know the exact way of installing those gems, our gut feeling is that they might have been resolved “incorrectly”. “Incorrectly,” however, indicates that they were resolved in an invalid way. At the same time, it may turn out that they were resolved exactly as expected but not as the end-user/programmer wanted. It might be, that it was due to this bug. That is, “dependency of my dependency will be checked in RubyGems.”

I use private gems, please help!

If you use private gems with private dependencies, you can do few things at the moment:

  1. Most important: Update Bundler once the patch is released.
  2. Most important: Update Bundler to the 2.2.10 version.
  3. Always use Bundler source blocks to ensure that private gems can only come from private sources. If your private gems depend on other private gems, you may need to declare those gems in the private source block as well.
  4. Setup a tool like Diffend and create rules that will block Bundler from using RubyGems as a source for gems that match your internal gems naming conventions.
  5. Mirror RubyGems and upstream only gems you are interested in + your own.
  6. [workaround] Book the gems names in RubyGems (althought this may also be a potential insight for hackers into internal naming conventions of your company).

Summary

So yes, there was a bug in Bundler (fixed in 2.2.10), but at the same time, if it was not for explicit permission from the RubyGems security team, those gems would have been yanked soon after they were released. That’s why in this particular case, I would rather say that those companies got “researched” rather than hacked.

This incident made RubyGems reconsider the “no harm” policy, and it may be subject to change in the near future.


Cover photo by Jan Hrdina on Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0).

How requiring a gem can mess up your already running application

Introduction

Ruby’s dynamic nature is both its advantage and disadvantage. Being able to reopen system classes during runtime, while useful, can also lead to unexpected behaviors. This article presents one such case: how just requiring a gem can mess things up in a completely different area of the application.

The bizzare error

Recently, after connecting the Diffend monitor into one of my systems, it started reporting a bizarre error:

uninitialized constant Whenever

whenever-1.0.0/lib/whenever/numeric.rb:3:in `respond_to?'
lib/ruby/2.7.0/bundler/settings.rb:368:in `=='
lib/ruby/2.7.0/bundler/settings.rb:368:in `=='
lib/ruby/2.7.0/bundler/settings.rb:368:in `converted_value'
lib/ruby/2.7.0/bundler/settings.rb:94:in `[]'
lib/ruby/2.7.0/bundler/fetcher.rb:80:in `'
lib/ruby/2.7.0/bundler/fetcher.rb:11:in `'
lib/ruby/2.7.0/bundler/fetcher.rb:9:in `'
diffend-monitor-0.2.36/lib/diffend/build_bundler_definition.rb:18:in `call'
diffend-monitor-0.2.36/lib/diffend/execute.rb:22:in `build_definition'
diffend-monitor-0.2.36/lib/diffend/execute.rb:12:in `call'
diffend-monitor-0.2.36/lib/diffend/track.rb:21:in `start'
diffend-monitor-0.2.36/lib/diffend/monitor.rb:42:in `block in '

the line in which it was happening was just a delegation to the Bundler API method:

::Bundler::Fetcher.disable_endpoint = nil

and in Bundler itself it is just an attr_accessor:

class << self
  attr_accessor :disable_endpoint, :api_timeout, :redirect_limit
end

So what does all of it has to do with the Whenever gem? Nothing.

We have nothing to do with Whenever but it does not mean Whenever has nothing to do with us.

Requiring a gem does not only mean that its code is being loaded. It also means that the gem can perform any operations it wants, whether legit or malicious.

When diffend-monitor is being required, it spins up its own Ruby thread and starts reporting data. And here is the moment when Whenever kicks in. It was being required after the monitor. Thus the monitor code was already running. In theory, those two should be separated entirely. Whenever and Diffend do entirely different things and they have their own namespaces.

It turns out, unfortunately, that Whenever is monkey patching Numeric class in an incorrect way:

Numeric.class_eval do
  def respond_to?(method, include_private = false)
    super || Whenever::NumericSeconds.public_method_defined?(method)
  end

  def method_missing(method, *args, &block)
    if Whenever::NumericSeconds.public_method_defined?(method)
      Whenever::NumericSeconds.new(self).send(method)
    else
      super
    end
  end
end

This patch seems to be safe, but there’s a really big assumption made: Whenever::NumericSeconds needs to be accessible. If we look into the Whenever code loading file, we will notice, that the patch is required before Whenever::NumericSeconds comes to existence:

require 'whenever/numeric'
require 'whenever/numeric_seconds'

This means that any action that would invoke #method_missing after the first file is loaded, but before the second one, will fail.

Can it even happen? Absolutely! Ruby’s require is not blocking. It means, that Ruby VM can stop the requiring after any of the files and switch context to do other things in other threads.

When the above is understood, building a reproduction code is just a matter of seconds:

Thread.new do
  while true
    begin
      1.respond_to?(:elo)
      sleep 0.00001
    rescue => e
      p e
    end
  end
end

sleep 0.2

require 'whenever'

Here’s how it behaves when executed:

I’ve created an issue in Whenever, and hopefully, its maintainers will address it. Meanwhile there’s one more question to ask: can we somehow address this problem, so it won’t break our code?

Mitigating the issue before the library is patched

There is no silver bullet for this type of problem. As any gem can introduce their own patches to other classes, the potential problems are endless. In this particular case, the code ends up being “ok” once everything is loaded. What we’ve decided to do was pretty trivial. We’ve decided to give the app enough time to require all the things that could potentially break the execution:

Thread.new do
  sleep 0.5

  while true
    begin
      1.respond_to?(:elo)
      sleep 0.00001
    rescue => e
      p e
    end
  end
end

This sleep ensures that as long as nothing heavy happens during gems requirement via Bundler, we don’t end up with partially loaded, broken monkey-patches while executing our own logic in a background thread.


Cover photo by Ruin Raider on Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) license.

Copyright © 2021 Closer to Code

Theme by Anders NorenUp ↑