Running with Ruby

Category: Ruby (page 2 of 101)

The hidden cost of a Ruby threads leakage

Bug hunting

Recently I’ve been working with one small application that would gradually become slower and slower. While there were many reasons for it to happen, I found one of them interesting.

To give you a bit of context: the application was a simple single topic legacy Kafka consumer. I rewrote it to Karafka, and all of the logic looks like this:

class EventsConsumer < Karafka::BaseConsumer
  def initialize(...)
    super
    @processor = Processor.new
  end

  def consume
    @processor.call(params_batch.payloads)
  end
end

And the processor looks like so (I removed all the irrelevant code):

class Processor
  def initialize
    @queue = Queue.new
    @worker = Thread.new { work }
  end

  def call(events)
    # do something with events
    results = complex_inline_computation(events)
    @queue << results
  end

  private

  def work
    while true
      result = @queue.pop
      # some sort of async storing operation should go here
      p result
    end
  end

  def complex_inline_computation(events)
    events.join('-')
  end
end

So, we have a Karafka consumer with a processor with one background thread supposed to flush the data async. Nothing special, and when putting aside potential thread crashes, all looks good.

However, there is a hidden problem in this code that can reveal itself by slowly degrading this consumer performance over time.

Karafka uses a single persistent consumer instance per topic partition. When we start processing a given partition of a given topic for the first time, a new consumer instance is created. This by itself means that the number of threads we end up with is directly defined by the number of topics and their partitions we’re processing with a single Karafka process.

If that was all, I would say it’s not that bad. While for a single topic consuming process, with 20 partitions, we do end up with additional 20 threads, upon reaching this number, the degradation should stop.

It did not.

There is one more case where our legacy consumer and Karafka would spin-up additional threads because of the processor re-creation: Kafka rebalance. When rebalancing happens, new consumer instances are initialized. That means that each time scaling occurred, whether it would add or remove instances, a new processor thread would be created.

Fixing the issue

Fixing this issue without a re-design is rather simple. As long as we can live with a singleton and we know that our code won’t be executed by several threads in parallel, we can just make the processor into a singleton:

class Processor
  include Singleton
  
  # Remaining code
end

class EventsConsumer < Karafka::BaseConsumer
  def initialize(...)
    super
    @processor = Processor.instance
  end

  # Remaining code
end

While it is not the optimal solution, in my case it was sufficient.

Performance impact

One question remains: what was the performance impact of having stale threads that were doing nothing?

I’ll try to answer that with a more straightforward case than mine:

require 'benchmark'

MAX_THREADS = 100
STEP = 10
ITERS = 50000000

(0..MAX_THREADS).step(STEP).each do |el|
  STEP.times do
    Thread.new do
      q = Queue.new
      q.pop
    end
  end unless el.zero?

  # Give it a bit of time to initialize the threads
  sleep 5

  # warmup for jruby - thanks Charles!
  5.times do
    ITERS.times do ; a = "1"; end
  end

  Benchmark.bm do |x|
    x.report { ITERS.times do ; a = "1"; end }
  end
end

I’ve run this code 100 times and used the average time to minimize the randomness impact of other tasks running on this machine.

Here are the results for Ruby 2.7.2, 3.0.0-preview2 (with and without JIT) and JRuby 9.2.13, all limited with time taskset -c 1, to make sure that JRuby is running under the same conditions (single core):

CRuby performance degradation is more or less linear. The more threads you have that do nothing, the slower the overall processing happens. This does not affect JRuby as JVM threads support is entirely different than the CRubys.

What worries me more, though, is that Ruby 3.0 seems to degrade more than 2.7.2. My wild guess here is that it’s because of Ractors code’s overhead and other changes that impact threads scheduler.

Below you can find the time comparison for all the variants of CRuby:

It is fascinating that 3.0 is slower than 2.7.2 in this case, and I will try to look into the reasons behind it in the upcoming months.

Note: I do not believe it’s the best use-case for JIT, so please do not make strong claims about its performance based on the charts above.

Summary

The more complex applications you build, the bigger chances are that you will have to have threads at some point. If that happens, please be mindful of their impact on your applications’ overall performance.

Also, keep in mind that the moment you introduce background threads, the moment you should introduce proper instrumentation around them.


Cover photo by Chad Cooper on Attribution 2.0 Generic (CC BY 2.0) license.

Diffend – OSS supply chain security and management platform for Ruby

I’m incredibly excited to announce a security platform for managing Ruby gems dependencies: diffend.io.

This platform is a result of my involvement in Ruby security matters for years. It all started in early 2018 with a tool to review gems versions diffs. While working on it, I’ve noticed that there’s much more that needs to be handled. Versions diffing while inevitable, by itself is insufficient, that’s why we’ve built this platform.

Getting started

If you’re just interested in the gems diffing, go to my.diffend.io and select any gem and versions you want to view. New releases for all the gems are computed in real-time, but for some of the older ones, you will have to wait a bit.

You can also use a shiny new link available on each RubyGems gems page to review changes against the previous release of the same gem:

If you would want to run a more thoughtful assessment, you can either run this script in your application main directory:

ruby <(curl -s https://my.diffend.io/api/setup/ruby)

or if you are like me and do not want to run scripts from the internet, you can just follow the super short manual with setup instructions here.

If something is not clear or you have any questions, please contact us at our Slack workspace with this invitation link or drop us a line at contact@diffend.io.

What does it do?

In short, Diffend allows you to:

  1. Review changes in between gems releases before you upgrade based on the gems content itself,
  2. Block attempts of even downloading potentially unwanted gems and their versions,
  3. Manage third party dependencies within your organization,
  4. Ensure OSS licensing consistency in your organization,
  5. Get insights on vulnerabilities, memory leaks and licensing problems of your dependencies,
  6. Make dependency audits a part of your workflow,
  7. Get real-time notifications about any new risks that occur in your production systems (coming soon)

It also runs certain types of heuristics and checks to pinpoint potentially “interesting” releases for further semi-manual inspection.

Why do we need it?

OSS supply chain attacks are becoming a more and more common thing. Looking at RubyGems or npm, there are plenty of examples of packages getting hijacked and malicious versions being uploaded. There were already several attacks that were detected and stopped thanks to Diffend and RubyGems close cooperation.

If you just update dependencies without checking them, you’re not actually sure of what you’re putting into production. You should not trust what’s on Github. An attacker can upload something to a registry without pushing it to Github. The only way to be sure is to look at what’s actually on the registry.

When it’s easy to work securely, people are more likely to do it. diffend.io, is another step towards improving Ruby’s security story by letting you generate diffs from any browser and share them as links. This also lends itself to automation: now you can connect Diffend with your Gemfile and make dependency audits a part of your workflow. We hope this will inspire the community with lots of new security ideas that don’t slow you down.

Is it secure?

Diffend was built with security in mind. Platform, plugin, and our gem collect the absolute minimum amount of data to provide you with the services. Both the Bundler plugin and the monitor will be open-sourced, but even now you can download and review their content.

On top of all of that, we’ve been super cautious about what we collect, that’s why:

  1. We do not collect credentials or environment variables;
  2. We do not execute any remote code from our plugin or gem. Never.
  3. We do not access anything except the Gemfile and Gemfile.lock content.
  4. We do not send to ourselves private access keys for any non-public gems.
  5. We are working on a fully anonymous mode where we do not track public IPs

Support us!

Diffend platform is free to use. You don’t even need an account to review the diffs (and you never will). If you like our platform, please consider convincing your company to support us with any amount of money. We’ll just invoice you for the service usage :)

This way, with a bit of funding, we might be able to push forward many security initiatives much faster.

What’s next?

At the moment we are working on several things:

  • Open-sourcing the plugin and the monitor,
  • Real-time production / staging based context aware Slack and e-mail notifications about new risks,
  • Improved heuristics and detection capabilities,
  • Modified Ruby VM for network tracking analysis with pre-execution permissions,
  • Ruby process behaviour tracking,
  • Open-sourcing several of the components for self-service,
  • Fully anonymous mode without collecting any public data.

Diffend is a platform in an alpha stage and under massive development. Some functionalities may not work on every operating system, and some other features may not be available or may be broken. We are working hard to fix and improve the platform, which is why we are counting on your feedback so that we can meet your exact needs faster!

Read more

Olderposts Newerposts

Copyright © 2021 Running with Ruby

Theme by Anders NorenUp ↑