Closer to Code

Tag: whenever

How requiring a gem can mess up your already running application

Introduction

Ruby’s dynamic nature is both its advantage and disadvantage. Being able to reopen system classes during runtime, while useful, can also lead to unexpected behaviors. This article presents one such case: how just requiring a gem can mess things up in a completely different area of the application.

The bizzare error

Recently, after connecting the Diffend monitor into one of my systems, it started reporting a bizarre error:

uninitialized constant Whenever

whenever-1.0.0/lib/whenever/numeric.rb:3:in `respond_to?'
lib/ruby/2.7.0/bundler/settings.rb:368:in `=='
lib/ruby/2.7.0/bundler/settings.rb:368:in `=='
lib/ruby/2.7.0/bundler/settings.rb:368:in `converted_value'
lib/ruby/2.7.0/bundler/settings.rb:94:in `[]'
lib/ruby/2.7.0/bundler/fetcher.rb:80:in `'
lib/ruby/2.7.0/bundler/fetcher.rb:11:in `'
lib/ruby/2.7.0/bundler/fetcher.rb:9:in `'
diffend-monitor-0.2.36/lib/diffend/build_bundler_definition.rb:18:in `call'
diffend-monitor-0.2.36/lib/diffend/execute.rb:22:in `build_definition'
diffend-monitor-0.2.36/lib/diffend/execute.rb:12:in `call'
diffend-monitor-0.2.36/lib/diffend/track.rb:21:in `start'
diffend-monitor-0.2.36/lib/diffend/monitor.rb:42:in `block in '

the line in which it was happening was just a delegation to the Bundler API method:

::Bundler::Fetcher.disable_endpoint = nil

and in Bundler itself it is just an attr_accessor:

class << self
  attr_accessor :disable_endpoint, :api_timeout, :redirect_limit
end

So what does all of it has to do with the Whenever gem? Nothing.

We have nothing to do with Whenever but it does not mean Whenever has nothing to do with us.

Requiring a gem does not only mean that its code is being loaded. It also means that the gem can perform any operations it wants, whether legit or malicious.

When diffend-monitor is being required, it spins up its own Ruby thread and starts reporting data. And here is the moment when Whenever kicks in. It was being required after the monitor. Thus the monitor code was already running. In theory, those two should be separated entirely. Whenever and Diffend do entirely different things and they have their own namespaces.

It turns out, unfortunately, that Whenever is monkey patching Numeric class in an incorrect way:

Numeric.class_eval do
  def respond_to?(method, include_private = false)
    super || Whenever::NumericSeconds.public_method_defined?(method)
  end

  def method_missing(method, *args, &block)
    if Whenever::NumericSeconds.public_method_defined?(method)
      Whenever::NumericSeconds.new(self).send(method)
    else
      super
    end
  end
end

This patch seems to be safe, but there’s a really big assumption made: Whenever::NumericSeconds needs to be accessible. If we look into the Whenever code loading file, we will notice, that the patch is required before Whenever::NumericSeconds comes to existence:

require 'whenever/numeric'
require 'whenever/numeric_seconds'

This means that any action that would invoke #method_missing after the first file is loaded, but before the second one, will fail.

Can it even happen? Absolutely! Ruby’s require is not blocking. It means, that Ruby VM can stop the requiring after any of the files and switch context to do other things in other threads.

When the above is understood, building a reproduction code is just a matter of seconds:

Thread.new do
  while true
    begin
      1.respond_to?(:elo)
      sleep 0.00001
    rescue => e
      p e
    end
  end
end

sleep 0.2

require 'whenever'

Here’s how it behaves when executed:

I’ve created an issue in Whenever, and hopefully, its maintainers will address it. Meanwhile there’s one more question to ask: can we somehow address this problem, so it won’t break our code?

Mitigating the issue before the library is patched

There is no silver bullet for this type of problem. As any gem can introduce their own patches to other classes, the potential problems are endless. In this particular case, the code ends up being “ok” once everything is loaded. What we’ve decided to do was pretty trivial. We’ve decided to give the app enough time to require all the things that could potentially break the execution:

Thread.new do
  sleep 0.5

  while true
    begin
      1.respond_to?(:elo)
      sleep 0.00001
    rescue => e
      p e
    end
  end
end

This sleep ensures that as long as nothing heavy happens during gems requirement via Bundler, we don’t end up with partially loaded, broken monkey-patches while executing our own logic in a background thread.


Cover photo by Ruin Raider on Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) license.

Ruby & Rails: Making sure rake task won’t slow the site down

If you don’t have multiple cores and/or you have a small VPN, you may end up with a huge slow down of your web app, when rake tasks are executed. This can be a big issue especially when you use something like whenever to perform periodic tasks. Luckily there’s a nice program:

nice is a program found on Unix and Unix-like operating systems such as Linux. It directly maps to a kernel call of the same name. nice is used to invoke a utility or shell script with a particular priority, thus giving the process more or less CPU time than other processes. A niceness of −20 is the highest priority and 19 or 20 is the lowest priority. The default niceness for processes is inherited from its parent process and is usually 0.

With nice you can tell your Linux kernel to give your rake tasks the lowest priority possible. Thanks to that, you can make sure, that any task with higher priority (like web UI server) won’t get stuck because of the background rake task.

Here’s how you can use it:

RAILS_ENV=:environment nice -n 19 bundle exec rake :your_task 

If you use a gem like whenever, you can easily integrate nice with it:

job_type :runner, "cd :path && nice -n 19 script/rails runner -e :environment ':task' :output"

every 30.minutes do
  runner 'Worker.new.perform'
end

or for a rake task with whenever, you can use this:

job_type :runner, "cd :path && :environment_variable=:environment nice -19 bundle exec rake :task --silent :output"

every 15.minutes do
  runner 'your_task'
end

Note: Of course this will help, when CPU is your bottleneck. Keep in mind that you’re rake task might slow down your website because of many more reasons (IO, DB, etc).

Copyright © 2021 Closer to Code

Theme by Anders NorenUp ↑