Running with Ruby

Category: Rails (page 1 of 78)

Diffend – OSS supply chain security and management platform for Ruby

I’m incredibly excited to announce a security platform for managing Ruby gems dependencies: diffend.io.

This platform is a result of my involvement in Ruby security matters for years. It all started in early 2018 with a tool to review gems versions diffs. While working on it, I’ve noticed that there’s much more that needs to be handled. Versions diffing while inevitable, by itself is insufficient, that’s why we’ve built this platform.

Getting started

If you’re just interested in the gems diffing, go to my.diffend.io and select any gem and versions you want to view. New releases for all the gems are computed in real-time, but for some of the older ones, you will have to wait a bit.

You can also use a shiny new link available on each RubyGems gems page to review changes against the previous release of the same gem:

If you would want to run a more thoughtful assessment, you can either run this script in your application main directory:

ruby <(curl -s https://my.diffend.io/api/setup/ruby)

or if you are like me and do not want to run scripts from the internet, you can just follow the super short manual with setup instructions here.

If something is not clear or you have any questions, please contact us at our Slack workspace with this invitation link or drop us a line at contact@diffend.io.

What does it do?

In short, Diffend allows you to:

  1. Review changes in between gems releases before you upgrade based on the gems content itself,
  2. Block attempts of even downloading potentially unwanted gems and their versions,
  3. Manage third party dependencies within your organization,
  4. Ensure OSS licensing consistency in your organization,
  5. Get insights on vulnerabilities, memory leaks and licensing problems of your dependencies,
  6. Make dependency audits a part of your workflow,
  7. Get real-time notifications about any new risks that occur in your production systems (coming soon)

It also runs certain types of heuristics and checks to pinpoint potentially “interesting” releases for further semi-manual inspection.

Why do we need it?

OSS supply chain attacks are becoming a more and more common thing. Looking at RubyGems or npm, there are plenty of examples of packages getting hijacked and malicious versions being uploaded. There were already several attacks that were detected and stopped thanks to Diffend and RubyGems close cooperation.

If you just update dependencies without checking them, you’re not actually sure of what you’re putting into production. You should not trust what’s on Github. An attacker can upload something to a registry without pushing it to Github. The only way to be sure is to look at what’s actually on the registry.

When it’s easy to work securely, people are more likely to do it. diffend.io, is another step towards improving Ruby’s security story by letting you generate diffs from any browser and share them as links. This also lends itself to automation: now you can connect Diffend with your Gemfile and make dependency audits a part of your workflow. We hope this will inspire the community with lots of new security ideas that don’t slow you down.

Is it secure?

Diffend was built with security in mind. Platform, plugin, and our gem collect the absolute minimum amount of data to provide you with the services. Both the Bundler plugin and the monitor will be open-sourced, but even now you can download and review their content.

On top of all of that, we’ve been super cautious about what we collect, that’s why:

  1. We do not collect credentials or environment variables;
  2. We do not execute any remote code from our plugin or gem. Never.
  3. We do not access anything except the Gemfile and Gemfile.lock content.
  4. We do not send to ourselves private access keys for any non-public gems.
  5. We are working on a fully anonymous mode where we do not track public IPs

Support us!

Diffend platform is free to use. You don’t even need an account to review the diffs (and you never will). If you like our platform, please consider convincing your company to support us with any amount of money. We’ll just invoice you for the service usage :)

This way, with a bit of funding, we might be able to push forward many security initiatives much faster.

What’s next?

At the moment we are working on several things:

  • Open-sourcing the plugin and the monitor,
  • Real-time production / staging based context aware Slack and e-mail notifications about new risks,
  • Improved heuristics and detection capabilities,
  • Modified Ruby VM for network tracking analysis with pre-execution permissions,
  • Ruby process behaviour tracking,
  • Open-sourcing several of the components for self-service,
  • Fully anonymous mode without collecting any public data.

Diffend is a platform in an alpha stage and under massive development. Some functionalities may not work on every operating system, and some other features may not be available or may be broken. We are working hard to fix and improve the platform, which is why we are counting on your feedback so that we can meet your exact needs faster!

Read more

Karafka framework 1.4.0 Release Notes (Ruby + Kafka)

This release mostly solves problems related to message deserialization and normalizes some of the naming conventions to ease during the upgrade to the upcoming 2.0 version.

Note: This release is the last release with ruby-kafka under the hood. We’ve already started the process of moving to rdkafka-ruby.

Note: If you are using Sidekiq-Backend plugin, please make sure that you’ve processed all the jobs from your Sidekiq queue before upgrading Karafka gems.

Changes (features, incompatibilities, etc)

consumer#metadata is now consumer#batch_metadata

This change is trivial: if you use batch consuming mode and you use the Consumer#metadata method, replace it with Consumer#batch_metadata.

# Karafka 1.3
class UsersConsumer < ApplicationConsumer
  def consume
    puts metadata
  end
end

# Karafka 1.4
class UsersConsumer < ApplicationConsumer
  def consume
    puts batch_metadata
  end
end

Message metadata available under #metadata method

Up to version 1.3, all the message metadata would be directly available under the root scope of the params object using both direct method reference as well as with #[] accessor.

While it felt like “The Rails way”, it had several side-effects, amongst which the biggest were the need of having a hash like API, issues with accessing metadata without payload deserialization, and a lack of clear separation between payload and the metadata.

From now on, you can use the params.metadata object to fetch all the metadata.

Note: we’ve preserved the direct metadata values fetching from the params object to preserve backwards compatibility.

# 1.3
params['partition'] #=> 0
params.partition #=> 0

# 1.4
params['partition'] #=> NoMethodError (undefined method '[]')

# This will work due to backward compatibility
params.partition #=> 0

# This is the recommended way of accessing metadata
params.metadata.partition #=> 0

# This will also work as metadata is a struct now
params.metadata[:partition] #=> 0
params.metadata['partition'] #=> 0

Message metadata access allowed without message deserialization

When accessing metadata, the payload is not being deserialized until #payload method is being used.

null message support in the default JSON deserializer

When the Kafka message payload is null / nil, deserialization won’t fail. Support for it was added as some of the Karafka users use log compaction with a nil payload. In case like that, #payload will return nil.

Karafka::Params::Params no longer inherits from a Hash

Karafka::Params::Params is now just a struct. This change is introduced to normalize the setup, limit the corner cases and simplify the interface only to methods that are really needed.

Documentation

Our Wiki has been updated accordingly to the 1.4 status. Please notify us if you find any incompatibilities.

Getting started with Karafka

If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:

git clone https://github.com/karafka/example-app ./example_app

then, just bundle install all the dependencies:

cd ./example_app
bundle install

and follow the instructions from the example app Wiki.

Olderposts

Copyright © 2021 Running with Ruby

Theme by Anders NorenUp ↑