Tag: caching

Dragonfly is a great on-the-fly processing gem, especially for apps that rapidly change and need files processed by multiple processors (for example, when you need to generate many thumbnails for the same attached image).

Since it is a on-the-fly processor, it has some downsides, and the biggest one, in my opinion, is keeping track of which thumbnails we have already created. There's a nice example of how to do it using only ActiveRecord in the dragonfly docs, but unfortunately this solution won't be enough for a heavy imaged website. Since each request for a thumb will require a single SQL query, you might end up having 50-60 and even more thumb related queries per page.

Of course, they would probably get cached, so they would not take much time, but still, there's a better way to do it.

Memcached (dalli) to the rescue

Memcached uses LRU algorithm to remove old and unused data, so no need to worry about it exceeding memory limits. Instead, we can focus on putting there our thumbnails details, and the more often we request them, the higher change there will be, that we will get a cache hit. We need to remember, then Memcached is an in memory data storage, so our data might dissapear at any point. That's why it is still worth adding second ActiveRecord layer.

Configuring Rails to use Memcached as a cache storage

Before we go any further, we need to tell our app, that we want to store cached data in Memcached. To do so, please follow Dalli docs instructions. Dalli is the best Memcached ruby client there is and it can be easily integrated with Ruby on Rails framework.

Creating ActiveRecord dragonfly cache storage

Dragonfly docs example names the ActiveRecord cache model Thumb. We will name it DragonflyCache, since we might use it to store info not only about images. Here's an sample migration for this resource:

class CreateDragonFlyCache < ActiveRecord::Migration
  def change
    create_table :dragonfly_caches do |t|
      t.string :uid
      t.string :job
      t.timestamps
    end

    add_index :dragonfly_caches, :uid
    add_index :dragonfly_caches, :job
  end
end

Adding Memcached layer to a DragonflyCache model

We now have a single layer ActiveRecord cache storage for our app. We will use two Rails cache methods to add a second layer:

Rails.cache.read(key)
Rails.cache.write(key, value)

Cache read (hit)

The whole flow for cache read will be pretty simple:

1. Check if job signature is in Memcached and if so - return it (do nothing else)
2. If not, check if it can be found in SQL database and if so, store it in Memcached and return it
3. If not found, return nil

# @return [::DragonflyCache] df cache instance that has a uid that we want
#   to get and use
# @param [String] job_signature that acts as a unique identifier for uid
def read(job_signature)
  stored = Rails.cache.read(key(job_signature))

  return new(uid: stored) if stored

  stored = find_by(job: job_signature)
  Rails.cache.write(key(job_signature), stored.uid) if stored

  stored
end

You might notice a method called key. When working with Memcached, I like to prefix keys, so there won't be any cache collisions (scenario when different data is there).

This method is pretty simple:

# @return [String] key made from job signature and a prefix
# @param [String] job_signature for which we want to get a key
def key(job_signature)
  "#{PREFIX}_#{job_signature}"
end

Cache write (store)

Cache write will be easier. We just have to store job details in both SQL database and Memcached:

# Stores a given job signature with uid in a DB and memcache it
# It is used as a fallback persistent storage, when memcached key
# is not found
# @param [String] job_signature that acts as a unique identifier
# @param [String] uid that is equal to our file path under which
#   we have this certain thumb/file
# @raise [ActiveRecord::RecordInvalid] raised when something goes
#   really wrong (should not happen)
def write(job_signature, uid)
  Rails.cache.write(key(job_signature), uid)

  create!(
    uid: uid,
    job: job_signature
  )
end

Now, we can incorporate the code above into DragonflyCache model.

Full DragonflyCache model

# DragonflyCache stores informations about processed dragonfly files that we have
# @see http://markevans.github.io/dragonfly/cache/
class DragonflyCache < ActiveRecord::Base
  # Prefix for all memcached dragonfly job signatures
  PREFIX = 'dragonfly'

  class << self
    # @return [::DragonflyCache] df cache instance that has a uid that we want
    #   to get and use
    # @param [String] job_signature that acts as a unique identifier for uid
    def read(job_signature)
      stored = Rails.cache.read(key(job_signature))

      return new(uid: stored) if stored

      stored = find_by(job: job_signature)
      Rails.cache.write(key(job_signature), stored.uid) if stored

      stored
    end

    # Stores a given job signature with uid in a DB and memcached
    # It is used as a fallback persistent storage, when memcached key
    # is not found
    # @param [String] job_signature that acts as a unique identifier
    # @param [String] uid that is equal to our file path under which
    #   we have this certain thumb/file
    # @raise [ActiveRecord::RecordInvalid] raised when something goes
    #   really wrong (should not happen)
    def write(job_signature, uid)
      Rails.cache.write(key(job_signature), uid)

      create!(
        uid: uid,
        job: job_signature
      )
    end

    private

    # @return [String] key made from job signature and a prefix
    # @param [String] job_signature for which we want to get a key
    def key(job_signature)
      "#{PREFIX}_#{job_signature}"
    end
  end
end

Connecting our DragonflyCache model with Dragonfly

This part is really easy. We have to add the following config options to our Dragonfly initializer:

Dragonfly.app.configure do
  define_url do |app, job, opts|
    cached = DragonflyCache.read(job.signature)

    if cached
      app.datastore.url_for(cached.uid)
    else
      app.server.url_for(job)
    end
  end

  before_serve do |job, env|
    DragonflyCache.write(job.signature, job.store)
  end
end

That's all. Now your Dragonfly cache should hit DB less often.