Tag: Pagination

How to use Ransack helpers with MongoDB and Mongoid on Rails 4.0

Ransack is a rewrite of MetaSearch. It allows you to create search forms and sorting for your ActiveRecord data. Somehow I really miss that, when working with Mongoid. To be honest I can even live without the whole Ransack search engine, but I liked the UI helpers for search forms and sorting data.

I've really wanted to be able to do something like that:

@users = current_app.users.order(params[:q]).page(current_page)

and

%tr.condensed{:role => "row"}
  %th= sort_link @search, :guid, 'User id'
  %th= sort_link @search, :email, 'Email address'
  %th= sort_link @search, :current_country, 'Country'
  %th= sort_link @search, :current_city, 'City'

with Mongoid documents. It turns out, that if you don't need really sophisticated stuff, you can use Ransack helpers even with Mongoid. In order to use search/sort UI features, you need @search Ransack model instance. It is based on ActiveRecord data and unfortunately we don't have any models that would suit our needs. That's why we need to create a dummy one:

module System
  # In order to use Ransack helpers and engine for non AR models, we need to 
  # initialize @search with AR instance. To prevent searching (this is just a stub)
  # we do it usign ActiveRecord dummy class that is just a blank class
  class ActiveRecord < ActiveRecord::Base

    def self.columns
      @columns ||= []
    end

  end
end

This is enough to use it in controllers and views:

# In order to use Ransack helpers and engine for non AR models, we need to 
# initialize @search with AR instance. To prevent searching (this is just a stub)
# we do it with none
def search
  @search ||= ::System::ActiveRecord.none.search(params[:q])
end

It is a dummy ActiveRecord class, that will always return empty scope (none), but it is more than enough to allow us to work with UI helpers. To handle ordering of Mongoid document collection, we could use following code:

module System
  module Mongoid
      # Order engine used to order mongoid data
      # It is encapsulated in segment module, because we store here all the
      # things that are related to filtering/ordering data
      #
      # Since it returns the resource/scope that was provided, modified
      # accordingly to sorting rules, this class can be used in defined
      # scopes and it will work with scope chainings
    class Orderable
      # @param resource [Mongoid::Criteria] Mongoid class or a subset 
      #   of it (based on Mongoid::Criteria)
      # @param rules [Hash] hash with rules for ordering
      # @return [Mongoid::Criteria] mongoid criteria scope
      # @example
      #   System::Mongoid::Orderable.new(current_app.users, params[:q])
      def initialize(resource, rules = {})
        @rules = rules
        @resource = resource
      end

      # Applies given sort order if there is one.
      # If not, it will do nothing
      # @example Ordered example scope
      #   scope :order, -> order { System::Mongoid::Orderable.new(self, order).apply }
      def apply
        order unless @rules.blank?
        @resource.criteria
      end

      private

      # Sets given order based on order rules (if there are any)
      # or it will do nothing if no valid order rules provided
      def order
        order = "#{@rules[:s]}".split(' ')
        @resource = order.blank? ? @resource : @resource.order_by(order)
      end
    end
  end
end

This can be used to create scopes like that:

class User
  include Mongoid::Document
  include Mongoid::Attributes::Dynamic

  # Scope used to set all the params from ransack order engine
  # @param [Hash, nil] hash with order options or nil if no options
  # @return [Mongoid::Criteria] scoping chain
  # @example
  #   User.order(params[:q]).to_a #=> order results
  scope :order, -> order { System::Mongoid::Orderable.new(self, order).apply }
end

Ruby, Mongoid and memory leaks – Identity map problem

Where is my memory?

Recently, when browsing large dataset from MongoDB using Padrino and Thin, Ruby started to have memory leaks. After each request it grew approximately 2-5 MB.

I've started debugging by putting following line in my action, to see memory usage increase per request:

puts 'RAM USAGE: ' + `pmap #{Process.pid} | tail -1`[10,40].strip

Results:

RAM USAGE: 796156K
RAM USAGE: 798284K
RAM USAGE: 798824K
RAM USAGE: 799088K
RAM USAGE: 799900K
RAM USAGE: 799900K
RAM USAGE: 812044K
RAM USAGE: 816152K
RAM USAGE: 816292K
RAM USAGE: 816836K
RAM USAGE: 818956K
RAM USAGE: 819088K
RAM USAGE: 830572K
RAM USAGE: 884604K
RAM USAGE: 887648K
RAM USAGE: 892800K
RAM USAGE: 897160K
RAM USAGE: 906960K

As you can see it grows rapidly. When looking at htop things get even worse:

88,4 MB
93,9 MB
97,5 MB
99,2 MB
109,4 MB
113,4 MB
122,7 MB
127,1 MB
...
1,2 GB!

It was definitely too much! Memory consumption reached it's limits and everything slowed down.

I knew that it had something to do with this line:

@analyses = Analysis.finished.page(params[:page] ||= 1).per(10)

Kaminari?

At the beginning I've suspected Kaminari and its pagination engine, however it is just a more complex layer covering some scopes. To check this I've removed Kaminari:

@analyses = Analysis.finished.skip(((params[:page] ||= 1)-1)*10).limit(10)

Unfortunately nothing good happened and memory consumption kept growing with same speed. Interesting is that, when I've turned off all MongoDB indexes:

db.collection1.dropIndexes()
db.collection2.dropIndexes()
...
db.collectionN.dropIndexes()

memory usage grew much slower than before. So WTF?

Identity map!

Finally I've discovered damn source of my problem. It was identity map in Mongoid. What is identity map?

The identity map in Mongoid is a current aid to assist with excessive database queries in relations, and is necessary for eager loading to work. (...) When a document is now loaded from the database, is is automatically added to the identity map by it's class and id. Subsequent request for that document by it's id will not hit the database, but rather pull the document back from the identity map itself. It's primary function in this capacity is to aid in cutting down queries for belongs_to relations when iterating over the parents.

Seems like identity map was never cleared (or it has a memory leak bug in it). Adding:

use Rack::Mongoid::Middleware::IdentityMap

didn't help at all so I've just turned identity map off:

mongo.identity_map_enabled = false

and everything went back to normal. Interesting thing is that identity map in ActiveRecord is by default turned off in Rails because it's known to cause similar problems.

Copyright © 2022 Closer to Code

Theme by Anders NorenUp ↑