Running with Ruby

Tag: SettingsLogic

Multiple Sidekiq processes for multiple Rails/Sinatra applications – namespacing

Sidekiq is a great background processing tool that can be used with many applications deployed on the same server. In order to make it work without any issues or collisions, you need to define namespace for each Sidekiq process group within single application. To do so, you need to create an initializer that will contain both: server connection details and a namespace name:

Sidekiq.configure_server do |config|
  config.redis = {
    url: 'redis://localhost:6379',
    namespace: 'my_app_name_production'
  }
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: 'redis://localhost:6379',
    namespace: 'my_app_name_production'
  }
end

Keep in mind, that you need to provide both url and namespace for client and server. I like to hook it up with SettingsLogic gem:

Sidekiq.configure_server do |config|
  config.redis = System::Settings.sidekiq.redis
end

Sidekiq.configure_client do |config|
  config.redis = System::Settings.sidekiq.redis
end

After you deploy all the apps, you can also easily check if everything is working, by simply executing redis console:

$: redis-cli

and running following command:

redis 127.0.0.1:6379> keys *NAMESPACE*

For example:

redis 127.0.0.1:6379> keys *api*
 1) "youtube_api_v2_production:stat:failed:2014-07-04"
 2) "youtube_api_v2_production:MIA-VPS-VM0974:770"
 3) "facebook_api_production:processes"
 4) "facebook_api_production:queues"
 5) "twitter_api_production:MIA-VPS-VM0974:32655"
 6) "youtube_api_v2_production:queues"
 7) "facebook_api_production:stat:processed"
 8) "youtube_api_v2_production:stat:processed:2014-07-04"
 9) "youtube_api_v2_production:stat:processed"
10) "youtube_api_v2_production:processes"
11) "facebook_api_production:MIA-VPS-VM0974:459"
12) "facebook_api_production:stat:processed:2014-07-04"
13) "twitter_api_production:stat:processed:2014-07-04"
14) "twitter_api_production:stat:processed"
15) "twitter_api_production:processes"
16) "twitter_api_production:queues"
17) "youtube_api_v2_production:stat:failed"

Above you can see multiple queues for multiple namespaced apps.

PhusionPassenger::UnknownError: Psych::SyntaxError – tab character that violate intendation

Yesterday I’ve started (after trying to restart Passenger) to see this exception coming out from Apache error.log:

[ pid=5098 thr=5896300 file=utils.rb:176 time=2012-08-31 19:00:45.293 ]:
*** Exception PhusionPassenger::UnknownError in 
PhusionPassenger::ClassicRails::ApplicationSpawner ((<unknown>): 
found a tab character that violate intendation while scanning a plain scalar at line 49 in 11
(Psych::SyntaxError)) (process 5098, thread #<Thread:0x00000000b3f0d8>):

Passenger ApplicationSpawner was endlessly creating new instances of my app, because the previous onces needed to be shutdown because of this error. It was quite obvious for me, that the error came from invalid yaml file (Psych::SyntaxError), however stack trace looked like this:

from /usr/local/lib/ruby/1.9.1/psych.rb:203:in `parse'
from /usr/local/lib/ruby/1.9.1/psych.rb:203:in `parse_stream'
from /usr/local/lib/ruby/1.9.1/psych.rb:151:in `parse'
from /usr/local/lib/ruby/1.9.1/psych.rb:127:in `load'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:114:in `initialize'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:71:in `new'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:71:in `instance'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:77:in `method_missing'
from /home/httpd/v5.senpuu.net/releases/20120829194758/config/initializers/resque.rb:5
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load'
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `block in load'
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in `load_dependency'
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load'
from /gems/railties-3.2.8/lib/rails/engine.rb:588:in `block (2 levels) in <class:Engine>'
from /gems/railties-3.2.8/lib/rails/engine.rb:587:in `each'
from /gems/railties-3.2.8/lib/rails/engine.rb:587:in `block in <class:Engine>'
from /gems/railties-3.2.8/lib/rails/initializable.rb:30:in `instance_exec'
from /gems/railties-3.2.8/lib/rails/initializable.rb:30:in `run'
from /gems/railties-3.2.8/lib/rails/initializable.rb:55:in `block in run_initializers'
from /gems/railties-3.2.8/lib/rails/initializable.rb:54:in `each'
from /gems/railties-3.2.8/lib/rails/initializable.rb:54:in `run_initializers'
from /gems/railties-3.2.8/lib/rails/application.rb:136:in `initialize!'
from /gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'

Ok, bot how to find the file with invalid yaml syntax? In my case, this exception was raised when SettingsLogic could not parse the given yaml file. So I’ve decided to hookup into it, just before the parsing, so I would be able to print out the invalid file. To do so I’ve edited /gems/settingslogic-2.0.8/lib/settingslogic.rb file. If you go to line 114 of this file, you will see something like this:

hash = YAML.load(ERB.new(File.read(hash_or_file)).result).to_hash
if self.class.namespace
  hash = hash[self.class.namespace] or return missing_key("Missing setting '#{self.class.namespace}' in #{hash_or_file}")
end

So in order to print the invalid file name we just need to add:

p hash_or_file

just before the YAML.load method execution. After this, it was really easy to find an invalid line in my yaml file.

Copyright © 2018 Running with Ruby

Theme by Anders NorenUp ↑