server | Closer to Code

Sometimes we want to perform some sort of tasks that consume whole available IO bandwidth. This may lead to some unexpected behaviours from our OS. OS might even kill the given process due to resource lack.

Lets take an example. We want to tar.gz a huge directory with a lot of files in it. Our machine also have a web-server which serves several sites. If we start "taring" our directory, it might lead to timeouts on server (it won't be able to respond as fast as we would expect). On the other hand, we don't care so much about the time needed to create archive file. We always can throw it in screen and detach it.

# Standard approach - will slow IO response time
tar -cvf ./dir.tar ./dir

pv to the rescue!

To slow things down to a tolerable level we will use pv tool. pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA. It can also limit the speed of incoming data, if used wisely.

To tar a file with a given speed (in MB) we need to use following command:

tar -chf - ./dir | pv -L 2m > ./dir.tar

Above example will allow us to tar ./dir with max speed 2MB/s.

We could use the same method to slow down a mysqldump method:

mysqldump --add-drop-table -u user -p password -h host db_name | bzip2 -c > ./dump.sql.bz2

Yesterday I've started (after trying to restart Passenger) to see this exception coming out from Apache error.log:

[ pid=5098 thr=5896300 file=utils.rb:176 time=2012-08-31 19:00:45.293 ]:
*** Exception PhusionPassenger::UnknownError in 
PhusionPassenger::ClassicRails::ApplicationSpawner ((<unknown>): 
found a tab character that violate intendation while scanning a plain scalar at line 49 in 11
(Psych::SyntaxError)) (process 5098, thread #<Thread:0x00000000b3f0d8>):

Passenger ApplicationSpawner was endlessly creating new instances of my app, because the previous onces needed to be shutdown because of this error. It was quite obvious for me, that the error came from invalid yaml file (Psych::SyntaxError), however stack trace looked like this:

from /usr/local/lib/ruby/1.9.1/psych.rb:203:in `parse'
from /usr/local/lib/ruby/1.9.1/psych.rb:203:in `parse_stream'
from /usr/local/lib/ruby/1.9.1/psych.rb:151:in `parse'
from /usr/local/lib/ruby/1.9.1/psych.rb:127:in `load'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:114:in `initialize'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:71:in `new'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:71:in `instance'
from /gems/settingslogic-2.0.8/lib/settingslogic.rb:77:in `method_missing'
from /home/httpd/v5.senpuu.net/releases/20120829194758/config/initializers/resque.rb:5
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load'
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `block in load'
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in `load_dependency'
from /gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load'
from /gems/railties-3.2.8/lib/rails/engine.rb:588:in `block (2 levels) in <class:Engine>'
from /gems/railties-3.2.8/lib/rails/engine.rb:587:in `each'
from /gems/railties-3.2.8/lib/rails/engine.rb:587:in `block in <class:Engine>'
from /gems/railties-3.2.8/lib/rails/initializable.rb:30:in `instance_exec'
from /gems/railties-3.2.8/lib/rails/initializable.rb:30:in `run'
from /gems/railties-3.2.8/lib/rails/initializable.rb:55:in `block in run_initializers'
from /gems/railties-3.2.8/lib/rails/initializable.rb:54:in `each'
from /gems/railties-3.2.8/lib/rails/initializable.rb:54:in `run_initializers'
from /gems/railties-3.2.8/lib/rails/application.rb:136:in `initialize!'
from /gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'

Ok, bot how to find the file with invalid yaml syntax? In my case, this exception was raised when SettingsLogic could not parse the given yaml file. So I've decided to hookup into it, just before the parsing, so I would be able to print out the invalid file. To do so I've edited /gems/settingslogic-2.0.8/lib/settingslogic.rb file. If you go to line 114 of this file, you will see something like this:

hash = YAML.load(ERB.new(File.read(hash_or_file)).result).to_hash
if self.class.namespace
  hash = hash[self.class.namespace] or return missing_key("Missing setting '#{self.class.namespace}' in #{hash_or_file}")
end

So in order to print the invalid file name we just need to add:

p hash_or_file

just before the YAML.load method execution. After this, it was really easy to find an invalid line in my yaml file.

Tag: server

Slowing down (limiting) tar, mysqldump or other processes to save IO bandwidth

pv to the rescue!

PhusionPassenger::UnknownError: Psych::SyntaxError – tab character that violate intendation