monitoring – Closer to Code

To monitor MongoDB you can use many tools, some like MongoDB Management Service (MMS) are cloud based, some like Munin might be installed locally.

Today we will focus on setting up Munin to monitor MongoDB.

Getting started with Munin

If you don't know what Munin is, or how to install it on your platform, please refer to following articles:

This is not a tutorial on how to install Munin itself. I assume that from this point, you have Munin and Munin-node running on your system and that you see basic Munin stats charts.

MongoDB configuration

MongoDB provides a simple http interface listing information of interest to administrators. This interface may be accessed at the port with a numeric value 1000 more than the configured mongod port. The default port for the http interface is 28017 (description copy-pasted from here). By default it is not enabled but it is required by Munin MongoDB plugins, so we need to turn it on.

Warning! Keep in mind, that if you don't block it, it will listed on your public interface and it will be accessible by default from internet. Please use iptables to make it work only from localhost.

To enable it, edit your /etc/mongod.conf file, find httpinterface line and uncomment it (or set to true if set to false):

# vim /etc/mongod.conf
# Enable the HTTP interface (Defaults to port 28017).

httpinterface = true

After that you need to restart MongoDB:

# As a root or using sudo
/etc/init.d/mongod restart

[ ok ] Restarting database: mongod.

To test it, open http://localhost:28017/ (remember to replace localhost with your server host). You should see page similar to this one:

Installing Munin MongoDB plugins

Some of those plugins won't work out of the box, but we will take care of that later. For now let's focus on the install process:

# as a root or using sudo
git clone git://github.com/erh/mongo-munin.git ~/mongo-munin
cd ~/mongo-munin

# We copy all the plugins into munin plugins
cp mongo_* /usr/share/munin/plugins/

# We need to activate them now
ln -s /usr/share/munin/plugins/mongo_btree /etc/munin/plugins/
ln -s /usr/share/munin/plugins/mongo_conn /etc/munin/plugins/
ln -s /usr/share/munin/plugins/mongo_lock /etc/munin/plugins/
ln -s /usr/share/munin/plugins/mongo_mem /etc/munin/plugins/
ln -s /usr/share/munin/plugins/mongo_ops /etc/munin/plugins/

cd ~
# We don't need this anymore
rm -rf mongo-munin

# Restarting munin-node...
/etc/init.d/munin-node restart

After restarting munin-node and waiting few minutes, we should have a new section in your munin web ui (mongodb). Part of the graphs won't display any data, but you should at least see the mongodb section.

Debugging and fixing Munin MongoDB broken plugins

Some of the plugins (like mongo_lock) won't work without a little tuneup. If you see graphs similar to this (without any data and -nan everywhere), then most likely those plugins aren't working.

To check each of the plugins, you need to run munin-run with appropriate plugin name (as root):

mongo_btree

# munin-run mongo_btree
Traceback (most recent call last):
  File "/etc/munin/plugins/mongo_btree", line 61, in <module>
    doData()
  File "/etc/munin/plugins/mongo_btree", line 35, in doData
    for k,v in get().iteritems():
  File "/etc/munin/plugins/mongo_btree", line 32, in get
    return getServerStatus()["indexCounters"]["btree"]
KeyError: 'btree'

mongo_conn

# munin-run mongo_conn
connections.value 0

mongo_lock

# munin-run mongo_lock
Traceback (most recent call last):
  File "/etc/munin/plugins/mongo_lock", line 54, in <module>
    doData()
  File "/etc/munin/plugins/mongo_lock", line 34, in doData
    print name + ".value " + str( 100 * getServerStatus()["globalLock"]["ratio"] )
KeyError: 'ratio'

mongo_mem

# munin-run mongo_mem
resident.value 37748736
virtual.value 376438784
mapped.value 83886080

mongo_ops

# munin-run mongo_ops
getmore.value 0
insert.value 1
update.value 0
command.value 1
query.value 53
delete.value 0

Errors summary

Based on the plugins output, we can see, that 2 out of 5 plugins aren't working:

mongo_btree
mongo_lock

They aren't working because the MongoDB HTTP interface response slightly differs from what it used to be when the plugins were developed.

Patching mongo_btree plugin

Apply following patch to /usr/share/munin/plugins/mongo_btree:

@@ -29,7 +29,7 @@ def getServerStatus():
     return json.loads( raw )["serverStatus"]
 
 def get():
-    return getServerStatus()["indexCounters"]["btree"]
+    return getServerStatus()["indexCounters"]
 
 def doData():
     for k,v in get().iteritems():

After patching, execute munin-run mongo_btree:

# munin-run mongo_btree
missRatio.value 0
resets.value 0
hits.value 2
misses.value 0
accesses.value 2

Patching mongo_lock plugin

Apply following patch to /usr/share/munin/plugins/mongo_lock:

@@ -31,7 +31,7 @@ def getServerStatus():
 name = "locked"
 
 def doData():
-    print name + ".value " + str( 100 * getServerStatus()["globalLock"]["ratio"] )
+    print name + ".value " + str( 100 * round(float(getServerStatus()["globalLock"]["lockTime"]["$numberLong"])/float(getServerStatus()["globalLock"]["totalTime"]["$numberLong"]), 8) )
 
 def doConfig():

After patching, execute munin-run mongo_lock:

# munin-run mongo_lock
locked.value 0.000785

After that (and few minutes) all of the stats should be working.

When I was moving Senpuu.net to a new location, I've wanted to preserve my Munin stats (since my new server have almost identical configuration they will be very useful to me). Unfortunately moving RRD files between different architectures is not possible. To copy Munin data to a new server with a different architecture, we need to use rrdtool (dump and restore).

# Error example from /var/log/munin/munin-node.log
This RRD was created on another architecture

rrd dump and restore

Lets quote a man for rrdtool:

Dump:

Dump the contents of an RRD in plain ASCII. In connection with restore you can use this to move an RRD from one computer architecture to another.

Restore:

Restore an RRD in XML format to a binary RRD.

Now everything looks better. We can easily dump RRD files to XML, move them to the other server and then easily restore them to RRD format.

To dump all RRD files for given node, you need to get (as root) to RRD files dir (by default it should be somewhere near /var/lib/munin) and execute following code:

cd /var/lib/munin/senpuu
# Create a dir for RRD dumps
mkdir ./../rrd_dump
 for i in ./*.rrd;do rrdtool dump $i ./../rrd_dump/$i.xml;done

The above code will create XML dumps in appropriate directory. When this is finished, you need to download all the files and upload them to your new server. Then execute:

# Execute this on a new server
cd /var/lib/munin/rrd_dump
for i in ./*.xml; do rrdtool restore "$i" "../senpuu/${i%.xml}"; done

after this you might reset munin-node just to be sure that everything will work fine:

/etc/init.d/munin-node restart

Now you should have your old charts up and running.

Tag: monitoring

MongoDB monitoring with Munin – Setting up Munin to work with MongoDB