Tag: architecture

Using multiple MongoDB databases instead of one – performance check

I'm starting to develop a new application. Can't say what it is, but it perfectly fits MongoDB Document Oriented Database approach. Everything is great. except of small detail - I don't want to store everything in one database. Of course I could use collections and embedded documents to organize whole nested structures and keep users stuff separated, although it would make source code much more complicated than it should be. Instead I've decided to use one MongoDB database per user. That way I can separate users data and I don't need to worry about scoping it out. There will be a gateway, that will authorize incoming requests to a proper database.

Gateway

Schema is pretty straightforward and the only thing that was bothering me was the multi-db switching performance. I've decided to make a simple benchmark that would test if there's a difference when using one or many databases. Results are promising. It seems, that there's only around 5% (5.3% exactly) performance loss when using many databases instead of one. 5% is a difference level that I can easily accept. To be honest I think, I will gain much more especially when I will have a lot of data. Lets say I have 100 customers with 100 000 000 records. With one database I would have to query all of it. With separate databases, I will have to query only 1% of it.

Below you can see performance difference when querying one vs many MongoDB databases.

Dbs I will definitely go with that approach and I will try to keep you posted.

Note: This is not a full-pro-extremely accurate long-time test - more like a proof of concept. Keep that in mind ;)

Munin: This RRD was created on another architecture – migrating between 32 and 64bit systems

When I was moving Senpuu.net to a new location, I've wanted to preserve my Munin stats (since my new server have almost identical configuration they will be very useful to me). Unfortunately moving RRD files between different architectures is not possible. To copy Munin data to a new server with a different architecture, we need to use rrdtool (dump and restore).

# Error example from /var/log/munin/munin-node.log
This RRD was created on another architecture

rrd dump and restore

Lets quote a man for rrdtool:

Dump:

Dump the contents of an RRD in plain ASCII. In connection with restore you can use this to move an RRD from one computer architecture to another.

Restore:

Restore an RRD in XML format to a binary RRD.

Now everything looks better. We can easily dump RRD files to XML, move them to the other server and then easily restore them to RRD format.

To dump all RRD files for given node, you need to get (as root) to RRD files dir (by default it should be somewhere near /var/lib/munin) and execute following code:

cd /var/lib/munin/senpuu
# Create a dir for RRD dumps
mkdir ./../rrd_dump
 for i in ./*.rrd;do rrdtool dump $i ./../rrd_dump/$i.xml;done

The above code will create XML dumps in appropriate directory. When this is finished, you need to download all the files and upload them to your new server. Then execute:

# Execute this on a new server
cd /var/lib/munin/rrd_dump
for i in ./*.xml; do rrdtool restore "$i" "../senpuu/${i%.xml}"; done

after this you might reset munin-node just to be sure that everything will work fine:

/etc/init.d/munin-node restart

Now you should have your old charts up and running.

Copyright © 2022 Closer to Code

Theme by Anders NorenUp ↑