Personal Workflow Blog

Spinoffs in the munin ecosystem

Steve Schnepp — Sat, 13 Apr 2013 14:42:00 +0200

KISS is the core design of Munin

Munin's greatest strength is its very KISS architecture. It therefore gets many things right, such as a huge modularity.

Each component (master/node/plugin) has a simple API to communicate with the others.

Spin-offs ...

I admit that the master, even the node, have convoluted code. In fact some rewrites already do exist.

... are welcomed ...

And they are a really good thing, as it enables rapid prototyping on things that the stock munin has (currently) trouble to do.

The stock munin is a piece of software that many depend upon, so it has to move at a much slower pace than one does want, even me. As much as I really want to add many many features to it, I still have to take extra care that it doesn't break stuff, even the least known features.

So I take munin off-springs very seriously and even offer as much help as I can in order for them to succeed.

... because they are very valuable in the long term

In my opinion competition is only short bad in the short term, and in the long term they usually add significant value to the whole ecosystem. That said, there's always a risk to become slowly irrelevant, but I think that's the real power of open-source's evolutionary paradigm : embrace them or become obsolete and get replaced.

Since, if someone takes the time to author a competitor that has a real threat potential, it mostly means that there's a real itch to scratch and that many things are to be learnt.

Different layers of spin-offs

The munin ecosystem is divided in 3 main categories, obviously related to the 3 main components of munin : master, node & plugin.

Plugins

That's the most obvious part as custom plugins are the real bread and butter of munin.

Stock plugins are mostly written in Perl or POSIX shell, as Perl is munin's own language and POSIX shell is ubiquitous. That fact is acknowledged by the fact that core munin provides 2 libraries (Perl & Shell) to help plugin authoring.

So, it's quite natural that each mainstream language has grown its own plugin library. Some language even have two of them.

C

Some plugins got even rewritten in plain C, as it was shown that shell plugins do have a significant impact on very under-powered nodes, such as embedded routers.

Node

This component is very simple. Yet, it has to be run on all the nodes that one wants to monitor. It is currently written in Perl, and while that's not an issue on UNIX-like systems, it can be quite problematic on embedded ones

Simple munin

The official package comes with a POSIX shell rewrite that has to be run from inetd. It is quite useful for embedded routers like OpenWRT, but still suffers from an hard dep on POSIX shell and inetd.

SNMP

SNMP is another way to monitor nodes. While it works really well, it mostly suffers the fact that its configuration is quite different of the usual way, so I guess some things will change on that side.

Win32 ports

Win32 has long been a very difficult OS to monitor, as it doesn't offer much of the UNIX-esque features. Yet the number of win32 nodes that one wants to monitor is quite high, as it makes munin one the few systems that can easily monitor heterogeneous systems.

Therefore, while you can install the stock munin-node, several projects emerged. We decided to adopt munin-node-win32.

Android

There's also a dedicated node for Android. It makes sense, given that the Android is yet Linux-derived, but lacks Perl, and is a Java mostly platform. This node also has some basic capabilities of pushing data to the master instead of the usual polling.

This is specially interesting given the fact that Android nodes are usually loosely connected, so the node spools values itself and pushes them when it recovers connectivity.

Note that this is specifically an aspect that is currently lacking in munin, and I'm planning to address it in the 2.1 series. So thanks to its author for showing a relevant use-case.

C

That's my last experiment. It started with a simple question : how difficult would it be to code a fairly portable version of the node ?

It turned out that it wasn't that difficult. I'm even asking myself about eventually replacing the win32 specific port with this one, as the code is much simpler. The win32 node has several plugin built-in mostly due to platform specifics. I still have to find a way to work my way around it, but it's in quite good shape.

This post was originally done to promote it, but while writing it I noticed that the ecosystem deserved a post on its own. So I'll write another one, specific to the C port of munin-node and plugins.

Master

The master is the most complex component. So rewrites of it won't happen as-is. They usually take the form of a bridge between the munin protocol and another graphing system, such as Graphite.

Clients

There are also client libraries that are able to directly query munin nodes, to be able to reuse the vast ecosystem. Languages are various, from the obvious Python to Ruby, along with a quite modern node.js one.

Do not fear git rebase : make snapshots !

Steve Schnepp — Thu, 04 Apr 2013 20:48:00 +0200

Git is a nice version system, but some commands are destructrive, such as rebase.

Here is a script to have a safety net, and free backups !

#! /bin/sh
# Script to snaphot a git repo
SNAP_VERSION=$(date +%s)
BUNDLE_NAME=$(basename $( pwd )).${SNAP_VERSION}.git.bundle
git bundle create ../${BUNDLE_NAME} --all
git remote add snap-${SNAP_VERSION} ../${BUNDLE_NAME}
git fetch -p snap-${SNAP_VERSION}

Usage is very easy. If you want to restore your current branch to the master one you made earlier.

git reset --hard snap-1365068411/master

When having good relationships with package maintainers can also be a curse

Steve Schnepp — Sun, 24 Feb 2013 13:23:00 +0100

I advise every user to only use the packaged version of munin. Here's a short article to explain the background of my reluctance to ask for users to directly use the official tarball.

I have become upstream of munin a while ago now. As such, I'm in contact with package maintainers. They take the official releases and cram it into their own distribution of choice^[1].

I have to admit that the various epic war stories read throughout the web about upstream vs packagers are very far from the truth here. They are a charm to work with. Often challenging and demanding, but always because there's a real need. And that's quite a good thing, as I'm still a rookie in term of open source software management. Therefore I'm quite grateful when they gently pinpoint my mistakes^[2].

Yet, this nice team comes with a price. Since we mostly hang out on IRC together, there is way much inter-distro communication than on other software. But I'm the sole owner of the tarball distro .

Yet, as I don't like to build everything from source, I obviously use a distro. There, since the packaging is very nicely done, I don't feel to take the hassle of using my own "tarball" to test them. I just build a package for my distro out of the release code.

That's also a curse, as I admit that I although I test the code, I only seldom test the packaging. This means that I cannot really advise someone on using the tarball, nor directly git code as even I don't do it.

But, that said, I still think I'm the luckiest upstream around. Thanks guys !

Notes

[1] Be it linux-based like Gentoo, Redhat..., BSD-based as FreeBSD, OpenBSD..., or even multi-kernel based as Debian

[2] Defaulting to CGI graphics was a move that was way too premature, end-user wise. So thanks to them, it defaults to cron again

Avoid those milli-hits in Munin

Steve Schnepp — Fri, 01 Feb 2013 01:47:00 +0100

A recurring question on IRC is : why do I have 500 million hit/s in my graph ?.

Turns out that they are really seeing 500m hit/s, and that lower-case m means milli, and not Mega as specified in the Metric system. This is automatically done by RRD.

To avoid this you should just specify graph_scale no as specified.

Waiting for Munin 2.0 - Break the 5 minutes barrier !

Steve Schnepp — Thu, 12 Jul 2012 21:58:00 +0200

Every monitoring software has a polling rate. It is usually 5 min, because it's the sweet spot that enables frequent updates yet still having a low overhead.

Munin is not different in that respect : it's data fetching routines have to be launched every 5 min, otherwise you'll face data loss. And this 5 min period is deeply grained in the code. So changing it is possible, but very tedious and error prone.

But sometimes we need a very fine sampling rate. Every 10 seconds enables us to track fast changing metrics that would be averaged out otherwise. Changing the whole polling process to cope with a 10s period is very hard on hardware, since now every update has to finish in these 10 seconds.

This triggered an extension in the plugin protocol, commonly known as supersampling.

Supersampling

Overview

The basic idea is that fine precision should only be for selected plugins only. It also cannot be triggered from the master, since the overhead would be way too big.

So, we just let the plugin sample itself the values at a rate it feels adequate. Then each polling round, the master fetches all the samples since last poll.

This enables various constructions, mostly around streaming plugins to achieve highly detailed sampling with a very small overhead.

Notes

This protocol is currently completely transparent to munin-node, and therefore it means that it can be used even on older (1.x) nodes. Only a 2.0 master is required.

Protocol details

The protocol itself is derived from the spoolfetch extension.

Config

A new directive is used, update_rate. It enables the master to create the rrd with an adequate step.

Omitting it would lead to rrd averaging the supersampled values onto the default 5 min rate. This means data loss.

Notes

The heartbeat has always a 2 step size, so failure to send all the samples will result with unknown values, as expected.

The RRD file size is always the same in the default config, as all the RRA are configured proportionally to the update_rate. This means that, since you'll keep as much data as with the default, you keep it for a shorter time.

Fetch

When spoolfetching, the epoch is also sent in front of the value. Supersampling is then just a matter of sending multiple epoch/value lines, with monotonically increasing epoch. Note that since the epoch is an integer value for rrdtool, the smallest granularity is 1 second. For the time being, the protocol itself does also mandates integers. We can easily imagine that with another database as backend, an extension could be hacked together.

Compatibility with 1.4

On older 1.4 masters, only the last sampled value gets into the rrd.

Sample implementation

The canonical sample implementation is multicpu1sec, a contrib plugin on github. It is also a so-called streaming plugin.

Streaming plugins

These plugins fork a background process when called that streams a system tool into a spool file. In multipcu1sec, it is the mpstat tool with a period of 1 second.

Undersampling

Some plugins are on the opposite side of the spectrum, as they only need a lower precision.

It makes sense when :

data should be kept for a very long time
data is very expensive to generate and it doesn't vary fast.

Enhance RRD I/O performance in Munin 1.4 and Scale

Steve Schnepp — Mon, 20 Jun 2011 09:42:00 +0200

As with most of the RRD-based monitoring software (Cacti, Ganglia, ...), it is quite difficult to scale.

The bad part is that updating lots of small RRD files seems like pure random I/O to the OS as stated in there documentation.

The good part is that we are not alone, and therefore the RRD developers did tackle the issue with rrdcached. It spools the updates, and flushs them to disk in a batched manner, or when needed by a rrd read command such as graphing. That's why it is scales well when using CGI graphing. Otherwise, munin-graph will read every rrd, and therefore force a flush on all the cache.

And the icing on the cake is that, although it is only fully integrated to munin 2.0, you can use it right away in the 1.4.x series.

You only need to define the environment variable RRDCACHED_ADDRESS while running the scripts accessing the RRDs.

Then, you have to remove the munin-graph part of the munin-cron and run it on its own line. Usually only every hour or so, to be able to accumulate data in rrdcached before flushing it all to disk when graphing.

Updating to 2.0 is also an option to have a real CGI support. (CGI on 1.4 is existing but has nowhere decent performance).

Autovivification in Perl : Great Idea but also Huge Trap - Another Leaking Abstraction...

Steve Schnepp — Thu, 16 Jun 2011 20:20:00 +0200

Autovivification is one of Perl's really great design success.

It all comes to you don't need to worry about existence before dereferencing something.

That means, for setting a nested hash, you only need to write :

$h->{foo}{bar} = "value";

And that will work out of the box. Perl will happily create all the data-structure for you.

So, now a little coding test, what does the following code output ?

my $a;

if ($a->{foo}{bar}) {
   print "Found foo/bar\n";
}

if ($a->{foo}) {
   print "Found foo\n";
}

Naively, it shouldn’t output anything, right ?

Not so fast. Upon a careful read of Perl will happily create all the data-structure for you, we can put some emphasis on one word : Perl will happily create all the data-structure for you.

That might be just perfect, except that Perl creates it whenever it needs it, even if it is only for reading.

And now you understand the catch : a read operation can result in a write one.

As Uncle Ben (from SpiderMan) said^[1] : With Great Power Comes Great Responsibility.

Dagfinn Ilmari Mannsåker showed me a nice autovivification module on CPAN that fixes this behavior, and enables a fine tuning of this process.

I really think the fact that creation also happen when querying the value is a real bug in Perl itself, or at least a bug in the design of the feature.

Notes

[1] Voltaire, Franklin D. Roosevelt and other said something very similar, but they are not as geeky.

Waiting for Munin 2.0 - Keep more data with custom data retention plans

Steve Schnepp — Mon, 23 Aug 2010 18:47:00 +0200

RRD is Munin's backbone.

Munin keeps its data in an RRD database. It's a wonderful piece of software, designed for this very purpose : keep an history of numeric data.

All you need is to tell RRD for how long and the precision you want to keep your data. RRD manages then all the underlying work : pruning old data, averaging to decrease precision if needed, ...

Munin automatically creates the RRD databases it needs.

1.2 - Only one set

In 1.2, every database creation was done with the same temporal & precision parameters. Since the output parameters were constant (day, week, month, year graphs), there were little need to have a different set of parameters.

1.4 - 2 sets : normal & huge

In 1.4, various users showed their need to have different graphing outputs, and began to hack around Munin's fixed graphing. It became rapidly obvious that the 1.2 preset wasn't a fit for everyone.

Therefore a huge dataset was available to be able to extend the finest precision (5min) to the whole Munin timeframe. This comes at a price though : more space is required, and the graph generation is slower, specially when generating the yearly one, since more data has to be read and analysed.

The switch is done for the whole munin installation by changing the system-wide graph_data_size, although already created rrd databases aren't changed. It is then even possible for a user to pre-customize the rrd file. Munin will then happily uses them transparently thanks to the RRD layer.

Manual overriding

Altering the RRD files after it is created is possible, but not as simple. Standard export & import from RRD take the structure with it. So data has to be moved around with special tools. rrdmove is my attempt to create such a tool. It copies data between 2 already existing RRD files, even asking RRD to interpolate the data when needed.

2.0 - Full control

Starting with 2.0, the parameter graph_data_size is per service. It also has a special mode : custom. Its format is very simple :

 
graph_data_size custom FULL_NB, MULTIPLIER_1 MULTIPLIER_1_NB, ... MULTIPLIER_NMULTIPLIER_N_NB
graph_data_size custom 300, 15 1600, 30 3000

The first number is the number of data at full resolution. Then usually it comes gradually decreasing resolution.

A decreasing resolution has 2 usages :

Limit the space consumption : keeping full resolution for the whole period (default : 5min for 2 years) is sometime too precise.
Increase performance : RRD will choose the best fitting resolution to generate its graphs. Already aggregated data is faster to compute.

Waiting for Munin 2.0 - Native SSH transport

Steve Schnepp — Mon, 12 Jul 2010 14:11:00 +0200

In the munin architecture, the munin-master has to connect to the munin-node via a very simple protocol and plain TCP.

This has several advantages :

Very simple to manage & install
Optional SSL since 1.4 enabling secure communications
Quite simple firewall rules.

It has also some disadvantages :

A new listening service means a wider exposure
The SSL option might add some administrative overhead (certificates management, ...)
A native protocol isn't always covered by all firewall solutions
Some organisations only authorize a few protocols to simplify audits (ex: only SSH & HTTPS)

Native SSH

Theses down points may be solved by encapsulation over SSH, but it can be a tedious task to maintain if the number of hosts increases.

Therefore 2.0 introduces the concept of a native SSH transport. Its usage is dead simple : replace the address with an ssh:// URL-like one.

The node still has to be modified to communicate with stdin/stdout instead of a network socket. For now, only pmmn and munin-async are able to provide such a node.

Configuration

The URL is quite self-explanatory as shown in the example below :

[old-style-host]
    address host.example.com

[new-style-host]
    address ssh://munin-node-user@host.example.com/path/to/stdio-enabled-node --params

Installation notes

Authentication should be done without password but via SSH keys. The connection is from munin-user@host-munin to munin-node-user@remote-node.

If you use munin-async, the user on the remote node might only be a readonly one, since it only needs to read spooled data. This implies that you use --spoolfetch and not --vectorfetch that updates the spool repository.

Upcoming HTTP(S) transport in 3.0

And the sweetest part is that since all the work has been done for adding another transport, adding a CGI-based HTTP transport one is possible (and therefore done) for 3.0.

Waiting for Munin 2.0 - Performance - Asynchronous updates

Steve Schnepp — Sat, 26 Jun 2010 16:07:00 +0200

munin-update is the fragile link in the munin architecture. A missed execution means that some data is lost.

The problem : updates are synchronous

In Munin 1.x, updates are synchronous : the value of each service^[1] is the one that munin-update retrieves each scheduled run.

The issue is that munin-update has to ask every service on every node for their values. Since the values are only computed when asked, munin-update has to wait quite some time for every value.

This very simple design enables munin to have the simplest plugins : they are completely stateless. While being one great strength of munin, it puts a severe blow on scalability : more plugins/node means obviously a slower retrieval.

Evolving Solutions

1.4 : Parallel Fetching

1.4 addresses some of these scalability issues by implementing parallel fetching. It takes into account that the most of the execution time of munin-update is spent waiting for replies. In 1.4 munin-update can ask max_processes nodes in parallel.

Now, the I/O part is becoming the next limiting factor, since updating many RRDs in parallel is the same as random I/O access for the underlying munin-master OS. Serializing & grouping the updates will be possible with the new RRDp interface from rrdtool version 1.4 and on-demand graphing. Tomas Zvala even offered a patch for 1.4 RRDp on the ML. It is very promising, but doesn't address the root defect in this design : a hard dependence of regular munin-update runs.

2.0 : Stateful plugins

2.0 provides a way for plugins to be stateful. They might schedule their polling themselves, and then when munin-update runs, only emit collect already computed values. This way, a missed run isn't as dramatic as it is in the 1.x series, since data isn't lost. The data collection is also much faster because the real computing is done ahead of time.

2.0 : Asynchronous proxy node

But changing plugins to be stateful and self-polled is difficult and tedious. It even works against of one of the real strength of munin : having simple & stateless plugins.

To address this concern, an experimental proxy node is created. For 2.0 it takes the form of a couple of processes : munin-async-server and munin-sync-client.

The proxy node in detail (`munin-async`)

Overview

These 2 processes form an asynchronous proxy between munin-update and munin-node. This avoids the need to change the plugins or upgrade munin-node on all nodes.

munin-async-server should be installed on the same host than the proxied munin-node in order to avoid any network issue. It is the process that will poll regularly munin-node. The I/O issue of munin-update is here non-existent, since munin-async stores all the values by simply appending them in a text file without any further processing. This file is later read by the client's munin-update, and it will be processed there.

Specific update rates

Having one proxy per node enables a polling of all the services there with a specific update rate.

To achieve this, munin-async-server forks into multiple processes, one for each proxied service. This way each service is completely isolated from the other, and therefore is able to have its own update rate, is safe from other plugins slowdowns, and it does even completely parallelize the information gathering.

SSH transport

munin-async-client uses the new SSH native transport of 2.0. It permits a very simple install of the async proxy.

Notes

[1] in 1.2 it's the same as plugin, but since 1.4 and the introduction of multigraph, one plugin can provide multiple services.

CGI on steroids with FastCGI, but on a CGI-only server - The FastCGI wrapper

Steve Schnepp — Mon, 21 Jun 2010 22:22:00 +0200

FastCGI is really CGI on steroids

FastCGI is very common way to increase performance of a CGI installation. It is based on the fact that usually the startup of CGI scripts is slow, whereas the response is quite fast.

So if you have a persistent process, you only have to take care of the startup once, and you then experience a real speedup.

FastCGI vs mod_perl (or mod_python, ...)

Once a big fan of mod_perl, I'm converted to FastCGI since. mod_perl was for a long time the answer for speeding up Perl CGI scripts. It has a very good track record of stability and has real hooks deep in the Apache processing requests.

FastCGI focuses on a different feature set that is more actual than mod_perl^[1] :

It is much simpler to install and configure, especially when having multiple applications.
Able to connect to a distant server (running as a different UID, chrooted or even on a remote host)
Able to mix scripting languages without any need to compile some other apache modules.
Able to be used with several webservers, even closed-source ones : FastCGI is a protocol, not an API.

But steroids do have some side effects

CGI issues

One downside is that your CGI script should be adapted to FastCGI and the fact that the script doesn't end with the end of the request.

In the real world that's quite easy. Every language that is commonly used for CGI offers CGI-wrapper libraries that works in a FastCGI context as well as a plain CGI one.

Webserver issues

Another issue can also come from the webserver. Since CGI is dead simple to implement even the micro-webserver thttpd implements it.

FastCGI on the other hand is a little more difficult to implement, since the webserver needs to create a container that monitors and calls the FastCGI-enabled script.

A standalone FastCGI container

Fortunately, the FastCGI team provided us with a ready-to-use container and a very simple client that acts a plain CGI script, but proxies it to a full-blown container.

Since the plain CGI part is a very small native executable its overhead is negligible compared to the reply time, even without comparison with the startup time of the whole script.

Its installation is also quite straightforward. I just installed the libfcgi package on Debian : it provides /usr/bin/cgi-fcgi.

I created a simple CGI wrapper for my previous munin benchmarking needs :

#! /bin/sh

exec /usr/bin/cgi-fcgi -connect /tmp/munin-cgi.sock \
     /usr/lib/cgi-bin/munin-cgi-graph

Notes

[1] who really need deep apache hooks ?

Waiting for Munin 2.0 - Performance - FastCGI

Steve Schnepp — Wed, 16 Jun 2010 23:25:00 +0200

1.2 has CGI, it is slow, unsupported, but it does exist.

1.4 has even an experimental FastCGI install mode.

Quoting from this page :

This is more a proof of concept than a recommended - it's slow. Also we do not test it before every release

In 2.0 lots of work has been done to take this experimental CGI mode into a supported one. It might even be the primary way of using munin since, when an install has a certain size, CGI becomes mandatory.

That's because munin-graph doesn't have time to finish its job when the next one is launched, and the new one doesn't run. It is not as dramatic as a missed munin-update execution, since the graphs will still be generated on the later round, but there will be random graph lags and it will put quite some stress on the CPU & I/O subsystem. This will slow munin-update down since it also uses the I/O subsystem much, and that's to be avoided at all costs.

Mainstream CGI has some consequences :

Only the FastCGI wrapper remained : the plain CGI one is dropped.
- The CPAN module CGI::Fast is compatible when launched as a normal CGI.
- Almost all HTTP servers support plain CGI, and with the cgi-fcgi wrapper from the FastCGI devkit (Debian package libfcgi), you can have the best of both worlds (a custom HTTP server & FastCGI). I even posted on how to have a working thttpd with FastCGI.
The old process limit mechanism is dropped also. The FastCGI server configuration is a much better way to control it. The old code was based on System V semaphores and was not 100% reliable.
A caching system has to be implemented, in order for each graph to be generated only once for its lifetime.
The CGI process is launched with the HTTP server user. Since it doesn't only read now, but also writes log files and images files, there is an extra step when installing it. But it's already described in the Munin CGI page given previously.
Since the process is launched only once, for now it read only once the config. So if some part of the config change, the FastCGI container MUST be restarted.

Some benchmarks

Now, the sweet part : I'm putting up some micro-benchmarks.

They should be taken with caution as every benchmark should be, but I think the general idea is conveyed. For the sake of simplicity I'm only doing 1 request in parallel and disabled IMS caching.

Basic 1.2 CGI

$ httperf --num-conns 10  --add-header='Cache-Control: no-cache\n' \
    --uri  /cgi-bin/munin-cgi-graph/localdomain/localhost.localdomain/cpu-day.png

Total: connections 10 requests 10 replies 10 test-duration 27.939 s

Connection rate: 0.4 conn/s (2793.9 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 1653.9 avg 2793.9 max 5217.0 median 1912.5 stddev 1487.8
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 0.4 req/s (2793.9 ms/req)
Request size [B]: 131.0

1.4 FastCGI

The munin-fastcgi-graph is only loaded once, but the munin-graph is reloaded each time.

$ httperf --num-conns 10  --add-header='Cache-Control: no-cache\n' \
    --uri  /cgi-bin/munin-fastcgi-graph/localdomain/localhost.localdomain/cpu-day.png

Total: connections 10 requests 10 replies 10 test-duration 13.807 s

Connection rate: 0.7 conn/s (1380.7 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 1141.3 avg 1380.7 max 1636.1 median 1381.5 stddev 173.7
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 0.7 req/s (1380.7 ms/req)

The response time is cut almost in half. That's expected, since only the top half of the processing isn't reloaded.

2.0 FastCGI

Here everything is loaded once.

$ httperf --num-conns 10  --add-header='Cache-Control: no-cache\n' \
    --uri  /cgi-bin/munin-cgi-graph-2.0/localdomain/localhost.localdomain/cpu-day.png

Total: connections 10 requests 10 replies 10 test-duration 1.668 s

Connection rate: 6.0 conn/s (166.8 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 123.0 avg 166.8 max 513.4 median 127.5 stddev 121.9
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 6.0 req/s (166.8 ms/req)

Now response time is cut almost by a ten factor ! That's quite good news, since it goes 20 times faster that the original CGI.

Waiting for Munin 2.0 - Performance - Architecture

Steve Schnepp — Thu, 10 Jun 2010 03:48:00 +0200

A little intro/refresh on munin's architecture on the master

Munin has a very simple architecture on the master : munin-cron is launched via cron every 5 minutes. Its only job is to launch in order munin-update, munin-graph, munin-html & munin-limits.

The various processes

munin-update

This process retrieves the values from the various nodes and to update the rrd files. This one should never take more than 5 minutes to run, otherwise there will be gaps since the next update will not be launched (lockfile-protected runs).

This process stresses the I/O on the master, and depends on the plugins execution time on the various nodes. On 1.4 the retrieval is multi-threaded^[1], so an slow node doesn't impact too much the whole process.

2.0 proposes asynchronous updates and vectorized updates.

munin-graph

This process generates all the image files from the rrd files.

It is usually a process that is quite CPU-bound, it generates also a fair load of I/O. Since 1.4 there might also be a parallel graphing generation in order to take advantage of multiple CPU / multiple I/O paths.

A simple optimization is to generate only needed graphs instead of all of them each time. This leads to CGI-generation of graphs. 1.2 & 1.4 took a first step in this direction, but it's quite a hack since it's only a very basic script that calls munin-update with the correct parameters.

A FastCGI port of the wrapper (munin-cgi-graph) removes the overhead of starting the wrapper for each call, but in 1.4 the code is quite experimental and has some serious bugs that would need extensive patching to be fixed.

2.0 completes the integration of CGI graphing with removing the overhead of calling munin-graph and does this extensive patching for bugs fixing

munin-html

This process generates all the html files from the rrd files. This one is quite fast for now.

munin-limits

This process checks the limits to see if there is a warning/alert to send via mail or nagios. This one is also quite fast for now.

Notes

[1] more multi-process actually

Waiting for Munin 2.0 - Introduction

Steve Schnepp — Tue, 08 Jun 2010 22:30:00 +0200

This is the first article of a series about the coming version 2.0 of Munin.

The idea came from the series Waiting from 8.5 about PostgreSQL.

The ironic part is that their 8.5 release has become a 9.0, just like our 1.5 will be a 2.0.

I'll post several small articles about new or enhanced-enough features. They will all be tagged munin20.

Planned summary :

Performance - Architecture context
Performance - FastCGI
Performance - Asynchronous updates
Performance - Misc
Native SSH transport
Custom data retention plans (keep more data)
Dynamic zooming

Don't use Excerpt... At least with DotClear.

Steve Schnepp — Thu, 01 Apr 2010 08:00:00 +0200

DotClear automatically generates a meta description tag from the blog entry, but it doesn't take the excerpt into account.

It just takes the beginning of the article content. Since the excerpt is also shown at the beginning of the article, I cannot just write 2 times the same content.

meta description is quite interesting since it is usually used for the little snipped under a search result in usual search engines, so having the beginning of the post in here is very nice.

This fact annihilates the good point of having excerpts.

I'm now falling back to removing progressively all the excerpts on my posts...

API Design: Avoid hidden costs of simple features

Steve Schnepp — Wed, 31 Mar 2010 21:00:00 +0200

Programmers are usually like water : they always use the path of least resistance.

Let's see how to use this fact to predict the usage of an API when you design it.

Initial API

Consider the very simple DB API that consumes a connected ResultSet and presents a disconnected version of it.

class DisconnectedResultSet{
        public DisconnectedResultSet (ResultSet rs);
        public boolean next();
        public Object getObject(int col_idx);
}

It's usage is quite easy :

while (drs.next()) {
        int col_idx = 1;
        drs.getObject(col_idx++); // Do something w/ 1st col
        drs.getObject(col_idx++); // Do something w/ 2st col
        //...
}

Just a little evolution...

Since the DisconnectedResultSet is disconnected, we can imagine that it should implement a rewind() method in order to use it several times without running the initial query again. We now have an updated class :

class DisconnectedResultSet{
        public DisconnectedResultSet (ResultSet rs);
        public boolean next();
        public Object getObject(int col_idx);   
        public void rewind(); // Be able to rewind it
}

And its classical usage :

while (drs.next()) {
        // do stuff...
}
// ...
drs.rewind();
while (drs.next()) {
        // do something else with the same data...
}
// ...
drs.rewind();
while (drs.next()) {
        // do something else with the same data...
}
// ...

A new need comes

A new need comes : see if the DisconnectedResultSet is empty or not in order to avoid sending header.

The usual way is to send them once when iterating like :

boolean is_headers_sent = false;
while (drs.next()) {
        if (! is_headers_sent) { 
                send_headers(); 
                is_headers_sent = true;
        }
        // do something else with the same data...
}

But since there is a nice rewind()method, just waiting to be used, the code might become :

if (drs.next()) {
        send_headers(); 
}
drs.rewind();
while (drs.next()) {
        // do something else with the same data...
}

Now, this code isn't generic anymore to accommodate a connected ResultSet.

So, as John Carmack said :

The cost of adding a feature isn't just the time it takes to code it. The cost also includes the addition of an obstacle to future expansion.

That's really true when you design APIs since their purpose is to last long and to be extended.

So, think twice when you propose an extension "just in case".

The little evolution, revisited...

To solve this case, don't propose a rewind() method, but offer a duplicate() one. It offers the same functionality, just in a new object.

The usage will be almost the same as shown below, but since it feels more performance-sensitive, it won't be used as lightly : the boolean is_headers_sent pattern has now more chances to be used.

while (drs.next()) {
        // do stuff...
}
// ...
drs = drs.duplicate();
while (drs.next()) {
        // do something else with the same data...
}
// ...
drs = drs.duplicate();
while (drs.next()) {
        // do something else with the same data...
}
// ...

It's an other example that immutable objects are the way to go, but for a different reason this time.

Note: Just finished my March 2010 article, even on time... I'm still trying to keep at least a one article per month blogging rate. So far so good for 2010, still 9 months to go !

Free Exception lunch : Use unchecked exceptions, but still announce which ones you might throw.

Steve Schnepp — Sat, 20 Feb 2010 16:01:00 +0100

In a previous article I choosed my side : Unchecked Exceptions are much simpler to use.

But, on the other side of this great division, there is a very valid point : You usually declare checked exceptions. Sure it's possible to only declare to throw Exception, but that would defeat the whole purpose of using checked exceptions.

The nicest thing is that you can also have a custom exception hierarchy, but based on RuntimeException instead of a plain Exception. This way it's like in C++. Everything might be thrown, and you don't need to handle them.

Declaring them, on the other side, is very interesting because you are documenting your interface for almost free.

So, use unchecked exceptions to free yourself of the checked catch-slavery, but still declare the custom ones you might throw.

Immutability of an URL

Steve Schnepp — Sat, 20 Feb 2010 15:18:00 +0100

In the pure spirit of Data is King I think that URL should never change. Even the W3C agrees with their Cool URIs don't change article.

But we all know that in IT never is only not in the foreseen future. So URL do change, at least after a while, and usually for technical reasons^[1].

Since you can update your website to update the URLs, but the inbound link cannot be easily updated. To handle this need, the HTTP protocol has specified the 301 response code.

The solution is that the site should remember all the urls that it generated and redirects accordingly. This way you'll never loose a potential reader to the infamous 404 (this page does not exist).

Some sites even try to approximate the page on a custom 404 page. That's another reason to have user-friendly urls : to be able to hint your reader to appropriate pages in case you don't find his initial destination.

Sadly, this redirect behavior isn't supported by my blog engine (dotclear)... That's for the eat your own dog's food, but I'm looking forward to do it on my current blogging platform.

Notes

[1] upgrade to another blog engine...

Avoid the Preprocessor : Use ''Compile-Time Polymorphism'' for Cross-platform Development

Steve Schnepp — Fri, 11 Dec 2009 23:00:00 +0100

When writing portable cross-platform code, don't litter your code with preprocessor macros, use compile-time polymorphism instead.

A flexible build system will enable you to use advanced OOP-like compile-time polymorphism. That way you can hide all the specifics of the different platform behind an interface firewall. It is the usual way that most cross-platform toolkits and frameworks (such as QT, GTK or wxWidgets) are designed.

We'll take a very simple file I/O subsystem here as an example (open, close). We'll also only take Linux and Windows, since that's usually the 2 most common platforms people want to develop for^[1].

The usual abstraction with the preprocessor

Usually preprocessor #ifdefs are used to compile specific parts for the underlying OS.

The file fileiosys.cpp looks^[2] like the one below :

fileiosys_filedesc open(char* filename) {
#ifdefine __WIN32__
    return OpenFile(
        filename, 
        GENERIC_READ | GENERIC_WRITE, 
        FILE_SHARE_READ | FILE_SHARE_WRITE, 
        NULL,
    );  
#else // UNIX
    return open(filename, O_READ | O_WRITE, 0666);
#endif
}

int close (fileiosys_filedesc fd) {
#ifdefine __WIN32__
    return CloseHandle(fd) ? 0 : -1;
#else // UNIX
    return close(fd);
#endif
}

The header fileiosys.h looks then like :

// Define a custom file descriptor
#ifdefine __WIN32__
    #define fileiosys_filedesc HANDLE
#else // UNIX
    #define fileiosys_filedesc int
#endif

fileiosys_filedesc open(char* filename); 
int close(fileiosys_filedesc fd);

Avoid the preprocessor Compile-time polymorphism

As the previous little example, the code isn't really easy to read. You have to always think which environnement you are in. All the specifics are multiplexed in the same file, and the programmer has to always demultiplex it in real time each time he reads the code.

The last part of the interface file is quite simple to read since it's already abstracted away. Now let's completely demultiplex the implementation in several implementation files.

The header file is the common interface

The interface is the most important part of the design. It should be high-level enough to mask the differences between the plateforms you want to support, but not too high-level, otherwise you'll end up duplicating to much code.

So, for our file I/O subsystem, we'll just abstract the usual syscalls open, close in the same way as before.

The parameters that are passed through the interface are also very important. You cannot usually leak a platform-specific structure. So here all the file descriptors are just an opaque handle, represented by a pointer to a structure defined only as a forward declaration in the header. This pattern, sometimes called the pimpl idiom, enables use to really share the representation while implementing it differently.

The header file is preserved as fileiosys.h, whereas the implementation is in the files linux/fileiosys.cpp and win32/fileiosys.cpp

`fileiosys.h`

// Define the forward declaration
struct fis_filedesc;
fis_filedesc* open(char* filename); 
int close(fileiosys_filedesc* fd);

`win32/fileiosys.cpp`

struct fis_filedesc {
    HANDLE handle;
};

fis_filedesc* open(char* filename) {
    fis_filedesc* fd = new fis_filedesc();
    fd->handle = OpenFile(
        filename, 
        GENERIC_READ | GENERIC_WRITE, 
        FILE_SHARE_READ | FILE_SHARE_WRITE, 
        NULL,
    );  

    return fd;
}

int close (fis_filedesc* fd) {
    int retval = CloseHandle(fd->handle) ? 0 : -1;
    delete(fd);
    return retval;
}

`linux/fileiosys.cpp`

struct fis_filedesc {
    int file_descriptor;
};

static fis_filedesc[32];

fis_filedesc* open(char* filename) {
    fis_filedesc* fd = new fis_filedesc();
    fd->file_descriptor = open(filename, O_READ | O_WRITE, 0666);
    return fd;
}

int close (fis_filedesc* fd) {
    int retval = close(fd);
    delete(fd);
    return retval;
}

The build system : `Makefile`

The makefile should take into account the different platforms, and only compile the needed implementation file. All the gluing magic will then be done at linking time instead of preprocessor time.

Conclusion

The interest of have multiple implementation files is obvious. It is much more straightforward to read and only marginally harder to write. But since most of the time code is read and not written, the choose is quite a no-brainer.

The nicest part is that all this is possible even without the expensive run-time polymorphism and RTTI, since the choose is done at compile-time.

Notes

[1] actually when targeting Linux, you usually target all Unix-like systems since they already have POSIX as a common abstraction

[2] The code is not real, it has been sweetened

Native SSH transport for Munin

Steve Schnepp — Thu, 26 Nov 2009 22:00:00 +0100

Update: This page is obsolete, since it has been merged in the 2.0 version of Munin. The syntax is a little bit different, but the idea is the same.

I blogged about the munin monitoring system a while ago.

The fact the Munin team did quite a remarkable job in cleaning up the 1.2 code for the 1.4 release enabled me to add a native SSH transport for Munin, and made be able to get rid of all SSH tunnels.

Actually the tunnel won't disappear, but they will be launched only when needed and, most importantly configured in munin.conf.

The native ssh for munin patch should applies quite cleanly on revision 3101 of the svn trunk.

Its use is quite straightforward : in /etc/munin/munin.conf, you just migrate address to the new configuration directive remote_connection_cmd that take the whole ssh command to launch a stdio munin-node such as pmmn.

If we take the examples from the previous post, it becomes clear it's much easier to configure.

munin.conf snippet - Inetd version

[server1]
    address localhost
    port 7001
[server2]
    address localhost
    port 7002

munin.conf snippet - Native SSH transport version

Right now, the address directive is still mandatory, but ignored when connecting.

[server1]
    address dummy
    remote_connection_cmd /usr/bin/ssh -- supusr@server1 /home/suprusr/pmmn/pmmn.pl
[server2]
    address dummy
    remote_connection_cmd /usr/bin/ssh -- supusr@server2 /home/suprusr/pmmn/pmmn.pl

Warning

Beware that the ssh process will now be launched by the munin user, so you have to update the key-based SSH authentication accordingly.

Personal Workflow Blog

Spinoffs in the munin ecosystem

KISS is the core design of Munin

Spin-offs ...

... are welcomed ...

... because they are very valuable in the long term

Different layers of spin-offs

Plugins

C

Node

Simple munin

SNMP

Win32 ports

Android

C

Master

Clients

Do not fear git rebase : make snapshots !

When having good relationships with package maintainers can also be a curse

Notes

Avoid those milli-hits in Munin

Waiting for Munin 2.0 - Break the 5 minutes barrier !

Supersampling

Overview

Notes

Protocol details

Config

Notes

Fetch

Compatibility with 1.4

Sample implementation

Streaming plugins

Undersampling

Enhance RRD I/O performance in Munin 1.4 and Scale

Autovivification in Perl : Great Idea but also Huge Trap - Another Leaking Abstraction...

Notes

Waiting for Munin 2.0 - Keep more data with custom data retention plans

RRD is Munin's backbone.

1.2 - Only one set

1.4 - 2 sets : normal & huge

Manual overriding

2.0 - Full control

Waiting for Munin 2.0 - Native SSH transport

Native SSH

Configuration

Installation notes

Upcoming HTTP(S) transport in 3.0

Waiting for Munin 2.0 - Performance - Asynchronous updates

The problem : updates are synchronous

Evolving Solutions

1.4 : Parallel Fetching

2.0 : Stateful plugins

2.0 : Asynchronous proxy node

The proxy node in detail (munin-async)

Overview

Specific update rates

SSH transport

Notes

CGI on steroids with FastCGI, but on a CGI-only server - The FastCGI wrapper

FastCGI is really CGI on steroids

FastCGI vs mod_perl (or mod_python, ...)

But steroids do have some side effects

CGI issues

Webserver issues

A standalone FastCGI container

Notes

Waiting for Munin 2.0 - Performance - FastCGI

Some benchmarks

Basic 1.2 CGI

1.4 FastCGI

2.0 FastCGI

Waiting for Munin 2.0 - Performance - Architecture

A little intro/refresh on munin's architecture on the master

The various processes

munin-update

munin-graph

munin-html

munin-limits

Notes

Waiting for Munin 2.0 - Introduction

The proxy node in detail (`munin-async`)

`fileiosys.h`

`win32/fileiosys.cpp`

`linux/fileiosys.cpp`

The build system : `Makefile`