Personal Workflow Blog

Originally a blog about PWKF, an easy-to-use workflow solution, but as my interest in PWKF faded, it evolved into a generic blog.

Therefore now it is more about random thoughts, Munin or even low-level techy stuff I’m doing with my sons.

Posts

Sep 21, 2026
Type Punning in C and C++
About 7 min read

I had a bug that took me a while to track down. The problem was type punning. A pointer cast worked fine at -O0 and silently broke at -O2. The C vs C++ distinction here is genuinely treacherous, and most blog posts on the topic get it wrong.
Read More...
Jul 10, 2026
GitHub Issue Discussions Belong in Code Comments
About 7 min read

I spent three hours last month hunting through deleted GitHub issues trying to understand why a particular loop had such a strange boundary condition. The issue was closed. The repo had moved. The original author had left. The context was gone.
Read More...
Mar 27, 2026
Amalgamate C Sources for Online IDE Compilers
About 6 min read

I use Compiler Explorer and PGE Tinker regularly. Yet both take only a single C or C++ file as input. Most of the projects do not. This tool closes that gap.
Read More...
Mar 17, 2026
Prompt Engineering Frameworks - A Practical Overview
About 7 min read

There is a recurrent question when starting to use LLMs seriously : “How do I structure my prompts ?”.
Read More...
Dec 19, 2025
Optimize for Size with Preprocessor Directives
About 6 min read

When working on embedded systems, optimizing for size can be crucial. Let’s even go beyond the compiler optimization flags by actually generating different code paths based on whether we are optimizing for size or speed.
Read More...
Dec 11, 2025
Jekyll Remote Theme in Debian 12
About 1 min read

In a previous post, we moved from the default minima theme version to a recent one that supports the dark theme, but I couldn’t test it anymore in my local Jekyll engine on my Debian 12.
Read More...
Dec 9, 2025
The Serenity Prayer, Revisited
About 3 min read

I keep running into the same mistake on projects: spending energy on things that won’t move. The Serenity Prayer is older than software, but it maps surprisingly well onto how to spend your time as a developer.
Read More...
Nov 12, 2025
Upgrading the LM358 OpAmp to be RRIO with a NE555
About 6 min read

This project is an attempt to understand how RRIO opamps work internally, and to replicate one using very common components. It was originally created in 2023 for Hackaday.io.
Read More...
Nov 10, 2025
Create SDL2 Debian Packages for mingw-w64
About 1 min read

SDL2 is an awesome library for a portable game. Yet cross-compiling with it for Windows on Linux is tricky.
Read More...
Oct 25, 2025
A Dark Theme
About 2 min read

Almost everyone uses Dark Themes by now, myself included. Either permanently or just at night. My blog didn’t have one, and it is actually very easy to setup on the minima theme of jekyll.
Read More...
Oct 19, 2025
Kartu Sama - Rules
About 4 min read

Kartu Sama is a card game I invented with my children. It’s similar to UNO but played with a regular 54-card deck. It’s designed to be quick and fun, but not as hectic as UNO since there are no time-sensitive actions.
Read More...
Sep 5, 2025
Installation of the very small LuckFox Micro SBC
About 1 min read

The LuckFox Micro SBC
Read More...
Aug 19, 2025
Workaround for deprecated gets()
About 1 min read

gets() is used a lot in ancien learning material for C. That said, it is unsafe and therefore removed in recent C versions. A simple #define can bring it back rather securely.
Read More...
Aug 17, 2025
RoboRally Mods - Handicap
About 3 min read

I introduced my kids to RoboRally some time ago. It’s an old but iconic game involving some programming and lots of fun dealing with bugs. We added our own additional rules set, effectively modding the game.
Read More...
Jul 16, 2025
The Rise of Fluid Teams
About 4 min read

Every project I have been on has the same unresolved tension: you need stable, trusted ownership of the codebase, and you also need to move fast on features. Agile solves the second, not the first. Fluid teams are my attempt at solving both at once.
Read More...
Jul 14, 2025
Using s3cmd in MSYS2
About 1 min read

s3cmd is the standard tool to access S3. Yet it isn’t packaged for MSYS2. Here are the steps to use it nonetheless.
Read More...
May 12, 2025
Unicode Symbols Sheet
About 1 min read

Unicode symbols are nice, but writing them is sometimes difficult specially since each OS and application has its own way. Here is a simple page where you can simply copy & paste them from.
Read More...
Apr 18, 2025
Replying Inline Emails Considered Harmful
About 15 min read

Do you feel drowning in useless emails that are difficult to follow ? Search no more, simply blame the “my replies are below”. Let’s see how you can improve the situation.
Read More...
Apr 17, 2025
Let Water Flow Down
About 2 min read

I’m always frustrated by heavy processes. They usually feel like unneeded overhead to get the job done. Let’s explore that overhead world, as some is actually desirable.
Read More...
Mar 20, 2024
Drive a WS2812B strip with an UART
About 11 min read

The WS2812B is the powerhorse of modern RGB leds. The protocol to drive them is very simple, but not to easy to implement as it involves precise timings. Since another protocol that involves precise timings is UART, it makes sense to try to combine them.
Read More...
Feb 4, 2024
CISC8 - A new ISA for a 8/16 bit MCU
About 5 min read

Philipp Klaus Krause (from the SDCC fame) is creating a new 8 bit CPU that would fit the gap that RISC-V has left when moving to 32/64 bit computing. It is the F8, and is heavily geared towards efficient code generation in C. I also wanted to explore that realm, mostly for learning purposes.
Read More...
Oct 25, 2023
Post Modern Cloud Native Development
About 4 min read

Moving everything to the cloud is now mainstream, and Cloud Native Development mindset took over.
- It has been successfully marketed as the only solution to every problem.
- No-one wants to work on boring tech anymore.
- It has become a Self-fulfilling prophecy.
Read More...
Aug 19, 2023
Adding some Color
About 3 min read

Small touches of color are always nice. It is actually very easy to do with Jekyll.
Read More...
Aug 18, 2023
Serverless MCU Lambdas - A Working Emulator - Part 1
About 6 min read

A follow-up of my Serverless MCU Lambdas idea. It is comprised of 3 phases :
- HTTP/CGI
- Generic RV32I emulation
- Specific RV32I emulation (curl, I/O, memory functions, …)
Read More...
Aug 17, 2023
1OPS - One Operation Per Statement
About 14 min read

The 1OPS principle is the single principle that had the biggest impact in my entire coding career. I gave some hints about it in a previous post, but I was suggested that it might warrant a post on its own to go more in depth.
Read More...
Jul 20, 2023
Immutability of a Post
About 2 min read

URLs shall be immutable. But what about the post content itself?
Read More...
Jul 16, 2023
Serverless MCU Lambdas
About 3 min read

I want to learn more about bare metal MCU programming. So I came with a crazy idea: instead of emulating webassembly, let’s emulate a RiscV MCU in serverless style.
Read More...
Jun 28, 2023
Cool Stuff versus Useful Stuff
About 15 min read

The main mindset driving development nowadays is web scale and modern stack. Most of the literature is therefore focusing on it, reinforcing the trend, as described in a previous post. Which makes them therefore very desirable, hence attractive to even more literature. And this usually leads to the creation of a hype-laden echo chamber.
Read More...
Jun 24, 2023
glCombat - Week 2 - Projectiles & Explosions
About 4 min read

Another week has passed. But this time, things have slowed down as real life took its toll. Focus was on the projectiles and their explosions.
Read More...
May 27, 2023
glCombat - Week 1 - Map
About 9 min read

One week has passed. As usually wih new projects lots has happened.
Read More...
May 21, 2023
Starting a GameDev Series
About 7 min read

I tried Ground Control some time ago with my sons, it was working very nicely on Windows but was horribly slow on Linux with Wine. Visuals were flawless, just very slow. I started to investigate the Wine codebase to fix it. Little did I know what I embarked on.
Read More...
Feb 15, 2023
Mission Logs are our Tracer Bullets - Always Use Them
About 6 min read

Exploratory work is often needed when trying a novel idea. As a good idea at first might end up highly impractical in reality. Which is why when doing a PoC, the outcome is not always a success. If it was, it would be a prototype.
Read More...
Dec 22, 2022
Micro-services are Mostly Harmful in Your Case
About 14 min read

Micro-services are very fashionable nowadays but the real questions are:
- What do they really bring to you?
- What is their hidden price?
- Can you avoid paying that price?
Read More...
Oct 16, 2022
USB-C Connectors Explained
About 6 min read

Having a USB-C device that only works when the cable is inserted on one side ? Having a USB-C device that only works with USB-A to USB-C cables ? While USB-C is a wonderful standard, it has its little quirks.
Read More...
Sep 18, 2022
Always Optimize for Junior Devs
About 16 min read

There is a single advice I would give to anyone writing software, and specially to great devs : Always optimize your code for your most junior developers. I agree that it is not a very popular advice, but it is the one that, in my experience, give the biggest bang for the buck.
Read More...
Sep 4, 2022
Emulating 8051 with AVR
About 3 min read

As stated before, 8051 is here to say. Very much like the 8086 is still alive with the newest x86-64 architecture. Since the Arduino Uno is the de-facto form factor for many things, let’s try to emulate 8051 on AVR.
Read More...
Sep 3, 2022
Sizes of SMD Components Explained
About 1 min read

Through-hole tech is very nice for hobbyists. But it is getting obsolete. And many components are not available anymore in anything else than SMD. The main problem is that there isn’t a single standard size anymore
Read More...
Aug 29, 2022
Enhance an 8051 Simulator
About 4 min read

As a followup of my clock kit, I’m growing a genuine interest into the 8051. While more modern MCU are vastly more powerful, the 8051 is just barely as complex as needed to be useful. That’s a sweet spot that makes it invaluable for learning purposes.
Read More...
Aug 27, 2022
Simple AT89C2051 programmer with Arduino
About 3 min read

Several AT89C2051 programmers exists. The most advanced one being the one from Paul Gallagher, as it is fully constrained. Even generating the 12V onboard with a charge pump.
Read More...
Aug 13, 2022
Adding Navigation Links
About 2 min read

Navigation inside a blog site is essential to discover related content.
Read More...
Aug 11, 2022
Adding Comments to a Github Pages Blog
About 2 min read

Jekyll makes it super easy to host a static blog on github pages. It feels like a match made in heaven, as it becomes really serverless. Just write some Markdown and do a git push for publication. Everything else is nicely handled for you by Github.

The biggest downside is, being static, it obviously lacks comments. Disqus fits that gap nicely, but I don’t like to rely on another service. Besides Github issues are already implementing a comment system, so let’s reuse that.
Read More...
Aug 7, 2022
A Tale of Two Programming Religions
About 4 min read

The obvious but overlooked secret of computing is that it is always about only 2 things:
Read More...
Aug 1, 2022
Test Markdown Formatting
About 1 min read

Formatting is important!
Read More...
May 30, 2022
Many ways to Blink a LED
About 4 min read

Blinking a LED is the electronics equivalent of the famous “Hello World!”. Therefore it has a very special meaning, and it is also the basis of “clocks” which are very imprtant in digital electronics.
Read More...
Jan 23, 2022
Minimal NE555 Blink
About 2 min read

My son recreated the “blink” with a NE555 and tried to remove the wire from RESET pin to VCC. It actually worked well, and we wondered “what’s the minimal BOM for a NE555 blink ?”.
Read More...
Jan 17, 2022
DIY 4-digit Clock Hacking
About 5 min read

I got my hands on a 4-digit clock DIY kit. Soldering it is rather easy, and it works as advertised.
Read More...
Sep 10, 2021
Emulating Logic Gates with an ATmega328P MCU - Part 1
About 6 min read

I’m discovering the world of electronics with my son. We encountered to some nice videos that are about recreating a whole CPU with only logic gates. The most well known one being the one from Ben Eater and the one from the NAND2Tetris project.
Read More...
Aug 31, 2021
Humble Beginnings in Electronics
About 3 min read

When I was this summer at my parents, I rediscovered a long-forgotten “electronics lab box”. I was super glad, since it was that box that enabled me to get into electronics. Not the digital kind with micro-controllers, but the old-school analogic one, with transistors, resistors & capacitors.
Read More...
Jun 6, 2021
Fix the HDMI output in the ODROID-HC4
About 7 min read

I bought an ODROID-HC4, but it didn’t work with my TV. Let’s fix it.
Read More...
May 24, 2021
Using Computer Modern Fonts
About 1 min read

I always tried to have a minimalist blog style. I almost totally agree with Fabien Sanglard’s 0x10 rules, except on the part with “monotype fonts”.
Read More...
Sep 27, 2020
Fun Yet Effective Meetings
About 9 min read

⚠️ This article is extreme & satirical on the purpose of being thought-provoking. Therefore, please, do take it with some grain of salt.

This is a followup of my Remote-mostly working is a paradigm shift.

Read More...
Sep 26, 2020
Remote-mostly working is a paradigm shift
About 4 min read

💡 As our daily life has been disrupted and work-from-home is slowly becoming the new normal, distributed teams are much more common. But normal work patterns don’t translate immediately, they have to be adjusted.

Read More...
Jan 19, 2014
How to try Munin 2.1.x easily in Debian-derivatives
About 2 min read

In my previous post I explained how to setup a simple development environment, but I feel that’s a little too much if you only want to try that new, shiny, munin version.
Read More...
Jan 14, 2014
Tutorial - Setup a dev environment for Munin
About 5 min read

I discovered some time ago the marvelous dev_scripts/ directory in the munin source code. So, as its usage is very easy, I’ll just write a tutorial about how to use it

Prerequisites

To use it, one has to install all the packages needed for munin, and to grab a copy of the source code. Easiest is to use either a tarball, or to clone the git repository.

Note that the guidelines on contributing back are specified directly in the git repo.

Now, I just assume you want to contribute back, otherwise you would not care much about the said dev environment. That means using the git way of doing it.

Download the source code

First step is to clone the git repository. We will use $HOME/src/munin as the development directory.
```
mkdir -p $HOME/src
cd $HOME/src
git clone https://github.com/munin-monitoring/munin munin
cd munin
```
Compile munin

Now, we have to compile the source code. I know that it sounds strange as the code is mostly Perl, but there are some templates that need to be filled with the environment specifics, such as the Perl interpreter path, a POSIX compatible shell, …
```
dev_scripts/install 1
```
Now all munin (and munin-node) should be compiled and installed in $HOME/src/munin/sandbox.

Note that the 1 at the end is explained below.

Using the dev tools

There are some different tools in dev_scripts/:

install

This is the one you used already. You have to use it every time you want to recompile & deploy the package.

The 1 argument, does a full re-install (wipe & install), so you don’t usually want to do that.

restart_munin-node

This is a tool to start the development node. Note that it listens on the port 4948, so you can use it alongside a normal munin-node.

run

The run command inside is used to launch all the executable parts of munin, such as munin-update, munin-html or munin-limits. It can also be used to launch munin-run and munin-node-configure.

The usage is very simple, just prefix the command to launch with dev_scripts/run, every environment variable and command line argument will be forwarded to the said command.
```
# launch munin-cron
dev_scripts/munin-cron

# launch manually some cron parts
dev_scripts/munin-update
dev_scripts/munin-limits
dev_scripts/munin-html
dev_scripts/munin-graph

# debug a plugin
dev_scripts/munin-run --debug cpu config
```
cgi

This is the same as run, only for CGI. It sets up the whole environment vars that emulates a CGI call. Usage is very easy:
```
dev_scripts/cgi munin-cgi-graph /localnet/localhost/cpu-day.png > out.dat
```
The out.dat will contain the whole HTTP output, with the HTTP headers and the PNG content. Everything that is sent to STDERR won’t be catched, so you can liberally use it while debugging.

query_munin_node

The query_munin_node is used to send commands to the node in a very simple way. Node commands are just args of the tool.
```
dev_scripts/query_munin_node list
dev_scripts/query_munin_node config cpu
dev_scripts/query_munin_node fetch cpu
```
Real CGI usage with your web browser

That’s the holy grail. You will have a development version that behaves the same as a real munin install.

First, let’s assume you have a working user cgi configuration (ie ~user/cgi/whatever is working). If not you should refer yourself to the local documentation of your preferred webserver. Note that nginx will not work, as it does not support CGI.

I wrote a very simple cgi wrapper script. The home dir is hard coded in the script.
```
#! /bin/sh

ROOT=/home/me/src/munin
eval "$(perl -V:version)"

PERL5LIB=$ROOT/sandbox/usr/local/share/perl/$version
#export DBI_TRACE=2=/tmp/dbitrace.log

exec perl -T -I $PERL5LIB $ROOT/sandbox/opt/munin/www/cgi/$CGI_NAME
```
Read More...
Apr 13, 2013
Experimenting with a C munin node
About 1 min read

As I wrote about it earlier, Helmut rewrote some core plugins in C. It was maintly done with efficiency in mind.

Core plugins are designed for simplicity…

As those plugins are only parsing one /proc file, there seemed no need to endure the many forks inherent with even trivial shell programming. It also acknowledges the fact that the measuring system shall be as light as possible

Munin plugin are highly driven towards simplicity. Therefore having shell plugins is quite logical. It conveys the educational sample purpose for users to write their own, while being quite easy to code/debug for the developpers. Since their impact on current systems is very small, there are not much incentive to change.

… but efficiency is coming !

Nonetheless, now monitored systems are becoming quite small.

It is mostly thanks to embedded systems like the RaspberryPi. This means that processing power available is much lower than on normal nodes¹.

Now the embedded C approach for plugins has a new rationale.
1. Usually datacenter nodes are more in the high end of the spectrum than the low-end. ↩
Read More...
Apr 13, 2013
Spinoffs in the munin ecosystem
About 7 min read

Munin’s greatest strength is its very KISS architecture. It therefore gets many things right, such as a huge modularity.

KISS is the core design of Munin

Each component (master/node/plugin) has a simple API to communicate with the others.

Spin-offs …

I admit that the master, even the node, have convoluted code. In fact some rewrites already do exist.

… are welcomed …

And they are a really good thing, as it enables rapid prototyping on things that the stock munin has (currently) trouble to do.

The stock munin is a piece of software that many depend upon, so it has to move at a much slower pace than one does want, even me. As much as I really want to add many many features to it, I still have to take extra care that it doesn’t break stuff, even the least known features.

So I take munin off-springs very seriously and even offer as much help as I can in order for them to succeed.

… because they are very valuable in the long term

In my opinion competition is only short bad in the short term, and in the long term they usually add significant value to the whole ecosystem. That said, there’s always a risk to become slowly irrelevant, but I think that’s the real power of open-source’s evolutionary paradigm : embrace them or become obsolete and get replaced.

Since, if someone takes the time to author a competitor that has a real threat potential, it mostly means that there’s a real itch to scratch and that many things are to be learnt.

Different layers of spin-offs

The munin ecosystem is divided in 3 main categories, obviously related to the 3 main components of munin : master, node & plugin.

Plugins

That’s the most obvious part as custom plugins are the real bread and butter of munin.

Stock plugins are mostly written in Perl or POSIX shell, as Perl is munin’s own language and POSIX shell is ubiquitous. That fact is acknowledged by the fact that core munin provides 2 libraries (Perl & Shell) to help plugin authoring.

So, it’s quite natural that each mainstream language has grown its own plugin library. Some language even have two of them.

C

Some plugins got even rewritten in plain C, as it was shown that shell plugins do have a significant impact on very under-powered nodes, such as embedded routers.

Node

This component is very simple. Yet, it has to be run on all the nodes that one wants to monitor. It is currently written in Perl, and while that’s not an issue on UNIX-like systems, it can be quite problematic on embedded ones

Simple munin

The official package comes with a POSIX shell rewrite that has to be run from inetd. It is quite useful for embedded routers like OpenWRT, but still suffers from an hard dep on POSIX shell and inetd.

SNMP

SNMP is another way to monitor nodes. While it works really well, it mostly suffers the fact that its configuration is quite different of the usual way, so I guess some things will change on that side.

Win32 ports

Win32 has long been a very difficult OS to monitor, as it doesn’t offer much of the UNIX-esque features. Yet the number of win32 nodes that one wants to monitor is quite high, as it makes munin one the few systems that can easily monitor heterogeneous systems.

Therefore, while you can install the stock munin-node, several projects emerged. We decided to adopt munin-node-win32.

Android

There’s also a dedicated node for Android. It makes sense, given that the Android is yet Linux-derived, but lacks Perl, and is a Java mostly platform. This node also has some basic capabilities of pushing data to the master instead of the usual polling.

This is specially interesting given the fact that Android nodes are usually loosely connected, so the node spools values itself and pushes them when it recovers connectivity.

Note that this is specifically an aspect that is currently lacking in munin, and I’m planning to address it in the 2.1 series. So thanks to its author for showing a relevant use-case.

C

That’s my last experiment. It started with a simple question : how difficult would it be to code a fairly portable version of the node ?

It turned out that it wasn’t that difficult. I’m even asking myself about eventually replacing the win32 specific port with this one, as the code is much simpler. The win32 node has several plugin built-in mostly due to platform specifics. I still have to find a way to work my way around it, but it’s in quite good shape.

This post was originally done to promote it, but while writing it I noticed that the ecosystem deserved a post on its own. So I’ll write another one, specific to the C port of munin-node and plugins.

Master

The master is the most complex component. So rewrites of it won’t happen as-is. They usually take the form of a bridge between the munin protocol and another graphing system, such as Graphite.

Clients

There are also client libraries that are able to directly query munin nodes, to be able to reuse the vast ecosystem. Languages are various, from the obvious Python to Ruby, along with a quite modern node.js one.
Read More...
Apr 4, 2013
Do not fear git rebase : make snapshots !
About 1 min read

Git is a nice version system, but some commands are destructrive, such as rebase.

Here is a script to have a safety net, and free backups!
```
#! /bin/sh
# Script to snaphot a git repo
SNAP_VERSION=$(date +%s)
BUNDLE_NAME=$(basename $( pwd )).${SNAP_VERSION}.git.bundle
git bundle create ../${BUNDLE_NAME} --all
git remote add snap-${SNAP_VERSION} ../${BUNDLE_NAME}
git fetch -p snap-${SNAP_VERSION}
```
Usage is very easy. If you want to restore your current branch to the master one you made earlier.
```
git reset --hard snap-1365068411/master
```
Read More...
Feb 24, 2013
When having good relationships with package maintainers can also be a curse
About 2 min read

I advise every user to only use the packaged version of munin. Here’s a short article to explain the background of my reluctance to ask for users to directly use the official tarball.

I have become upstream of munin a while ago now. As such, I’m in contact with package maintainers. They take the official releases and cram it into their own distribution of choice¹.

I have to admit that the various epic war stories read throughout the web about upstream vs packagers are very far from the truth here. They are a charm to work with. Often challenging and demanding, but always because there’s a real need. And that’s quite a good thing, as I’m still a rookie in term of open source software management. Therefore I’m quite grateful when they gently pinpoint my mistakes².

Yet, this nice team comes with a price. Since we mostly hang out on IRC together, there is way much inter-distro communication than on other software. But I’m the sole owner of the tarball distro.

Yet, as I don’t like to build everything from source, I obviously use a distro. There, since the packaging is very nicely done, I don’t feel to take the hassle of using my own “tarball” to test them. I just build a package for my distro out of the release code.

That’s also a curse, as I admit that I although I test the code, I only seldom test the packaging. This means that I cannot really advise someone on using the tarball, nor directly git code as even I don’t do it.

But, that said, I still think I’m the luckiest upstream around. Thanks guys !
1. Be it linux-based like Gentoo, Redhat…, BSD-based as FreeBSD, OpenBSD…, or even multi-kernel based as Debian ↩
2. Defaulting to CGI graphics was a move that was way too premature, end-user wise. So thanks to them, it defaults to cron again ↩
Read More...
Feb 1, 2013
Avoid those milli-hits in Munin
About 1 min read

A recurring question on IRC is : “why do I have 500 million hit/s in my graph ?”.

Turns out that they are really seeing 500m hit/s, and that lower-case m means milli, and not Mega as specified in the Metric system. This is automatically done by RRD.

To avoid this you should just specify graph_scale no as specified.
Read More...
Jun 20, 2011
Enhance RRD I/O performance in Munin 1.4 and Scale
About 1 min read

As with most of the RRD-based monitoring software (Cacti, Ganglia, …), it is quite difficult to scale.

The bad part is that updating lots of small RRD files seems like pure random I/O to the OS as stated in there documentation.

The good part is that we are not alone, and therefore the RRD developers did tackle the issue with rrdcached. It spools the updates, and flushs them to disk in a batched manner, or when needed by a rrd read command such as graphing. That’s why it is scales well when using CGI graphing. Otherwise, munin-graph will read every rrd, and therefore force a flush on all the cache.

And the icing on the cake is that, although it is only fully integrated to munin 2.0, you can use it right away in the 1.4.x series.

You only need to define the environment variable RRDCACHED_ADDRESS while running the scripts accessing the RRDs.

Then, you have to remove the munin-graph part of the munin-cron and run it on its own line. Usually only every hour or so, to be able to accumulate data in rrdcached before flushing it all to disk when graphing.

Updating to 2.0 is also an option to have a real CGI support. (CGI on 1.4 is existing but has nowhere decent performance).
Read More...
Jun 16, 2011
Autovivification in Perl : Great Idea but also Huge Trap - Another Leaking Abstraction...
About 2 min read

Autovivification is one of Perl’s really great design success.

It all comes to you don’t need to worry about existence before dereferencing something.

That means, for setting a nested hash, you only need to write:
```
$h->{foo}{bar} = "value";
```
And that will work out of the box. Perl will happily create all the data-structure for you.

So, now a little coding test, what does the following code output?
```
my $a;

if ($a->{foo}{bar}) {
   print "Found foo/bar\n";
}

if ($a->{foo}) {
   print "Found foo\n";
}
```
Naively, it shouldn’t output anything, right?

Not so fast. Upon a careful read of “Perl will happily create all the data-structure for you”, we can put some emphasis on one word : “Perl will happily create all the data-structure for you”.

That might be just perfect, except that Perl creates it whenever it needs it, even if it is only for reading.

And now you understand the catch : a read operation can result in a write one.

As Uncle Ben (from SpiderMan) said¹ : “With Great Power Comes Great Responsibility”.

Dagfinn Ilmari Mannsåker showed me a nice autovivification module on CPAN that fixes this behavior, and enables a fine tuning of this process.

I really think the fact that creation also happen when querying the value is a real bug in Perl itself, or at least a bug in the design of the feature.
1. Voltaire, Franklin D. Roosevelt and other said something very similar, but they are not as geeky. ↩
Read More...
Jun 13, 2011
Waiting for Munin 2.0 - Break the 5 minutes barrier !
About 4 min read

Every monitoring software has a polling rate. It is usually 5 min, because it’s the sweet spot that enables frequent updates yet still having a low overhead.

Munin is not different in that respect : it’s data fetching routines have to be launched every 5 min, otherwise you’ll face data loss. And this 5 min period is deeply grained in the code. So changing it is possible, but very tedious and error prone.

But sometimes we need a very fine sampling rate. Every 10 seconds enables us to track fast changing metrics that would be averaged out otherwise. Changing the whole polling process to cope with a 10s period is very hard on hardware, since now every update has to finish in these 10 seconds.

This triggered an extension in the plugin protocol, commonly known as supersampling.

Supersampling

Overview

The basic idea is that fine precision should only be for selected plugins only. It also cannot be triggered from the master, since the overhead would be way too big.

So, we just let the plugin sample itself the values at a rate it feels adequate. Then each polling round, the master fetches all the samples since last poll.

This enables various constructions, mostly around streaming plugins to achieve highly detailed sampling with a very small overhead.

Note: This protocol is currently completely transparent to munin-node, and therefore it means that it can be used even on older (1.x) nodes. Only a 2.0 master is required.

Protocol details

The protocol itself is derived from the spoolfetch extension.

Config

A new directive is used, update_rate. It enables the master to create the rrd with an adequate step.

Omitting it would lead to rrd averaging the supersampled values onto the default 5 min rate. This means data loss.

Note: The heartbeat has always a 2 step size, so failure to send all the samples will result with unknown values, as expected.

The RRD file size is always the same in the default config, as all the RRA are configured proportionally to the update_rate. This means that, since you’ll keep as much data as with the default, you keep it for a shorter time.

Fetch

When spoolfetching, the epoch is also sent in front of the value. Supersampling is then just a matter of sending multiple epoch/value lines, with monotonically increasing epoch. Note that since the epoch is an integer value for rrdtool, the smallest granularity is 1 second. For the time being, the protocol itself does also mandates integers. We can easily imagine that with another database as backend, an extension could be hacked together.

Compatibility with 1.4

On older 1.4 masters, only the last sampled value gets into the rrd.

Sample implementation

The canonical sample implementation is multicpu1sec, a contrib plugin on github. It is also a so-called streaming plugin.

Streaming plugins

These plugins fork a background process when called that streams a system tool into a spool file. In multipcu1sec, it is the mpstat tool with a period of 1 second.

Undersampling

Some plugins are on the opposite side of the spectrum, as they only need a lower precision.

It makes sense when:
- data should be kept for a very long time
- data is very expensive to generate and it doesn’t vary fast.
Read More...
Aug 22, 2010
Waiting for Munin 2.0 - Keep more data with custom data retention plans
About 3 min read

Munin keeps its data in an RRD database. It’s a wonderful piece of software, designed for this very purpose : keep an history of numeric data.

RRD is Munin’s backbone.

All you need is to tell RRD for how long and the precision you want to keep your data. RRD manages then all the underlying work : pruning old data, averaging to decrease precision if needed, …

Munin automatically creates the RRD databases it needs.

1.2 - Only one set

In 1.2, every database creation was done with the same temporal & precision parameters. Since the output parameters were constant (day, week, month, year graphs), there were little need to have a different set of parameters.

1.4 - 2 sets : normal & huge

In 1.4, various users showed their need to have different graphing outputs, and began to hack around Munin’s fixed graphing. It became rapidly obvious that the 1.2 preset wasn’t a fit for everyone.

Therefore a huge dataset was available to be able to extend the finest precision (5min) to the whole Munin timeframe. This comes at a price though : more space is required, and the graph generation is slower, specially when generating the yearly one, since more data has to be read and analysed.

The switch is done for the whole munin installation by changing the system-wide graph_data_size, although already created rrd databases aren’t changed. It is then even possible for a user to pre-customize the rrd file. Munin will then happily uses them transparently thanks to the RRD layer.

Manual overriding

Altering the RRD files after it is created is possible, but not as simple. Standard export & import from RRD take the structure with it. So data has to be moved around with special tools. rrdmove is my attempt to create such a tool. It copies data between 2 already existing RRD files, even asking RRD to interpolate the data when needed.

2.0 - Full control

Starting with 2.0, the parameter graph_data_size is per service. It also has a special mode : custom. Its format is very simple:
```
graph_data_size custom FULL_NB, MULTIPLIER_1 MULTIPLIER_1_NB, ... MULTIPLIER_NMULTIPLIER_N_NB
graph_data_size custom 300, 15 1600, 30 3000
```
The first number is the number of data at full resolution. Then usually it comes gradually decreasing resolution.

A decreasing resolution has 2 usages:
- Limit the space consumption : keeping full resolution for the whole period (default : 5min for 2 years) is sometime too precise.
- Increase performance : RRD will choose the best fitting resolution to generate its graphs. Already aggregated data is faster to compute.
Read More...
Jul 9, 2010
Waiting for Munin 2.0 - Native SSH transport
About 2 min read

In the munin architecture, the munin-master has to connect to the munin-node via a very simple protocol and plain TCP.

This has several advantages:
1. Very simple to manage & install
2. Optional SSL since 1.4 enabling secure communications
3. Quite simple firewall rules.
It has also some disadvantages:
1. A new listening service means a wider exposure
2. The SSL option might add some administrative overhead (certificates management, …)
3. A native protocol isn’t always covered by all firewall solutions
4. Some organisations only authorize a few protocols to simplify audits (ex: only SSH & HTTPS)
Native SSH

These down points may be solved by encapsulation over SSH, but it can be a tedious task to maintain if the number of hosts increases.

Therefore 2.0 introduces the concept of a native SSH transport. Its usage is dead simple : replace the address with an ssh:// URL-like one.

The node still has to be modified to communicate with stdin/stdout instead of a network socket. For now, only pmmn and munin-async are able to provide such a node.

Configuration

The URL is quite self-explanatory as shown in the example below:
```
[old-style-host]
    address host.example.com

[new-style-host]
    address ssh://munin-node-user@host.example.com/path/to/stdio-enabled-node --params
```
Installation notes

Authentication should be done without password but via SSH keys. The connection is from munin-user@host-munin to munin-node-user@remote-node.

If you use munin-async, the user on the remote node might only be a readonly one, since it only needs to read spooled data. This implies that you use --spoolfetch and not --vectorfetch that updates the spool repository.

Upcoming HTTP(S) transport in 3.0

And the sweetest part is that since all the work has been done for adding another transport, adding a CGI-based HTTP transport one is possible (and therefore done) for 3.0.
Read More...
Jun 17, 2010
CGI on steroids with FastCGI, but on a CGI-only server - The FastCGI wrapper
About 3 min read

FastCGI is very common way to increase performance of a CGI installation. It is based on the fact that usually the startup of CGI scripts is slow, whereas the response is quite fast.

FastCGI is really CGI on steroids

So if you have a persistent process, you only have to take care of the startup once, and you then experience a real speedup.

FastCGI vs mod_perl (or mod_python, …)

Once a big fan of mod_perl, I’m converted to FastCGI since. mod_perl was for a long time the answer for speeding up Perl CGI scripts. It has a very good track record of stability and has real hooks deep in the Apache processing requests.

FastCGI focuses on a different feature set that is more actual than mod_perl¹:
- It is much simpler to install and configure, especially when having multiple applications.
- Able to connect to a distant server (running as a different UID, chrooted or even on a remote host)
- Able to mix scripting languages without any need to compile some other apache modules.
- Able to be used with several webservers, even closed-source ones : FastCGI is a protocol, not an API.
But steroids do have some side effects

CGI issues

One downside is that your CGI script should be adapted to FastCGI and the fact that the script doesn’t end with the end of the request.

In the real world that’s quite easy. Every language that is commonly used for CGI offers CGI-wrapper libraries that works in a FastCGI context as well as a plain CGI one.

Webserver issues

Another issue can also come from the webserver. Since CGI is dead simple to implement even the micro-webserver thttpd implements it.

FastCGI on the other hand is a little more difficult to implement, since the webserver needs to create a container that monitors and calls the FastCGI-enabled script.

A standalone FastCGI container

Fortunately, the FastCGI team provided us with a ready-to-use container and a very simple client that acts a plain CGI script, but proxies it to a full-blown container.

Since the plain CGI part is a very small native executable its overhead is negligible compared to the reply time, even without comparison with the startup time of the whole script.

Its installation is also quite straightforward. I just installed the libfcgi package on Debian : it provides /usr/bin/cgi-fcgi.

I created a simple CGI wrapper for my previous munin benchmarking needs:
```
#! /bin/sh

exec /usr/bin/cgi-fcgi -connect /tmp/munin-cgi.sock \
     /usr/lib/cgi-bin/munin-cgi-graph
```
1. who really need deep apache hooks? ↩
Read More...
Jun 13, 2010
Waiting for Munin 2.0 - Performance - Asynchronous updates
About 5 min read

munin-update is the fragile link in the munin architecture. A missed execution means that some data is lost.

The problem : updates are synchronous

In Munin 1.x, updates are synchronous : the value of each service¹ is the one that munin-update retrieves each scheduled run.

The issue is that munin-update has to ask every service on every node for their values. Since the values are only computed when asked, munin-update has to wait quite some time for every value.

This very simple design enables munin to have the simplest plugins : they are completely stateless. While being one great strength of munin, it puts a severe blow on scalability : more plugins/node means obviously a slower retrieval.

Evolving Solutions

1.4 : Parallel Fetching

1.4 addresses some of these scalability issues by implementing parallel fetching. It takes into account that the most of the execution time of munin-update is spent waiting for replies. In 1.4 munin-update can ask max_processes nodes in parallel.

Now, the I/O part is becoming the next limiting factor, since updating many RRDs in parallel is the same as random I/O access for the underlying munin-master OS. Serializing & grouping the updates will be possible with the new RRDp interface from rrdtool version 1.4 and on-demand graphing. Tomas Zvala even offered a patch for 1.4 RRDp on the ML. It is very promising, but doesn’t address the root defect in this design : a hard dependence of regular munin-update runs.

2.0 : Stateful plugins

2.0 provides a way for plugins to be stateful. They might schedule their polling themselves, and then when munin-update runs, only emit collect already computed values. This way, a missed run isn’t as dramatic as it is in the 1.x series, since data isn’t lost. The data collection is also much faster because the real computing is done ahead of time.

2.0 : Asynchronous proxy node

But changing plugins to be stateful and self-polled is difficult and tedious. It even works against of one of the real strength of munin : having simple & stateless plugins.

To address this concern, an experimental proxy node is created. For 2.0 it takes the form of a couple of processes : munin-async-server and munin-sync-client.

The proxy node in detail (munin-async)

Overview

These 2 processes form an asynchronous proxy between munin-update and munin-node. This avoids the need to change the plugins or upgrade munin-node on all nodes.

munin-async-server should be installed on the same host than the proxied munin-node in order to avoid any network issue. It is the process that will poll regularly munin-node. The I/O issue of munin-update is here non-existent, since munin-async stores all the values by simply appending them in a text file without any further processing. This file is later read by the client’s munin-update, and it will be processed there.

Specific update rates

Having one proxy per node enables a polling of all the services there with a specific update rate.

To achieve this, munin-async-server forks into multiple processes, one for each proxied service. This way each service is completely isolated from the other, and therefore is able to have its own update rate, is safe from other plugins slowdowns, and it does even completely parallelize the information gathering.

SSH transport

munin-async-client uses the new SSH native transport of 2.0. It permits a very simple install of the async proxy.
1. in 1.2 it’s the same as plugin, but since 1.4 and the introduction of multigraph, one plugin can provide multiple services. ↩
Read More...
Jun 9, 2010
Waiting for Munin 2.0 - Performance - FastCGI
About 5 min read

1.2 has CGI, it is slow, unsupported, but it does exist.

1.4 has even an experimental FastCGI install mode.

Quoting from this page :

This is more a proof of concept than a recommended - it’s slow. Also we do not test it before every release

In 2.0 lots of work has been done to take this experimental CGI mode into a supported one. It might even be the primary way of using munin since, when an install has a certain size, CGI becomes mandatory.

That’s because munin-graph doesn’t have time to finish its job when the next one is launched, and the new one doesn’t run. It is not as dramatic as a missed munin-update execution, since the graphs will still be generated on the later round, but there will be random graph lags and it will put quite some stress on the CPU & I/O subsystem. This will slow munin-update down since it also uses the I/O subsystem much, and that’s to be avoided at all costs.

Mainstream CGI has some consequences :
1. Only the FastCGI wrapper remained : the plain CGI one is dropped.
  - The CPAN module CGI::Fast is compatible when launched as a normal CGI.
  - Almost all HTTP servers support plain CGI, and with the cgi-fcgi wrapper from the FastCGI devkit (Debian package libfcgi), you can have the best of both worlds (a custom HTTP server & FastCGI). I even posted on how to have a working thttpd with FastCGI.
2. The old process limit mechanism is dropped also. The FastCGI server configuration is a much better way to control it. The old code was based on System V semaphores and was not 100% reliable.
3. A caching system has to be implemented, in order for each graph to be generated only once for its lifetime.
4. The CGI process is launched with the HTTP server user. Since it doesn’t only read now, but also writes log files and images files, there is an extra step when installing it. But it’s already described in the Munin CGI page given previously.
5. Since the process is launched only once, for now it read only once the config. So if some part of the config change, the FastCGI container MUST be restarted.
Some benchmarks

Now, the sweet part : I’m putting up some micro-benchmarks.

They should be taken with caution as every benchmark should be, but I think the general idea is conveyed. For the sake of simplicity I’m only doing 1 request in parallel and disabled IMS caching.

Basic 1.2 CGI
```
$ httperf --num-conns 10  --add-header='Cache-Control: no-cache
' \
    --uri  /cgi-bin/munin-cgi-graph/localdomain/localhost.localdomain/cpu-day.png

Total: connections 10 requests 10 replies 10 test-duration 27.939 s

Connection rate: 0.4 conn/s (2793.9 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 1653.9 avg 2793.9 max 5217.0 median 1912.5 stddev 1487.8
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 0.4 req/s (2793.9 ms/req)
Request size [B]: 131.0
```
1.4 FastCGI

The munin-fastcgi-graph is only loaded once, but the munin-graph is reloaded each time.
```
$ httperf --num-conns 10  --add-header='Cache-Control: no-cache
' \
    --uri  /cgi-bin/munin-fastcgi-graph/localdomain/localhost.localdomain/cpu-day.png

Total: connections 10 requests 10 replies 10 test-duration 13.807 s

Connection rate: 0.7 conn/s (1380.7 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 1141.3 avg 1380.7 max 1636.1 median 1381.5 stddev 173.7
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 0.7 req/s (1380.7 ms/req)
```
The response time is cut almost in half. That’s expected, since only the top half of the processing isn’t reloaded.

2.0 FastCGI

Here everything is loaded once.
```
$ httperf --num-conns 10  --add-header='Cache-Control: no-cache
' \
    --uri  /cgi-bin/munin-cgi-graph-2.0/localdomain/localhost.localdomain/cpu-day.png

Total: connections 10 requests 10 replies 10 test-duration 1.668 s

Connection rate: 6.0 conn/s (166.8 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 123.0 avg 166.8 max 513.4 median 127.5 stddev 121.9
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 6.0 req/s (166.8 ms/req)
```
Now response time is cut almost by a ten factor ! That’s quite good news, since it goes 20 times faster that the original CGI.
Read More...
Jun 9, 2010
Waiting for Munin 2.0 - Performance - Architecture
About 3 min read

Munin has a very simple architecture on the master : munin-cron is launched via cron every 5 minutes. Its only job is to launch in order munin-update, munin-graph, munin-html & munin-limits.

The various processes

munin-update

This process retrieves the values from the various nodes and to update the rrd files. This one should never take more than 5 minutes to run, otherwise there will be gaps since the next update will not be launched (lockfile-protected runs).

This process stresses the I/O on the master, and depends on the plugins execution time on the various nodes. On 1.4 the retrieval is multi-threaded¹, so an slow node doesn’t impact too much the whole process.

2.0 proposes asynchronous updates and vectorized updates.

munin-graph

This process generates all the image files from the rrd files.

It is usually a process that is quite CPU-bound, it generates also a fair load of I/O. Since 1.4 there might also be a parallel graphing generation in order to take advantage of multiple CPU / multiple I/O paths.

A simple optimization is to generate only needed graphs instead of all of them each time. This leads to CGI-generation of graphs. 1.2 & 1.4 took a first step in this direction, but it’s quite a hack since it’s only a very basic script that calls munin-update with the correct parameters.

A FastCGI port of the wrapper (munin-cgi-graph) removes the overhead of starting the wrapper for each call, but in 1.4 the code is quite experimental and has some serious bugs that would need extensive patching to be fixed.

2.0 completes the integration of CGI graphing with removing the overhead of calling munin-graph and does this extensive patching for bugs fixing

munin-html

This process generates all the html files from the rrd files. This one is quite fast for now.

munin-limits

This process checks the limits to see if there is a warning/alert to send via mail or nagios. This one is also quite fast for now.
1. more multi-process actually ↩
Read More...
Jun 8, 2010
Waiting for Munin 2.0 - Introduction
About 1 min read

This is the first article of a series about the coming version 2.0 of Munin.

The idea came from the series Waiting from 8.5 about PostgreSQL.

The ironic part is that their 8.5 release has become a 9.0, just like our 1.5 will be a 2.0.

I’ll post several small articles about new or enhanced-enough features. They will all be tagged munin20.

Planned summary :
1. Performance - Architecture context
2. Performance - FastCGI
3. Performance - Asynchronous updates
4. Performance - Misc
5. Native SSH transport
6. Custom data retention plans (keep more data)
7. Dynamic zooming
Read More...
Mar 31, 2010
Don't use Excerpt... At least with DotClear.
About 1 min read

DotClear automatically generates a meta description tag from the blog entry, but it doesn’t take the excerpt into account.

It just takes the beginning of the article content. Since the excerpt is also shown at the beginning of the article, I cannot just write 2 times the same content.

meta description is quite interesting since it is usually used for the little snipped under a search result in usual search engines, so having the beginning of the post in here is very nice.

This fact annihilates the good point of having excerpts.

I’m now falling back to removing progressively all the excerpts on my posts…
Read More...
Jan 14, 2010
API Design: Avoid hidden costs of simple features
About 6 min read

Programmers are usually like water : they always use the path of least resistance.

Let’s see how to use this fact to predict the usage of an API when you design it.

Initial API

Consider the very simple DB API that consumes a connected ResultSet and presents a disconnected version of it.
```
class DisconnectedResultSet{
	public DisconnectedResultSet (ResultSet rs);
	public boolean next();
	public Object getObject(int col_idx);
}
```
It’s usage is quite easy :
```
while (drs.next()) {
	int col_idx = 1;
	drs.getObject(col_idx++); // Do something w/ 1st col
	drs.getObject(col_idx++); // Do something w/ 2st col
	//...
}
```
Just a little evolution…

Since the DisconnectedResultSet is disconnected, we can imagine that it should implement a rewind() method in order to use it several times without running the initial query again. We now have an updated class :
```
class DisconnectedResultSet{
	public DisconnectedResultSet (ResultSet rs);
	public boolean next();
	public Object getObject(int col_idx);
	public void rewind(); // Be able to rewind it
}
```
And its classical usage :
```
while (drs.next()) {
	// do stuff...
}
// ...
drs.rewind();
while (drs.next()) {
	// do something else with the same data...
}
// ...
drs.rewind();
while (drs.next()) {
	// do something else with the same data...
}
// ...
```
A new need comes

A new need comes : see if the DisconnectedResultSet is empty or not in order to avoid sending header.

The usual way is to send them once when iterating like :
```
boolean is_headers_sent = false;
while (drs.next()) {
	if (! is_headers_sent) {
		send_headers();
		is_headers_sent = true;
	}
	// do something else with the same data...
}
```
But since there is a nice rewind()method, just waiting to be used, the code might become :
```
if (drs.next()) {
	send_headers();
}
drs.rewind();
while (drs.next()) {
	// do something else with the same data...
}
```
Now, this code isn’t generic anymore to accommodate a connected ResultSet.

So, as John Carmack said :

The cost of adding a feature isn’t just the time it takes to code it. The cost also includes the addition of an obstacle to future expansion.

That’s really true when you design APIs since their purpose is to last long and to be extended.

So, think twice when you propose an extension “just in case”.

The little evolution, revisited…

To solve this case, don’t propose a rewind() method, but offer a duplicate() one. It offers the same functionality, just in a new object.

The usage will be almost the same as shown below, but since it feels more performance-sensitive, it won’t be used as lightly : the boolean is_headers_sent pattern has now more chances to be used.
```
while (drs.next()) {
	// do stuff...
}
// ...
drs = drs.duplicate();
while (drs.next()) {
	// do something else with the same data...
}
// ...
drs = drs.duplicate();
while (drs.next()) {
	// do something else with the same data...
}
// ...
```
It’s an other example that immutable objects are the way to go, but for a different reason this time.

Note: Just finished my March 2010 article, even on time… I’m still trying to keep at least a one article per month blogging rate. So far so good for 2010, still 9 months to go !
Read More...
Dec 11, 2009
Compile-Time Polymorphism for Cross-platform Development
About 8 min read

When doing cross-platform development, the common usage is preprocessor #ifdef macros to compile specific parts. This quickly makes the code unreadable.
Read More...
Nov 26, 2009
Native SSH transport for Munin
About 1 min read

Actually the tunnel won’t disappear, but they will be launched only when needed and, most importantly configured in munin.conf.

The native ssh for munin patch should applies quite cleanly on revision 3101 of the svn trunk.

Its use is quite straightforward : in /etc/munin/munin.conf, you just migrate address to the new configuration directive remote_connection_cmd that take the whole ssh command to launch a stdio munin-node such as pmmn.

If we take the examples from the previous post, it becomes clear it’s much easier to configure.

munin.conf snippet - Inetd version
```
[server1]
    address localhost
    port 7001
[server2]
    address localhost
    port 7002
```
munin.conf snippet - Native SSH transport version

Right now, the address directive is still mandatory, but ignored when connecting.
```
[server1]
    address dummy
    remote_connection_cmd /usr/bin/ssh -- supusr@server1 /home/suprusr/pmmn/pmmn.pl
[server2]
    address dummy
    remote_connection_cmd /usr/bin/ssh -- supusr@server2 /home/suprusr/pmmn/pmmn.pl
```
Warning

Beware that the ssh process will now be launched by the munin user, so you have to update the key-based SSH authentication accordingly.
Read More...
Nov 12, 2009
Sed is much slower than Perl, or not...
About 2 min read

I wanted to do some text replacement with a huge file (think ~18GiB), filled with huge lines (think ~2MiB per ligne)¹.

I naïvely piped it through sed and I was quite shocked that it was CPU bound, and not I/O bound. The average rate was about 5 MiB/s (measured with pv, and the CPU was at almost 100%.The text file was gzipped on the filesystem, but with a 1/100 ratio, so the gzip process just took less than 2% CPU. I replaced then the sed -e with the Perl one-liner perl -lnpe, and …. tadaa, it was flying at a rate of 50MiB/s !

While I’m a big fan of Perl, and know its effectiveness to handle text streams, I’m was still astonished : being 10x faster than sed was something.

But in the good old saying “Too good to be true means suspect”, I remembered something about the character encoding of the regular expression. Since the system is entirely configured in UTF8, I suspected the infamous UTF8 overhead over plain ASCII.

I was right : a little LANG=C in front of the sed command line restored the rate to 50MiB/s.

So, beware of the performance impact of UTF8 strings, and try to avoid it if you can.
1. For the record, it was a MySQL dump ↩
Read More...
Sep 11, 2009
Synchronize clock between hosts with SSH
About 1 min read

NTP is very handy for server clock synchronisation, but it can be cumbersome to deploy.

Sometimes you just need to do a one-shot clock synchronisation, so you use the standard date command. But there isn’t a flag to easily copy a setting to another.

From a remote host

Quite easy :
```
date `ssh remoteuser@remotehost date +%m%d%H%M%Y.%S`
```
To a remote host

It’s also very easy¹ :
```
ssh root@remotehost date `date +%m%d%H%M%Y.%S`
```
1. Yes, I do know that logging remotely as root is a security pitfall… ↩
Read More...
Sep 8, 2009
Overloading a method is hard : a common pitfall
About 1 min read

As I said in my equality article, overloading in Java¹ is resolved by the static type of the argument, not the run-time type.

It’s a generic problem of most compiled OO languages since usually overloading resolution happens at compile-time and not at runtime.

Now, that militates for the well known idiom :

Never overload a method with one that has the same number of parameters.

Actually, it should be enough to overload a method with one that accept parameters that are not inheritance-related : String and Number would be OK, but MyClass and Object would not.
1. It’s not really a Java-ism, it’s the same in other languages, such as C++ . ↩
Read More...
Sep 3, 2009
Free Exception lunch : Use unchecked exceptions, but still announce which ones you might throw.
About 1 min read

In a previous article I choosed my side : Unchecked Exceptions are much simpler to use.

But, on the other side of this great division, there is a very valid point : You usually declare checked exceptions. Sure it’s possible to only declare to throw Exception, but that would defeat the whole purpose of using checked exceptions.

The nicest thing is that you can also have a custom exception hierarchy, but based on RuntimeException instead of a plain Exception. This way it’s like in C++. Everything might be thrown, and you don’t need to handle them.

Declaring them, on the other side, is very interesting because you are documenting your interface for almost free.

So, use unchecked exceptions to free yourself of the checked catch-slavery, but still declare the custom ones you might throw.
Read More...
Aug 12, 2009
Databases: Efficient Case-insensitive searches with Function-based Indexing
About 4 min read

Case-insensitive search is sometimes very useful, but a naive approach can be very harmful to your performance.

Special case of case-insensitive search

In Oracle10g, you might use the new case-insensitive search with a NLS_SORT=BINARY_CI command.

Pro
- Designed for this purpose, so it’s very straightforward to use
Con
- Limited to case-insensitive searching by design
Native functional indexes

Some databases provides native functional indexes.

On these databases optimization is done simply by creating an index on UPPER(MY_COLUMN) and letting the query optimizer transparently¹ use the newly created index.

It usually work by applies a function to the data just before handing it to the index, so the function output doesn’t exist in the database.

Pro
- Very easy to use : it just feels right (you can naïvely create an index on the WHERE clause)
- Doesn’t take any extra space in the database (only the index).
- Generic, can be used for something else than just case-insensitive searches.
Con
- Since the data isn’t stored in the database, a call to the function has to be made when
- Functions have to be from the immutable category in the function volatility categories
Generated columns

DB2 provides something called generated columns. It’s almost the same than the native indexes, except that the functional column is explicit.

Pro
- Quite easy to use, since the column is updated and used transparently.
- Generic, can be used for something else than just case-insensitive searches. Just make sure the optimizer uses the extra column. You might have to rewrite the request a little.
Con
- Requires extra space in the table.
- Removing a column can be cumbersome (in DB2 you have to recreate the whole table for example), whereas removing a simple index is much easier.
- The extra column is returned when doing a SELECT * FROM ...².
Trigger-based Generated columns Emulation

If nothing else is provided, you always have the option to emulate. The solution will be trigger-based since it’s one of the few perfect match for them.

So, the base idea is derived from the Generated columns : have a special extra column that represents the output of the function. An index will be created on this column and used via a manual update of the involved requests (Adding an extra WHERE clause should be more than enough, this way you might even benefit from a partial match).

Pro
- Universal. Useful if portability is paramount.
- Very simple : there is no need to understand advanced database features.
Con
- Only a poor man’s solution : everything is manual
- The same than Generated columns since it’s the same idea, just manually implemented.
1. You might have to update some kind of statistics for the new index ↩
2. But you don’t do that anyway, do you ? ↩
Read More...
Jul 3, 2009
Databases: Better Defer Constraints than Avoid Them
About 4 min read

Constraints are a very important tool in a programmer’s belt. But they come with a price, which can usually be mitigated simply.

ACIDity of transactions as rescue…

Let’s examine what all of the A, the C, the I and the D have for implications on our current subject.

Transactions are atomic (A)

The database should not really care about validating the constraints all the time Only the beginning and end state is really important. Inside a transaction, the data may be inconsistent : “Dust hasn’t yet settled”.

Let’s study an example. You have an unique index on an ordering.
```
TABLE ITEMS (
	ITEM_ID SERIAL,
	GROUP_ID INTEGER,
	ORDER INTEGER,
	LABEL VARCHAR
)

ADD UNIQUE INDEX ON ITEMS(GROUP_ID, ORDER)
```
Here, if you want to swap 2 items, you have to use an unused temporary value, otherwise the check won’t be valid at all times.

Transactions are consistent (C)

The check needs to be done at least when the transaction ends. No special need here except that you need to re-enable before ending the transaction.

Transactions are isolated (I)

We can also easily imagine that the check only needs to be done when the transaction ends, just before the commit. Normally no one should be able to see the changing data meanwhile it’s not completed.

One very important thing to know is that if you are using a READ UNCOMMITTED isolation for other transactions you will see inconsistent data.

Anyway, if you are doing that, you know what you are doing and are obviously taking special care about it, don’t you ?

Transactions are durable (D)

This final property has nothing to do with our current issue. Good.

… But sometimes early warning is quite nice

When interacting with external systems, that don’t participate in our transaction, extra care should be taken in order to cope with the exceptional case of a constraint failure. Distributed transactions is a very complex subject, and usually it’s not ~~supported~~ done¹.

Conclusion

With the principle of least surprise in mind, we can easily understand why deferred is not the default behavior, but it makes a very nice addition to our toolbox.

So now there isn’t any good reason anymore not to use (and abuse) constraints in your databases.

Remember, your data is you most precious asset, protect it at all cost from evil misbehaved bug-ridden software² !

Wow… Just in time before the end of the month… I would have failed my motto on URLs ;-)
1. Do you always implement a double-phase commit on every SOAP/REST/XML-HTTP/POST interface that you expose ? ↩
2. Yeah, I have chosen my side in the coming war… ↩
Read More...
Jun 1, 2009
Equality in Java is a Hot Topic, but a Hazardous one.
About 1 min read

It seems that comparing two objects isn’t as a simple task to do as it seems at first.

My concept for smart comparison still holds, but is completed perfectly with the How to Write an Equality Method in Java article posted on Artima soon after mine¹.

2 main points that you have to be careful to (Others are also detailed in this article) :
- hasCode() has to be redefined usually since otherwise the equality contract on Object.hascode() would be broken.
- the equals() has to take an Object as parameter since overloading in Java is resolved by the compile-time type of the argument, not the run-time type.
I just got bitten by the second one, but not much (yet) since I mostly rely on compile-time overloading. My purpose here was only to compare objects either to constants or to simple variables. But in the generalization I would certainly have overlooked this and be bitten much more deeply.
1. But as much as I wished it to be, it is not related :-) ↩
Read More...
May 29, 2009
Synthetic Style for Blog Posts : Presentation Style Blogging
About 2 min read

Blogging has become so mainstream it is fading away to more modern media, not with some drawbacks. With some enhancements, we can have the best of both worlds.

The good
- With SMS, IM and now Twitter becoming more and more predominant : ideas might be given as bullet points
- A presentation is much more dense in meaning than a big blob of text, Less time is required to read the post and be inspired by the content
- Blogging is much less time consuming. As Jeff Artwood said Quantity Always Trumps Quality, it is the commitment that is important (hence the schedule).
- You might divide a long article in several short and related posts.
The bad
- Blog articles are usually a medium-depth analysis of a problem whereas presentation are usually a written support of a more detailed oral presentation.
- Presentation sentences are usually hard to understand since they are just headlines without the underlying context.
- You can always do a fast-reading version by putting the important sentences in bold for all those hasty readers.
The ugly
- Longer don’t always mean more interesting. Voltaire said “Perfection is attained by slow degrees; it requires the hand of time.” and Antoine de Saint-Exupery completed “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”.
Conclusion

As concluded by Pascal, “I made this letter longer than usual because I lack the time to make it shorter”, a good and terse article is therefore much more difficult to achieve. Therefore Presentation Style Blogging may be a false good idea.
Read More...
May 20, 2009
Compare Efficiently in Java : Embrace Smart Comparison
About 7 min read

Bill the Lizard shows us a nice trick to avoid this NPE when used with string literals : call the method on the string literal. It’s possible because it’s also an object, and cannot be null.

But, as much as this trick code seems sweet, but it just feels like a red herring on the long run, First it only works for comparing strings and even only when comparing a string with a string literal.

To Equality …

It is what he said at the end that is much more interesting :

The lesson to be learned here is that if you have to resort to “clever” tricks to get something done, with just a little bit of lateral thinking you can probably find a cleaner, simpler way.

Let’s take this lesson even further.

I had a Perl & C++ background before coming to the Java world, and I am sometimes puzzled that Java makes things quite unnecessary complicated for the programmer from time to time.

My little suggestion : create a quite simple helper static function :
```
static boolean isEqual(String a, String b) {
   if (a == b) return true;

   // a & b cannot be null at the same time
   if (a == null || b == null) return false;

   // Now none can be null
   return a.equals(b);
}
```
Now, the NPE is avoided for any kind of string, with any order. We could also have a smarter equality comparison function by first converting to a string, and comparing the resulting string instead of comparing the original numbers.
```
static boolean isEqualSmart(Object a, Object b) {
   if (a == b) return true;

   // a & b cannot be null at the same time
   if (a == null || b == null) return false;

   // Now none can be null
   return a.toString().equals(b.toString());
}
```
With the wonders of Java’s function overloading, you can even write specialized functions that convert even more smartly their arguments. And then be able to compare a string with a number (even a primitive type).

… and Beyond

We began about testing equality, but for comparison such as <= or >, you have to decode the output of the quite ugly compareTo() method. The helper static function trick comes also handy here.

First, we design a complete API that enable alphanumeric, and numeric comparison : the so-called smart comparison. The API should take any Object, or even primitive types, and convert it at will to enable hybrid comparison. This will greatly enhance the comparisons abilities of Java.
```
class OpAlpha {

   // Alphanumeric comparison
   public static boolean lt(Object a, Object b) {
      /* convert to string and compare */
   }
   public static boolean gt(Object a, Object b) {
      /* convert to string and compare */
   }

   // ... rest of the implementation ...
}


class OpNum {

   // numeric comparison
   public static boolean lt(Object a, Object b) {
      /* convert to numeric and compare */
   }

   // ... rest of the implementation ...
}

class Op {
   // smart comparison, depends on the 2nd argument type
   public static boolean lt(Object a, int b) {
      /* convert to int and compare */
   }
   public static boolean lt(Object a, String b) {
      /* convert to string and compare */
   }
   public static boolean lt(Object a, Number b) {
      /* convert to Number and compare */
   }

   // ... rest of the implementation ...
}
```
The usage is like this :
```
String a = "124";
assertTrue( Op.eq(a, 124) );
assertFalse( Op.lt(a, 1) );
assertTrue( OpAlpha.lt(a, 1) );
```
We can see that all the tedious part of handling the conversions is done under the hood. Choosing the comparison to is just a matter of choosing the right class. A little more work could even be done to take the Op classes to implement Comparator in a complete OO way in addition of the static helpers.

Update (01/06/2009)

It seems that equality in Java is quite hazardous, and that you have to be extra careful not to set a comparison-land-mine off by accident.
Read More...
May 18, 2009
Are Excerpts a Good Thing ?
About 1 min read

One thing I’m really wondering is : “Should I use excerpts” ?

On the plus side :
- The list is easier to read & shorter to parse. You can then just click on the link that are interesting to you
- I can manage to see which posts are mostly read (useful ?) since the comments aren’t a good indicator¹
- Mostly all my readers comes from Google. Having a little list and only then content on a dedicated page seems to help being nicely referenced on more specifics, hence ususally more relevant topics.
On the down side :
- The RSS flow and the front page list are truncated. You are one click further to the whole story.
- It seems quite arrogant to force the reader to come to your site.
I just edited my articles to have an excerpt when it is quite long.

What do you think ?
1. It is very far from being a very famous blog with hordes of readers :-p ↩
Read More...
May 14, 2009
Databases: Efficient Denormalization with Views
About 3 min read

On the other side, to denormalize is sometimes seen as a way to :
- optimize development : you do not need to write (and debug) complex queries since all the data is nicely located in the same table
- optimize performance : the data has a better locality (no need to fetch or compute data from elsewhere). You can even pre-compute order totals¹.
Denormalize correctly is quite difficult since one change snowballs in multiple updates to keep the redundant data coherent. It is therefore usually done on the application side with ready-to-use ORM frameworks. But managing it at the application level comes with an ugly cost : it has to be reimplemented over and over each time the application switches technology, version or when a new application is connecting to the database.

In almost every programming language, code reuse is encouraged. Here, at the database level, we can also apply the same principles, and use views to transparently present a denormalized API (read tables) on a more normalized schema.

The main points are :
- Views are the main code reuse vector at the database level. Current database usually correctly optimize (by rewrite and recombination) the simple queries so that performance is on par with a hand crafted query that doesn’t use the views
- Sometime even faster since the data is nicely organized, so there is much less data to transfer, and I/O bandwidth is a usual suspect.
- The application part doesn’t even need to know that normalization happens under-hood since updates to views are possible in many modern RDMs with the @@INSTEAD OF@@ trigger (or something equivalent).
- Theses updates can be created with Materialized views are a step even further on the denormalizing road, since it provides the common benefits of denormalize without the implementation caveats. We can even hand-craft these Materialized views directly in an aggregation table in order to have the space benefits of normalisation and the performance benefits of denormalization.
So, nothing stops you from normalizing at will, and denormalizing when needed. Where to put the cursor is yours to decide, but moving it afterwards is finally easier that what is commonly admited.
1. That article also explains why denormalization maintenance must stay at the database level with a very interesting metaphor ↩
Read More...
May 11, 2009
Surrogate Keys : Globally Unique, Application Unique or Type Unique ?
About 8 min read

Type unique : One sequence per table

The most common idiom out there. Just define a surrogate key per table, usually done via an auto_increment field, serial field or a sequence per table.

It is the most natural way of thinking, since the ID is usually seen as a monotically incrementing counter. Therefore many applications use even the evil¹ SELECT MAX(ID)+1 construct, because they specifically require that there should be no missed rows and that the ID have to be strictly in order.

The usual implementation of this is to have a kind of thread-local function that hands unique ID allocated in group and never trying to reuse an already issued/allocated ID. If you reserve 50 ID at once you directly level the cost of locking the ressource to only 1 in 50 requests, and not trying to reuse them enables you to just have to lock the end of the spectrum and not the whole spectrum. That’s what the CACHE keyword is about in Oracle and PostgreSQL. Others RDBMS usually also have a similar option

But this optimisation leaves us with a big side-effect : the ID that are generated are not strictly in order anymore, and furthermore there can be huge gaps in them. Since one of the main purpose of one ID per table is now gone, why not directly bite the bullet and use the same ID generation on the the whole application ?

Application unique : One sequence to rule them all

This is something I first experienced with Subversion, coming from a CVS background : the revision number is repository-wide and not file-wide anymore. This has some mental drawbacks at first, since you still know that a file of version 5324 is more recent than version 3200, but not how much more since the commits could have been done in another place. The difference is more visible with 1.4 versus 1.5.

But as I was more used to it, that thinking was obviously a red herring. It did divert us that a file could have been completely rewritten in just one revision, and that one signle line could have be flipped back and forth in several revisions,

So the number, and the gap between them, as the basis of the amount of modifications has its value decreased. What could be quite interesting though is that the revision number is globally incrementing². That enables us to know that a file in revision 1000 was commited before another one with revision 2000.

Back in our database, if we just define one sequence for all the ID in the database we have 2 advantages :
1. One ID represents only one row. Debugging is easier and when you see an ID in the wild, you can think of which row it represents.
2. You can establish an almost exact history of sequencing the creations of the rows.
The inevitable locking performance issue that arises often is made painless through bigger block allocations.

One of the lesser known issue is that, usually when using an ID per table, after a while you begin to know which ID belongs to wich table since all the tables are filled at different rates : 50083 is an order ID, whereas 13650239 looks more like a message ID.

Globally unique :

If you push this logic a little bit further, you can even imagine that all the ID produced could be unique. Then you will have the benefits of unicity, but even on the whole IT level. It makes interapplication debugging easier, since the leaked keys can be nicely tracked down to their origin. How to generate such an unique ID is tricky question, since the applications usually are coded in heterogeneous environemments and technologies.

A central ID service

The most obvious way is to have a central ID service, the same as the unique sequence in the application. It has to be accessed in a distributed way, and the implementation can be quite cumbersome thoughout all the different technologies that exists.

This service has also to have the highest SLA as the applications that depends on it. It becomes then quickly a SPOF if not designed carefully.

GUID

Since online ID generation has some serious drawbacks, the easiest offline way is to generate GUIDs, there are many hooks in each technology/application to be a quite educated move.

It also has drawbacks, but more in the semantic fields. A GUID is quite semantically opaque since looking at it does not tell you anything about where/when the ID was generated. You have to do a full key scan of all the applications to find its origin.

Almost globally unique : Application ID + Application Unique

Earlier, we just found out the application-unique ID. If you pair it with an unique application ID, you just managed to have a globally unique ID. In the generation of this ID, you just prefix it with the ID of the application, and tada…. you have your globally unique ID.

The net effect is that if you see this ID in the wild, you know where it was created. If you have an increasing one, you can even guess almost when.

Parallels between applications and networks

The more I think about it, the more I can draw parallels between rows in applications and adresses in networks, but I’ll write more on that later in another article.
1. Evil because it’s usually not thread safe and quite horridly inefficient ↩
2. Actually in subversion it has a strict incrementing policy ↩
Read More...
Apr 27, 2009
Immutability of an URL
About 1 min read

In the pure spirit of Data is King I think that URL should never change. Even the W3C agrees with their Cool URIs don’t change article.

But we all know that in IT never is only not in the foreseen future. So URL do change, at least after a while, and usually for technical reasons¹.

Since you can update your website to update the URLs, but the inbound link cannot be easily updated. To handle this need, the HTTP protocol has specified the 301 response code.

The solution is that the site should remember all the urls that it generated and redirects accordingly. This way you’ll never loose a potential reader to the infamous 404 (this page does not exist).

Some sites even try to approximate the page on a custom 404 page. That’s another reason to have user-friendly urls : to be able to hint your reader to appropriate pages in case you don’t find his initial destination.

Sadly, this redirect behavior isn’t supported by my blog engine (dotclear)… That’s for the eat your own dog’s food, but I’m looking forward to do it on my current blogging platform.
1. upgrade to another blog engine… ↩
Read More...
Apr 27, 2009
Bringing C++ Const to Java
About 17 min read

In Java, the commonly accepted way is to use the final keyword. But it has a major drawback : the object cannot be redefined, but can be modified by calling mutable members. You have to convert it to an immutable type. This is a simple task, but radically different ways exists.

Achieve Immutable in Java

We’ll see some differents methods with a common example. Let’s start with a very simple Point class :
```
public class Point {
	private int x;
	private int y;
	void setX(int i) { this.x = i; }
	void setY(int i) { this.y = i; }
	int getX() { return this.x; }
	int getY() { return this.y; }
}
```
The Java Collection way

It is the easiest way. It is designed to be a drop-in, like in the Java Collection API. You just create a read-only class that derives your read-write class just like the PointReadOnly that mimics the r/o List. It overrides all the methods either to delegate or to forbid calling by throwing a run-time exception.
```
public class PointReadOnly extends Point {
	private final Point inner;
	public PointReadOnly(Point p) { this.inner = p };

	void setX(int i) {
		throw new UnsupportedOperationException("PointReadOnly is read-only");
	}
	void setY(int i) {
		throw new UnsupportedOperationException("PointReadOnly is read-only");
	}

	int getX() { return inner.getX(); }
	int getY() { return inner.getY(); }
}
```
Advantages

You can use it right-now : Your code can slowly evolve to progressively generate and integrate this new object without really knowing it’s here. The misuses, that are changing the object, will be catched in the testing phase when exceptions are being thrown. The use of a good logging system in the setters is a nice addition.

Disadvantages

Any pedantic compiler should warn you that the setter parameter isn’t read. I tried to be quite clever and find ways to ignore it. Just rewrite it as :
```
	void setX(int i) {
		if (i == 0) { /* Do nothing but keep the compiler happy */ }
		throw new UnsupportedOperationException("PointReadOnly is read-only");
	}
```
The performance penalty is minimal since any compiler should optimize the if-case away and even if not, this function should not be called anyway.

But as a user on StackOverflow pointed, that warning definitively indicates a code smell : this hierarchy blatantly violates Liskov’s principle of substitution since a class receiving an instance of a Point expects setX() to work, and may not handle this exception correctly. Obviously a read-writeable class is readable, but a readable class is not necessarly a read-writeable class. This may introduce subtle bugs that would only be catched at runtime, and therefore testing, as simple as it may be, has to be quite thorough in order not to be bitten at a inconvenient time.

A good solution that respects Liskov’s principle is to redesign the hierarchy, we’ll talk about it just later. But right now, don’t despair, you can have almost the best of the both world : simple, yet almost respecting Liskov’s principle.

Circus time : ride two horses at once

Actually what violates Liskov’s principle is that the setters don’t do the job they are entitled to do. Throwing an exception is a sure way to ensure that the object won’t be modified, but that is usually overkill¹. If you just log the erroneous call, but still do what it should do, you have the warnings, but not the nasty consequences. Just take care to log the callstack also, in order to be able to fix it much more easily.

The new method becomes then :
```
void setX(int i) {
	Log.warn(new UnsupportedOperationException("PointReadOnly is read-only"));
	return inner.setX(i);
}
```
A notable side effect is that your class isn’t really read-only anymore though, just complain-only.

The Object-Oriented way

A much better solution is to create a read-only super class that the read-write one would inherit. The read-only class only implements getters (or any non-changing method). The read-write complement them with the setters.

It would lead to code like :
```
public class PointReadOnly {
	private final Point inner;
	public PointReadOnly(Point p) { this.inner = p };

	int getX() { return inner.getX(); }
	int getY() { return inner.getY(); }
}

public class Point extends PointReadOnly {
	private int x;
	private int y;
	void setX(int i) { this.x = i; }
	void setY(int i) { this.y = i; }
	int getX() { return this.x; }
	int getY() { return this.y; }
}
```
Advantages

Now Liskov’s principle is not violated anymore and code that expects a read-only class can seamlessly have a r/o or a r/w class.

The changing will also spread naturally all throughout the codebase, just like the C++ const modifier and failures to comply would be caught at compile time and not at runtime.

The r/o is a nice wrapper for the read-only class, and a r/w can even be used in place of the r/o place.

Disadvantages

It doesn’t feel right :The r/o class is just a placeholder for a r/w class.

Comments

An interface-base inheritance scheme can even be better. Point as the r/o interface, PointMutable as the r/w and PointBase as the base implentation class. This way you can avoid the wrapper class. You also get the C++ const_cast<>() for free (by just downcasting as PointMutable).
```
interface Point {
	int getX();
	int getY();
}
interface PointMutable extends Point {
	void setX(int i);
	void setY(int i);
}
class PointBase implements Point {
	private int x;
	private int y;
	void setX(int i) { this.x = i; }
	void setY(int i) { this.y = i; }
	int getX() { return this.x; }
	int getY() { return this.y; }
}
```
I realize this does not answer your question about getting rid of the warnings. But warnings can either be suppressed, ignored, or addressed. An unused parameter is often a bad smell that indicates your method might not be doing what it’s expected to do. Methods should only get essential parameters. If the parameter is not used, the parameter is not essential, and therefore something needs to be changed.

The String way

Another way is just to avoid the read-only and mutable API all together and go the functional way. Since it’s a completely different paradigm, it doesn’t exactly fits the bill here but it’s a quite important way of cutting the gordian knot. Its concepts are rooted in both functional programming and RAII. It is quite practical only if you have an efficient garbage collection system, since it has the tendency to create lots of temporary objets that are mostly created, used and forgot.

Do do thing, you have to redesign the whole class, in order to be able to defined it only once, and when you call a modifying method, you end up with an other object, also immutable. The String, Integer and Class class make extensive usage of this.

So, our class will become :
```
class final Point {
	private final int x;
	private final int y;
	Point(int x, int y) { this.x = x; this.y = y; }
	int getX() { return this.x; }
	int getY() { return this.y; }

	void setX(int i) { return new Point(i, this.y); }
	void setY(int i) { return new Point(this.x, i); }
}
```
As you can see, every object is not modifiable, if a method recieves the object as an argument, it cannot modify it, since it would have a new object. The properties are final, and the class has usually to be made final in order not to be able to mutate the class by inheritance.

This enables programming without side-effect and therefore is quite interesting for multi-thread programming.

Conclusion & thoughts

The UnsupportedOperationException is a very practical idiom, that even the Java Collection API is based on it. It may not be very nice in theory, but in practice it has its usages. It should not be abused thought, and the OO-way should be prefered when possible, usually when you are in control of all the code base.

The logging approch doesn’t really solve the problem since it’s only advisory, but can fit the bill when the code base is huge and you want to be able to understand what is going wrong.

The OO-way is a nice compromise between the abstract beauty of the String-way and the quite hugly kludge of the Collection-way, especially when used with the interfaces scheme : it spreads out as the const param, and force the coder to think if a modifying is really needed.

The String-way is very tempting, as a true create-and-forget way. If you could even enforce to only have one instance of each representation you’ll have a free == (almost) overloaded operator².

So we have at least 3 ways to learn and adapt idioms from C++ in Java. The major issue is that it’s not as itegrated as const : every object needs custom code except maybe the String-way.
1. It all depends on the indent, specific policies on specific usages. ↩
2. Just be aware of classloader issue with static. Hmmm that will be covered in another article. ↩
Read More...
Apr 19, 2009
Checked or Unchecked Exceptions for Legacy Code ?
About 3 min read

In theory checked exceptions are quite nice since the called code can communicate with it’s caller when something unexpected happened.

Checked Exceptions are breaking encapsulation

The key word here is unexpected. If you have to explicitly know the exceptions that could occur, it’s not really unexpected. And if it’s not unexpected, using exception handling just add an out-of-band data path. It’s on par with transporting data in a private class field when calling a member instead of using its arguments. This leads to breaking encapsulation as Alan Griffiths wrote in Exceptional Java. This vision is also shared by Bruce Eckel in his article entitled Does Java need Checked Exceptions?

Checked Exceptions are quite painful to use

Local Exception Handling is hard to manage

I personally find checked exceptions quite painful to use. By definition, you have to catch every exception that is thrown by the underlaying code.

And if the underlaying code doesn’t know what to do with the exception, chances are, that you don’t know either, so you just pass the exception to the caller. And so on…

Too much code to change

Therefore exceptions are usually caught at the top level with a generic catch-all structure that logs the error, since no layer could sensibly do something clever with the exception.

Then you just have to change all the signature of the whole stack, just to be able to catch them at the top. Using unchecked exception lets you have this for free, and conveys the meaning that nothing is caught until the top.

Unchecked Exceptions might be dangerous…

Obviously, unchecked means not checked. So you might fail to catch them at the top level and then the whole application crashes. Checked exceptions are a safeguard against that. Just like strong static typing is.

You trade compile-time safety (checked) for development-time speed and ease (unchecked).

… but are not really.

On the other hand, if you have a good design, you don’t have much different top levels, and then the risk is somewhat limited.

Moreover, since you always have to take unchecked one into account, why don’t use them ?

By the way, in C++, the exceptions are unchecked by default.
Read More...
Apr 8, 2009
Email Ping to Comments Reply on Blogs
About 1 min read

As a blog writer I can receive the comments written on my blog via email.

I’m wondering why this service isn’t implemented for replies to my comments on foreign blogs. I don’t really want to remember all the blogs I left a comment on in order to poll them to see if there is a reply to my comment.

Actually, a quick hack would be to subscribe to the RSS feed of this particular entry and let the RSS feeder handle all the remembrance and polling for us, but that seems so inneffective. Just imagine all the wasted traffic induced by polling if this habit becomes mainstreams.

I thought about turning it the other way around, since it seems that I’m forced to leave an email adress on each comment I leave, why don’t the blogging engines just use this adress to send a notification (maybe opt-in) if there is a activity ?

I wanted to activate this feature on my blog and it seems non-existant. I therefore searched around quickly only to discover that noone seems to be having this particular need.

Strange…

Edit(15/04/2009): I just found out that WordPress seems to support this functionality via a plugin : Subscribe to Comments
Read More...
Apr 7, 2009
Should The URL Include a Date, or Not ?
About 3 min read

A common issue is that a blog entry takes time to be written, refined and then published. Therefore the date contained in the URL isn’t the publication time. But since the publication time is the time the entry is created in the point of view of the public the date in the URL conveys only little meaning, just the time is took to create it.

Another point is that some blogger, i’m no exception, just create an entry to make a note. Theses entries are composed of bare notes without any relation or phrases. It serves as a unique receptacle of related raw ideas, useful not to loose them or to be obliged to scan through many medium just to make sure we didn’t forget something that we thought important earlier. Then the date in the URL conveys very little meaning, mostly when we first had the idea about it.

On the opposite side, a date in the URL is quite interesting to know the relative age of a post without having to look in detail inside. This information is especially important in a technical field such as IT where the world is turning around quite fast.

We can see now that the timing information is important, just that the standard precision of one day is a bit overkill. All depends on the time taken to write the blog entry, and on the time scope it we be relevant.

For a article about a precise event, a precision of one day is interesting. At this end of the scale, blog entries that relates a event throughout a day would even have a hour precision. If most of the articles had a shift of 1 to 2 days in the creation time and the publication time, a precision of a day would be a fair deal. And the other side of the spectrum, an article about a slowing evolving subject such as demographic studies are fairly represented by a yearly url. For timeless subject such as philosophy, I still think that yearly would be adequate since one year represent quite a long time on a human scale.

Since usually my blog entries are in gestation for about 1 to 3 weeks, and are about IT subjects, a monthly precision is the most interesting since that conveys an adequate timing information in a changing world without having a false impression of exactitude.

Therefore I’m changing my URL format to a monthly rule, meanwhile allowing myself a different precision for special entries on specific subjects.

Unfortunatly it is very discouraged to change the URL once it’s published since otherwise you have to redirect all the links¹ that are currently pointing on it.
1. from your other articles, from other blogs or from search engines ↩
Read More...
Apr 7, 2009
Databases: Partial Indexing
About 7 min read

Let’s take the table that contains orders that we created in my previous post to show you some examples.

Orders have usually a state, Let’s create it¹ :
```
ALTER TABLE ORDERS ADD COLUMN STATE
```
Now usually a application makes use of the state of the order by querying it, even quite often. The main problem is that this query will do a full scannof the table unless the state field is indexed. Here comes the other issue : indexes take very much space since they have to be ordered and cannot be trimmed (except throw an expensive DROP/CREATE or REINDEX).

The space used is adequate if the index has many different values, but usually a state has only a finite set of values, and usually some are very transient and other quite long. Compare for example the state CANCELLED or SHIPPED that are mostly final to the very transient state INVOICE_TO_BE_PRINTED.

The number of orders that are in the state INVOICE_TO_BE_PRINTED are quite a few (especially compared to the final states). However these temporary states are the mostly queried (usually by batches, pollers or even web pages that present the workload to the users for manual polling).

Another overlooked usage is to have an ALMOST-UNIQUE index. I mean an unique index, but only on selected entries. In our ORDERS table, we can add a field that represent the PRINTING_PID of the process that is responsible for printing the invoice. It is only unique for the orders that are in the INVOICE_TO_BE_PRINTED state. A normal index isn’t much of a help here since either you have to clear it afterwards, but you loose information, or you cannot enforce any uniqueness.

So there comes the PARTIAL INDEX. It is native in some databases² but in most it isn’t. Triggers comes then to the rescue. Some are not really fond of triggers since many things can happen undercover, but my moto is to use trigger in order to, and only to, maintain business-agnostic extra datas (usually to cope automatically with denormalization). Triggers shall not change the data, merely copy and/or move it around.

The new almost-unique column is also created :
```
ALTER TABLE ORDERS ADD COLUMN PRINTING_ID
```
Implementations

Reducing the index size and the overhead

So, a new table is created to contain the partial index :
```
TABLE PI_ORDERS_STATE (
   INT ORDERS_ID FOREIGN KEY ON ORDERS(ORDERS_ID),
   STATE,
   INDEX ON (STATE)
)
```
And the insertion/deletion for the table is done via triggers. I do it a little rapidly since the real creation is a little more tedious than the pseudo-code here :
```
CREATE TRIGGER ON ORDERS WHERE INSERT, UPDATE
  WHEN o.STATE not in (PENDING, CANCELLED, SHIPPED)
    INSERT INTO PI_ORDERS_STATE(o.ORDERS_ID, o.STATE) IF ORDERS_ID NOT PRESENT
  ELSE
    DELETE FROM PI_ORDERS_STATE
      WHERE ORDERS_ID = o.ORDERS_ID
```
One should rewrite queries to take advantage of this partial index since when it’s integrated in the database the optimizer just does it himself. A query like this

Old :
```
SELECT * FROM ORDERS WHERE ORDERS.STATE = INVOICE_TO_BE_PRINTED
```
New :
```
SELECT o.* FROM ORDERS o
INNER JOIN PI_ORDERS_STATE p ON p.ORDERS_ID = o.ORDERS_ID p.STATE = INVOICE_TO_BE_PRINTED
```
The query optimizer should now be able to use the small index in PI_ORDERS_STATE, and retrieve only the relevant orders via their primary key in ORDERS.

Almost-unique colums

To handle the almost-unique column, it’s basically the same idea : a new table, new triggers :
```
TABLE PI_ORDERS_PRINTING_PID (
   INT ORDERS_ID FOREIGN KEY ON ORDERS(ORDERS_ID),
   PRINTING_PID,
   UNIQUE INDEX ON (PRINTING_PID)
)
```
```
CREATE TRIGGER ON ORDERS WHERE INSERT, UPDATE
  WHEN o.STATE = INVOICE_TO_BE_PRINTED
    INSERT INTO PI_ORDERS_PRINTING_PID(o.ORDERS_ID, o.PRINTING_PID) IF ORDERS_ID NOT PRESENT
  ELSE
    DELETE FROM PI_ORDERS_PRINTING_PID
      WHERE ORDERS_ID = o.ORDERS_ID
```
No need to rewrite any query here.
1. Notice that here I don’t create all the meta-data stuff. That is left as an exercise to the reader - Yeah, I always wanted to say that… ↩
2. MS-SQL has it but calls it Filtered Index. PostgreSQL calls it PARTIAL INDEX. ↩
Read More...
Mar 31, 2009
Databases: Version History
About 4 min read

One concept gaining huge momentum lately is file versionning (mostly Git and Subversion). It is quite interesting to track the evolution of the data contained in the files, and not only the last time the file was updated.

On the last post Databases: Meta-Data I was discussing about the merits of having a modification timestamp among with other various informations and more generally about the parallels between a database and a filesystem.

We can also adapt this model to our database by not only storing the last modification date, but also the old data.

Two main options are :
- replicate the whole row in a history table
- have a generic history table that only store the old values of the column that have changed
The approach is done in the way this ddj database article, but in a much less intrusive manner. We just have our trigger-based system evolve with every modification logging an insertion in the derived history table.

The history table and trigger become :
```
TABLE MD_ORDERS_HISTORY (
   SERIAL MD_ORDERS_HISTORY_ID PRIMARY KEY,
   TIMESTAMP CTIME,
   CHAR TYPE DEFAULT 'M',
   INT ORDERS_ID FOREIGN_KEY ON ORDERS(ORDERS_ID),
   MONEY PRICE_ADJUSTMENT
)

CREATE INSERT TRIGGER ON ORDERS o
BEGIN
  INSERT INTO MD_ORDERS (ORDERS_ID) VALUES (o.ORDERS_ID)
END

CREATE UPDATE TRIGGER ON ORDERS o
BEGIN
  UPDATE MD_ORDERS SET MTIME=NOW(), MLOGIN=CURRENT_USER()
    WHERE ORDERS_ID = o.ORDERS_ID
  -- Log the change
  INSERT INTO MD_ORDERS_HISTORY (ORDERS_ID, PRICE_ADJUSTMENT)
    VALUES (o.ORDERS_ID, o.PRICE_ADJUSTMENT)
END

CREATE DELETE TRIGGER ON ORDERS o
BEGIN
  -- Log the delete
  INSERT INTO MD_ORDERS_HISTORY (ORDERS_ID, PRICE_ADJUSTMENT, TYPE)
    VALUES (o.ORDERS_ID, o.PRICE_ADJUSTMENT, 'D')
END
```
The interesting point is that you can now travel back in time in order to see what happened to a row in case of debugging the application or its (mis)usage.

The main issue of this system is that your database usage will be larger by several orders of magnitude, depending of how often your application update its data.

Several possibilities to limit the size exists :
- Partitioning the historical data, and storing it on a slower (cheaper) array.
- Pruning the old data since depending on the application, traveling back 1 whole year may be overkill
- Live aggregating changes. If the data was modified less than 1 hour ago, just update the last history line. We can even have a history_start timestamp and history_stop timestamp to show that a aggregation has taken place.
- Deferred aggregating changes. Like the live one, but on a scheduled basis. It can even have a dynamic granularity (a granularity of 1 day if the data is 1 year old, 1 hour if it is 1 month old).
Read More...
Mar 19, 2009
A Simple Dns Server for a SOHO Network
About 1 min read

I’m in search of a very simple DNS Server for a small network. It should be :
- recursive & caching (can be used as a proxy)
- very simple administration (parsing /etc/hosts would be perfect, raw DNS zones like BIND would be a little bit overkill)
- quite lightweight (aka no dependency on an SQL engine like MySQL, such as MyDNS)
- Seamless integration to Windows lookups (nmblookup) via proxying functions (DNS to/from NMB)
Read More...
Dec 16, 2008
Databases: Meta-Data (ctime & mtime)
About 6 min read

Having this, knowing the last modification of an item is as easy as querying this extra information that is updated automatically by the system.

Adaptation to RDBMS is quite easy. Either the RDBMS comes already with the necessary tools to records this kind of extra information, or it is just a matter of ghosting the schema with tables that will only contain the meta-data. The information inside is updated automatically via triggers on the main table.

An example would be for the ORDERS and ORDER_ITEMS tables (I don’t use a specific SQL dialect, it’s just for illustrate purposes. Just feel free to adapt the code to your favorite database) :
```
TABLE ORDERS (
   SERIAL ORDERS_ID PRIMARY KEY,
   MONEY PRICE_ADJUSTEMENT
)

TABLE ORDER_ITEMS (
   SERIAL ORDER_ITEMS_ID PRIMARY KEY,
   INTEGER ORDERS_ID FOREIGN_KEY ON ORDERS(ORDERS_ID),
   INTEGER PRODUCT_ID,
   INTEGER QUANTITY,
   MONEY UNIT_PRICE
)
```
We have to create 2 ghost tables MD_ORDERS and MD_ORDER_ITEMS :
```
TABLE MD_ORDERS (
   INT ORDERS_ID FOREIGN_KEY ON ORDERS(ORDERS_ID),
   TIMESTAMP CTIME DEFAULT NOW(),
   TIMESTAMP MTIME DEFAULT NOW(),
   VARCHAR CLOGIN DEFAULT CURRENT_USER(),
   VARCHAR MLOGIN DEFAULT CURRENT_USER()
)

TABLE ORDER_ITEMS (
   INT ORDER_ITEMS_ID FOREIGN_KEY ON ORDER_ITEMS(ORDER_ITEMS_ID),
   TIMESTAMP CTIME DEFAULT NOW(),
   TIMESTAMP MTIME DEFAULT NOW(),
   VARCHAR CLOGIN DEFAULT CURRENT_USER(),
   VARCHAR MLOGIN DEFAULT CURRENT_USER()
)
```
The colums CTIME and MTIME are obviously like their corresponding part in file-systems. CLOGIN and MLOGIN are either the login used to connect to the database or for a web application that usually have only one DB login the current application user connected (To communicate this information to the database system, tt could be inserted in a special temporary table at the beginning of each request, scoped at the current connection/transaction/end-user-request/… in a way shared by all applications).

Updating theses 2 tables is as easy as adding several TRIGGERS. (The examples given below are only for ORDERS, for ORDER_ITEMS it’s the same pattern)
```
CREATE INSERT TRIGGER ON ORDERS o
BEGIN
  INSERT INTO MD_ORDERS (ORDERS_ID) VALUES (o.ORDERS_ID)
END

CREATE UPDATE TRIGGER ON ORDERS o
BEGIN
  UPDATE MD_ORDERS SET MTIME=NOW(), MLOGIN=CURRENT_USER()
    WHERE ORDERS_ID = o.ORDERS_ID
END
```
The obvious avantage to this that you can use this technique right now on your application, since it doesn’t require any applicative change. The new columns are completely transparent. Even the locking scheme is the same : if the shadow row has to be locked for updating, the underlaying base row is currently also locked anyway. The only side effect is that you will have effectively 2 times more updates in the database system, and therefore you have to be careful at where you put your new tables (in order not to be bitten by the I/O increase cost) if you have a clever table/tablespace layout.

Another advantage is that if you relax your foreign keys constraints you can even log the deletion date of a row in the underlying base table.
```
TABLE MD_ORDERS (
   INT ORDERS_ID ON ORDERS(ORDERS_ID) INDEXED,
   TIMESTAMP CTIME DEFAULT NOW(),
   TIMESTAMP MTIME DEFAULT NOW(),
   TIMESTAMP DTIME DEFAULT NULL,
   VARCHAR CLOGIN DEFAULT CURRENT_USER(),
   VARCHAR MLOGIN DEFAULT CURRENT_USER(),
   VARCHAR DLOGIN DEFAULT NULL
)
```
The DELETE trigger is obvious, but the INSERT one has to be careful to handle the insertion of a deleted entry.
Read More...
Oct 4, 2008
A Poor Man's Munin Node to Monitor "Hostile" UNIX Servers
About 4 min read

Munin is a nice monitoring system. Simple but quite effective. It’s main selling point is the UNIX-esque simplicity of the architecture. You can just create a new plugin in a matter of minutes to monitor whatever you can imagine.

There is even a comprehensive collection of plugins ready to use (admittedly of various quality).

Various platform are supported

Usually the main issue is the MuninNode, an agent (daemon) that runs on the server to be monitored, since it is responsible of translating the request of the munin server (the one with the graphs) to the various plugins, build-ins or external. The ease of installing this agent depends on the OS and the access you have on the server :

Windows

For Windows you can install munin-node-win32.

Unix (with root access)

For Unix when you are root, usually there is a package ready to install in your distribution, or from the source.

Others (Hostile servers)

On hostile servers, you don’t usually have a root access and no easy acces to a compiler.

I wrote pmmn (Poor Man’s Munin Node), a little vanilla Perl script that emulates the core functionality of the real munin-node script, but without having to install many Perl CPAN modules.

It has also a nice functionality : it is possible to communicate via stdin/stdout instead of a TCP port. This way it is very easy to monitor hosts that are behind a firewall without opening (and monitoring) many tunnels.

Installation of pmmn

Suppose you have access to the server via a supervision user (let’s say supusr). Installation of pmmn is quite easy : just copy the files somewhere on the disk where you have access, for example (/home/supusr).

TCP installation

Just launch the server with -p 4949 and declare it in the munin.conf file on your munin-server. This solution is quite equivalent to a regular munin-node installation.

Tunnel installation

Same as the TCP, but you have to create a TCP tunnel via SSH to be able to reach the munin-node.

Inetd+SSH installation

It is a mix between port forwarding via inetd and the Tunnel-based previously discussed.

You first have to established a key-based SSH authentication without passphrase (you will not be there to type it) from supusr on the inet server (usually the munin-node one) to the user supusr on the server to be monitored.

For example, to monitor server1 and server2, in the file /etc/inetd.conf, you have to add lines :
```
7001   stream  tcp     nowait  supusr /usr/bin/ssh -- supusr@server1 /home/suprusr/pmmn/pmmn.pl
7002   stream  tcp     nowait  supusr /usr/bin/ssh -- supusr@server2 /home/suprusr/pmmn/pmmn.pl
```
Then, in the munin.conf file of the MuninServer, you just have to declare the new nodes :
```
[server1]
    address localhost
    port 7001
[server2]
    address localhost
    port 7002
```
The MuninServer will now set up a stdin/stdout SSH tunnel transparently and launch the pmmn server when needed. You are now free to write plugins like if a real munin-node where installed. The only restriction is that y ou don’t have a root access, so you are limited in the information you may collect.
Read More...
Jul 31, 2008
Daisy Chain Setters and Handle Optional Parameters Effectively
About 5 min read

The Pure RAII way

You use many different constructor signatures. It’s quite suboptimal if you have many parameters that have nothing in common except their type : you have to use the infamous null, or have a special value that conveys a not specified meaning.

Code usage is something like this :
```
MyFile fileRead = new MyFile("in.txt", Flags.Read);
MyFile fileTemp = new MyFile("tmp.txt",
    Flags.Write, Boolean.TRUE, Boolean.FALSE);
MyFile fileWrite = new MyFile("out.txt",
    Flags.Write, null, Flags.Boolean.FALSE);
```
The JavaBean way

The JavaBean standard militates for a simple constructor : the default one. The client uses then setters to initialize the object. This isn’t RAII anymore, but is a very common idiom in Java. But in my opinion the code produced (like the one below) is quite cluttered.

It produces code like this :
```
MyFile file = new MyFile();
file.setFileName(fileName);
file.setOpenFlags(openFlags);
file.setShouldLock(shouldLock);
file.setIsSync(isSync);
file.setMaxFileSize(maxFileSize);
file.setReadAheadSize(readAheadSize);
```
The StringBuffer way

With the same trick as the StringBuffer.append(), it is possible to chain all the setters just like this :
```
MyFile file = new MyFile()
    .setFileName(fileName)
    .setOpenFlags(openFlags)
    .setShouldLock(shouldLock)
    .setIsSync(isSync)
    .setMaxFileSize(maxFileSize)
    .setReadAheadSize(readAheadSize)
  ;
```
In my opinion this is more typo-proof, since you don’t need to repeat the variable name each time. It may also been seen as easier to read since the whole initializing part is done in one block.

The main design point with this construct is that the setters must honor exactly the same contract than the constructor does :
- They cannot return null (otherwise chaining them will throw an uninformative NPE). Only exceptions are allowed to signal a failure while setting the value.
- In case they throw an exception, they have to cleanup any non-memory resource, since the caller has no reference on the object anymore.
These rules are quite natural if you think of the setters as a extension to the constructor. The object is still responsible for the resources it manages, although the resources itself can change when calling setters (by setting a different filename for example).

The Immutable way

Since every setter returns a MyFile, you can use an immutable design. It has many benefits, specifically when multi threading. The concept is quite easy, and inspired from the String object. Each constructor creates a brand new object that copies every properties from the parent except the one that is changing. The downside is it might create many unnecessary objects but it is a good concept for factories that are seldom created, but used many times.

Factories, can then create object that have already all their properties set with the factory default. It can then reuse pooled objets that have the same properties.

An example for a factory can be :
```
private final static MyFileFactory fileFactory =
    new MyFileFactory().setIsSync(true);
...
    MyFile file =fileFactory.create("filename.txt");
```
The MyFileFactory.create() must honor the same contract than the MyFile constructor for the same reasons than the setters do.
Read More...
Jul 27, 2008
RAII in Java to clean your code
About 4 min read

RAII is a very common idiom in C++ and some other languages that don’t have an integrated garbage collection management.

Java has GC, therefore this idiom is not as popular. But the main problem of Java is that although the GC system has become quite efficient, it only handles the memory management. For other resources (database connections, sockets or file descriptors for exemple), this system is not really adequate. The release of these resources has always to be explicit, and handling this via the finalize() method is not satisfactory.

In short the finalize execute itself when the object is about to be garbaged. The main problem is that this garbaging does only take into account the memory limits, not the resources limits (max number of open file descriptors for example). So you can run out of open file descriptors way before running out of free memory.

So, the usual construction is like this :
```
MyResource res = null;
try {
  res = new MyResource();
  res.setSomething(someValue);
  /* Use the resource */
  res.close();
} catch (Exception e) {
  // release the resource if needed
  if (res != null) { res.close(); }
}
```
But hey, that’s many code lines, and in case of a Throwable, you don’t release the resource. The concept of releasing the resources with a try { } finally { } construct is much better (actually, it’s one of the most common usage of finally).

The construction becomes :
```
MyResource res = null;
try {
  res = new MyResource();
  res.setSomething(someValue);
  /* Use the resource */
} finally {
  if (res != null) { res.close(); }
}
```
But here we can see that Java is not quite different from C++ for that matter, so we can just adapt the C++-ism that is RAII, and write a much cleaner version that aquire the ressource in the constructor, so most failure conditions can be checked immediatly.

The construction becomes finally :
```
MyResource res = new MyResource(someValue);
try {
  /* Use the resource */
} finally {
  res.close();
}
```
Since a constructor never returns a null value, there is no need to test. And if the constructor throws an exceptions, the general contract is that the object does not exists. Therefore no resource has been allocated since it would be impossible for the caller to release it (remember, no object was created). So there is no need to release it.

The setter is also integrated in the constructor, since the whole RAII concept is that the constructor returns a completely initialized object. It also enables to write cleaner code since when calling the close() there is no need to do some if() to know the object initialisation-state.
Read More...
May 26, 2008
Why Negative comments are better than positive ones
About 3 min read

Why Negative comments are better than positive ones If you are reading this blog and thinking “Hey ! What a He’s just talking nonsense…” please don’t walk away in horror. It would be very much appreciated if you drop me a comment about what is wrong in my post instead. I actually prefer to have negative comments than positive ones… at least when they are well argumented.

As a matter of fact, we mostly learn by our mistakes. If you do something and it works, you are really happy but you don’t know why it works. The next time you have to do something quite similar you are tempted to change the least possible in order not to be disappointed. This leads to the infamous Cargo Cult Programming effect. Whereas if it doesn’t work you spend some times, but just learned something that you can subsequently reuse.

It’s actually by been challenged that you make the most interesting progress. I nevertheless agree that when in isolation your raw productivity is much bigger than when being part of a team. The reason if quite obvious : since you don’t have to argue with others, you can spend all your time doing useful stuff. The main problem with this approach is that the real goal (where you really should go), isn’t necessarily where you think it is. So you just might go very fast, but aiming the wrong goal. Sometimes the very fact explaining something to someone (that didn’t even disagree) can show you the internal problems of your way of thinking.

It’s actually the convincing-battle that you have to fight with your audience/co-workers/etc that leads to the most interesting solutions. We are all humans, and each had different experiences, hence different point of views. So the real cleverness is to be able to take the ladder of your opinions, climb with them on the shoulder of giants and give the feed back your fellow giants what your new point of view gives you to see. I really insist on the feed-back stage, knowledge is something you can even increase by giving it, since it usually makes you think ways that you would not have explored normally.

So I do write in this blog with my own convictions. It is certainly not the universal truth, but it’s my very own vision of it. If you feel that I’m wrong, feel free to tell me : I don’t say I’ll agree with you, but it may be a very interesting battle that might even elevate both of us.
Read More...
Mar 28, 2008
Speed Up OpenOffice with LeapHeap
About 1 min read

Using a new heap manager like Leap Heap under windows takes OpenOffice to blazing speeds. Tried the same with FireFox with the same conclusions.

Would it be possible to learn from this little add-in ?
Read More...
Jan 5, 2008
A Little History of PWKF
About 6 min read

It all began with a scratching need as I just felt that workflows (mostly BPM engines to be more precise) were the way to do many things in IT of the future. There are even many workflows out here already. My main feeling was that they put too much emphasis on how much of the standard they support, mostly in order to be “Entreprise Grade”.

That felt just plain wrong to me : WF where at first designed (at least that’s was the usual original marketing scheme) to put back the design of the business rules in the hands of our beloved users, in order to mostly bypass the IT departement, and be able to change and adapt the process very rapidly as described on ACM. With the current implementations of WF, i just have to feeling that it involves a lot more of XML and/or BPEL files to be written than a non-IT worker can manage.

I think that WFs are a very good concept, but the emphasis should be on the modeling, on the reporting, and on the operational part (in that order). The number of constructs that the implementation recognise isn’t that important : it could support a much smaller instruction set, but doing it in a user-friendly way.
- The modeling is the most important part of the equation.
It’s always über-fustrating that we have to attend to meetings, capturing the informations on how to model the business rules, tranlate it into something that the systems understand, and to try to explain it to the users. The most important part of the modeling is that the model is accurate enough. Modeling is usually the difficult part of the job. I don’t advocate that the business users should be able to model it themselves : it’s usually a recipe for a disaster since they mostly don’t have the mind for this. But the opposite is also true : the user has to understand easily what is modeled. Modeling a workflow is like writing a book : it takes a different skillset to write it and to read it to be able to spot inaccuracies. Since users aren’t very good at decyphering computer languages, having a good bijection between the modelling and a random-human-readable form is very important. That enables the fact that you users don’t have to model their business through you, but with you.
- Reporting is the second most important part
The debugging part is also very important. Since everyone makes mistakes, it’s very important to be able to spot them. So we have to see what happened to a particular order. Usually that’s the most difficult part of the system, but with workflow, it’s quite easy to log everything that happened, and be able to show it later, even to show a snapshot of the current state and history of a particular workcase. If you also manage to show it with a graphical form that is the same than the modeling one, you’ll have bug reports that would be much more accurate. The users will be able to tell you : “This workcase has gone through here and there, but since it’s the special case A, it should have gone here instead.” and then bugfixing would be usually like a piece of cake.
- Operating the workflow is also important.
You don’t use the workflow for a living. you design them. But that’s not a reason to make the life of operators miserable :-). And then they should have a nice list of tasks to be done. Tasks forms should be standardized, and it should be possible to have a wizard-like approach so they don’t need to fill submit, reopen the case on another task, fill and resubmit anbd so one. The current workcase could be left opened, with the forms “advancing” without need to reselect them again. And as Alan Kay said : “Simple things should be simple, complex things should be possible. “, once all those 3 priorities are done, it’s very easy to divide task that’s isn’t defined in the core workflow to an external “plugin”, such as “signaling an other application to do something, via a webservice for example”.

So, that laid the initial approach to PWKF, which was at first named “Perl Workflow” since I was planning to do it in Perl & wxWidgets. I migrated to Java since i wasn’t that fluent in Perl anymore since I mostly use Java now at work and therefore rename PWKF in “Personal Workflow”, in the way PHP is for ‘Personal Home Page’.
Read More...
Dec 3, 2007
Use Immutable Objects to Avoid Synchronisation
About 4 min read

With the future and its multiple core environnements as stated in a previous post about workflows, efficient locking will be more and more an issue.

My previous way to cut this gordian knot was to :
1. multiply the objects that could be locked to reduce contention : have many multiple elementary objects . These can be workcases in the workflow theory.
2. cheat to minimize the time spend on locking : use something called software transactional memory that only locks at aquiring the ressource (actually, here it means taking a copy) and only updating it at the end of the processing (remember those infamous access EJB ?). This can be that at be beginning of computing a task, every data from the workcase is copied in a new, non-shared, worktask. All the task work will then be done on the privately copied data. It can surely be optimised in copying only the data that “might” be used (read and/or write). And at the end of the task, the workcase is just “commited” (updated) in the main data storage. The nice thing is that you only need to synchronise the beginning, the end and to prevent concurrent modifications (usually done with a incrementing version counter).
Now, if you cross this with another previous post about caching cleverly and sparingly, you can also have another way of having for exemple a configuration that is at the same time :
1. fast
2. can be updated at runtime
3. transactional (once you access it once, you will have all the properties that are coherent together)
The idea is to use immutable objects (such as java.lang.String). They are usually despised as memory eaters since you have to create a whole bunch of objects since you cannot modifiy them, only recreate them with the updated values. But they have a very nice property : there are completly thread-safe, since no one can modify them, so they are lock-free.

So, just imaging that the first time you ask for a configuration, you just load the whole in an immutable config object into something like a singleton. You just hand a reference to it to the called after you stored the reference in the caller’s context (could be a HttpServletRequest). The second time the caller asks for the configuration it’s already in its HttpServletRequest, so you take it from there.

Meanwhile some other thread just asks to refresh the configuration, a new immutable config object is created, it’s swapped with the old one (only the reference is updated, not the object). This swap and the handing are to be synchronized together (it’s not even always mandatory, since usually if there are several handings that give the old value, it’s often not that problematic : the whole ole value is coherent). When all the old contexts will go out of scope, so will the old config object.

The use of immutable objects has becomed much easier with GC, since we don’t have to track the scoping anymore (usually it was done through a pseudo-immutable object, that was only mutable on the reference counting).

It’s also one application of COR (Copy On Read) instead of the more usual COW (Copy On Write).
Read More...
Oct 9, 2007
Keep your caches coherent : Scope them !
About 3 min read

You can mostly have these kind of scoping
- Sub-routine (typically a function or a statement bloc)
- Request (typically in a HttpServletRequest)
- Session (typically in a HttpSession)
- Node (typically in a JVM)
- Application (typically on a specific node)
- Time (typically a filenumber per day / most cached values)
Here we encounter at last the Time scope. This one is the hardest to cope with when it come to cached values. Usually a quick and dirty caching is done with a combinaison of a Map that contains Keys linked to a TimedValue. The last part is composed of the value itself and a sort of expiration date (can also be implemented with a manufactured date).

The biggest problem with these time-scoped values are that they are not easily updated to maintain coherency with the value they should be a copy of. It involves that for each update, we manage to find the cached value and either delete it (so that it can be recomputed later) or update it. It’s not really acceptable since :
1. it is really coding-wise intrusive (you have to seek every place that you update an element of the calculation of the cached value) to put triggers everywhere, and therefore not really practical, not to say sometime not even possible (you don’t own the whole code)
2. it is really performance-wise disastrous : for every write you do, you have to fire the trigger to update all your caches that impacted from this update.
So, usually this triggering is just avoided, and the cache is left in a non-coherent state. And in reality it ends up in a fighting of speedup vs accuracy.

But this fight sometimes can be avoided if you scope correctly your caches by storing it in an appropriatly scoped variable and not as usually done in a static variable. It will have the same effect as a transaction in a database since it’s basically a copy-on-read scheme. Putting it in an appropriatly scoped variable has the net effect of relieving of pruning the cache with data that are not valid anymore without any extra effort.

This lays the premice of STM that will be something that will really count in the era of massive-multi-core-computing (MMCC) that we are beginning to enter with processors like Sun’s Niagara T2 and its 64 thread per CPU.
Read More...
Sep 28, 2007
Clean-Up Your Code and Boost Your Google Ranking at the Same Time : Post-process Your Links
About 2 min read

The search engine ranking of a web site is very important nowadays. As usually every optimisation means an complexification of the code, many developpers just pay that price. It’s mostly paid with a javascriptification of the links that we don’t want to be followed, and javascriptification of functional dialog popups.

There’s an other approch to this. It still makes heavy use of Javascript, but :
- your site will still work as intended even with javascript disabled
- your site will be much more maintenable
The idea is to post-process the links with a Javascript function that is triggered just before the end of the loading of the page. Usually it can be done with a body-onLoad, or a <script/> element at the bottom of the page. This script parses all the <a/> elements, and decodes the classes added to it. If you encode the classenames like this code :
```
<a
href="my_popup.html"
class="medium_popup pp/popup">
	explain these words
</a>
```
We can see that the classes of the link are medium_popup, that is handled normally by the browser, and pp/popup is used by the post-processing code to add a onClick() handler that could be expressed as onClick="javascript:open_popup('my_popup.html');return false;".

The big advantage is that you can very simply add popup to your links, web-spiders knows how to go to your popup with all your shiny glossary, and so can users without javascript or just open in a new tab with a middle-click on some browsers.

If it’s a dialog box to ask some more informations, you can encode the link like that :
```
<a
href=""
class="medium_popup pp/link/my_dialog.html">
	choose the date
</a>
```
This way pp/link/my_dialog.html will be parsed and the href will be dynamically replaced with my_dialog.html.

You can even later add a pp/popup after the pp/link/my_dialog.html, and have the dialob box magically opened in a popup.

I’ll post later some example code to parse it.
Read More...
Sep 18, 2007
Convert your Log Files into Gold
About 8 min read

Log files are a necessary evil on a live system. A few rules can transform your log files from a useless heap of textfiles to a gold mine.

I’ll focus mostly on Log4J since it’s available on Java, but is ported on many languages also as it sets some kind of logging standard.

The logger is configured per class with a :
```
private static Logger logger =
Logger.getLogger(MyClass.class);
```
The logger is private since each class should have its logger, especially the derived ones (that way you can very nicely debug the virtual function calls), and static since it’s either thread-safe and that way you have it even in the <init> and <cinit> of your class (you did put it in the first line of the class, didn’t you ?).

This is very useful since you can configure a per-package or a even per-class level of logging. It is very useful since all your classes do only one thing, don’t they ?

I usually use only 5 levels of logging : ERROR, WARN, INFO, DEBUG, TRACE
- TRACE is used to help with a dump of many internal variables (it’s a last resort debug, since usually it’s very verbose)
- DEBUG is used to debug this class, with keypoints inside the class, in order to see the inside flow of execution.
- INFO is used to debug other classes with the help of this one. Usually it emits only one line per public call, with the incoming parameters and the result displayed in a synthetised form.
- WARN is used when an exceptional situation happens (usually a catched Exception that is triggered by the caller data) and there is a known path to recover.
- ERROR is used when an exceptional situation happens, but there is no known path to recover. Usually this line is send by email to the administrator. If there is a problem, and an ERROR is logged, there should be no other ERROR logged for this problem : it will help you to keep a high signal/noise ratio.
In a live production system, I just log INFO for terminal business classes (the ones that represent actions), and WARN on technical ones (the one that actions class uses). I configure it to send an email on every ERROR.

It’s also very important to have a reminder of a synthetised form of the arguments in the INFO, WARN & ERROR log messages (such as an ORDER_ID if it’s an ordering action or the PRODUCT_ID if it’s a deleting product action). It’s also a good thing to put the exception that triggered the WARN/ERROR log. That way you can just grep through your logs to see if, when and what happened to that famous product that everyone is so excited about

Always use a RollingFileAppender. Always. If you’re scared about loosing some logs, just put an insane number of backup files since nothing is worse than not enough space on the log filesystem : you won’t have the logs anyway. Note that if you have a different kind of rolling mecanismed you can use it, the point is that you should never leave a growing log file without control.

So, here is a exemple of 2 classes (one business, one technical) that present the logging system I talked about :
```
public class PublishProductAction extends Action {

private static Logger logger = Logger.getLogger(AddProductAction.class);
private int product_id;
public PublishProductAction(int product_id) { this.product_id = product_id; }
public void execute() {
	logger.info("publishing product[product_id:"
			+ product_id + "]");
	try {
		// Do things...
	} catch (Exception e) {
		logger.error("Cannot publish product["
				+ product_id + "] : ", e);
	}
	logger.debug("done publishing product[product_id:"
			+ product_id + "]");
}

}

public class MyPreparedStatement {

private static Logger logger = Logger.getLogger(MyPreparedStatement.class);
private String sql;
public MyPreparedStatement(String sql) { this.sql_id = sql; }
public void execute(Collection params) {
	startTimer();
	try {
	if (sql == null) {
		// Log in warning, since it should not happen,
		// but we can handle it gracefully
		logger.warn("The SQL statement is null, we do nothing");
		return;
	}
		try {
		// Do things...
	} catch (Exception e) {
		// log only in info since we don't catch
		// the Exception.
		logger.info("Cannot execute SQL[" + sql + "] : ", e);
		thow e;
	}
	} finally {
		stopTimer();
		// help the outside class to see what happened
		if (logger.isInfoEnabled()) {
			logger.info("executed sql ["
					+ parseSQL(sql, params) + "] in "
					+ getTime() + " ms");
		}
	}
}

}
```
Read More...
Sep 6, 2007
Are Workflows the Future of IT Computing ?
About 3 min read

We are about to change from a “CPU power is cheap”-paradigm to a “CPU are cheap”-one. Everyone knows now that the 10Ghz barrier will not be that easily broken like said Herb Sutter in his famous The Free Lunch Is Over DDJ article.

I also think it will be much cheaper to produce a 3Ghz 128-core CPU than a 10Ghz 1-core one, and that’s a huge problem in our current way to code. Current common code just isn’t adapted to a many-cpu world as said Jeff Atwood in his comparison of 4- and 2-way systems. It’s because real parallel programming is not easy. Even our current practices of locking are quite difficult to get right.

A good way do it efficiently, is to delegate the plumbing of parallelizing the tasks to an automated system. It seems at first less efficient to do, but noone really want to go back to ASM programming. We really have to define a new level of language that auto-parallelize the treatements, like the way we code in a high-level language that is either interpreted or even compiled.

This move in the programming area is of same order than the move from ASM to scripting or compiled language. And here you have 2 differents approaches :
- the static (compiled) way
- the dynamic (interpreted) way
In the static way, you have also mostly 2 choices :
- a language (usually compiled) that handle the dividing & regrouping of tasks, like the model followed by OpenMP using mainly threads, shared memory and locking.
- a language (usually fonctionnal) that handle the differents tasklets with a message-passing like interface (Erlang is a good example of this)
The dynamic road is less defined. But usually it involves a state-flow machine, that can coordinate all the tasks to do on all the processing units. That is usually done via a Workflow (also called BPM-engine in this particular area), like the Master-Worker pattern Alex Miller speaks about.

This last item is, in my opinion the future. If you put everything in a database, and the BPM-engine interprets each item independantly, wich a high focus on a runtine resolution of the next thing to do. It gives you a highly flexible design that you can change at runtime without much of trouble. Its flexibility is quite the same the one you would have with scripting languages (think Ruby) versus compiled one (think Java).
Read More...
May 30, 2007
Should SQL die, and can it ?
About 3 min read

I just draw a comparison of the data language and the general languages :
- Creating our data structures by hand feels like ASM and its multiple wheels reinvented everytime.
- Vanilla SQL feels like C and its bare instruction set.
- Advanced SQL (Stored Proc et al) feels just like C++ and all those almost compatible implementations.
- A statically compiled ORM (think Hibernate) feels like Java and it’s strong typing system with its truckload of runtime casts.
- A dynamic ORM (think ActiveRecord) feels like a dynamic scripting language and its “wow” factor at the beginning, rapidly followed by the “is-this-the-fastest-cpu-available” factor.
I have some grudges against the ORM i’m using now :
- It’s not really easy to reverse-engineer a database. Everyone seems really happy to start with a fresh new one.
- Joins are optimised for only the simplest joins. (If it’s not just a plain foreach-like application loop)
- If you have some complex things to do, the usual “you-always-can-write-a-custom-mapping” applies.
Not writing any SQL can be a nice goal, but i don’t think that writing SQL is specially that hard. It’s a quite good language for what it’s designed for actually.

I think that today’s ORM solutions are a little half-baked. It puts too much hassle on the user for the result it provides. He has to describe the whole database structure in order to manage to generate the differents objects that impersonate the undelaying tables.

Why can’t this tedious process happen in runtime ? Ok, maybe not really in runtime, but either on compile time or on launch time. Two directions could be explored :
- The ORM could automatically adapt itself on the undelaying database schema by introspecting the schema (many DBA tools did that for years).
- The ORM could also adapt the database schema to runtime changes by doing some runtime ALTER TABLE.
In contrast, writing SQL yourselfs is often the easiest and fastest solution, since you directly and only ask what you want. It also allow you not to think too much about concurrency since that’s usually what good RDBMS do the best. With an ORM, either you explain a lot your intentions, and then you often do the SQL optimiser job, or you hope that your ORM is clever enough to guess what you want to do. Usually it’s not that good at it, and tries to hide it behind some agressive caching mecanism.

So I think SQL is here to stay a little while, just like C/C++ is. The ORM have no really catched up since. (We’re like in the old days of non-JIT Java).

Btw, I even dreamed of an ORM in bijection with XML or YAML with a XPath query language, but I will write later on that.
Read More...
May 26, 2007
Avoid the runtime penaly of Singletons
About 3 min read

A common way to do the Singleton pattern involves a getInstance() method which does a runtime test like this :
```
private static Singleton instance;
private static Object mutex;

public static Singleton getInstance() {
	if (null == instance) {
		// First call
		synchronized (mutex) {
			if (null == instance) {
				instance = new Singleton ();
			}
		}
	}
	return instance;
}

private Singleton() {
}
```
This method works well, since it uses the double-null-check. You always have to check it while beeing under a synchronized belt, and you check it just before in order to avoid the synchronized cost after its initialisation. So we have a method that still makes a runtime check, and cannot be nicely inlined by the compilator.

If you can afford to :
- not be able to deregister the instance afterwards with a freeInstance() method (We’ll speak about this later)
- not defering the creation of the instance the first time you need it (also called lazy creation)
you can avoid that extra cost by creating the instance in class loading time with a static block.

The code then becomes :
```
public static Singleton getInstance() {
	return instance;
}

private Singleton() {
}

private static Singleton instance = new Singleton();
```
The getInstance() method can now be inline quite easily and there is no more extra check cost. Note that here the default constructor is empty. If it’s not, I suggest to put the static initialisation at the end of the class, in order for the rest of the class to be initialised and to be ready for use in the constructor.

If you want to be able to deregister you have to use the first version, and write a freeInstance() like :
```
public static void freeInstance() {
	if (null != instance) {
		// First call
		synchronized (mutex) {
			if (null != instance) {
				// Do some closing work here if required like instance.close()
				instance = null;
			}
		}
	}
}
```
And now you just wrote a primitive form of pooling (with only one instance)..But that’s another story…
Read More...
Mar 18, 2007
Inheritance of a public method considered harmful
About 3 min read

I’m always wondering why all textbooks and reference material on the web always describe inheritance with the same kind of example :
```
class A {
	public execute() {
		// do something...
	}
}
```
```
class B extends A {
	public execute() {
		// initialize for B
		super.execute();
	}
}
```
```
class C extends A {
	public execute() {
		// initialize for B
		super.execute();
	}
}
```
One thing struck me : if I want to have some code executed before execute() in every class, such as preparing the graphic context for example, I have to write a protected method initializeContext() on the base class and call it from every derived execute() method. This is a good candidate for errors since it’s manual and therefore not automatic.

I prefer to go for an IoC-like inheritance scheme, we could call it “Inversion of Inheritance”. The code would be something like :

class A {
```
   protected executePre() {
		// Default implementation
		// does nothing
	}
```
```
   protected executePost() {
		// Default implementation
		// does nothing
	}
```
```
   protected executeInternal()     {
		// do something
	}

	public final execute() {
		executePre();
		executeInternal();
		executePost();
	}
}
```
```
class B extends A {
	protected executeInternal() {
		// initialize for B
	}
}
```
```
class C extends A {
	protected executeInternal() {
		// initialize for C
	}
}
```
There are 2 problems with this approach :
1. There is a little performance penalty since every calls issues 3 virtual calls.
2. If you want to add another interception function, you have to edit the base class.
But the main purpose is that you are in control of the derived classes of your class since the creativity of the reuser is severly hampered. I think of it as the most easy way to enforce the Liskov substitution principle.
Read More...
Mar 17, 2007
First (Real) Post
About 1 min read

This is my first attempt at making a live blog. It should have been a way of expressing my progress of one of my pet project : Personal Workflow.

As with every pet projects, there is such an excitation at the beginning, but after a while it drops rapidely down to zero.

It’s quite the same effect when your tendency to postpone is too high as Andy Hunt and Dave Thomas said : Don’t Live with Broken Windows !. It will always be “If I could postpone it at that time, I can still postpone it.”, and then it’ll take an enormous amount of willpower to be even started.
Read More...

Posts

Prerequisites

Download the source code

Compile munin

Using the dev tools

install

restart_munin-node

run

cgi

query_munin_node

Real CGI usage with your web browser

Core plugins are designed for simplicity…

… but efficiency is coming !

Spin-offs …

… are welcomed …

… because they are very valuable in the long term

Different layers of spin-offs

Plugins

C

Node

Simple munin

SNMP

Win32 ports

Android

C

Master

Clients

Supersampling

Overview

Protocol details

Config

Fetch

Compatibility with 1.4

Sample implementation

Streaming plugins

Undersampling

RRD is Munin’s backbone.

1.2 - Only one set

1.4 - 2 sets : normal & huge

Manual overriding

2.0 - Full control

Native SSH

Configuration

Installation notes

Upcoming HTTP(S) transport in 3.0

FastCGI is really CGI on steroids

FastCGI vs mod_perl (or mod_python, …)

But steroids do have some side effects

CGI issues

Webserver issues

A standalone FastCGI container

The problem : updates are synchronous

Evolving Solutions

1.4 : Parallel Fetching

2.0 : Stateful plugins

2.0 : Asynchronous proxy node

The proxy node in detail (munin-async)

Overview

Specific update rates

SSH transport

Some benchmarks

Basic 1.2 CGI

1.4 FastCGI

2.0 FastCGI

The various processes

munin-update

munin-graph

munin-html

munin-limits

Initial API

Just a little evolution…

A new need comes

The little evolution, revisited…

munin.conf snippet - Inetd version

munin.conf snippet - Native SSH transport version

Warning

The proxy node in detail (`munin-async`)