Once I passed the cultural shock I managed to come up with a working solution and presented a dynamically assembled table with my own content using this howto. I was developing on the command line and my code was presented correctly. When moving over to net-snmp and snmpwalk however the output became stale. No matter how often I snmpwalked and how long I waited the result table always contained the same values. I was looking for some caching or expiration but without success. In the same time the output from my command line version [1] was always showing correct output.

It took me a while and actually I’m quite embarrassed about it. My SNMP handler function was working with a globally defined hash as a source for the SNMP table. The hash was populated with proper values on script invocation – which happend every time on command line, but only once on snmpd startup!

So important lesson learned: embedded languages get evaluated on startup and work from their on with their data structures. You require updated information? Make sure the function you call (the handler) updates it.

1: In case you wonder how to call the code from command line, simply replace the following section:

$agent->register($program, $regoid, \&my_snmp_handler);

with

my $rootOID = ".1.3.6.1.4.1.8072.998";
my $regoid = new NetSNMP::OID($rootOID);
$regat = '.1.3.6.1.4.1.8072.998';
$extension = '1';
$mibdata = '/root/passwd';
$delimT='';
$delimV=':';
&my_snmp_handler

from the tutorials example.

To solve this we added another 12 GB RAM into the machine and put the jrb files on a ram disk. While this solved the immediate problem to write the data collected by the OpenNMS to disk it opened another can of problem for us: power (and data) loss. We agreed on backing up the ram disk to local disks every 30 minutes, so a small gap during a power outage was acceptable. This also went fine, but eventually the load on the machine went above 20 every now and then during disk backup and the machine became unavailable for minutes.

At that point we had three options:

1. move out /var/opennms to our NetAPP FAS3020
2. attach two Dell PV220s each with 12 15k 300GB disks
3. look out for those fast-as-hell PCIe SLC flash drives

Extending the workload of the monitoring system to our primary storage backend device was not a very popular idea (and had its chicken-egg problems on its own). Attaching some old stinky hardware via deprecated SCSI just because “we have it on stock” also seemed ill fated. After some internal discussion we came up with some promising TCO arguments regarding power consumption, rack space and vendor support contract costs towards management and convinced them to buy us a FusionIO 80GB ioDrive for storing our performance data. This beast is fast enough to do some 100k IO Ops with Single Level Cell (SLC) flash NAND chips connected directly to your PCIe bus. Woah.

So we went out to look for a FusionIO distributor, starting with our bread and butter vendor Dell. However, as always when you need something more than a cheap server with some additional network cards you suffer. They indeed offer the FusionIO on their website, but it is not available within Germany nor China where we have offices. So we went out and found the guys from HamburgNET who had the card on stock and delivered it to us within a couple of days. Too bad for Dell.

So finally the card arrived and we put it into a Dell r620 quad-xeon with 48GB RAM for playing around. After booting CentOS 5.4 we found a new device with lspci:

[root@xxx ~]# lspci
...
04:00.0 Mass storage controller: Unknown device 1aed:1005 (rev 01)
...

We checked the fusionio website and downloaded a driver source rpm and run rpmbuild over it. Our kernel is 2.6.18-164.15.1.el5xen, the latest supported kernel release by fusionio is 2.6.18_128.1.6.el5xen, however that did not pose a problem for the driver:

# install needed software
[root@xxx ~]# yum install kernel-devel kernel-headers rpm-build gcc
# build driver
[root@xxx ~]# rpmbuild --rebuild iodrive-driver-source-1.2.7.2-1.0.src.rpm
# install driver
[root@xxx ~]# rpm -ivh /usr/src/redhat/RPMS/x86_64/iodrive-driver-1.2.7.2-1.0_2.6.18_164.15.1.el5xen.x86_64.rpm
# load driver
[root@xxx ~]# depmod -a
[root@xxx ~]# modprobe fio-driver
[root@xxx ~]# lspci
...
04:00.0 Mass storage controller: Fusion-io ioDIMM3 320GB (rev 01)
...

There it is, /dev/fioa, 80 GB in size.

We will start doing some heavy benchmarking now, let’s get some numbers for that beast. A first shot to see local write performance with ext2:

[root@xxx fusionio]# time dd if=/dev/zero of=./bigfile bs=64k
dd: writing `./bigfile': No space left on device
1206304+0 records in
1206303+0 records out
79056330752 bytes (79 GB) copied, 106.389 seconds, 743 MB/s

For quite some time I’m watching our big postgresql datawarehouse onlinebackup being compressed for 4 days, so when I read this it came instantly to my mind that we can drastically reduce the amount of time it takes to bzip the backup file. The changes to our backup script were easy. Instead of tar’ing everything to one big file I piped the output to split and put it into nice 512MB chunks.

tar c –ignore-failed-read –numeric-owner –exclude ‘lost+found’ –exclude ‘pg_xlog’ -f – /mnt/myPGDATA | split -b 536870912 -d – myPGDATAbackup.tar.

This command will create myPGDATAbackup.tar.000, myPGDATAbackup.tar.001, myPGDATAbackup.tar.002, etc. files in your current work directory, all maximum 512MB in size.

Afterwards you just execute xargs on this files with CoreCount being the number of cores you want parallel to compress the files:

ls -1 myPGDATAbackup.tar.*|xargs -r -n 1 -P $CoreCount bzip2 -9

You can assign as many cores as you want to your backup job (use CoreCount=0 to use spawn as many processes as files), however you should be careful not to shoot your IO-backend down. After increasing the memory to 512*$CoreCount we were able to hold all currently compressed backup files in the buffer cache, so that the cores do not have to wait for the io subsystem to catch up with read requests.

The performance advancement is amazing. Our onlinebackup compression time went from 49 hours to 5,5 hours! Now we should speed up the data gathering via tar, bit that is another story.

00:00000:00000:2010/04/17 10:19:58.70 kernel  Using config area from primary master device.
00:00000:00000:2010/04/17 10:19:58.72 kernel  Detected 16 physical CPU’s
00:00000:00000:2010/04/17 10:19:58.73 kernel  Locking shared memory into physical memory.

The system started to behave very weirdly, a ‘ps xa’ was hanging, a ‘ls /proc’ also got unresponsive. We investigated this issue further with strace and found out that the ASE was hanging while calling mlock(2):

6128  shmget(0x1107fde5, 34359738368, IPC_CREAT|IPC_EXCL|0600) = 98307
6128  shmat(98307, 0, 0)                = ?
6128  mlock(0x2aaaabddd000, 34359738368

After several rounds of config switch flipping we were able to track down the problem to the ramdisk we were using. Since the ramdisk is empty on every reboot we must create the sybase tempdb device files prior to the start of the ASE. If we do not do this, the database will not be able to finish its startup recovery and not boot up properly. However, if we do not create the device files, the database will go past its critical mlock call and startup normally within seconds (up to the point where it tries to read the tempdb device files..). This behavior cropped up regardless whether we use ramfs or tmpfs for ramdisk. To add more confusion to this issue the behavior was only reproducible with shared memory allocations > 16GB.

The big surprise came after we left the database in this state in the evening and came back to work the next day just to discover that the database was up and running happily. Indeed, the mlock call was not hanging forever but it took it 2 hours to finish until the startup of the ASE continued. At this point we involved the sybase technical support, but they weren’t very helpful as this problem seems to have been new to them as well. Several rounds of strace runs and Q&A mails went around without any result.

Eventually we were able to work around this issue by starting the sybase ASE without the tempdb device files created and putting the database immediately in the background. We executed a “sleep 4″ on the shell and created the device files just in time for the sybase to start recovery on them. Obviously this is a very crappy workaround and the timing is dependent on a number of factors; But it bought us some time to find the real issue and we were able to hand over the database to the application developers for testing.

I asked a friend of mine to write me a small program which does nothing else then to allocate an arbitrary amount of shared memory and release if afterwards. And indeed, the problem was fully reproducible with this program! The memory allocation was dog slow when grabbing the memory with mlock, so sybase itself was out of the game.
A couple of google rounds later I found an interesting article on LWN which was describing the behavior we faced. Together with kernel/Documentation/sysctl/vm.txt I found the kernel tunable /proc/sys/vm/zone_reclaim_mode which was set 2 on our system. Setting this value to 0 made the kernel process mlock way faster and giving us the normal fast startup speed of sybase.

To make a long story short, the kernel tries to allocate my 32GB shared memory sequentially from its memory zone pool. Since we already allocated 10GB RAM to a ramdisk it hits a lot of non-reclaimable, non-swapable pages. Iterating through all 10GB of occupied RAM takes time – in our case 2 hours. Setting zone_reclaim_mode = 0 makes the kernel skip allocated pages during its iteration, hence the shared memory allocation is faster -> the database starts quickly.

It seems that the kernel developers also think that zone_reclaim_mode = 0 is a good idea and it will be default in future kernel versions. Let’s hope sybase and redhat (centos) take this change from upstream and integrate it into their products / solving guides. I also think that this issue could probably hit you on oracle, postgres or other databases as well – probably anything that allocated shared memory.