Linuxha.net and DRBD 8.4

So ... I was all set to perform a standard set of tests on the latest version of Scientific Linux (RHEL Clone) and push out linuxha 1.4.11 - and it failed!

The problem was that the command line handling between DRBD 8.3 and below simply was radically different for 8.4 which was what SL linux packages used.

So.. back to the drawing board! The results are finished now and Linuxha.net now works with DRBD 8.4 and should thus work with DRBD 9 stable releases too when they become available.

I've not yet packaged the code since I need to regression test the changes with older DRBD releases - but fingers crossed the packages should be available within the next 7 days ... and the documentation updated shortly afterwards.

Linuxha.net 1.4.11 nearly ready

Whilst I'm still working in the background on the next truecl clustering software release I've taken a bit of a time-out on it to refresh the Linux 2 node DRBD clustering software I called "linuxha.net".

For it's 10th birthday I'm updating it to take account of the fact that recent kernels and Linux distributions include DRBD software and so I don't need to. The end result is a further simplification of the installation requirements - always a bonus.

Of course that doesn't make an exciting release does it? Hence I've also taken the opportunity to do the following:

1. Strip out many bundling Perl modules (not all) - relying on the distribution or the administrator to provide these. The result - more up to date versions will probably be used and should improve security/performance and reliability.
2. Reorganised the installation structure. Previously the software placed files in about 6 different directory structures; not ideal. Now the software lands in /opt/linuxha14 - though /etc/cluster is still used for configuration and /var/log/cluster for log files.
3. Simplified the event monitoring system; typically no longer need to edit a configuration file; the default "just works".
4. Improved Network handling; the infrastructure better handles network failures and IP fail-over - for example it is possible to run application adds and rebuilds even if the standard node IP addresses are not in operation.
5. IPv6 support. It is possible to use the software totally using IPv6 functionality - both for the software itself and the application IP addresses it presents. Of course it will work with IPv4 too if you wish; or both at the same time if you really want to!

 As an example - here is a ping6 of an application IP address during a fail-over - lost about 40 seconds before it was visible on the remaining node in the cluster.

64 bytes from fec0::192:168:1:200: icmp_seq=220 ttl=64 time=0.164 ms
64 bytes from fec0::192:168:1:200: icmp_seq=221 ttl=64 time=0.201 ms
64 bytes from fec0::192:168:1:200: icmp_seq=222 ttl=64 time=0.133 ms

64 bytes from fec0::192:168:1:200: icmp_seq=266 ttl=64 time=0.281 ms
64 bytes from fec0::192:168:1:200: icmp_seq=267 ttl=64 time=0.123 ms
64 bytes from fec0::192:168:1:200: icmp_seq=268 ttl=64 time=0.112 ms
A snapshot of the cluster application whilst a single node was running:

# clstat -A apache
Cluster: test - UP

 Application       Node      State  Runnnig  Monitor  Stale  Fail-over?
      apache   lubuntu2    STARTED  0:00:09  Running      2          No

 File Systems

 Mount Point              Valid   Type      State   % Complete  Completion
 /apache/admin            local   drbd     Unsync
 /apache/data             local   drbd     Unsync

 General Monitors

            Type          Name    Status
      FS Monitor     fsmonitor   Running

An subset of the DRBD configuration - showing it running over IPv6 in this cluster:



protocol C;
_this_host {
    device            minor 0;
    disk            "/dev/apachevg/apache2";
    meta-disk        "/dev/apachevg/apache2_meta" [ 0 ];
    address            ipv6 [fec1::192:168:100:25]:9906;
}
_remote_host {
    address            ipv6 [fec1::192:168:100:26]:9906;
}

So when will it be released? Soon is the best answer I can give. I need to validate the functionality across recent Redhat Linux configurations and update the documentation - including a 1.4.10 to 1.4.11 upgrade guide.

Original Generic Reporting Intranet Release

Although it has only been about a month since the last release one feature that was really missing was "back-end security". Of course the support for SSL encryption of the traffic between the web server and the client is a given; however the web server must communicate with the authentication service and any server that actual host reports that are available via this tool-set.

Hence the latest release (available from here) - version 0.9.0 - adds RSA Encryption support for those communications. This is entirely optional but is strongly recommended. However it is fairly easy to set up; but once enabled all servers that provide reports must make use of it.

The actual protocol in use attempts to be quite lightweight - so only the initial exchange to generate a random key using RSA encryption - once key has been securely exchanged the remainder of the communication will make use symmetric compression for improved performance.

Perl provides two basic modules for providing RSA encryption:

Crypt::RSA

Crypt::OpenSSL::RSA

OGRI can make use of either - however whichever is used must be used consistently across all machines! This is simply because the format and content of the public/private key files are not inter-changeable.

New TruePackager2 Release

It's taken a good while to release this significant update to the packaging suite; but hopefully it will be worth it. This software package consists of over 35,000 lines of Perl code and around 200 pages of documentation; hence the time between releases!

As normal there are a whole host of fixes but some notable additional functionality too:

• The "tp2 list --namespaces --all" functionality will list the capabilities the current user has in the namespaces listed.

• A new "allow_list" capability has been added. This allows the system administrator to restrict UNIX permissions for a namespace, but still allows users access to view the content if they are allowed.

• The "tp2 show_ns" has been extended to show additional details, including namespace package signing requirements and also lists known DSA signatures for the namespace.

• The functionality offered by the TP2 daemon has been enhanced and so the install with "--diffs" works via the daemon and current repository settings are passed across for installs to.

If you've not come across TP2 now might be a time to look; it has been written to provide robust, audited package installation facilities aimed at corporations when require cross-environment deployments. It offers some unique features including "preview installs/removes", a complete audit log; flexible dependency handling; automatic file system growth; access control lists; flexible package signature enforcements; package bundles; atomic installs, etc.

Ogri Source Refreshed

Introduction

After quite a bit of refinement the latest version of Ogri has been pushed up the google projects server (link below). This is classed as only version 0.0.4 but the code in reality is probably really 1.0.4 but will only truly be numbered in that manner once a couple of kinks are finally ironed out.

This is the first release where some of the portal access management can be delegated to "owners" of the portal by site administrators. Next comes the concept of "voters" for portals allowing the democratic decisioning (if you want it) for defining whether access to certain portals for requesting users should be granted or not.

Documentation

This release ahs been delayed whilst I complete the user guide and supplements the installation guide - both available on the Google Code downloads page. As always I've veered towards simplicity but welcome any feedback.

User Management

The tool can be quite easily integrated into "Single sign-on" services and also shows how a interface to LDAP-based group access can be performed too.

Currently site administrators can also manage users alerting to ensure they get emailed when certain new reports come into existence.

What Next?

I'll probably get the "portal voting" functionality in place soon and then call that version 1.0.0. Along the way it will probably pick up a few more preferences and administration functions.

When will this appear? Possibly note until I've done some minor updates of some of my other projects:

• cm2 - improvements to mass code check-ins, change-set generation/application improvements, optional relaxation of the check in/out permissions.

• truecl - work through some of the enhancements currently logged. This is a huge project (think 100,000 lines of code for a single person), of great complexity and I've enjoyed a break from the brain-ache, but time has come to jump back into the code and push this forward again!

Links

Ogri - the "Original Generic Reporting Intranet" can be found here.

Original GRI Source Uploaded

Before spending a fair proportion of my coding time working on the Javascript/Web-2/AJAX interface known as "GRI 2" I had the original as a rudimentary series of pages linked together with a bit of Perl.

It was fairly successful as far as it went; a basic web-based reporting portal, but didn't offer the glitz and interactivity that some seemed to crave. Hence I put it aside and worked on a different front-end whilst keeping the server side as compatible as possible (with extensions to facilitate the asynchronous nature of the requests driven by AJAX).

That is ongoing (not a small task). However I came to realise that the simplicity of the original code was a strength not to be over-looked. Yes the interface is basic; but the requirements for viewing the reporting application are very modest.

With that taken into account I've taken the code from my local virtual environments, and have begun the process of improving portability, adding some more features, and also improving GUI-based administration and posted to Google code.

The install guide has been published too - a user guide is in the works. The code is usable as it stands so if you need a web-based report portal with good security, distributed report management and low system requirements take a look!

http://code.google.com/p/old-generic-reporting-intranet/

Skulker Update Out

Skulker is pretty mature software these days; not much goes wrong and in the environments I run it on a daily basis it quietly goes about its work; handing 1,000,000 files a month and saving 6Tb of disk space in one organization I work in.

However recently I was made to consider a requirement for auditing that required logs to be integrity checked and since Skulker manages the aspect on compressing logs and removing them once old enough it only seemed sensible to extend Skulker to provide the integrity checking facilities too.

Hence for the last few weeks my spare evening time has been spent adding these features and some of the others that had been collecting in the issues log. The coding, testing and documentation changes are now complete and the updates can be found on the relevant Google code page here.

I'll probably write a separate post to describe a typical use of the new checksumming functionality soon; suffice to say it attempts to remain true to my mantra of keeping all things as simple as possible.

Technically the only really interesting thing about the new checksumming functionality is that the two new features (chksumadd and chksum) both support multi-threading. Given that checksum calculation can requires a full sequential read of the file it made sense to provide this facility. Of course whether this makes a difference depends on your Perl implementation, the size and number of files covered (larger files make better sense for threading) and of course the IO and CPU facilities of the machine on which you are running it.