Skulker v1.0.2 New Features

Over the weekend version 1.0.2 of Skulker will appear. This has a couple of interesting features:

• time_limit facility - it is possible to indicate the maximum amount of time a rule should run for before continuing with the next one. This is an additional "filter" over existing ones so you can indicate that, for example, up to 5000 files should be compressed, oldest version but only taking a maximum elapsed time of two hours.
• Random data scrubbing. The delete "file scrubbing" functionality has been extended to include a "random data" overwrite option (including multiple passes) if you might think it may be of use. A lot will depend on the OS and file system as to whether this does anything useful - absolutely not for copy-on-write file systems of course.

Code Manager 2 version 1.3.2 released.

The code is out there ... but not mentioned on freshmeat as yet since I've got to tidy up the user guide and push that out first. Documentation is never much fun, but most software needs at least a bit so I want to get that out too. Should happen in the next day or two.

1.3.2 has a few minor improvements but the main aim was to fix some problems. Nothing exciting or extras shiny to look at!

Code Manager 2 Work

Code Manager 2 is a strange beast really. Is it a cross between a single site version control system and a release management system. As with a lot of my work the key defining aim is to ensure the target audience finds the tool easy to use yet powerful enough to be genuinely useful.

Oodles of documentation on the Google code pages for it, but suffice to say it works with a single working directory structure shared by developers and a repository stored elsewhere. Various different repositories are supported and it works on UNIX and Windows.

All the standard features you might expect are there - working at the individual file level; but so are a lot more. It is possible to compare "projects" (a collection of directories making a single development environment), and even tag releases and automatically create packages from them for various package managers.

If you fancy a simple version control system, but powerful enough to work with a largish collection of sources and like the idea of easy package creation, take a look.

http://code.google.com/p/codemanager2/

It is currently at version 1.3.1 and a version 1.3.2 is probably going to happen in a few days time with numerous small fixes and tweaks.

Skulker v1 Up and Running

Wow - it has been some time since I last posted! The good news is that the software has continued to develop despite the lack of news here. Firstly Skulker...

Well 1.0.1 is actually out at present and the key features I really wanted have made it and are working well. These are:

• Multi-threading - the core code has been written to be thread friendly and the compress functionality supports threading across all recent (5.6 and above I reckon) versions of Perl. What this did show me is that lightweight threading in Perl is pretty inefficient but it seems to work OK for the compression module.
• Internal compression types - it does now support use of "internal" compression - that is rather than running external programs it can optionally (if configured and available) make use of Perl modules greatly improving performance - especially when handling a lot of files.
• Windows support - Working, including threading. Tested with Strawberry Perl and running in productions environments now just fine.
• Scrubbing - zero's out files before deleting them. Aim is to help with storage reclaims on sparse storage (certain high-end environments).

Skulker Version 1.0.0 nearly here!

It has taken quite a bit longer than expected to get round to polishing Skulker for a 1.0.0 release! The main reason in this instance was not lack of time - but instead adding of a feature that had not originally been intended for inclusion - native Windows support!

Yep - you read that correctly. A couple of people have asked for this and so the file handling was radically overhauled to handle the differences between Windows and UNIX environments. Not as straightforward as it may sound.

The support for threading is there too - for compression rules at the moment. This was not easy and required the whole of the software source to be modified significantly to ensure it ran with or without threads using the same software.

Threading forced out two further changes; the requirement to run Skulker in "strict" mode in Perl - forcing out several unknown bugs, and also to support internal compression functionality available with Perl.

The Google code page has been refreshed with the new code and at present documentation and testing continues. However the bulk of the work is done and version 1.0.0 will be with us very soon.

Linuxha.net 1.4.8 Out

Well the code has been released anyway ... the documentation update will be out sometime over the next week. The big news is that applications do not need to use file systems - raw logical volumes are supported.

Although that might have some people happy the real reason for the above is that it opened the most flexible way of supported live migration for virtual machines that are running as applications under linuxha.net. I didn't want to have to tie admins with setting up clusters file systems; raw volumes required no setup.

And live migration is working via the new "clmigrateapp" command. It typically takes 20 seconds to migrate a Xen VM - though of course to the application and it's clients there is no interruption in server - at all.

Obviously there are some particular setup requirements - so these will be documented in the admin guide - an update which should appear shortly.

New Skulker Features under development

A couple of new features have been started for Skulker, both requirements for improving Skulker when dealing with enterprise environments.

Null Data Overwrite Support
More and more people are making use of virtualised storage - and particularly "thin provisioning". Whether this is provided at the file system level [think Solaris ZFS] or dedicated hardware storage arrays, one feature is becoming more common is the ability of the storage provider to reclaim storage that is no longer used. Several now support this implicitly if pages of storage are over-written with null data. Hence this null data support, initially for the delete action, will overwrite files will NULL's and then remove it. This will obviously incur a performance hit, so it is optional.

Parallel Processing Support
A couple of customers have rules which match a large number of files each time Skulker is run. Since these people have Skulker running on multi-cpu machines it makes sense to attempt to harness the additional processing power and perform several actions in parallel.

The mechanism now being developed will optionally allow allow functions that are applied to files to run across multiple "dispatchers". The key here will be flexibility to support 1-N dispatchers whilst not requiring much effort to make use of.