Using Skulker to Improve Security

Isn't Skulker used to save disk space?
That is indeed the case; but one feature that can be used is that of the "setperms" function - it can be used to change the permissions, owner and group of matched files (UNIX platforms only unfortunately).

Example 1
A few examples might clarify the use. Consider the following requirement:

Find all world-writeable files in the directory "/data" and remove that permission bit.

Doing this via a rule is easy:


<?xml version="1.0" standalone="yes"?>
<skulker_rules>
<defaults
interval="0D"
/>

<rule

n="10"

type="setperms perms=o-w"

match_pattern="/data/.../.*"

                match_by="has_perm('o','w')"

/>

</skulker_rules>

Notice the use of the "has_perm" function to ensure that only files which have world write permissions are matched. Actaully since the "setperms" function is selecting removing the permissions bit the match_by clause in this case is not needed, and so the above can be simplified a little:

<?xml version="1.0" standalone="yes"?>
<skulker_rules>
<defaults
interval="0D"
/>

<rule

n="10"

type="setperms perms=o-w"

match_pattern="/data/.../.*"

/>

</skulker_rules>

In this case all files in the "/data" directory structure are passed to the "setperms" routine - but only files that are world-writeable will be modified (and shown in the log output).

Example 2

Consider the following requirement:

Ensure all gzip compressed log files in the directories "/data/dir1" and "/data/dir2" are changed to only have owner permissions.

<?xml version="1.0" standalone="yes"?>
<skulker_rules>
<defaults
interval="0D"
/>

<rule

n="10"

type="setperms perms=o-wrx,g-rwx"

match_pattern="/data/dir[12]/.*\.gz$"

/>

</skulker_rules>

A couple of points to note about this rule...

1. The match_pattern shows the use of directory-level pattern matching (both file and directories can contain regular expressions)
2. The "perms" argument to "setperms" has two sets of changes; firstly "o-rwx" removes all permissions from others, and then "g-rwx" removes all permissions from group attributes.

Skulker v1.1.0 Nears Release

It has been a while since the previous release of Skulker and so a few queued requests and an annoying but have been dealt with. See the Skulker page for full details, but the most important being improvements for pattern matching, a new option to set permissions and a working history log.


It has been completed and shortly packages and updated documentation will be posted on the relevant code page. 

Skulker v1.0.2 New Features

Over the weekend version 1.0.2 of Skulker will appear. This has a couple of interesting features:

• time_limit facility - it is possible to indicate the maximum amount of time a rule should run for before continuing with the next one. This is an additional "filter" over existing ones so you can indicate that, for example, up to 5000 files should be compressed, oldest version but only taking a maximum elapsed time of two hours.
• Random data scrubbing. The delete "file scrubbing" functionality has been extended to include a "random data" overwrite option (including multiple passes) if you might think it may be of use. A lot will depend on the OS and file system as to whether this does anything useful - absolutely not for copy-on-write file systems of course.

Skulker v1 Up and Running

Wow - it has been some time since I last posted! The good news is that the software has continued to develop despite the lack of news here. Firstly Skulker...

Well 1.0.1 is actually out at present and the key features I really wanted have made it and are working well. These are:

• Multi-threading - the core code has been written to be thread friendly and the compress functionality supports threading across all recent (5.6 and above I reckon) versions of Perl. What this did show me is that lightweight threading in Perl is pretty inefficient but it seems to work OK for the compression module.
• Internal compression types - it does now support use of "internal" compression - that is rather than running external programs it can optionally (if configured and available) make use of Perl modules greatly improving performance - especially when handling a lot of files.
• Windows support - Working, including threading. Tested with Strawberry Perl and running in productions environments now just fine.
• Scrubbing - zero's out files before deleting them. Aim is to help with storage reclaims on sparse storage (certain high-end environments).

Skulker Version 1.0.0 nearly here!

It has taken quite a bit longer than expected to get round to polishing Skulker for a 1.0.0 release! The main reason in this instance was not lack of time - but instead adding of a feature that had not originally been intended for inclusion - native Windows support!

Yep - you read that correctly. A couple of people have asked for this and so the file handling was radically overhauled to handle the differences between Windows and UNIX environments. Not as straightforward as it may sound.

The support for threading is there too - for compression rules at the moment. This was not easy and required the whole of the software source to be modified significantly to ensure it ran with or without threads using the same software.

Threading forced out two further changes; the requirement to run Skulker in "strict" mode in Perl - forcing out several unknown bugs, and also to support internal compression functionality available with Perl.

The Google code page has been refreshed with the new code and at present documentation and testing continues. However the bulk of the work is done and version 1.0.0 will be with us very soon.

New Skulker Features under development

A couple of new features have been started for Skulker, both requirements for improving Skulker when dealing with enterprise environments.

Null Data Overwrite Support
More and more people are making use of virtualised storage - and particularly "thin provisioning". Whether this is provided at the file system level [think Solaris ZFS] or dedicated hardware storage arrays, one feature is becoming more common is the ability of the storage provider to reclaim storage that is no longer used. Several now support this implicitly if pages of storage are over-written with null data. Hence this null data support, initially for the delete action, will overwrite files will NULL's and then remove it. This will obviously incur a performance hit, so it is optional.

Parallel Processing Support
A couple of customers have rules which match a large number of files each time Skulker is run. Since these people have Skulker running on multi-cpu machines it makes sense to attempt to harness the additional processing power and perform several actions in parallel.

The mechanism now being developed will optionally allow allow functions that are applied to files to run across multiple "dispatchers". The key here will be flexibility to support 1-N dispatchers whilst not requiring much effort to make use of.

Minor tweaks to Skulker

Since the last upload of Skulker a couple of minor problems were found out in the real world that did not show up in the test suite. I've now corrected the conditions and the source code and package downloads are available from Google code.

To an extent this obviously shows that the testing suite needs to be expanded; something I intend to turn my attention to on an intermittent basis over the next few months.

Skulker 2 v0.6.4 released

This is the first version of Skulker 2 to be made available via code.google. The aim is to make the project more 'open'. A fair amount of my other work will also be hosted there if things go well.

Skulker 2 on Google can be found at http://code.google.com/p/skulker2/