Skulker Update Out

Skulker is pretty mature software these days; not much goes wrong and in the environments I run it on a daily basis it quietly goes about its work; handing 1,000,000 files a month and saving 6Tb of disk space in one organization I work in.

However recently I was made to consider a requirement for auditing that required logs to be integrity checked and since Skulker manages the aspect on compressing logs and removing them once old enough it only seemed sensible to extend Skulker to provide the integrity checking facilities too.

Hence for the last few weeks my spare evening time has been spent adding these features and some of the others that had been collecting in the issues log. The coding, testing and documentation changes are now complete and the updates can be found on the relevant Google code page here.

I'll probably write a separate post to describe a typical use of the new checksumming functionality soon; suffice to say it attempts to remain true to my mantra of keeping all things as simple as possible.

Technically the only really interesting thing about the new checksumming functionality is that the two new features (chksumadd and chksum) both support multi-threading. Given that checksum calculation can requires a full sequential read of the file it made sense to provide this facility. Of course whether this makes a difference depends on your Perl implementation, the size and number of files covered (larger files make better sense for threading) and of course the IO and CPU facilities of the machine on which you are running it.

TrueCL Update finally out

Ok ... so it is about 6 months late, but hopefully worth waiting for. This release has over 100 fixes and enhancements as can be seen from the following link:

0.9.5 Fixes/Enhancements

Still not really ready for a 1.0.0 release as such ... but getting pretty close on Linux. The 2nd-class architecture supported of Solaris, AIX, HP-UX and BSD are a bit further behind unfortunately, though they should still work.

This is a big project; with around a 300 page manual updated and 100k code lines across multiple operating systems is a lot of work; but I'm pleased with the results and are greatly looking forward to a 1.0.0 release later in 2012.

The latest packages and documentation can be found in the download tab on the projects source code page:

TrueCL Source Code

Another TP2 Release

Although only a few days back a release of tp2 was made available, an update has followed. This improves the format of the "tp2 verify" package verification output ... but the main change is support for two new namespace ACL's:

ALLOW_PREVIEW_INSTALLERS - given a list of names these users are able to perform "tp2 install --preview ..." commands to perform preview installs.

ALLOW_PREVIEW_REMOVERS - specifies a list of users that are able to perform "tp2 remove --preview ..." to perform preview remove commands.

Note that membership of the "ALLOW_INSTALLERS" assumes membership of "ALLOW_PREVIEW_INSTALLERS - the same being true for the remove ACL too.

True Packager 2 Release 1.3.3 Features - Part 2

Passing Environment Variables

When you are running as the owner of a namespace when you attempt to install or remove a package the software will run the relevant programs natively; and hence have access to the current environment.

However if you are running as a user that is authorised (but does not own) the namespace in question, all activity is spawned via the daemon process. This has the following impact:

• The actual process doing all the work is initiated by the daemon; the process in the terminal is simply relaying the information it receives in this instance.
• Since the process installing the package is doing so via the daemon it is no access to the environment variables that the caller has available currently.

However from 1.3.3 onwards this problem can be overcome by making use of an additional optional section in the package configuration file, namely:

  <env>

    ENV_VAR1

    ENV_VAR2 

  </env>

The "env" element allows you to list a series of environment variables that the caller and the spawned install/remove session will pass between them. When such is part of the package those variables will be communicated from the calling shell and made available to the four following scripts:

preinstall

postinstall

preremove

postremove

Getting User Responses

This is still in the early stages of development but the idea is that the package should be able to configure a series of queries which the user can respond to and these responses are then made available to the two installation scripts; "preinstall" and "postinstall".

Again this is achieved via a new section in the package configuration file, namely the optional "response_info" element. This will have a separate "q" element for each query that the user is asked. Consider the following example:

  <response_info>

    <q text="Please enter your age" default="5" 

       type="integer" range="1 99" label="get_age"/>

    <q text="Please enter your email address" default="%USERNAME%@%HOSTNAME%" 

       type="text" label="get_email" pattern="^\S+@\S+" 

       hint="Please enter valid email address format"/>

  </response_info>

In this case two questions will be asked. The utility will attempt to perform validation and range checking for numbers, and for text queries it can optionally include a "pattern" attribute to ensure the response entered matches the pattern in question.

The "default" attribute is mandatory - this will allow installations to take default values if the user responses are not available. This is designed for headless installs; even though that is not yet available.

In such cases the preinstall and postinstall scripts have these values entered as part of the environment - these are in variables TP2_RESPONSE_label. Hence in the above example:

TP2_RESPONSE_get_age

TP2_RESPONSE_get_email

For more information see the available documentation on the google code downloads page: true packager 2 downloads.

Javascript simple Game writing - main loops

Basic Main Loop
Firstly the demo site is here: main_loop_demo_1

Here the main-character is animated using the following code:

setInterval( function() {game_loop(maze1,ctx2,my_pacman);},50);

The "game_loop" function being:

function game_loop(maze,ctx,pacman) {
move_pacman(maze,pacman);
draw_sprites(maze,ctx,pacman);
}

So every 50ms the game_loop function is called to move the pac-man object and then re-draw it. This is equivalent to 20 frames-per-second - the ideal is 50.

Of course you could simply change the "50" in the "setInterval" command to 1000/50 to "20", but some things should be considered:

• How to we know whether the client browser can cope with this animation rate?
• Should our game take all resources it can or take account of the load on the client browser?

My approach is to always take account of the end-user environment - and so if attempting 50 frames per second puts stress on the client system we should not do so. Hence a better approach is to try for 50 frames and reduce if not possible.

An Aside - Closures
Like most things, it took me a while to get to grips with closures. Put simply, they allow references to variables and functions to "hang around" after a calling function has returned. Consider the following code:

function set_up_x() {
var x=20;
  return function next_x() {
    return x++;
  };
}

If you have Node.js installed (very useful) you can run the above interactively and calling "y()" see the returned value increase:

$ node

> function set_up_x() {
... var x=20;
...   return function next_x() {
...     return x++;
...   };
... }

> y=set_up_x(44);
[Function: next_x]

> y();

20

> y();

21

> y();

22

So in the above example the function "set_up_x" has the variable "x" defined locally - and this is visible to the anonymous function (as returned by the function call). The result is the "x" remains visible to the anonymous function ... hence the function gains a "static" variable - most useful!

Notice that we must call "set_up_x" and use the return value (the function) to actually perform the work!

More Advanced Main Loop
We can use the concept of a closure for our main loop - here we don;t actually use one function to return another; but use one function to define another and then use setInterval to call that function - effectively in the closure.

The concept is:

function call_me(var1,var2) {
var closure_var1=xx;
var timer=setInterval(function() {my_hidden_loop(var1,var2);},interval);


  function my_hidden_loop(my_var1,my_var2) {
   // do what we want and alter / remove timer if necessary..
  clearInterval(timer);
  timer=setInterval(function() {my_hidden_loop(var1,var2);},new_interval);
  }
}

So if the code calls "call_me(x,y)" - then the function "my_hidden_loop" will start running at regular intervals ... with access to the current values of "x" and "y". More importantly it can change/access the current values for the variables enclosed by "call_me" - including "closure_var1" and "timer" itself.

Firstly the loop in all it's glory:


function handle_timed_mainloop(maze1,ctx2,my_pacman) {
var want_fps=50;
var min_fps=20;
var current_fps=50;
var calc_interval=1000/current_fps;
var timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);


function game_loop(maze,ctx,pacman) {
var start=new Date().getTime();
move_pacman(maze,pacman);
draw_sprites(maze,ctx,pacman);
var end=new Date().getTime();
if( (end-start*2) < calc_interval && current_fps < want_fps) {
// Not taken much time so increment frame rate
clearInterval(timer);
current_fps++;
calc_interval=1000/current_fps;
timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);
console.log("Increased frame rate to ",current_fps);
} else if((end-start*2) > calc_interval && current_fps > min_fps) {
// Asking too much of system, so slow down...
clearInterval(timer);
current_fps--;
calc_interval=1000/current_fps;
timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);
console.log("Decreased frame rate to ",current_fps);


}
// console.log("Duration in ms = ", end-start);
}
}

Firstly notice that the only things that are actually executed in this function are:

var want_fps=50;
var min_fps=20;
var current_fps=50;
var calc_interval=1000/current_fps;
var timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);

The rest of the function is actually the "game_loop" function which is executed by the final line above. The result is that the call to the "handle_timed_mainloop" function will quickly exit - after ensuring that "game_loop" is started.

What the code above does is attempt to start the game with 50 frames per second - as per the "current_fps" setting. The "want_fps" (which is also 50) defines the ideal and the minimum defines what we should not slow down beyond. This information is used to set "calc_interval" and call "game_loop":


var calc_interval=1000/current_fps;
var timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);

Now of course the aim of "main_loop" is to perform all game logic - for the moment just moving the pacman around the maze and listening to events. The events have actually been defined prior as:

document.onkeydown=function() {handle_key(event,my_pacman)};

This updates the "my_pacman" object as appropriate for the "main_loop" programming to act on.

In essence the "main_loop" is actually just two function calls:

move_pacman(maze,pacman);

draw_sprites(maze,ctx,pacman);

The first function call takes in the current position, current direction and next direction and updates the position of the pacman. The second actually draws the pacman at the new position (erasing the old canvas content first of course).

Now the interesting bit... notice that we wrap the above two calls with getting the current time (in milliseconds)?

var start=new Date().getTime();

move_pacman(maze,pacman);

draw_sprites(maze,ctx,pacman);

var end=new Date().getTime();

So by taking the "end" from the "start" we can estimate the time our current browser takes to perform a frame. We can then use this to increment the frame rate if the time is small enough, or decrease it if the time it takes is too long:


if( (end-start*2) < calc_interval && current_fps < want_fps) {
// Not taken much time so increment frame rate
clearInterval(timer);
current_fps++;
calc_interval=1000/current_fps;
timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);
console.log("Increased frame rate to ",current_fps);
} else if((end-start*2) > calc_interval && current_fps > min_fps) {
// Asking too much of system, so slow down...
clearInterval(timer);
current_fps--;
calc_interval=1000/current_fps;
timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);
console.log("Decreased frame rate to ",current_fps);
}

The only points beyond the obvious this the above code are that if we are changing the interval we clear it, calculate a new interval (based on the new attempted frame rate) and set the timer off again:

clearInterval(timer);
...

calc_interval=1000/current_fps;
timer=setInterval( function() {game_loop(maze1,ctx2,my_pacman);},calc_interval);

So there you have it - a game loop with a dynamic frame rate based on the host performance of the previous frame update.

The code in action can be seen here: main_loop_2nd_demo

Current Problems with Advanced Main Loop

Although the adaptive animation frame rate works ... so does have some issues:

• The pac-man sprite appears to speed up (or slow down)
• The rate of pac-man animation appears to speed up (or slow down)

The basic problem is the rate of movement and animation is naive; every time the move (and thus animate) routine is called it will perform the action. What really needs to happen is to separate the two; and only make changes after a particular time has past since the other change.

The next post will alter the game to add these features...

Using Skulker to Improve Security

Isn't Skulker used to save disk space?
That is indeed the case; but one feature that can be used is that of the "setperms" function - it can be used to change the permissions, owner and group of matched files (UNIX platforms only unfortunately).

Example 1
A few examples might clarify the use. Consider the following requirement:

Find all world-writeable files in the directory "/data" and remove that permission bit.

Doing this via a rule is easy:


<?xml version="1.0" standalone="yes"?>
<skulker_rules>
<defaults
interval="0D"
/>

<rule

n="10"

type="setperms perms=o-w"

match_pattern="/data/.../.*"

                match_by="has_perm('o','w')"

/>

</skulker_rules>

Notice the use of the "has_perm" function to ensure that only files which have world write permissions are matched. Actaully since the "setperms" function is selecting removing the permissions bit the match_by clause in this case is not needed, and so the above can be simplified a little:

<?xml version="1.0" standalone="yes"?>
<skulker_rules>
<defaults
interval="0D"
/>

<rule

n="10"

type="setperms perms=o-w"

match_pattern="/data/.../.*"

/>

</skulker_rules>

In this case all files in the "/data" directory structure are passed to the "setperms" routine - but only files that are world-writeable will be modified (and shown in the log output).

Example 2

Consider the following requirement:

Ensure all gzip compressed log files in the directories "/data/dir1" and "/data/dir2" are changed to only have owner permissions.

<?xml version="1.0" standalone="yes"?>
<skulker_rules>
<defaults
interval="0D"
/>

<rule

n="10"

type="setperms perms=o-wrx,g-rwx"

match_pattern="/data/dir[12]/.*\.gz$"

/>

</skulker_rules>

A couple of points to note about this rule...

1. The match_pattern shows the use of directory-level pattern matching (both file and directories can contain regular expressions)
2. The "perms" argument to "setperms" has two sets of changes; firstly "o-rwx" removes all permissions from others, and then "g-rwx" removes all permissions from group attributes.

More on GRI 2 - Meta Reports

Although the key aim of GRI 2 is to allow a single web-interface across a collection of hosts to gather reports in a secure manner (customised for each user), the advantage of the JavaScript front-end is that some processing can be handled by the user system and not the server.

This functionality allows GRI 2 to provide "meta reports". Meta reports basically capture raw data from the hosts in question and format the content dynamically as part of the web-interface. This is best demonstrated with by a couple of examples.

Example 1 - Pie Chart
Consider the following data file on a server that is to be presented via GRI 2 to the user:


Object,Number Sold
pies,22
rolls,33
fish,44

This is saved as a report file name and has a type of "csv". Now if it was considered as a standard report then the content when presented to the user would just be as above (opened via a spreadsheet application).

However if this is defined as a meta-report (with a bit of XML) instead it can be presented to the user as:

Example 2 - Line Chart

In a similar manner consider the following report content - again stored as a CSV file:

Total,Used

Date,Total Space,Used Space

17/01/12,100,40

18/01/12,100,45

19/01/12,100,35

20/01/12,100,55

This content is a bit more complex but still GRI 2 allows a simple XML configuration to describe this data in a graphical manner for representation to the user - this time as a line graph:

Summary

Meta-Reports are a key feature of GRI 2. Although still very much in development the prospect of providing end-users with attractive graphical representations of data with out having to generate those images on the end servers is a considered a most useful feature.