FileEvent Example 2 - Basic Archiving Support

Introduction

FileEvent is a utility designed to allow event-driven handling of files when they come into existence based on simple rules stored in XML configuration files. This is the second example of how FileEvent might be used and extends the facilities shown based on the initial example.

Why Archive?

In the previous example it was shown how FileEvent is stateful in that (by default) will not process the same file twice. This means that when multiple files are matched in a directory and not all files are processed it will “skip” the files in question, for example re-capping the previously shown command:

2013/08/12 22:25:23.443 0000013-W File '/home/venture/projects/SOURCE/fileevent/testing/example1/file1_20130822.txt' already sent - removing from list to send or fetch.

2013/08/12 22:25:23.443 0000014-W File '/home/venture/projects/SOURCE/fileevent/testing/example1/file1_20130821.txt' already sent - removing from list to send or fetch.

2013/08/12 22:25:23.443 0000015-D Actual cut-down list to send now:

2013/08/12 22:25:23.443 0000015-D file1_20130819.txt

2013/08/12 22:25:23.443 0000016-D send_file of '/home/venture/projects/SOURCE/fileevent/testing/example1/file1_20130819.txt' called.

2013/08/12 22:25:23.444 0000017-D xfer_destination raw: test@lubuntu1:/tmp/%{f}

2013/08/12 22:25:23.444 0000018-D xfer_destination cooked: test@lubuntu1:/tmp/file1_20130819.txt

This is all well and good, but this scanning the files and reordering them is additional processing which is best avoided, and hence we come to another feature of FileEvent – archiving. 

FileEvent Archiving

Archiving will move any successfully sent files to an archive directory and compress it to, saving space. Just a single additional line is usually enough, for example:

<?xml version="1.0" standalone="yes"?>

<FileEvent>

<settings>

  <db>/tmp/testing.db</db>

</settings>

<event>

  <description>scp example1 transfer</description>

  <file_pattern>file1_%{4year}%{2month}%{2day}.txt</file_pattern>

  <directory>/home/venture/projects/SOURCE/fileevent/testing/example2</directory>

  <xfer_job_type>scp</xfer_job_type>

  <xfer_destination>test@lubuntu1:/tmp/%{f}</xfer_destination>

  <post_archive>true</post_archive>

</event>

</FileEvent>

Notice the bolded line. Now look what happens when a file is sent in this instance (a sub-set of output since we are running with --debug):

$ fileevent.pl --action=process --debug --config example2.xml

2013/08/12 22:47:38.575 0000014-D send_file of '/home/venture/projects/SOURCE/fileevent/testing/example2/file1_20130822.txt' called.

2013/08/12 22:47:38.575 0000015-D xfer_destination raw: test@lubuntu1:/tmp/%{f}

2013/08/12 22:47:38.576 0000016-D xfer_destination cooked: test@lubuntu1:/tmp/file1_20130822.txt

2013/08/12 22:47:38.761 0000017-D Archiving file in directory '/home/venture/projects/SOURCE/fileevent/testing/example2/archive'.

2013/08/12 22:47:38.761 0000018-I Directory to archive to [/home/venture/projects/SOURCE/fileevent/testing/example2/archive] does not exist - attempting create,

2013/08/12 22:47:38.761 0000019-I Archive directory created successfully.

2013/08/12 22:47:38.763 0000020-I File archived '/home/venture/projects/SOURCE/fileevent/testing/example2/file1_20130822.txt' as '/home/venture/projects/SOURCE/fileevent/testing/example2/archive/file1_20130822.txt.000000.gz'.

Notice that it automatically created a directory called archive and once the file had been sent to the remote destination it moved and compressed the file to that new directory.

Also note the filename FileEvent uses – it has added a “.000000” before the “.gz” compression extension. This is a feature that ensures that even if you process the same named file (which must be explicitly indicated), it will still be given a unique name in the archive directory to ensure any file previously sent and residing in the archive directory is not over written.

As before one file is processed for each run time until all files have been processed.