Introduction
FileEvent is a utility designed to allow event-driven handling of files when they come into existence based on simple rules stored in XML configuration files. This is the second example of how FileEvent might be used and extends the facilities shown based on the initial example.Why Archive?
In the previous
example it was shown how FileEvent is stateful in that (by default)
will not process the same file twice. This means that when multiple
files are matched in a directory and not all files are processed it
will “skip” the files in question, for example re-capping the
previously shown command:
2013/08/12
22:25:23.443 0000013-W File
'/home/venture/projects/SOURCE/fileevent/testing/example1/file1_20130822.txt'
already sent - removing from list to send or fetch.
2013/08/12
22:25:23.443 0000014-W File
'/home/venture/projects/SOURCE/fileevent/testing/example1/file1_20130821.txt'
already sent - removing from list to send or fetch.
2013/08/12
22:25:23.443 0000015-D Actual cut-down list to send now:
2013/08/12
22:25:23.443 0000015-D file1_20130819.txt
2013/08/12
22:25:23.443 0000016-D send_file of
'/home/venture/projects/SOURCE/fileevent/testing/example1/file1_20130819.txt'
called.
2013/08/12
22:25:23.444 0000017-D xfer_destination raw: test@lubuntu1:/tmp/%{f}
2013/08/12
22:25:23.444 0000018-D xfer_destination cooked:
test@lubuntu1:/tmp/file1_20130819.txt
This is all well
and good, but this scanning the files and reordering them is
additional processing which is best avoided, and hence we come to
another feature of FileEvent – archiving.
FileEvent Archiving
Archiving will move
any successfully sent files to an archive directory and compress it
to, saving space. Just a single additional line is usually enough,
for example:
<?xml
version="1.0" standalone="yes"?>
<FileEvent>
<settings>
<db>/tmp/testing.db</db>
</settings>
<event>
<description>scp
example1 transfer</description>
<file_pattern>file1_%{4year}%{2month}%{2day}.txt</file_pattern>
<directory>/home/venture/projects/SOURCE/fileevent/testing/example2</directory>
<xfer_job_type>scp</xfer_job_type>
<xfer_destination>test@lubuntu1:/tmp/%{f}</xfer_destination>
<post_archive>true</post_archive>
</event>
</FileEvent>
Notice the bolded
line. Now look what happens when a file is sent in this instance (a
sub-set of output since we are running with --debug):
$ fileevent.pl
--action=process --debug --config example2.xml
2013/08/12
22:47:38.575 0000014-D send_file of
'/home/venture/projects/SOURCE/fileevent/testing/example2/file1_20130822.txt'
called.
2013/08/12
22:47:38.575 0000015-D xfer_destination raw: test@lubuntu1:/tmp/%{f}
2013/08/12
22:47:38.576 0000016-D xfer_destination cooked:
test@lubuntu1:/tmp/file1_20130822.txt
2013/08/12
22:47:38.761 0000017-D Archiving file in directory
'/home/venture/projects/SOURCE/fileevent/testing/example2/archive'.
2013/08/12
22:47:38.761 0000018-I Directory to archive to
[/home/venture/projects/SOURCE/fileevent/testing/example2/archive]
does not exist - attempting create,
2013/08/12
22:47:38.761 0000019-I Archive directory created successfully.
2013/08/12
22:47:38.763 0000020-I File archived
'/home/venture/projects/SOURCE/fileevent/testing/example2/file1_20130822.txt'
as
'/home/venture/projects/SOURCE/fileevent/testing/example2/archive/file1_20130822.txt.000000.gz'.
Notice that it
automatically created a directory called archive and once the file
had been sent to the remote destination it moved and compressed the
file to that new directory.
Also note the
filename FileEvent uses – it has added a “.000000” before the “.gz”
compression extension. This is a feature that ensures that even if
you process the same named file (which must be explicitly indicated),
it will still be given a unique name in the archive directory to
ensure any file previously sent and residing in the archive directory
is not over written.
As before one file
is processed for each run time until all files have been processed.
