FileEvent Example 3 - Multiple File Support

Introduction

This builds on the previous two posts regarding the FileEvent tool and builds out the example to handle multiple copies of the file.

Working with lots of files

If we ignore that sent files shown previously and start from a clean configuration again with a directory containing the following files in this case:

-rw-rw---- 1 venture venture 488 Aug 12 22:30 example2.xml

-rw-rw-r-- 1 venture venture 25944 Aug 12 22:52 file1_20120723.txt

-rw-rw-r-- 1 venture venture 25944 Aug 12 22:53 file1_20130201.txt

-rw-rw-r-- 1 venture venture 27125 Aug 12 22:28 file1_20130819.txt

-rw-rw-r-- 1 venture venture 27125 Aug 12 22:28 file1_20130821.txt

-rw-rw-r-- 1 venture venture 27125 Aug 12 22:28 file1_20130822.txt

What if you want to process all matching files in a directory and don't want to call “fileevent.pl” multiple times until no files are sent? This is easy – a simple addition to the configuration file:

<?xml version="1.0" standalone="yes"?>

<FileEvent>

 <settings>

  <db>/tmp/testing.db</db>

 </settings>

 <event>

  <max_send_count>10</max_send_count>

  <description>scp example1 transfer</description>

  <file_pattern>file1_%{4year}%{2month}%{2day}.txt</file_pattern>

  <directory>/home/venture/projects/SOURCE/fileevent/testing/example3</directory>

  <xfer_job_type>scp</xfer_job_type>

  <xfer_destination>test@lubuntu1:/tmp/%{f}</xfer_destination>

  <post_archive>true</post_archive>

 </event>

</FileEvent>

The “max_send_count” means send between 1 and this number of files depending on what matches. Running the utilty is “list” mode using the command gives:

$ fileevent.pl --config example3.xml --action=list

Event #0 matched files:

Directory: /home/venture/projects/SOURCE/fileevent/testing/example3

Pattern : ^file1_%{4year}%{2month}%{2day}.txt$

Actual cut-down list to send now:

file1_20130822.txt

file1_20130821.txt

file1_20130819.txt

file1_20130201.txt

file1_20120723.txt

And running it in process mode (with verbose output):

$ fileevent.pl --config example3.xml --action=process --verbose

2013/08/12 23:05:24.196 0000000-I Events to load from configuratione file: 1

2013/08/12 23:05:24.217 0000001-I Event #0 [scp example1 transfer] processing.

2013/08/12 23:05:24.402 0000002-I Successfully sent '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130822.txt'.

2013/08/12 23:05:24.405 0000003-I Directory to archive to [/home/venture/projects/SOURCE/fileevent/testing/example3/archive] does not exist - attempting create,

2013/08/12 23:05:24.405 0000004-I Archive directory created successfully.

2013/08/12 23:05:24.406 0000005-I File archived '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130822.txt' as '/home/venture/projects/SOURCE/fileevent/testing/example3/archive/file1_20130822.txt.000000.gz'.

2013/08/12 23:05:24.596 0000006-I Successfully sent '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130821.txt'.

2013/08/12 23:05:24.600 0000007-I File archived '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130821.txt' as '/home/venture/projects/SOURCE/fileevent/testing/example3/archive/file1_20130821.txt.000000.gz'.

2013/08/12 23:05:24.841 0000008-I Successfully sent '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130819.txt'.

2013/08/12 23:05:24.845 0000009-I File archived '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130819.txt' as '/home/venture/projects/SOURCE/fileevent/testing/example3/archive/file1_20130819.txt.000000.gz'.

2013/08/12 23:05:25.036 0000010-I Successfully sent '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130201.txt'.

2013/08/12 23:05:25.047 0000011-I File archived '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20130201.txt' as '/home/venture/projects/SOURCE/fileevent/testing/example3/archive/file1_20130201.txt.000000.gz'.

2013/08/12 23:05:25.238 0000012-I Successfully sent '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20120723.txt'.

2013/08/12 23:05:25.242 0000013-I File archived '/home/venture/projects/SOURCE/fileevent/testing/example3/file1_20120723.txt' as '/home/venture/projects/SOURCE/fileevent/testing/example3/archive/file1_20120723.txt.000000.gz'.

So all 5 files have been sent in a single call, all in the order based on the date from the parsed file names.