Howtos.md

Howto's

Create a Trigger for a Task

There are different methods to trigger a Task execution process:

• Using the tag RunOnTrigger
• Using the tags SartDate and StopDate
• Using the filtering tag FileType

The most flexible method is actually the FileType filter, as everything is treated as a file being processed

Using FileType filter method

The presence of a dedicated trigger file (eg. TR$PROCESSOR.*.trigger) , or also any other file, filtered by the expression, acts as a driver for the Task execution. This allows than to create a list of trigger files that the DES can then elaborate in a FIFO queue.

Because the Date for example is often used as input parameter for a processing time series for example, the Date has to be passed as input parameter to the Task executing a particular job.

One approach to implement this, is to ensure that the trigger file TR$PROCESSOR.*.trigger includes the date at a certain position, i.e position 5 (. separated):

TREURAC_RADBEAM_P.southtyrol_highres.6120_6120.hour.8.2017-02-10T15:00:00.Eurac.01.00.trigger

So the implemented wrapper Script (Perl, Python, Shell) or Java Class for the Task can extract the processing parameters as needed, i.e. The date to be processed. The trigger file itself might contain even a more structured meta information in form of XML for example.

By convention the trigger file should have the following naming convention:

TR$PROCESSOR.$AOI.TileID.Type.Value.Date.Eurac.$versionMajor.$versionMinor.trigger

Any other naming convention can be adapted by providing an appropriate wrapper script that is called.

Example

Processor configuration:

<TaskGroup name="Processing" type="serial" active="yes" priority="NORM">
	<Task name="createMap" type="cmd" active="yes" priority="NORM">
		<EMail>mail@host.yourdomain</EMail>
		<NotifyEmail>OnError</NotifyEmail>
		<FileType>TREURAC_RADBEAM_P.*.trigger</FileType>
		<Command>sh</Command>
		<Parameter>/Algorithms/runProc.sh</Parameter>
	</Task>
	<Task>
		...
	</Task>
</TaskGroup>

Task implementation: (runProc.sh)

#!/bin/sh
arg=($*)
BASE_PATH=$1
AOI_NAME=$2
PROCESSOR_NAME=$3
TRIGGER_FILE_NAME=$4
 
echo "creating environment vars.."
# User specific aliases and functions
export PATH=$PATH:/Algorithms/
 
echo "start processing..."
 
file=$TRIGGER_FILE_NAME
echo "  processing file: $file"
 
DATE_FROM_FILE=$(echo "$file" | cut -d '.' -f 6)
echo "date: $DATE_FROM_FILE"
 
DATE=`date -d "$DATE_FROM_FILE"  +"%Y-%m-%d"`
HOUR=`date -d "$DATE_FROM_FILE" +"%H"`
HOUR2=$HOUR
HOUR=`expr $HOUR2 + 1`
MIN=`date -d "$DATE_FROM_FILE" +"%M"`
 
echo "running algorithm for date: $DATE"

Note: The Date format contained in the Trigger file naming convention should comply if possible with the format yyyymmddThhmmss

The trigger files may than be generated manually or via a script called by a Cronjob (see example below):

/Cronjobs/createDESTaskTriggers.sh /raid0/DES/data southtyrol_highres EURAC_RADBEAM_P 6120_6120 [ 20170101T091500 ]

Cronjob Script: (createDESTaskTriggers.sh)

#!/bin/sh
arg=($*)
BASE_PATH=$1
AOI_NAME=$2
PROCESSOR_NAME=$3
TILE_ID=$4
 
echo `date`
echo "creating trigger..."
TRIGGER_PATH=$BASE_PATH/$AOI_NAME/$PROCESSOR_NAME
DATE=`date +"%Y-%m-%dT%H:%M:%S"`
echo "Date: $DATE"
TR_FILE="TR$PROCESSOR_NAME.$AOI_NAME.$TILE_ID.year.2017.$DATE.Eurac.01.00.trigger"
echo "creating trigger $TRIGGER_PATH/$TR_FILE"
touch $TRIGGER_PATH/$TR_FILE
touch $TRIGGER_PATH/$TR_FILE.md5
echo "done"
echo "------------"

Run a DES instance in Test Mode

When testing a new processor it could be useful to have a test environment where it is possible to check the output. To do this is straightforward because for each installed DES instance there is a test_data folder where test can be performed.

Steps:

• Create a symbolic link to your processor: ln -s DES.jar DES_test.jar to have a better control on your DES_test (if you decide to call it in this way) processor
• Copy your YOUR_PROCESSOR.processor_config.xml file in the folder /DES/test_data/processor_config/
• Do your changes to this file. For example you can enable or disable AOI, Task, TaskGroup, Distribution, Download, etc.
• Open the file /DES/config/DistributionServerTest.ini and change the value of LOGS_PATH variable from LOGS_PATH=LOG4J to LOGS_PATH=STDOUT. This will allow to have the log in the standard output to facilitate the debug.
• Change to the DES folder: cd /DES
• Run the following command:

java -jar DES_test.jar -c ./config/DistributionServerTest.ini -p YOUR_PROCESSOR.processor_config.xml