Conventions.md

# Conventions #
 
 
## Logical directory hierarchy ##
 
The DES starts it's operations (eg. <FileType> filtering) considering a logical hierachy of directories starting form a given BASE_PATH that 
reflects the configuration within a given processor configuration file.

The directories, if not already present, are created automatically with the following structure:
```
$BASE_PATH/$AOI/$PROCESSOR
```
 
 
 
## FileType filtering ##
 
The files to be handled by a give *Task* or that are used for a given *Task* triggering, are filtered by regular expressions defined in the *FileType* tag of the *Task*

Files are detected by the *FileType* filtering IFF:
- No stamp file exists (eg. yourfile.txt.stamp) in the path defined by *STAMPS_PATH/$PROCESSOR*
- An equivalend .md5 file exists in the logical directory hierarchy (ex.yourfile.txt.md5), unless the tag *MD5Filter* is set explicitly to value *false* 

Stamp files are used to track that a file has already been processed by a given Task.

IFF a given *Task* returns either *EXIT_SUCESS* (0) or *EXIT_WARNING* (1), the DES will create a corresponding stamp file for the file processed.

IF a given *Task* does not comply to the *EXIT_CODES* defined above, the *Task* implementation, upon completion,
should therefore either delete the input file and it's equivalent md5, and if necessary, to avoid a reprocessing of a given file, 
create a corresponding stamp file in *STAMPS_PATH/$PROCESSOR* directory
 
 
 
## Stamp files ##
 
A Stamp file is an empty file that is created iff a given Task has been completed for a given input file filtered by a *FileType* expression.
In presence of a stamp file a subsequent *FileType* filtering will not detect the file that has an equivalent stamp file.

Stamp files are stored in the directory specified by *STAMPS_PATH/$PROCESSOR*

Stamp files have the following naming convention:
```
[PUSH | GET | TASK]_[Consumer name | Task name]_[Procesor name]_[AOI name]_[File name].stamp
```
 
Example:
```
TASK_createMap_EURAC_RADBEAM_P_southtyrol_TREURAC_RADBEAM_P.southtyrol.demST25m.year.2017.2018-06-19T16:45:00.Eurac.01.00.trigger.stamp
```
 
 
 
## Exit Codes ##

The *Task* entities registered within a *Processor* definition  should use the following ExitCodes when returning from execution 

```
EXIT_NONE = -999; // No files are available in the DES task pipeline
EXIT_SUCESS = 0;
EXIT_WARNING = 1; //1 - 127
EXIT_ERROR = -1;
EXIT_ERROR_SHELL = 255; // Shell scripts that exit -1 are converted to 255
EXIT_NOPROC = -666; // Task was executed but, either did not any job at all, OR the Task plugin did skip the files 
EXIT_MISSING_PARAM = 2; // Task did not receive all or valid parameters
```