Newer
Older
The DES startup configuration contains initialization variables that are loaded at DES startup or restart
**RDS_ID** DES identifier (ex. DES_yourID@yourdomain.org)
**RDS_HOME** HOME directory of DES
**CONFIG_FILE_PATH** Path where processor configuration files are located ( $PROCESSOR.processor_config.xml)
**LOGS_PATH** STOUT print to stdout | LOG4J : use log4j2| Path of the log file dir (ending with / , ex: ./testdata/logs/ ) directory will be created
**BASE_PATH** Basepath at which logical directory tree starts: $BASE_PATH/$AoiName/$ProcessorName
**STAMPS_PATH** Directory for Stamp files
**COMPILECODE_PATH** Temporary directory where class files are compiled on the fly. Content will be deleted at every restart
**MAX_PROCESSOR_THREADS** Max processor Threads
**MAX_AOI_THREADS** Max Threads per AOI
**MAX_AOI_TASKGROUP_THREADS** Max Threads per TaskGroup
**AOI_THREAD_SLEEP** Time interval for AOI Thread
**MAX_NR_FILTER_FILES** Max number of files filtered in each iteration
**DEBUG** Write to log files
**STDOUT_STDERR** Print Tasks output streams to STDOUT/STDERR)
[DB]
**USE_STATUS_DB** Enable Task status logging in DB
**DB_SERVER** = Database server
**DB_DATABASE** = Database name
**DB_PORT** = Database port
**DB_USER** = Database user
**DB_PWD** Database password
[Mail]
**FROM_EMAIL** e-mail for delivery
**SMTP** = URL for SMTP
**SMTP_PORT** Port for SMTP
[Plugin]
**CONFIG_PLUGIN** Configuration file for Plugins. Java class variable: DistributionServerConfig.CONFIG_PLUGIN
```
[System]
RDS_ID=DES_YOURIDC@yourdomain.org
RDS_HOME=/DES
CONFIG_FILE_PATH=./test_data/config/
LOGS_PATH=LOG4J
BASE_PATH=./test_data/data/
STAMPS_PATH=./test_data/stamps
COMPILECODE_PATH=./test_data/tmp
MAX_PROCESSOR_THREADS=50
MAX_AOI_THREADS=50
MAX_AOI_TASKGROUP_THREADS=50
AOI_THREAD_SLEEP=10000
MAX_NR_FILTER_FILES=5
DEBUG=true
STDOUT_STDERR=false
[DB]
USE_STATUS_DB=false
DB_SERVER=server.yourdomain
DB_DATABASE=des
DB_PORT=5432
DB_USER=user
DB_PWD=pwd
[Mail]
FROM_EMAIL=des@yourdomain.org
SMTP=mailsubmit.yourdomain.org
SMTP_PORT=25
[Plugin]
CONFIG_PLUGIN=./plugins/plugin.ini
```
## Processors
The DES executes concurrently a given set of *Processor* definition files that are defined in XML format.
A *Processor* configuraiton file ( *.processor_config.xml ) is the main executable entity for the DES.
- There can be 1 - n *Processor* configuration files running on a given DES instance.
- Each processor configuration file can have 1 - n *AOI* (Areas Of Interest).
- Each *AOI* can have 1 - n functional entities (*Distribution, Download, DataCleaner, Task, TaskGroup*)
- The same xml format is used also by EGPF (Eurac Generic Processor Framework)
A typical DES *Processor" may have the following layout:
```xml
<EURAC_GENERIC_P>
<Processing>
<AOI name="myAOI" active="yes">
<!-- Download data -->
<Download type="sftp" active="yes" priority="NORM">
...
</Download>
<!-- Process data -->
<TaskGroup name="processAOI" type="serial" active="yes" priority="NORM">
<Task name="task1" type="class" active="yes" priority="NORM">
...
</Task>
<Task name="task2" type="class" active="yes" priority="NORM">
...
</Task>
</TaskGroup>
<!-- Distribute data -->
<Distribution type="sftp" active="yes" priority="NORM">
...
</Distribution>
</AOI>
</Processing>
</EURAC_GENERIC_P>
```
The *Processor* configuration options available are documented here:
https://gitlab.inf.unibz.it/css-public/des_documentation/blob/master/Processors.md
Authentication parameters used in the operations **<Distribution>** and **<Download>** are stored within the file :
**DistributionAuth.xml:** ./config/DistributionAuth.xml
Sample entry:
```xml
<Auth>
<AuthRef name="testuser@server.yourdomain" active="yes">
<Host>server.yourdomain</Host>
<User>testuser</User>
<Pwd>test</Pwd>
<Param>nativejava</Param>
</AuthRef>
</Auth>
```
In the DES processor operation declarations the authentication entry can be references as follows:
```xml
<consumer name="testuser@server.yourdomain" active="yes">
<ProductType>*.txt</ProductType>
<EMail>notifymail@yourdomain</EMail>
<NotifyEmail>false</NotifyEmail>
<Recursive>true</Recursive>
</Consumer>
```
Inside a *Processor* definition configuration file some build-in parameter can be used to abstract some configurations.
The build-in parameters are replaced at runtime in all <Parameter> and in all custom tags (eg. *$DAY*)
*$BASEPATH* The base path variable configured in *DES.ini*
*$PROCESSOR* The name of the executing *Processor*
*$CWD* The current working directory, that expands to *$BASEPATH/$AOI/$PROCESSOR*
*$STAMPS_PATH* The path for Stamp files configured in *DES.ini*
*$FILE* Iff *<FileType>* is configured, references the current File being processed (*<ProductType>* is deprecated but still valid for backward compatibility)
1. Edit the ./config/DES.ini to fit your environment
2. Create the set of *.processor_config.xml files that you want to run and place them in the path CONFIG_FILE_PATH as defined in DistributionServer.ini config file
3. Choose a suitable operation mode
print program arguments:
```
java -jar DES.jar –h
```
1) Start a DES instance with default configuration defined in DES.ini and run all Processors (*.processor_config.xml files) located in CONFIG_PATH
2) Start a DES instance with default configuration defined in DES.ini and run a single Processor
```
java -jar DES.jar -p yourtest.processor_config.xml
```
3) Start a DES instance with a different configuration (eg. DES_test.ini) and run all Processors (*.processor_config.xml files) located in CONFIG_PATH
```
java -jar DES.jar -c ./config/DES_test.ini
```
4) Start a DES instance with a different configuration (eg. DES_test.ini) and run a given Processor
```
java -jar DES.jar -c ./config/DES_test.ini -p yourtest.processor_config.xml
```
### Passing command line startup variables to be referenced in DES Processor ###
Custom arguments can be passed to the startup command line
```
java -jar DES.jar -v HOSTNAME=des.yourdomain -v IP=192.168.2.2
```
The defined variables can than be referenced in a given Processor definition with the '@' reference (eg. @HOSTNAME)
```xml
<Task name="testCmd" type="cmd" active="no" priority="NORM">
<EMail>support.rs@eurac.edu</EMail>
<NotifyEmail>false</NotifyEmail>
<Command>ping</Command>
<Parameter>@HOSTNAME</Parameter>
</Task>
```