Newer
Older
For some tasks and processes it might be necessary to choose some optimizations for running a given processor, for examples when reprocessing
a large set of files:
- Setup a didicated indipendent DES instance for executing given tasks or processor
- Ensure that $BASE_PATH and $BASE_PATH are placed on a local disc rather than a shared network filesystem that might have extra latencies
- Make sure that the $CWD contains a subset of the files at a given time, and possibly only symbolic links to the actual data (See DES_Plugin recursiveSymlinkSetup.pl)
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
## Use of symbolik links when referencing files
In certain situations, for better management and more security, it might be more appropriate to have just the references (Symbolic Links)
to the data inside the $BASEPATH/$AOI/$PROCESSOR directory, instead of the data itself.
For security reason, the use of symbolic link is strongly recommended because it will avoid a potential, not foreseen, deletion of files by the *Task* implementation.
A dedicated *Task* plugin can be used to create the symbolic links and prepare the data environment for subsequent Tasks:
```xml
<AOI name="southtyrol" active="yes" sleepFactor="10">
<Task name="createSymlinks" type="cmd" active="yes" priority="NORM">
<RunOnTrigger>null</RunOnTrigger>
<EMail>mail@host.yourdomain</EMail>
<NotifyEmail>OnError</NotifyEmail>
<Command>perl</Command>
<Parameter>./plugins/recursiveSymlinkSetup.pl</Parameter>
<Parameter>$CWD</Parameter>
<Parameter>/mnt/COMPUTE/GeoServer/coverages/PROCESSING/$PROCESSOR_$AOI/data
</Parameter>
<Parameter>EURAC_RADBEAM_P.*.tif</Parameter>
</Task>
</AOI>
```
The *Task* entity defined above calls a perl script (*recursiveSymlinkSetup.pl*), which creates the symbolic links with respect to the parameters indicated:
1) the first parameter represents the folder where the symbolic links will be created i.e. *$CWD*
2) the second parameter represents the folder where the data to be pushed towards the FTP server are located i.e. */mnt/COMPUTE/GeoServer/coverages/PROCESSING/$PROCESSOR_$AOI/data*
3) the third parameter indicates a regular expression that can be used to filter the data i.e. EURAC_RADBEAM_P.*.tif
Note: The sleep factor has been set to 10 to let the *Task* run with a different time frequency in order to optimize the overall DES execution performance