Skip to content
Snippets Groups Projects

Optimization

Processing of large set of files

For some tasks and processes it might be necessary to choose some optimizations for running a given processor, for examples when reprocessing a large set of files:

  • Setup a didicated indipendent DES instance for executing given tasks or processor
  • Ensure that $BASE_PATH and $BASE_PATH are placed on a local disc rather than a shared network filesystem that might have extra latencies
  • Make sure that the $CWD contains a subset of the files at a given time, and possibly only symbolic links to the actual data (See DES_Plugin recursiveSymlinkSetup.pl)