T
Tall Egret
On many HPC systems it can be difficult to work round walltime and CPU usage limits for long running, persistent tasks. Possible solutions for this are to have either a special workflow queue with limited compute resources, but extended walltimes, or to allow long running tasks to run on dedicated 'workflow' login nodes. At NCI Australia, we are considering both potential solutions to facilitate long workflows for multiple scientific disciplines. To add more context, the standard maximum walltime for batch jobs at NCI is 48 hours and any unix process that consumes more than 30 minutes of core time on a login node is killed. 
This feature request would add an option into the Seqera Platform/Tower interface so that Nextflow runs in the same location as the agent. This would support two scenarios:
  1. A user submits a batch queue request to the 'workflow' queue for a multi-week allocation. Both the agent and Nextflow itself are run from this batch job. Workflows that take more than 48 hours can run without difficulty provided that individual workflow processes require less than 48 hours.
  2. A user starts the agent on a dedicated 'workflow' login node that allows long running unix processes. When workflows are started, Nextflow is also run on this node getting round the need to have a special queue. This may also be helpful at HPC facilities where compute nodes don't have the external network access required for Nextflow to communicate with the Seqera Platform/Tower.