Seqera Platform Feature Requests

On many HPC systems it can be difficult to work round walltime and CPU usage limits for long running, persistent tasks. Possible solutions for this are to have either a special workflow queue with limited compute resources, but extended walltimes, or to allow long running tasks to run on dedicated 'workflow' login nodes. At NCI Australia, we are considering both potential solutions to facilitate long workflows for multiple scientific disciplines. To add more context, the standard maximum walltime for batch jobs at NCI is 48 hours and any unix process that consumes more than 30 minutes of core time on a login node is killed. This feature request would add an option into the Seqera Platform/Tower interface so that Nextflow runs in the same location as the agent. This would support two scenarios: A user submits a batch queue request to the 'workflow' queue for a multi-week allocation. Both the agent and Nextflow itself are run from this batch job. Workflows that take more than 48 hours can run without difficulty provided that individual workflow processes require less than 48 hours. A user starts the agent on a dedicated 'workflow' login node that allows long running unix processes. When workflows are started, Nextflow is also run on this node getting round the need to have a special queue. This may also be helpful at HPC facilities where compute nodes don't have the external network access required for Nextflow to communicate with the Seqera Platform/Tower.

Seqera Platform currently supports Prometheus-based metrics but does not export metrics or log messages related to pipeline or workflow failures. Customers are therefore required to poll the Platform API to detect elevated failure rates, which may have performance impact. This feature request tracks the work to add basic pipeline status metrics (e.g. counters for pipelines launched, failed, and succeeded) to support monitoring and alerting without API polling.

With the introduction of new workflow output syntax, users are requesting if we can include a field on Platform to explicitly set the outputDir (instead of setting it at the top-level in the nextflow.config file). This is primarily due to the order in which Seqera Platform reads in parameters, and the potential conflict in the values for outputDir and params.outdir

Currently, Run names are auto-populated by a random two-word string. It would be helpful for organizations if naming conventions for resources in Seqera Platform (such as Runs) could be enforced via a pattern/policy defined by an administrator.

When a Nextflow pipeline fails and gets restarted, only costs for successful jobs are passed over to the new run. I would like to account for failed jobs also so we can estimate how much a pipeline cost in total, including failed runs.

Nextflow metering is great for memory and CPU - do you have any plans to add storage utilization to your metrics? We find that tmp file storage can account for a significant additional cost in our pipeline runs and we would like to have some way to measure and optimize.

Will it be possible for users to host their own registry to deploy private plugins in their org?

We would like to extend our use of Seqera Platform/Tower for communication between people and want the ability to comment on Runs within the platform. This tab should keep track of the comments and who commented based on user. This will allow people to put comments on failed runs so that people can know if someone is already looking at it. This could be used for whatever purpose we like, allowing user details to be put into the run and stay with the run.

Currently it seems you need to append the .dump() operator to each channel you're interested in debugging, but this means debugging pipelines you've pulled in requires you chase down every channel in the source and append .dump to it to find the cause of the problem

As far as I'm aware, there is no way to specify these behaviours inside a .config file, they can be only done through the CLI flags. However the web GUI doesn't let you add custom CLI flags. There is a toggle for "Stub Run". I think there should be more toggles for these debugging features.