Seqera Platform Feature Requests

AWS Batch provides some great compute resource sharing capabilities via scheduling policies and fair share job queues. This would allow within-queue job priority and resource allocation across different workflows in a compute environment. Minimal ask : for manual CEs add a parameter that defines the share identifier name . Bigger ask : for Batch Forge allow creation of scheduling policies within the job queue.

GCP has a new feature designed for AI/ML use, zonal buckets, which promise lower read/write latency but are pinned to one specific zone. We can accomplish this by adding a clause to the nextflow.config override for the compute environment, but it would be good to be able to just set this on the compute environment directly.

I want to launch jobs directly on shared workstations without having to install a scheduler. Nextflow should simply run the job as "local" instead of "slurm", "pbs", etc. This feature would be extremely useful as it would help managing those workstations (I would create a workspace for each one) without having to provide access to each user. It would work either with ssh or tw agent.

Now that support for network tags in Google Batch has been merged into Nextflow via PR #5951 , the Platform should expose a corresponding option in the UI to allow users to define custom network tags when configuring a Google Batch CE. Network tags are critical for GCP environments where firewall rules, routing configurations, or other networking behaviors are scoped based on these tags. Without the ability to specify these in the Platform UI, users will have to specify them at the Nextflow config level for each workflow/execution. Proposed Feature: Add a new input field to the Google Batch CE Advanced Options panel in the Platform UI. This field should accept one or more comma-separated network tags, which will be passed through to Nextflow and included in the google.batch.networkTags` setting in config. Validation should ensure the tags follow GCP's allowed syntax (e.g., lowercase letters, digits, hyphens; must start and end with a letter or number). You can find more info here on network tags: https://cloud.google.com/vpc/docs/add-remove-network-tags

At the moment if you want to fix a Nextflow version, you have to use a pre-run script to export the NXF_VER environment version. This is fine, but it's not intuitive for what is a fairly central part of the experience of running Nextflow on Seqera Platform. Suggestion is to have a new field in the UI to set the Nextflow version. This could potentially be when creating a new Compute Environment, ideally with a secondary field which could be set when launching a workflow (either to set new or to override the default in the CE). The field should default to using the latest stable release of Nextflow, but give the option of using the latest edge release instead, or a specified exact release version.

Currently, Seqera Platform can authenticate to Azure using a service principal. While fine, this requires long-lived keys. An enterprise (on premise) installation of Seqera Platform could use a managed identity for credential-less authentication. This would be more secure and provide significantly more security but is only available when running on an Azure hosted system. This is supported on AWS using the equivalent IAM roles: https://docs.seqera.io/platform/24.2/enterprise/advanced-topics/use-iam-role Azure supports a resource having >1 Managed Identity attached, therefore there are two ways this could work: Add a system assigned managed identity, users would still need to add Azure "credentials" for the Azure Storage and Batch account name, however they would not need to add any service principal details. Using multiple user-assigned managed identities, a user would still need to add Azure "credentials" for the Azure Storage and Batch account name, however they would also need to specify which managed identity they would like to use to authenticate 1 is easier to configure and more secure, however 2 provides far more control for system administrators. It might require more work on the user end to know which ID to use.

When Seqera Platform submits jobs to Azure Batch and encounters a 409 HTTP status code (indicating the batch quota limit has been reached), the run fails. Instead of failing, can we modify the behavior so that the platform continues attempting to submit the job until the Azure Batch pool can accept it? Caused by: Status code 409, { "odata.metadata":"https://babixanaldevm42aen.uaenorth.batch.azure.com/$metadata#Microsoft.Azure.Batch.Protocol.Entities.Container.errors/@Element", "code":"ActiveJobAndScheduleQuotaReached","message": { "lang":"en-US","value":"Active job and job schedule quota for the account has been reached.\nRequestId:e9dede46-9ab1-463b-b797-cdfee3d4db8f\nTime:2024-10-18T06:36:49.8081952Z" } } Instead of returning a 409 Status Code error and halting the run, Seqera Platform should retry submitting the job continuously until the quota is available, ensuring that the job is eventually accepted. This should be done without manual intervention and in a way that does not overload the system.

When users submit jobs via Seqera Cloud (particularly with Google Batch executions), if there are underlying API request errors or IAM permission issues (e.g., request is denied or returns 403 errors), the Platform currently surfaces a generic message: "Unexpected error while processing request - Error ID: 6F1eRTX5WuqySP3zS6E5W6" This generic response is uninformative and forces users to reach out to Support to understand what went wrong. To enhance the user experience and reduce unnecessary support inquiries, we need to improve how these errors are surfaced to users. Infrastructure and DevOps teams at larger organizations also often need access to these error details to debug permission issues and misconfigurations on their end. Lack of clear error messages slows down troubleshooting and increases dependency on Seqera support. Instead of a generic error, display something like and potentially the response as well: "Error: Request denied due to insufficient IAM permissions. Please check your service account roles or quota settings."

We have issues where user submitted pipelines running in AWS Batch end up with misconfigured resource requests. This is especially the case when Nextflow pipelines use dynamic directives to automatically increase their CPU or Memory resource requirements on subsequent retried pipeline tasks. User's Nextflow tasks end up eventually requesting more CPU or Memory resources than is available by any of the EC2 types in the AWS Batch Compute Environment. This causes the jobs to get stuck in a RUNNABLE state. However, worse is that this causes the entire AWS Batch Job Queue to grind to a halt and stop running any Jobs at all. This is not a mistake on AWS's part, this is the accepted and expected behavior for AWS Batch; a single misconfigured Batch Job halts all job execution on the entire Job Queue. See the docs here; https://docs.aws.amazon.com/batch/latest/userguide/job_stuck_in_runnable.html The "solution" for this from AWS is that you need to update your AWS Batch Job Queue with jobStateLimitActions.action in order to have jobs stuck in RUNNABLE state automatically get canceled after some time limit if they are stuck due to MISCONFIGURATION:COMPUTE_ENVIRONMENT_MAX_RESOURCE or MISCONFIGURATION:JOB_RESOURCE_REQUIREMENT . I am not aware of any alternative method for handling this, as this is the procedure recommended by AWS Support themselves. This is a problem because Seqera Platform's TowerForge interface for creating Compute Environments with AWS Batch via Platform does not appear to have any way to add these jobStateLimitActions. We cannot apply the jobStateLimitActions in a post-hoc manner ourselves because the TowerForge CE's are being actively managed by Platform, via the Platform UI, tw , and seqera-kit . We cant go making custom changes to the CE outside the scope of what TowerForge has set up since that breaks any notion of consistency in our deployment via "Infrastructure as Code". We also cannot even make these Job Queue changes ourselves since the AWS environment is managed and requires escalation and support tickets to IT to make such changes, which is infeasible if we need to re-deploy our CE's repeatedly via tw / seqerakit , etc.. Ultimately these features need to be baked into TowerForge so that we can apply this in our scripted Platform configs.

Currently, when adding an AWS Batch compute environment and setting it to use Forge, if an SSH key pair is configured, the pipelines will fail right from the beginning, since the roles created by Forge do not include the necessary permissions to use the keys.