Pipeline pre-flight check
acknowledged
Andrew Dawson
We’d love to see a “pre-flight check” feature in Seqera Platform that helps users catch common issues before a pipeline is submitted. This would improve the success rate of runs by validating key parts of the configuration and cloud environment up front.
Specifically, this check could:
- Generate a clear, consolidated summary of the final configuration (combining Nextflow config, Platform overrides, workspace settings, etc.)
- Verify authentication and authorization to external resources (e.g., object storage, compute queues)
- Flag missing or potentially conflicting settings
- Suggest fixes or helpful links if something looks off
This would help users answer questions like:
- “What credentials will my job use?”
- “Am I allowed to access this S3 bucket / Azure container?”
- “Which compute environment is actually going to run this?”
It could be surfaced as:
- A button in the UI before submitting a pipeline
- A CLI command (e.g., nextflow nf-launch --check)
- Or even integrated into the pipeline submission flow as an optional step
This kind of validation would be especially valuable in cloud or enterprise environments with complex policies and configurations. It would reduce frustration, support burden, and unnecessary trial-and-error.
Anton Tsyganov-Bodounov
Merged in a post:
Provide better warning messages for schema default and config default mismatches
Edmund Miller
If no default should be set, completely omit this key from the schema. Do not set it as an empty string, or null.
However, parameters with no defaults should be set to null within your Nextflow config file.
We should do a better job of warning users when there's no default set.
Rob Newman
Merged in a post:
Warn users when their schema and nextflow.config are out of sync
Edmund Miller
From Nextflow docs:
When creating a schema using nf-core schema build, this field will be automatically created based on the default value defined in the pipeline config files.
Generally speaking, the two should always be kept in sync to avoid unexpected problems and usage errors. In some rare cases, this may not be possible (for example, a dynamic groovy expression cannot be encoded in JSON), in which case try to specify as "sensible" a default within the schema as possible.
We warn users about this in the docs but it would be nice if we gave feedback to them before they clicked launch
Rob Newman
Merged in a post:
Improving credentials validation in Seqera Platform/Tower
B
Brass Wildcat
The inability to apply correct Git credentials is frequently reported by Seqera Platform users. This can cause pipeline executions to fail because Nextflow cannot successfully pull the pipeline code from the Git repository.
Additionally, Seqera platform currently lacks real-time updates for compute environment (CE) statuses, resulting in CE being marked as "Available" even when they are not. This leads to a number of failed jobs or jobs remaining in a submitted status indefinitely.
We aim to enhance this user experience by adding a credentials validation feature to the platform:
- Validation on Credentials Creation: When a user adds new Git credentials, the system must provide a test URL. The system will use this test URL to validate the credentials and either accept or reject them.
- Add a Validate Action in the Credentials List Page: Users will have the ability to validate their credentials and check if they remain valid over time.
- Implement System Periodic Checks: The system will periodically validate the credentials to ensure they are not expired. When the system detects non-functional credentials, a warning alert will be displayed on the list page, and, optionally, an email notification can be sent.
- Pipeline Git Credentials Reporting: When launching a pipeline, the platform will report which Git credentials have been applied to access the pipeline Git URL or display an error message if no credentials are found.
- Add a "refresh/check connectivity" button on the CE page to validate the actual availability of each compute environment. Additionally, leveraging periodic checks can ensure status accuracy without placing the entire responsibility on administrators.
Rob Newman
Merged in a post:
Pipeline launch validation
B
Brass Wildcat
The successful launch of a pipeline involves access to various critical resources. These resources include:
- A Git repository hosting the pipeline code.
- Data input files stored across different storage solutions (e.g., HTTP, S3, Azure).
- One or more storage buckets where the pipeline's work and output data will be stored.
- Container images stored in public or private registries.
- Access credentials required for storage and computing resources.
- Secrets that may be necessary for pipeline execution.
Having one or more of these parameters missing or incorrectly configured can prevent successful pipeline execution, causing frustration for users and making troubleshooting complex.
To improve the user experience and prevent runtime errors, the system could proactively validate the launch parameters at launch time. Potential improvements include:
- Parse the pipeline input form and configuration to determine if all input files are accessible.
- Verify that the pipeline's working directory is accessible and writable.
- Check the accessibility of pipeline container images.
- Confirm that the pipeline's Git repository is reachable.
- Validate the compute environment required for the pipeline execution is available.
- Verify the accessibility of any secrets declared by the pipeline.
Andrew Dawson
acknowledged
Andrew Dawson
Merged in a post:
Pre-execution Nextflow version checker
B
Brass Wildcat
Add a pre-execution validation check that verifies the Nextflow version matches the Seqera Platform/Tower requirement and returns an actionable error message if the wrong version is used.
For context, when using a grid based compute environment, the Nextflow runtime needs to be pre-installed in the target environment. A common problem is using a Nextflow version that does not match the one required by Seqera Platform/Tower, causing an unexpected error.
Andrew Dawson
Merged in a post:
Pre-flight checks to compare requested resources to available AWS Batch resources
Ken Brewer
It would be helpful to include Seqera Platform-based pre-flight checks to compare requested system resources versus the maximum capacity of EC2 instances available to the AWS Batch cluster. This could prevent jobs with excessive resource demands locking up a compute environment.
S
Sapphire Canidae
We ran into a similar issue where we had requested 1000 GB RAM for a process, but the largest instance type in the autoscaling group we had provisioned was an r6id.32 xlarge (128 CPU and 1024 GiB RAM). This resulted in a blockage in our queue, with lots of jobs stuck in "RUNNABLE". It was only when we lowered our process resource request to 900GB that we were able to unblock the queue. Having a preflight check or warning would have been an incredibly useful feature.
Y
Yellow sunshine Firefly
We have run into issues where tasks requesting more resources than the cluster could ever give cause other jobs to get stuck in line,