Pipeline Versioning - Create, track, and launch multiple pipeline configurations
in progress
Anton Tsyganov-Bodounov
This project will introduce versioning capabilities to Seqera Platform, allowing users to create, save, and reference different versions of pipelines based on their configuration and parameters.
Planned Features
Automatic Version Tracking
- Automatic snapshot creation whenever a pipeline is created, modified, or launched with edited versioned parameters
- Checksum-based tracking for pipeline integrity and provenance
- All configuration changes preserved in version history
Version Management
- Users with pipeline edit capabilities can assign custom names to snapshot versions
- Ability to set any named version as the default for launch
- Option to save completed workflow runs as new named versions
- Named versions can be reassigned to different snapshots
Versioned Parameter Control
- By default, parameter changes trigger automatic version creation
- Custom nextflow_schema.jsonfiles can define which parameters should not trigger versioning
- Launch users can modify parameters and automatically create snapshots
Version Selection at Launch
- All users see the default named version in the launch form
- Users with pipeline edit capabilities can select and launch any version (named or snapshot)
- Launch users can view and launch named versions from the pipeline details page
- Launch users create snapshots through parameter modifications but cannot publish named versions
Commit ID Tracking
- Commit IDs are stored alongside branch/tag information for reproducible pipeline execution
- Users can pin to specific commits or update to the latest branch version
- Pull latesttoggle sets commit ID for dynamic branch tracking
Target Users
This functionality is intended for bioinformaticians who customise platform pipelines for themselves or their users, applicable to all customers including Enterprise requiring pipeline execution control.
Rob Newman
marked this post as
in progress
Anton Tsyganov-Bodounov
Merged in a post:
Run history of pipeline
B
Bronze Crow
In workspaces with many pipelines that have been active for a long time, it could be useful to be able to see the history of runs of a specific pipeline. As far as I know, currently there is no direct way to do so in the GUI. One possibility is to just search for the pipeline name in the Runs page, but this is only applicable when not using random naming of the workflows.
There are at least two places where this could be implemented:
- A History tab next to the Detail one when viewing a pipeline
- A filter in the Runs page
Anton Tsyganov-Bodounov
marked this post as
planned
Anton Tsyganov-Bodounov
Merged in a post:
Customization of pipelines' launch interfaces
C
Concrete Hornet
We would like to build custom launch interfaces for nf-core pipelines in our Seqera Platform instance. By that I mean a restricted, minimal interface that only allows to set a minimal set of parameters while the rest is either frozen to a predefined default, or derived from these parameters.
For instance, for the nf-core/demultiplex pipeline, the only thing we'd like launch users to be able to set is the flowcell ID, the filename of the samplesheet and to which project this flowcell belongs. The input and output path shall be derived from the flowcell and project ID, and other options (e.g.
demultiplexer
) shall not be editable by the user. For instance, to derived the outdir from the project_id
, we'd envisage something like params.outdir = "s3://some-company-namespace-${params.project_id}/demultiplex"
.To achieve this, for now, we built a custom Nextflow pipeline that wraps the nf-core pipeline with a custom schema. Example. However, ideally, the same could be achieved directly in Seqera Platform. For that, I believe the following would be necessary:
- support for custom schemas outside the pipeline repo, see also
- support for custom nextflow.configoutside the pipeline repo
- some sort of scripting to manipulate params and possibly input samplesheets (i.e. generate params and samplesheets from whatever is specified in the custom schema).
Anton Tsyganov-Bodounov
Merged in a post:
Pipeline details: versions support
B
Brass Wildcat
In the platform, each pipeline is linked to a specific repository version and a predefined set of parameters. This feature will enable associating multiple configurations and/or versions to a single pipeline to boost flexibility and simplify user interaction.
Rob Newman
Merged in a post:
Allow custom schemas from a second source
Rike Hanssen
At the moment custom schemas can be set in the pipeline launch form, but they have to be in the same repository as the pipeline code.
When adding community pipelines (like nf-core) that custom schema would need to be available in the same repository.
To make this process easier, it would be helpful to be able to add a schema from another location like a raw github URL or s3 path.
Rob Newman
Bronze Crow: Does applying a filter on the Runs page in a workspace and hitting search not provide this information? For example: https://cloud.seqera.io/orgs/seqeralabs/workspaces/showcase/watch?search=proteinfold
Rob Newman
Merged in a post:
Improved support of schemas of multi-revision workflows
M
Mighty Bird
Dear Seqera-Team,
As of Version 23.2.0, the Seqera Platform has poor support for launching multi-revision pipelines and the UI facilitates the misconfiguration of parameters.
Therefore, I wanted to point out a potential UI/UX improvement.A user launching a run via Seqera Platform’s browser interface is first presented with a form for entering pipeline parameters. Similar to nf-core launch, this form is based on the
nextflow_schema.json
file. Only if all mandatory parameters were provided, it is possible to proceed to the Launch settings. Here, a different workflow revision, e.g. an older version, may be selected. Unfortunately, it is rather the rule than the exception that parameters and schema change between workflow revisions. Hence, the current UI does not allow configuring launches of older revisions properly.
For example,
nf-core/taxprofiler
uses the parameter kraken2_save_readclassification
up to revision and 1.0.1 and kraken2_save_readclassifications
for newer releases. Even if a user is aware of this change, it is not possible to provide the correct parameter kraken2_save_readclassification
when launching older revisions, since it will be validated against the most recent schema. When providing kraken2_save_readclassifications
instead, it is ignored for older revisions and the pipeline’s default is used. Moreover, it happens silently, so a user can easily fail to notice that the workflow is not executed as planned.To no longer promote misconfiguration via the UI, I suggest putting the revision choice first and thereafter displaying the parameter form, based on the actual
nextflow_schema.json
of that particular revision: bash git --git-dir="</path/to/pipeline.git>" show <revision>:nextflow_schema.json
If the schema is for technical reasons only read when adding or updating pipelines, and looping through all revisions is not feasible, it should be possible to disable the parameter form in favour of a general text area to enter all parameters at once.
Thanks a lot for consideration.
Ticket attachments:
- SchemaRender.png
- RevisionSelector.png