Template resource labels | Voters | Seqera Feedback Forum

Template resource labels

acknowledged

Brass Wildcat

Users frequently request the ability to associate cloud service provider tags with workflow runs. This feature aims to annotate resources utilized by workflow runs with cloud-specific tags, enhancing cost allocation, monitoring and tracking capabilities from the cloud provider's administration console.

The current limitations include the inability to add tags to Seqera Platform elements in shared workspaces. Additionally, resource labels lack parametric behavior (such as

seqera:workflowId={workflow_id}

, making it challenging to track users and individual workflows for cost allocation reporting.

The proposed change is two-fold:

Add resource labels to items in shared workspaces.
Introduction of template resource labels, allowing fixed names with flexible values at launch (e.g.,
user_name
or
workflow_id
). This enhancement significantly improves cost allocation tracking capabilities and management experience.

Implementing these enhancements aligns the resource label system with user expectations, providing a more flexible and powerful solution for resource annotation and tracking.

November 23, 2023

Flamingo pink Python

not sure if this was covered in the described scope already, but in addition to the other comments, an important consideration for my own purposes has been the ability to include dynamic values in the resource labels. This is a code snippet I have been playing with for this;

// in the nextflow.config file
process.resourceLabels = { [
            pipelineProcess: task.process.toString(),
            pipelineUser: workflow.userName.toString(),
            pipelineSessionId: workflow.sessionId.toString(),
            pipelineResume: workflow.resume.toString(),
            pipelineRevision: workflow.revision.toString(),
            pipelineCommitId: workflow.commitId.toString(),
            pipelineRepository: workflow.repository.toString(),
            pipelineRunName: workflow.runName.toString(),
            pipelineName: workflow.manifest.name.toString()
    ] }

However, I also want to expand this to include some of the following;

metadata about the Tower execution, including the Tower username, Tower run ID, Tower pipeline used, etc..
an arbitrary list of key:value pairs

The arbitrary list of resources labels would be provided outside of the Nextflow pipeline by the Tower User launching the pipeline from the web dashboard, via pipeline launch with the

cli tool, and even by launching Nextflow directly without the involvement of Tower. Also, the intention of this "arbitrary list of resource labels" is to encompass things such as Cost Centers, as someone commented on, along with any other business-specific labels that are needed.

Right now, I could hack it together by using some

params

field that accepts a list of mappings to append to the

process.resourceLabels

. I can also hack it together by using a wrapper-script to dynamically create a Nextflow config file and then passing that in with the pipeline execution. However the code to implement these things gets pretty hairy. Some kind of consistent, built-in approach as described in this Feature Request sounds preferable.

this is related to the ongoing efforts to allow for applying configs at the Compute Environment level ( https://feedback.seqera.io/feature-requests/p/allow-setting-nextflowconfig-settings-on-compute-environments-or-seqera-platform ) since some of these resource labels would also be specific per-compute environment, while others would not be and would be applied based on different criteria.

Rob Newman

marked this post as

acknowledged

Rob Newman

Merged in a post:

Is it possible to introspect a tower user in a workflow for tagging in Resource Labels?

Aquamarine Gibbon

Love the new resourceLabels directive. Curiosity question... Do you think it's possible when launching from tower to have users "username" added as a resourceLabel directive?

Maybe something along the line of:

process my_task {
    resourceLabels user: "$workflow.userName"
    
    '''
    <task script>
    '''
}

I guess I'm just not sure if/how one would access the users name when things were launched via Seqera Platform/Tower?

February 22, 2024

Amber Marlin

We would like to add cost centers to workflows/resources. Moreover we have a list (~70) such cost centers and would like our users to be

required

to enter one of the list of values when provisioning.

Prospective Parrotfish

+1 on this request. This would be a very useful feature to have and make tracking more robust.