Enable read access to pipeline secrets in the head node | Voters

Enable read access to pipeline secrets in the head node

closed

Enable users to use pipeline secrets in the head node, for example in the post-run script so to enable automation and integration with 3rd party tools and APIs

December 1, 2023

Rob Newman

updated the status to

closed

Secrets aren't broadly available in the environment on the head node and this is not a behavior we intend to change.

Phil Ewels

Hi folks, this is a short feature request but surprisingly complex in the details.
Pipeline secrets are a Nextflow feature. We try to keep their scope as narrow as possible, with Nextflow loading them as-needed at launch (for use in config, plugins) and generally restricting them to task nodes only where possible.
As of Nextflow 25.10, secrets.*
 can be used in the config, the workflow body and the onComplete
 block. Secrets within a process (on task nodes) should work as usual with AWS / GCP - we're currently working on extending to Azure as well.
What doesn't work is using secrets on the head node outside of Nextflow (eg. a post-run script). Secrets aren't broadly available in the environment on the head node and this is not a behaviour we intend to change.

Rob Newman

updated the status to

acknowledged