Select platform compute environment via nextflow config without retrieving the ID from the console
acknowledged
Adam Talbot
It's quite common to point a Nextflow process at a different compute environment than the one the platform submits to. E.g when we create a second compute environment called "myGpuComputeEnvironment", we can point a process at it with a config file like so:
process {
withLabel: "GPU" {
queue = "TowerForge-3d21D1WBJ80hWpvZ9x7ks1"
}
}
But you have to rummage around in the AWS/Azure/GCP console and find that unique ID which can be frustrating. Instead, it would be much better if we could use the compute environment names from Seqera Platform, like so:
process {
withLabel: "GPU" {
queue = "myGpuComputeEnvironment"
}
}
The platform performs a switch under the hood to point
myGpuComputeEnvironment
to TowerForge-3d21D1WBJ80hWpvZ9x7ks1
, but the user has a much easier interface for re-directing a process to a different queue. In addition, if I delete and re-make myGpuComputeEnvironment
it will automatically re-point to the right compute environment.This would enable more complex deployments of compute environments while still using the same interface.
S
Silky Porpoise
Hey, Seqera, where are we on this? We can't migrate our CEs to Forge until this happens, and we'd really like to let Tower manage our CEs for us.
Rob Newman
Silky Porpoise: Thanks for note, Jim. This will be assessed once we have shipped Studios to General Availalbility.
Adam Talbot
Hey Silky Porpoise, what's the blocking issue for you guys? Everything here is possible but it has to be done manually in the nextflow configuration. The purpose of this feature request is to make it more convenient.
S
Silky Porpoise
Adam Talbot What's the blocking issue? It's as you've said. We have developers who've hard-coded the Batch queue names into their processes. As a result, moving to Forge is impossible without making them fix their code. Am I misunderstanding your remarks?
S
Silky Porpoise
To be more clear: Forge compute environments can come and go, and there's no contract with Seqera that it won't re-create something in Batch. We're assuming Batch names, though currently just hard to remember, are unchanging. That's not an assumption I want to make. People should stop hard-coding queue names in the first place, but this makes it worse.
Adam Talbot
Silky Porpoise thanks Jim, that's great. One thing to note is your developers can override the process.queue directive in configuration at runtime and I would generally recommend this over hard coding it into the pipeline, so the pipeline is portable. Although this might not work in your situation, it's a useful thing to note for anyone else who finds this issue.
Perhaps this is the wrong feature request though. We could imagine a way a user could assign certain processes to different batch queues while setting up a pipeline, e.g a UI screen where you can override the 'default' batch queue for specific processes, no config required.
Rob Newman
acknowledged