C
Charcoal Mandrill
Hi as the Seqera folks are aware we've had a lot of issues lately migrating to scripted "Infrastructure as Code" deployments with Seqera Platform, and I think the root of a lot of the problems we have hit is that when you use things like tw / seqerakit / Seqera Terraform Provider for deploying Compute Environments, there are a LOT of settings, most all of them independently toggle-able, yet if you want things like Wave + Fusion to work its critical that you use the correct combinations of settings with the correct combinations of values for those settings. This has lead to us making many rounds of thinking we deployed a working CE with Seqera based on small testing, only to find with larger scale testing (when the users start running pipelines) that something is wrong and things break.
I think the real solution to this is for Seqera to host some clear and repeatable and re-usable examples of full configurations for various use cases. Similar to how we have the "Nextflow Patterns" webpage that documents various end-to-end scripted methods for Nextflow, I feel that we really need something analogous for the Seqera Platform scripted deployments, because its too easy to miss settings that you did not realize were required for your CE to work.
I think some of the following would be good examples to start with, described in both tw and seqerakit and Seqera Terraform Provider;
  • deploying a Compute Environment with EBS AutoScale
  • the above but with Wave included
  • the above but with Wave + Fusion included
  • launching a test pipeline on the CE just created above
  • doing all of the above with both TowerForge created CE's and with Manually Added CE's
I think some examples like this are already in the seqerakit GitHub repo, not sure if there are equivalents already available for
tw-cli
. Ultimately it will be needed for the Terraform Provider as well, and they will all benefit from being available in one place so that users can see everything at once.