Intelligent Compute should have configs to prefer dedicated EC2's for some tasks
acknowledged
C
Charcoal Mandrill
Related to the Feature Request about NVMe cache in the EC2 here https://feedback.seqera.io/feature-requests/p/nvme-free-disk-space-should-be-a-consumable-resource-for-intelligent-compute-sch and related to some past discussions, just wanted to get a dedicated Request for this as well.
In many cases we are favoring "stability" of Nextflow task execution moreso than "optimization". To this end, I think a lot of headaches could be avoided if we were able to more easily force Nextflow jobs in AWS Batch to be scheduled on dedicated EC2 instances, instead of attempting to pack tons of jobs on fewer EC2's. This especially becomes a concern when things like NVMe starts getting heavily utilized and there are concerns that multiple jobs e.g. using Fusion heavily for large data could end up colliding and potentially cause degraded performance or failures if the NVMe fills up from multiple jobs running at once, or for other reasons.
So overall, we would be interested to have the option to configure Intelligent Compute in a way that favors giving jobs their own dedicated EC2 instances, along with taking any other potential measures, to attempt to ensure stability of task execution.
Thanks
Rob Newman
updated the status to
acknowledged