How can we change the default process.scratch = true for AWS Batch e.g. when using FSx for Lustre?
acknowledged
B
Beige Wildcat
It seems the value of
process.scratch
is set to true
by default for cloud-based executors such as AWS Batch. We are experimenting with using FSx for Lustre to provide scratch space for certain workflows, but we don't seem to be able to create a Compute Environment in which writing to the FSx for Lustre mount (within the workDir
) is the default. Is there a way to do this? I understand that for every run, we could override
process.scratch = false
in nextflow.config
, but it would be far preferable to set this in the Compute Environment since it is the Compute Environment configuration in which we decide to provide FSx for Lustre as scratch space instead of selecting nodes with substantial/fast local scratch space.
Rob Newman
acknowledged
Rob Newman
Beige Wildcat Currently, the only method to set the scratch option to
false
is to use Fusion 2. We can investigate the feasibility for adding this option to the Compute Environment configuration. To help guide us, can you provide some parameters or examples of the workflows you wish to run (size, number of jobs, etc) and the size of the scratch space needed? Thank you!