We have been using Fusion for a while now and overall its fantastic. However, it seems like one of the more common edge-cases we ended up debugging is for tasks and processes that are reading or writing data to/from disk at incredibly high rates. In these cases, the data IO speed seems like it can sometimes degrade when Fusion is in use, and one of the "work arounds" can sometimes end up being to try to force the Nextflow task to manually download the entire S3 file locally first, so that the scripts and tools used in the Nextflow task can take advantage of the full speed of the local NVMe cache without the overhead of streaming data in from S3 in the background.

This goes against the core purpose of Fusion, and I know the Fusion dev team is working to deal with these situations better, but it also makes me wonder if its worth having some sort of "stage in method" for Fusion (possibly configured from the Nextflow script?) in order to indicate that some specific input files should just automatically get fully downloaded ahead of time in the background by Fusion instead of trying to stream them in as normal.

I know this would surface other issues, such as the NVMe cache potentially filling up, so I am not sure if this is an ideal solution to the problem. However at least for debugging, it would actually be very helpful to be able to easily enable a feature like this. Let me know what you guys think. Maybe some alternatives that keep Fusion's core functionality better, might be the ability to ensure that it is pre-cache'ing larger chunks of data than might be the default setting.