Support S3 conditional writes (If-None-Match) for publish operations
acknowledged
Jon Manning
Add support for AWS S3 conditional writes on publish operations in Nextflow and Fusion, so bucket policies requiring
If-None-Match: *
can be used to enforce object immutability at the storage layer.Why?
Strongest guarantee that published outputs are never overwritten, since enforcement lives in S3 itself and also defends against non-Nextflow writers. The publish
overwrite: false
option is the current workaround but is set in pipeline code (per-process on the classic publishDir directive, or per-output on the new workflow output block), can't be enforced globally at the run or platform level, and only protects against Nextflow.Current gap
Neither Nextflow nor Fusion sends
If-None-Match
on S3 writes today, so a bucket policy requiring the header would reject every write, including workDir
staging, making the policy unusable as-is.C
Clever Orca
Looking at AWS docs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html), I have seen this note that can produce undesired effects:
"If you use a bucket policy to enforce conditional writes, you can't perform copy operations to the bucket or prefix specified in your bucket policy. CopyObject requests without an If-None-Match or If-Match HTTP header fail with a 403 Access Denied error. CopyObject requests made with those HTTP headers fail with a 501 Not Implemented response."
In Nextflow, copyObject is used when publishing files where the working directory and publish directory are in S3 buckets. Not sure if the same could happen in Fusion
M
Michael Tansini
marked this post as
acknowledged