Simpler Execution Log Persistence
acknowledged
Mattia
After a pipeline completes on Seqera Platform it can be tricky to clean up execution work folders (to save space) while preserving useful log files.
This workflow could be improved by enabling Seqera Platform/Nextflow to send all logs to separate storage – outside of the pipeline run work folder, but still accessible by Seqera Platform.
N
Net Ox
This is crucial since otherwise there is little point in a scatch directory. All files required to render the tower interface such as log files and the files required to render reports should be preserved in a place other than scratch.
Rob Newman
acknowledged
I
Indigo Wildcat
This would be great--we apply lifecycle rules to clear out our working directory after a few weeks, but would be great to retain nextflow logs elsewhere (not concerned about storage costs for them, likely to be negligible)
Mattia
Merged in a post:
Archive logs & files from jobs
T
Tangerine yellow Ladybug
We would like to see logs and files that are generated from the Nextflow jobs (launched manually or from the platform) to be archived into a dedicated Tower S3 bucket. The reason for this is working directories are purged are a certain amount of time, and so is cloud watch. If the run stays in tower, the logs should too.
This should include logs, timeline, reports, etc. Anything shown in the platform should survive work directory purging.
Y
Yellow sunshine Firefly
I strongly agree. Here is a design idea:
As part of the "Tower Platform" infrastrucutre is creates and owns an S3 bucket. When a pipeline is completed, it copies the log files from the work directory, to thw Tower managed S3 bucket. It then assoicates all log references for the run in Tower, to the files in S3. This will allow the work directory to be deleted, but all logging information is retained. When the run is deleted from Tower, Tower would then delete the log files from the Tower S3 Logs Bucket.
M
Mighty Bird
I strongly second this one.