My new data analysis pipeline code
First, I write a recipe file, 'metagenome.recipe', laying out my job description for, say, sequence trimming and assembly with Velvet:
fasta_file soil-data.fa qc_filter min_length=50 remove_Ns=true graph_filter min_length=400 velvet_assemble k=33 min_length=1000 scaffolding=True
Then I specify machine parameters, e.g. 'bigmem.conf':
[defaults] n_threads=8 [graph_filtering] use_mem=32gb [velvet] needs_mem=64gb
And finally, I run the pipeline:
% ak-run metagenome.recipe -c bigmem.conf
If I have cloud access (and who doesn't?) I can tell the pipeline to spin up and down nodes as needed:
% ak-aws-run metagenome.recipe -c bigmem.conf
(Bear in mind most of these tasks are multi-hour, if not multi-day, operations, so I'm not too worried about optimizing machine use and re-use.)
Hadoop jobs could be spawned underneath, depending on how each recipe component was actually implemented.
As for testing reproducibility of pipeline results, which is the short-term motivation here, I can store results for regression testing with later versions:
% ak-run metagenome.recipe -c bigmem.conf --save-endpoint=/some/path
and then compare:
% ak-run --check-endpoint=/some/path
Now, does anyone know of a package or packages that actually do this, so I/we don't have to write it??