Exposing the --chunk_size parameter to users #419

Hello,

Would it be possible to expose --chunk_size to users at the pipeline level, e.g. pipeline_singleRef? I noticed that for large files (e.g. 20,000+ reference sequences), chunk_size is usually the memory bottleneck. I know I can execute the individual commands to set the parameter there, however, that also comes with managing temp files that may or may not be created. For example, one error that I got with the deconstructed workflow was that a critical input was missing when executing:

Rscript /path_to/Rscripts/castor_hsp.R arc_reduced_placed_seqs.tre \
    /tmp/tmps2nqnxv4/subset_tab_12 mp 0.5 FALSE FALSE \ 
    /tmp/tmp_xu1vdam/predicted_counts.txt /tmp/tmp_xu1vdam/predicted_ci.txt 100

because /tmp/tmps2nqnxv4 was not created by the previous step despite the previous step completing successfully. When I manually changed the source code to reduce chunk_size to 100 and rebuilt, I was able to run the full pipeline with pipeline_singleRef. I think if chunk_size has this much of an effect on memory, then perhaps the full pipeline should allow users to set it as well as number of parallel processes.

Thanks in advance!