kmtricks uses all RAM

[szachn-u]

hello

I launched this kmtricks command for a file of files with ~ 37000 fastq files

kmtricks pipeline --file "$list_fq" --run-dir "$outDir/out" --kmer-size 31 --hard-min 1 --mode kmer:count:bin --until count --cpr -t 20

i runs nicely but at some point uses all ram available (i have 128Gb) and the script stops

./analysis_10x_MDAMB468_kmtricks.sh : ligne 30 : 21185 Processus arrêté "$kmtricks" pipeline --file "$list_fq" --run-dir "$outDir/out" --kmer-size 31 --hard-min 1 --mode kmer:count:bin --until count --cpr -t 20

Is there a way to limit the RAM usage ?

thanks a lot

[tlemane]

Hello,
Sorry for my late reply.

Unfortunately, you are encountering a known kmtricks problem. Memory grows with the number of files, regardless of their sizes, which is not expected. I have started investigating and this will be corrected in future releases.

It seems you use kmtricks only to count k-mers without building the matrix (because of --until count), right? In this case, you can simply do it in several runs by splitting your fof into smaller ones. If you need a matrix, you can use the same strategy and then combine all runs using kmtricks combine. See https://github.com/tlemane/kmtricks/wiki/combine.

Hope this helps
Teo

[MorillonLab]

Hi Teo
thanks for the answer, i'll try it.
Ugo