Killed after receive Segmentation fault:SIGSEGV(11) signal

[dosshra]

running:
kmtricks pipeline --file fof2 --run-dir ./kmer_pa --kmer-size 31 --mode kmer:pa:text -t 10
I get:

Killed after receive Segmentation fault:SIGSEGV(11) signal

running 120 samples, 420 read files (~14GB each) on Ubuntu 22.04, kernel 6.5.0-21-generic, kmtricks version v1.4.0. installed in conda environment. 125GB of RAM, 50BG sawp, 25 CPU. I monitored CPU and RAM using htop and did not see overuse of either CPU or RAM see log file:

Backtrace:
1 0x00007fe97f642520 (null) + 140640841377056
2 0x000055a6bf40c531 (null) + 94174661625137
3 0x000055a6bf40c33d (null) + 94174661624637
4 0x000055a6bf3ee1a3 gatb::core::kmer::impl::RepartitorAlgorithm<32ul>::computeRepartition(gatb::core::kmer::impl::Repartitor&) + 563
5 0x000055a6bf3eee6a gatb::core::kmer::impl::RepartitorAlgorithm<32ul>::execute() + 138
6 0x000055a6bf290914 (null) + 94174660069652
7 0x000055a6bf291098 (null) + 94174660071576
8 0x000055a6bf2917d0 (null) + 94174660073424
9 0x000055a6bf044260 main + 3312
10 0x00007fe97f629d90 (null) + 140640841276816
11 0x00007fe97f629e40 __libc_start_main + 128
12 0x000055a6bf0465e5 (null) + 94174657668581

Please advise? Thank you

[tlemane]

Hello,

Sorry for this issue.
The problem seems to occur at an early stage of the process in an external library which basically reads a few sequences from each input file to compute a partitioning scheme. I have never encountered this kind of problem, can you post an example of your data ? Just a few lines from one of your fastq ?

Teo

[dosshra]

Thank you @tlemane for the response. See attached file. I should mention that a toy example of 3 fastq files from a different source worked well on the same machine.
Thank you
ERR_example.fastq.txt

[dosshra]

Hello,
I Downloaded a different version of the fastq files with a limit of 1M reads. kmtricks pipeline was running successfully on 20 files. I will try up-scaling.

[dosshra]

Hello
I tried to run with all samples and data. 120 samples, 420 read files (~14GB each). kmtricks pipeline --file ./fof1 --run-dir ~/18tb/kmer_pa_bin --kmer-size 31 --mode kmer:pa:bin -t 10 --hard-min 2 2>kmtricks.err
I got these running messages:

Compute SuperK [=======> ] [09h:33m:15s]
Compute SuperK [================================> ] [01d:11h:10m:14s]
Compute SuperK [==================================================] [02d:04h:29m:01s]
Count partitions [======================Compute SuperK [==================================================] [02d:04h:29m:01s] Count partitions [==================================
================] [02d:04h:29m:01s]
Merge partitions [> ] [00:00s]

This back trace:

Backtrace:
1 0x00007f6987042520 (null) + 140091213489440
2 0x00007f69870969fc pthread_kill + 300
3 0x00007f6987042476 raise + 22
4 0x00007f69870287f3 abort + 211
5 0x00007f6987476f00 __gnu_cxx::__verbose_terminate_handler() + 192
6 0x00007f698747543c (null) + 140091217892412
7 0x00007f698747548e (null) + 140091217892494
8 0x000055bde4189705 (null) + 94274063996677
9 0x00007f6987045495 (null) + 140091213501589
10 0x00007f6987045610 on_exit + 0
11 0x000055bde424b98a (null) + 94274064791946
12 0x00007f6987042520 (null) + 140091213489440
13 0x00007f69870e57f8 clock_nanosleep + 200
14 0x00007f69870ea677 __nanosleep + 23
15 0x000055bde425d66b (null) + 94274064864875
16 0x000055bde43bd7f0 (null) + 94274066307056
17 0x000055bde4170260 main + 3312
18 0x00007f6987029d90 (null) + 140091213389200
19 0x00007f6987029e40 __libc_start_main + 128
20 0x000055bde41725e5 (null) + 94274063902181

And these error log:

[2024-04-02 08:42:46.390] [info] Run with Kmer<32> - uint64_t implementation
[2024-04-02 08:42:46.615] [info] Compute configuration...
[2024-04-02 08:42:46.616] [info] 204 samples found (408 read files).
[2024-04-02 08:43:47.104] [info] Use 169 partitions.
[2024-04-02 08:43:47.108] [info] Compute minimizer repartition...
terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
what(): Unable to open /home/user/18tb/kmer_pa_bin/counts/partition_6/TA2703.kmer
terminate called recursively
terminate called recursively
[2024-04-06 07:04:10.765] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log

The file TA2703.kmer is there and about the same size of the other kmer files.
It seems that some of the first steps wee completed. How can I run kmtricks to try and complete only the missing steps?
Thank you

[dosshra]

Hello
I ran the following code:
kmtricks pipeline --file ./fof1 --run-dir ~/18TB/kmer_pa_bin --kmer-size 31 --mode kmer:pa:bin -t 10 --hard-min 2 --until count 2>kmtiks.err
This part runs successfully. And then I ran the following code:
kmtricks merge --run-dir kmer_pa_bin --mode kmer:pa:text -t 15 -v debug 2>merge_kmtrics.err
The command terminates immediately while generating the files listed below. Any help would be much appreciated
kmtricks_backtrace.log.txt
merge_kmtrics.err.txt
.