Graphtyper vcf_merge failing with error code 139 for VCFs containing SVs

[sroener]

Hi,

I have a set of vcf files containing SVs and want to merge them. When I run the following command, I receive an error code 139 after a while.

graphtyper_v2.7.7 vcf_merge -v --sv --log=output/graphtyper_SV_batches_495-499.vcf_log.txt --output output/graphtyper_SV_batches_495-499.vcf.gz input/*.vcf.gz

There is no further information in the log files. Code is executed in a Ubuntu 24.04 VM/container with the following specifications:

• 8 CPU cores
• 64 GB of RAM
• 300GB SSD storage

Graphtyper version is 2.7.7 of the pre-compiled 64bit binary. Memory was at ~33% 2 minutes before the error, so I assume a "hidden" OOM is unlikely.

Help would be appreciated.

Edit:

I tryed the same with fewer samples (2 VCF files, each around 1.3 GB). Same result, but Memory consumption is similar. Is there any way of avoiding OOMs?

[hannespetur]

Hi Sebastian,

How were the input files produced?

vcf_merge merges VCFs produced with graphtyper and assumes that the input VCFs have the exact same variant sites in the same order but with different samples. It is useful for when you have a large cohort and want split the input samples into batches due to memory limitations. These files are typically covering SVs in 1Mbp regions so they are usually much smaller than 1.3 GB.

My first guess would be that those input files contain different variants and this is causing a segfault. It's very unlikely due to OOM error. If you think this is the problem it would be best if you can extract and share me with a subset of the VCFs to reproduce the error.

Best,
Hannes

[sroener]

Hi Hannes,

thank you for your reply.

The VCFs were created using graphtyper and concatenated using the vcf_concatenate subcommand to reduce the number of files that need to be merged. I tried conatenating the 1Mbp regions by chromosome or whole batch. The results were the same for both approaches, with most jobs failing due to some errors.

Do you think the issue can be fixed by sorting the concatenated VCFs? Otherwise, I will merge by 1Mbp region and see, if the errors persist.

Best,

Sebastian

[hannespetur]

I tried conatenating the 1Mbp regions by chromosome or whole batch. The results were the same for both approaches, with most jobs failing due to some errors.

Alright, yeah try not using vcf_concatenate since it is possible it has some problems - I have not been using it for awhile.

Do you think the issue can be fixed by sorting the concatenated VCFs?

They should be sorted already.

Best,
Hannes

[sroener]

Ok, good to know.

So after merging, you would concatenate the files using bcftools? Is there anything to keep in mind when using the VCF files created by graphtyper?

Best,

Sebastian

[hannespetur]

Yes bcftools concat should work fine and I use that probably more than graphtyper vcf_concatenate. The --file-list parameter in bcftools concat is useful when concatenating large amount of files.

[jjfarrell]

I am also running out of memory using graphtyper vcf_merge. We are merging using a 256GB compute node. It finishes for chromosome 3 to 22 which a merged file. When monitoriing the chr1 and 2 merges, the memory slowly increases and approaches 100% memory over the run and eventually terminates. The number of files merged varies from 900 (chr22) to 5550 (chr1) for merging. It looks like some sort of a memory leak. There are 68,713 samples in total. bcftools could be used to merge the chr1 and chr2 vcfs but we would miss the updated INFO and Filter fields present in the other vcfs.
5550 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_1.vcfs.list
3780 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_2.vcfs.list
3030 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_3.vcfs.list
2970 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_4.vcfs.list
2730 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_5.vcfs.list
2670 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_6.vcfs.list
2460 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_7.vcfs.list
2190 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_8.vcfs.list
2145 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_9.vcfs.list
2475 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_10.vcfs.list
2010 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_11.vcfs.list
2130 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_12.vcfs.list
1515 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_13.vcfs.list
1350 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_14.vcfs.list
1320 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_15.vcfs.list
1335 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_16.vcfs.list
1365 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_17.vcfs.list
1380 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_18.vcfs.list
855 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_19.vcfs.list
1230 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_20.vcfs.list
1020 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_21.vcfs.list
900 /restricted/projectnb/cadre/sv.pipeline/graphtyper/n68713/batch_graphtyper_22.vcfs.list

[hannespetur]

Are you also concatinating and then merging on the chromosomal level? If so, could you try merging first? If not, I am really not sure what is causing high memory usage. I have not seen vcf_merge do that in my runs. I checked my runs for the UKB 500k release, and all 1Mbps regions (500 files containing 1k samples each) were merged using less than 8GB RAM.

[jjfarrell]

@hannespetur The pipeline is concatenating first with bcftools and then merging with graphtyper. I increased the memory enough that I was able to eventually get chr1 and chr2 merged. I will try the merge first and then concatenate approach and test it on the two chromosomes. It will be great to get the memory requirement reduced down to 8GB!

John