Assistance in training a 3D denoise model

[Erik-Collet]

Hello, I'm reaching out to request assistance with training a 3D model. I'm working with high-resolution Cryo-EM tomograms, the pretrained denoising models work well for most of my tomograms but there are a couple which I can't get working. So far here are the steps I've taken.

I have two folders, Denoise_Model_Training/even and Denoise_Model_Training/odd which have 39 tomograms in them. The tomograms in each folder are named the same (xxx.mrc), and were generated by picking the even or odd frames from the same initial tomogram. In cases where there were an odd number of frames to begin with, I trimmed the volume so each even and odd tomogram has the same number of slices.

I used the gui to generate this command:

topaz denoise3d -a /Denoise_Model_Training/even -b /Denoise_Model_Training/odd --optim adagrad --lr 0.001 --criteria L2 --crop 48 --batch-size 10 --weight_decay 0 --momentum 0.8 --N-train 1000 --N-test 200 --num-epochs 100 --save-interval 5 --device 0 --num-workers 1 --save-prefix /Denoise_Model_Training/outputs

After running, I receive this output:

using device=0 with cuda=True

loading pretrained model: unet-3d-10a-v0.2.4.sav

Traceback (most recent call last):
File "/programs/x86_64-linux/topaz/0.3.7/bin/topaz", line 8, in
sys.exit(main())
File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/main.py", line 148, in main
args.func(args)
File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/commands/denoise3d.py", line 78, in main
train_data, val_data = make_tomogram_datasets(args.even_train_path, args.odd_train_path,
File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/denoising/datasets.py", line 535, in make_tomogram_datasets
data = TrainingDataset3D(even_path, odd_path, tilesize, N_train, N_test)
File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/denoising/datasets.py", line 209, in init
train_idxs, test_idxs = self.sample_coordinates(mask, N_train, N_test, vol_dims=(tilesize, tilesize, tilesize))
File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/denoising/datasets.py", line 300, in sample_coordinates
sample_inds = np.random.choice(len(foreground_pos[0]), 2, replace=False)
File "numpy/random/mtrand.pyx", line 968, in numpy.random.mtrand.RandomState.choice
ValueError: a must be greater than 0 unless no samples are taken

I would appreciate any help you are able to give me, thank you in advance.

[tbepler]

Thanks @Erik-Collet for sharing. We'll look into this.

Are you able to share a minimal dataset that reproduces this error? E.g., do you have one or two tomograms from your dataset that still give this error if you try to fit o them alone?

[Erik-Collet]

Thank you for replying! Yes, I can supply a couple example tomograms which reproduce the error. I've included a dropbox link to a couple of the files which, when I try to run the denoise command above, reproduce the error. Please let me know if I can provide more information, thank you so much for helping me with this!

https://www.dropbox.com/scl/fo/wp51nhkigrenhn4dzifz4/AKKeCWClNkc-7mAtqG3qp2U?rlkey=8hmtkpfx9blai4xyplqcbd519&st=vec13ac3&dl=0

[DarnellGranberry]

Hi @Erik-Collet, it looks like there are two issues here. First, the default --patch_size (-s) is 96, which is larger than your tomogram's thickness, which stops Topaz from being able to sample the coordinates properly. If you reduce this to anything < 80, that error goes away.

There is a second error after that about the use_cuda argument that I believe is related to how arguments are ingested, so I will have that fixed and updated today (provided no new issues arise).

[DarnellGranberry]

@Erik-Collet, I've pushed fixes for those issues and related changes. Can you update your topaz installation to 0.3.8 through conda or from source (I am having a PyPI issue) and try again?

[Erik-Collet]

Thank you for your help! I am working on a cluster using the sbgrid version, when I get a chance I will download the newer version to my own machine and can try the updates. Thank you again!