denoise 3d GPU error

[mdjflores]

I am using denoise 3d on my tomograms and have ran into an error when trying to run the denoise 3d job on my GPU. I have successfully ran it using CPU denoising, so I don't think its a problem with my data or installation. Currently- NVIDIA GA102 [GeForce RTX 3080 Ti] and Cuda 11.7

Is this an incompatibility with my graphics card?

I am getting the following error :

File "/speed/jargroup/conda-envs/topaz/bin/topaz", line 33, in
sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main
args.func(args)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise3d.py", line 773, in main
, total_volumes=total
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise3d.py", line 671, in denoise
x = model(x)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 876, in forward
p1 = self.enc1(x)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/speed/jargroup/conda-envs/topaz/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 476, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

[DarnellGranberry]

Could you share your installation details? In particular,

• what version of cuDNN you have installed
• what version of PyTorch
• what OS you are using
• are you in a server environment where other module might be loaded?

I mainly suspect that there are versioning mismatches between PyTorch, CUDA, and cuDNN, but also maybe memory issues. Are there any other tasks running on the GPU or are all ~12GB free?

Finally, could you post the topaz command you used that gave you this error?