🐞 OOM of patchcore and padim on Intel iGPU

[zyz9740]

Describe the bug

When training padim and patchcore on Intel iGPU, out-of-memory occurs, killing the training process.

Dataset

MVTecAD

Model

PatchCore

Steps to reproduce the behavior

pip install anomalib[full]==2.1
pip install torch torchvision --index-url https://download.pytorch.org/whl/xpu --force-reinstall

import os
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"

import argparse
from pathlib import Path
from anomalib.data import MVTecAD
from anomalib.models import Patchcore
from anomalib.engine import Engine
from anomalib.engine import Engine, SingleXPUStrategy, XPUAccelerator
from anomalib.data.utils import ValSplitMode


def arg_parser():
    parser = argparse.ArgumentParser(description='Train PatchCore model on MVTec bottle dataset', formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument('-o', '--output', type=Path, default='ovmodels', required=False, help='Path to save the output OpenVINO IR.\n(default: %(default)s)')
    return parser.parse_args()

if __name__ == "__main__":
    args = arg_parser()
    
    print("Starting training PatchCore model on MVTec bottle dataset...")
    
    # Initialize the datamodule, model and engine for bottle category
    dataset_path = Path("../datasets/MVTecAD")

    datamodule = MVTecAD(
        root=dataset_path, 
        category="bottle", 
        train_batch_size=1, 
        eval_batch_size=1, 
        val_split_mode=ValSplitMode.FROM_TEST, 
        val_split_ratio=0.0001,
    )

    model = Patchcore()

    engine = Engine(
        strategy=SingleXPUStrategy(),
        accelerator=XPUAccelerator(),
    )
    
    # Train the model
    engine.fit(datamodule=datamodule, model=model)
    
    print("Training completed!")
    

OS information

OS information:

• OS: ubuntu 22.04
• Python version: 3.12
• Anomalib version: 2.1.0 （v2.0.0 work well, no oom on both models）
• PyTorch version：2.8
• CUDA/cuDNN version: NA
• GPU models and configuration: 1270P iGPU
• Any other relevant information: 32G memory

Expected behavior

Work well just as same as the behavior on v2.0.0.
I believe the 2 models should work well on MVTecAD dataset when training on Intel iGPU

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

main

Configuration YAML

NA

Logs

(anomalib_2_1) intel@intel-O-E-M:~/zyz/bmk_openvino/unsupervised/patchcore$ python train.py 
/home/intel/zyz/bmk_openvino/unsupervised/anomalib_2_1/lib/python3.12/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
Starting training PatchCore model on MVTec bottle dataset...
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
Zero subset length encountered during splitting. This means one of your subsets
            might be empty or devoid of either normal or anomalous images.
Zero subset length encountered during splitting. This means one of your subsets
            might be empty or devoid of either normal or anomalous images.
/home/intel/zyz/bmk_openvino/unsupervised/anomalib_2_1/lib/python3.12/site-packages/lightning/pytorch/core/optimizer.py:183: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer

  | Name           | Type           | Params | Mode 
----------------------------------------------------------
0 | pre_processor  | PreProcessor   | 0      | train
1 | post_processor | PostProcessor  | 0      | train
2 | evaluator      | Evaluator      | 0      | train
3 | model          | PatchcoreModel | 24.9 M | train
----------------------------------------------------------
24.9 M    Trainable params
0         Non-trainable params
24.9 M    Total params
99.450    Total estimated model params size (MB)
19        Modules in train mode
174       Modules in eval mode
/home/intel/zyz/bmk_openvino/unsupervised/anomalib_2_1/lib/python3.12/site-packages/lightning/pytorch/utilities/data.py:106: Total length of `DataLoader` across ranks is zero. Please make sure this was your intention.
Epoch 0:  44%|████████████████████████████████████████████████████████████████████████████████████████████████████                                                                                                                             | 93/209 [00:10<00:12,  9.28it/s]Killed
Killed

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[rajeshgangireddy]

Hi @zyz9740
We are currently testing anomalib models on iGPUs. While most models work out of the box for the newer iGPUs (from Lunar lake and Arrow lake), some models fail on iGPUs like Intel® Iris® Xe Graphics (I believe this is the one you are using).
However, 64 GB of shared memory is still good enough for these models to run.
Moreover, for patchcore, OOM usually occurs in the next step (not the one where you are facing the error).
Are all the drivers installed properly ?

Please try the following :

1. Could you check if torch is actually using xpu ?

import torch 
torch.xpu.is_available() 
torch.xpu.get_device_properties()

2. Patchcore is memory bank based model. More images mean more memory required. Could you try with a smaller category of mvtec like "toothbrush" (this has only ~60 images).

3. Can you send the output log with padim as well, if possible.

[zyz9740]

Thanks for your quick response !

For your first question. In 1270P, the configuration is as following.

>>> import torch

>>> torch.xpu.is_available()

True
>>> torch.xpu.get_device_properties()
_XpuDeviceProperties(name='Intel(R) Iris(R) Xe Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero', type='gpu', driver_version='1.6.31294+20', total_memory=29467MB, max_compute_units=96, gpu_eu_count=96, gpu_subslice_count=6, max_work_group_size=512, max_num_sub_groups=64, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=0, has_atomic64=1)

In addition, I have found another ARL-H device for testing, with the configuration listed below. I can run code on this device for the following discussion if needed.

OS information:

OS: windows 11
Python version: 3.13.5
Anomalib version: 2.1.0
PyTorch version：2.8
CUDA/cuDNN version: NA
GPU models and configuration: ARL-H 265H
Any other relevant information: 32G memory * 2 = 64G

>>> import torch
... torch.xpu.is_available()
... torch.xpu.get_device_properties()
...
_XpuDeviceProperties(name='Intel(R) Arc(TM) 140T GPU (32GB)', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero', type='gpu', driver_version='1.6.34177', total_memory=34376MB, max_compute_units=128, gpu_eu_count=128, gpu_subslice_count=16, max_work_group_size=1024, max_num_sub_groups=128, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=1, has_atomic64=1)

I make sure both the 2 devices use xpu because you can find that the GPU usage is nearly 100%

[zyz9740]

For second question. I don't think the memory consumption for xpu is reasonable, because whether I run the same code on anomalib v2.0.0 or on CPU. The memory consumption is only ~2G

If I run it on xpu, it use nearly 30G and then is killed.

If it helps, on 265H iGPU, it use nearly 40G memory whether running on CPU or iGPU. I'm not sure whether it is reasonable.

If I use a small datasets, it works well. But I don't understand why the memory consumption varies a lot on different devices and different version and what is corrent.

[zyz9740]

The log of padim is listed below. ( on Iris iGPU）

(anomalib_2_1) intel@intel-O-E-M:~/zyz/bmk_openvino/unsupervised/padim$ python train.py
/home/intel/zyz/bmk_openvino/unsupervised/anomalib_2_1/lib/python3.12/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
Starting training PaDiM model on MVTec bottle dataset...
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
Zero subset length encountered during splitting. This means one of your subsets
            might be empty or devoid of either normal or anomalous images.
Zero subset length encountered during splitting. This means one of your subsets
            might be empty or devoid of either normal or anomalous images.
/home/intel/zyz/bmk_openvino/unsupervised/anomalib_2_1/lib/python3.12/site-packages/lightning/pytorch/core/optimizer.py:183: `LightningModule.configure_optimizers` returned `None`, this fit will run with no optimizer

  | Name           | Type          | Params | Mode
---------------------------------------------------------
0 | pre_processor  | PreProcessor  | 0      | train
1 | post_processor | PostProcessor | 0      | train
2 | evaluator      | Evaluator     | 0      | train
3 | model          | PadimModel    | 2.8 M  | train
---------------------------------------------------------
2.8 M     Trainable params
0         Non-trainable params
2.8 M     Total params
11.131    Total estimated model params size (MB)
19        Modules in train mode
69        Modules in eval mode
/home/intel/zyz/bmk_openvino/unsupervised/anomalib_2_1/lib/python3.12/site-packages/lightning/pytorch/utilities/data.py:106: Total length of `DataLoader` across ranks is zero. Please make sure this was your intention.
Epoch 0:  92%|███████████████████████████████████████████████████████████████████████▋      | 192/209 [00:08<00:00, 21.98it/s]
Killed

Similar to patchcore, it works well on CPU

On 265H， CPU

On 265H，iGPU

[zyz9740]

For this question. I find that torch xpu module doesn't formally support Iris iGPU so I can use Arc iGPU for reproduction. But I think it is a common problem on both patchore and padim on iGPU（whether the iGPU is Iris or Arc）

Besides, I only reproduced it on the latest version v2.1.0 instead of v2.0.0 so that I choose to use the older version now. I'm not sure whether this problem is related to #2813 merged in v2.1.0

[rajeshgangireddy]

@zyz9740
Thank you so much for the detailed information. This is helpful.

I find that torch xpu module doesn't formally support Iris iGPU

This is true, even if the training starts, some of the operations might not be supported on Iris iGPU
For example, pytorch/pytorch#149953 (comment)
From what i gather, the intel-pytorch team is actively fixing this.

Besides, I only reproduced it on the latest version v2.1.0 instead of v2.0.0 so that I choose to use the older version now. I'm not sure whether this problem is related to #2813 merged in v2.1.0

This is interesting. When we made those changes, we haven't tested them on iGPU. But now, as we want to support anomalib on xpu as well, I will look into it and see if we can fix it.