Why normalization and randomerasing are done in data prefetcher but other data transformations are not?

[yxchng]

Just curious, how do you decide what transformations/augmentations are done in data prefetcher and what are not? Understanding this will help me to decide where to put my extra transformations. Thanks.

[rwightman]

@yxchng for performance, the early transforms that are handed by Pillow (I always use the SIMD version!) are fairly efficient in uint8 mode. The goal is to move the image onto the GPU in uint8 and then convert to float32/float16 (which is done as part of the normalization). Random erasing should be done after norm for proper centering and current impl works well on GPU. This balanced the CPU vs GPU use well.

If you want to add new transforms, consider if it makes sense to do on the CPU w/ Pillow or other ops while it's still in uint8. If you need float and the transforms can utilize the GPU I'd put them in the prefetcher.

I'm currently working on some variations of this in the bits_and_tpu branch because TPU has some diff requirements, but that's still pretty experimental.