Does simultaneous use of NPU, GPU, and CPU increase memory bandwidth usage on SoCs?

[hotstone1993]

In the paper “HeteroLLM,” the authors claim that using the NPU together with other components (such as the CPU and GPU) allows the system to utilize memory bandwidth closer to its theoretical maximum, thereby improving performance.

However, in the following link, it is stated that the CPU alone can already saturate the available memory bandwidth, meaning the NPU’s peak performance is inherently determined by the memory bandwidth limit. This seems to contradict the HeteroLLM paper. Furthermore, if multiple components each process parts of the model and share the bandwidth, I would expect interference, potentially resulting in lower effective bandwidth for each component.

So my question is: Does using multiple components (NPU, CPU, GPU, etc.) together actually allow higher effective bandwidth utilization, or can contention between them reduce bandwidth?

Both the paper and the link discuss QNN, so it doesn’t seem to be a matter of platform differences.

[tim-shea]

In general, the gap between the peak bandwidth of the interface and effective bandwidth is explained by the accumulation of small periods of memory underutilization over the course of executing a large model. See this excellent blog for a discussion of some of these effects.

At the hardware level, each of the chips is normally capable of saturating the memory interface (with the exception of some poorly designed, older NPUs) but scheduling everything in a way that the memory is constantly busy, without introducing some other undesireable complexity is not easy. I haven't read the paper you shared yet (will take a look for sure) but I'm guessing that using two chips is not providing any fundamental bandwidth advantage, and more likely just achieves better bandwidth in practice due to quirks of how the operations and reads get scheduled.

[hotstone1993]

Thank you so much for your response!

In general, the gap between the peak bandwidth of the interface and effective bandwidth is explained by the accumulation of small periods of memory underutilization over the course of executing a large model. See this excellent blog for a discussion of some of these effects.

As you mentioned, I can see how it could be difficult to fully utilize bandwidth in typical decoding scenarios.
However, the situation discussed in the current paper is a shared memory environment where multiple chips on the SoC share and use memory together. In this case, could the latency issues mentioned in the blog also occur?

and more likely just achieves better bandwidth in practice due to quirks of how the operations and reads get scheduled.

Finally, could you please give me an example of “quirks of how the operations and reads get scheduled”?

[zhouwg]

I have to say that this is a real good tech question! thanks for your good tech question: this is a real practical tech question in real codes. also many thanks for the answer from CS scientist.

I have been found some weird issues with memory usage on 8Elite/8Gen3, e.g. the same dsp codes can running well on 8Elite but failed on 8Gen3:

I’m also curious to know how they(the authors in that paper) take full advantage of the shared memory between the ARM AP and the NPU on 8Gen3 and avoid memory cache issue/memory barrier issue/performance loss/latency issue at the same time.

[tim-shea]

Finally, could you please give me an example of “quirks of how the operations and reads get scheduled”?

If we have a sequence like Linear -> Elementwise GeLU -> Linear in our model (very common in transformers), the two Linear ops represent the memory bottleneck, but because we (generally) can't fuse those kernels with GeLU, we end up with 2 synchronization barriers (either side of the GeLU) where memory reads are going to be stalled waiting on compute, even though in principle the chip should be able to hide some of the latency of the second linear inside the GeLU.

If we distribute work across two or three different processors with coarse-grained parallelism, they are going to (probably) be issuing read requests and waiting for data irrespective of what the other processors are doing, and that could allow us to sometimes "fill in" these gaps.