ESPNow - Disabling ACK packets increases latency?

[charlielee206]

Hi, I am trying to send audio over ESPNOW, and for more throughput and less latency, I have decided to try disabling acknowledgement packets. My theory was that by skipping the whole handshake phase, we could achieve more throughput and less latency for the cost of less audio quality or artifacts.
However, when I actually implemented the changes, something strange happened. Although I have achieved more throughput, the latency increased dramatically.

I also noticed that the latency increased over time. Over a period of about 20 seconds right after power cycling both the transmitter and the receiver at the same time, I noticed that there was a gradual increase in delay.

For my tests, the default latency (when ACK packets were enabled) was about 25ms. When disabled the ACK packets, the latency jumped to about 60ms on startup, which gradually increased to about 80ms.

I have disabled the ACK packets by doing cfg.use_send_ack = false; for the config of the ESPNowStream in the main sketch. I measured the throughput by communication-espnow-send and receive-measure scripts, and I measured the latency by actually sending audio over espnow, and comparing the time between the source and the outputs.

Also, while trying to solve this issue, I have made a small change in default_recv_cb, within AudioTools/Communication/ESPNowStream.h. I removed the delay while waiting for the buffer. I think this just drops the packets when there is no space in the buffer, but doing this reduced the latency buildup without noticeable audio defects. (I guess most of the dropped packets were defective?)

within `default_recv_cb` in `ESPNowStream.h`:

changed from

while (bufferAvailableForWrite() < data_len) {
      delay(2);
    }

to

    if (ESPNowStreamSelf->cfg.use_send_ack) {
      while (bufferAvailableForWrite() < data_len) {
        delay(2);
      }
    } else {
      if (bufferAvailableForWrite() < data_len) {
        return;
      }
    }

Although this did remove most of the latency buildup, It did not remove the original increase in latency. Now, at least in my setup, the latency stays at around 60ms.

I think the latency comes from a blocking function somewhere that waits for the now non-existant ACK packets. However, although I might be missing something, I could not find any blocking functions that expect ACK packets within the audio-tools library. I also could not look through the main ESPNOW code within the core library, since they exist in a pre-compiled archive file. (All i could find that was directly read and editable were just header files, which all point to functions within the precompiled archive file. libespnow.a to be exact)

How would I remove the extra latency that happens when use_send_ack is set to true? I have tried everything I could think of, and I have no idea where I should look next. Some tips would be appreciated.

[pschatzmann]

If the latency is too big, you should try to minimize the buffers in the whole processing chain: e.g. the buffer_size and buffer_count in I2S and ESPNow are a bit big to be on the save side and you can try to optimize this considerably.

The use_send_ack in ESPNow is critical to prevent a buffer overflow assuming the sending speed is above the playback speed. If you remove it you need to do 2 things:

• Make sure that you limit the sending speed to the sample rate
• Add some additional logic on the output side to prevent buffer over and underruns because of the slight clock differences between the sender and receiver: e.g. Monitor the buffer and do some resampling to speed up or slow down the playback slightly based on the filling grade of this buffer.

Alternatively you could try to save the received data in PSRAM if the sending file is small enought. Maybe it is also an option to save it in a File, but I don't know about the limitations of writing and reading from a file at the same time.

ps. I think the delay(1) is important to process some other tasks.
You can try to inclrease the thruput by selecting a higher wifi_phy_rate_t

[charlielee206]

I have minimized the buffers. There is probably some more that I could shave off, but what I'm questioning is why disabling ACK would increase latency.
Why would buffer overflow create latency, even when I made it so that the code would drop extra packets? I would understand if I heard static, cuts, or general audio defects, but I don't understand why latency would increase.
Since the sample frequency and channels stayed the same across the experiments, and I'm dropping additional packets, I don't think it would be a problem where the CPU gets overwhelmed with too much data.
I'm still trying to wrap my head around this problem, since I still think that I should have less latency with lower quality.
Is there something obvious I'm missing?

[pschatzmann]

My guess is, that thisis rather caused by your removal of the delay as mentioned above.

[charlielee206]

If I keep the delay in the code, the Initial latency stays the same as when I removed the delay in the code, but the latency ramps up over time. My thought was that the initial spike in latency is caused by something else, and the additional latency that ramps up is due to the buffer waiting longer and longer as more packets stack in the buffer.
Removing the delay in the code solved the increasing latency problem, but not the initial spike in latency.
My current thought is that the initial spike is caused by something else, but the additional latency rise was caused by buffers. Removing the delay will remove latency problems related to the buffer, at the cost of audio quality.
And my test results seem to back this hypothesis.

Results

Setting	Latency-on boot	Latency-after 60seconds
Stock	25ms	25ms
No ACK	60ms	90ms
No ACK, Modified Library (delay removal)	60ms	60ms

I'm sure that removing the delay will cause many problems in the long run, and generally compromise the stability of the code. But I'm not sure why it would cause extra latency problems. And if removing the delay will create more latency, why did the latency after 60 seconds drop?

[pschatzmann]

I guess that is the time it takes to fill up the buffer and then instead of receiving any data most of the effort is spent on waiting

[pschatzmann]

I changed the logic to use a semaphore to manage the confirmations and the BufferRTOS class for the buffer.

[charlielee206]

After a month, I have given up. But I have found out some things about this problem. I will document my findings here, and maybe someone smarter than me could pick up from where I left off.

The latency comes from queue buildup. This explains why the latency starts small and rises over time. Using the new RTOS buffered library did take a chunk off of the initial latency spike, but the fact that the latency increases to an unuseable state still remains true.

The queue buildup seems to originate from the transmitter, not the receiver.
I monitored the espnow queue on the receiver side, and the data came in with the queue already built up/delayed. Also, resetting/rebooting the receiver did not get rid of the latency. Once the latency was built up, it did not go away.
On the other hand, resetting/rebooting the transmitter did temporarily reset the latency buildup.
I believe this latency buildup is due to an internal timeout from the espnow library itself.

I tried my best to monitor the queue buildup before the final esp_now_send function, and I could not detect any meaningful buildup. This would mean that the latency buildup originates from somewhere after the espnow_send function is called, and before the receiver processes the sent packet.
This kind of makes sense, since when I set the cfg.use_ack to false, I think I am only telling the audio-tools library that I'm not using ACK packets. I think that the core espnow library still expects ack packets, waits for the nonexistant packets, times out, and creates latency buildup.
However, I am still not certain if this is true. I could not find and measure any internal buffers that back up my hypothesis. But for me, I think that this is the most likely cause of the problem.

I do not know enough about the core library to start diagnosing this problem, nor am i smart enough to confirm/deny anything.
Also, the core espnow library seems like a pre-compiled archive library, and I could not look into it.
(I could tell that the archive file had espnow related function symbols, but that was about it.) I also could not find the actual source code for the precompiled files.

I tried to "spoof" the core library by setting the address to FF:FF:FF:FF:FF:FF, but nothing notable changed. I don't think that i successfully spoofed the library, and it still knows that it's in unicast mode.

I did find a band-aid solution to this problem. I added a function to flush the queue if the queue got too long. Running this function on the receiver side "prevented" the latency buildup and decreased the latency drastically.
However, this causes audible noise artifacts. I tried to minimize the artifacts by regulating how fast the packets are dropped and by adding filters before the speaker, but the artifacts still remain. I'm still not happy about it, but it is what it is. I can now stream 48khz 16bit adpcm encoded audio with minimal latency this way. (Also, the size of the data dropped must match the adpcm block size. If not, the codec will go out of sync and start screaming.)

If someone stumbles upon this problem, I hope that this will help. Also, sorry about the rambling.

[hammerheaddown]

nice! if you get a chance show us how you did it, i really want to have a intercom type system out to the garage this summer so this would be a fun lil project i think.