Wifi in tasks -- What's needed in a PR?

[jeffreyabecker]

The documentation states

FreeRTOS is supported only on core 0 and from within setup and loop, not tasks, due to the requirement for a very different LWIP implementation. PRs always appreciated!

Is this about needing locks around the underlying lwip calls but only when the calls are not coming from a lwip callback?

[earlephilhower]

It's a bit more work than that, unfortunately.

LWIP doesn't do multithreading and really doesn't do SMP. We've got both here, so it's very important to be strict about keeping things proper. The way LWIP crashes is you get a failure well after memory is corrupted so debugging is a real PITA.

There's already pretty good locking around the LWIP calls. It's automagically done by GCC and lwip_wrap.cpp. The issues with FreeRTOS come in with interrupts and periodic tasks (same thing).

Two big issues come into play w/FreeRTOS and LWIP (and the WiFi driver):

1. LWIP operates as a callback under an IRQ context. In the main core that's not really an issue (it's used as a poor man's task switcher and doesn't block other IRQ processing like SPI/etc.). See the async_contect_threadsafe_background used by the SDK. You can get a periodic IRQ and LWIP might process a TCP resend or something, or you could get an Ethernet-level IRQ which then sends a packet to LWIP. Still in the IRQ context, LWIP will then process the Ethernet packet into TCP/UDP and then do a callback to user code (i.e. "here's a GET request") which will do some work, potentially do more LWIP work, and then return up a couple levels and return from IRQ. IRQs are per-core and not per-MCU, so that IRQ could come in while doing LWIP code on core1 and bad stuff would happen (sometime in the future...this often shows a memory corruption and not an immediate crash).
2. FreeRTOS doesn't allow mutex acquires w/o special casing stuff from IRQs. Meaning doing memory allocations from inside the LWIP IRQ-level processing will cause a programmatic panic. Their reasoning for this is sound in general, but by construction in our specific case malloc will always be able to get the mutex and finish processing (IRQs and SysTick task swaps disabled). Bare metal, the current setup has been proven solid for multiple years now.

The suggested architecture for using LWIP with FreeRTOS from the LWIP folks is to implement a single task that does all LWIP calls. Set up a work Queue and let it process calls as they become available. This way only 1 task ever calls LWIP and it's effectively the nice and single core, single thread it wants. I did some hacking on this (we can use the wrappers to replace calls with a xQueueSend and wait on a response) but there's also a need to redo the way the Ethernet drivers work.

Now, any Ethernet device can raise an IRQ when an Ethernet packet is ready for processing. These are processed, obviously, in an IRQ context (see wall of text above). But they'd all need to be broken into multiple stages for full FreeRTOS integration. You still have the IRQ callback, but all it does is disable the IRQ (but not clear it!) and xTaskNotify to a real full-fat task to do the actual processing and then clear the IRQ notification and re-enable the IRQ.

Getting both these would remove 1. and 2. above since the only thing in IRQ-time is a small "wake up a task" shim. I did something similar in the ESP32 BackgroundAudio-L271 since we couldn't actually decode in the IRQ without killing the WiFi. Latency is worse, but if you're using FreeRTOS latency is not generally something you care about.

Speaking of WiFi, the CYW43 WiFi driver in the SDK would also need to be changed for this because the bare metal version handles the IRQs/PIO-SPI/etc. itself. The 3 other WiFi and 6 other Ethernet drivers here in the core are fully self-contained and could be modified w/o opening up libpico.

[earlephilhower]

FWIW, I've just pushed the branch I was working on if you want to eyeball what I was thinking of. It's rather out of date but does implement the single LWIP task. It does not do the IRQ 2-phase processing. I also don't like the way queue messaging structs are allocated, but it works for a subset of configurations. I got bogged down on integrating the CYW43 async_context and IRQ processing since they're SDK and not from the core proper.

[jeffreyabecker]

I am digging into this -- its just a lot because of the surface area

[earlephilhower]

Good luck! You might want to start with one of the wired Ethernet interfaces (W5100, etc.) which have every line of code under our own control in the core. They have the same issue as the CYW43 WiFi related to LWIP/etc. but without the SDK internals.

[jeffreyabecker]

Just to verify, you're using platform IO as the IDE?

…

On Sat, Jun 28, 2025, 1:00 PM Earle F. Philhower, III < ***@***.***> wrote: Good luck! You might want to start with one of the wired Ethernet interfaces (W5100, etc.) which have every line of code under our own control in the core. They have the same issue as the CYW43 WiFi related to LWIP/etc. but without the SDK internals. — Reply to this email directly, view it on GitHub <#3013 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAZH3STDIY77OO65EENVL3F3J27AVCNFSM6AAAAACABKNTDSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNRQGYYTEMY> . You are receiving this because you authored the thread.Message ID: <earlephilhower/arduino-pico/repo-discussions/3013/comments/13606123@ github.com>

[earlephilhower]

Is VIM an IDE? 😆 Plus GDB from the command line when things go really pear shaped.

I didn't normally run p.io, just a mix of adrduino-cli and IDEs 1.8 and 2.x. The libs are cmake based since that's what the SDK requires.

That shouldn't really affect any coding, though.

[jeffreyabecker]

Now I need to go digging. I can't stand platform io's build system screwing with mu vscode configuration.

…

On Sat, Jun 28, 2025, 1:11 PM Earle F. Philhower, III < ***@***.***> wrote: Is VIM an IDE? 😆 Plus GDB from the command line when things go really pear shaped. I didn't normally run p.io, just a mix of adrduino-cli and IDEs 1.8 and 2.x. The libs are cmake based since that's what the SDK requires. That shouldn't really affect any coding, though. — Reply to this email directly, view it on GitHub <#3013 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAZH2TV3EEFUJ3LKSMW4D3F3LF5AVCNFSM6AAAAACABKNTDSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNRQGYYTOMY> . You are receiving this because you authored the thread.Message ID: <earlephilhower/arduino-pico/repo-discussions/3013/comments/13606173@ github.com>

[earlephilhower]

Yeah, sorry for the exposition. It's logically simple but the devil is in the details.

Conceptually we need to make LWIP (i.e. OSI layer 4) FreeRTOS safe as well as the Ethernet layer (OSI layer 2). They both need to be ported or it won't work. LWIP is (was) almost there a few months ago when I stopped hacking. Ethernet was not started. Things "work" for a while if you're lucky with this config, but it's not stable because there will eventually be a race condition hit and LWIP will lost its mind.

Is any arduino-pico project always going to use FreeRTOS?

Not sure I grok this in all fullness, but I know there are some users of it already (check the closed bugs 😆 ). This specific core itself, probably not. A fork with only FreeRTOS might be the way fwd, where things like the peripherals don't busy wait, etc.

are WiFiClient and BearSSL::WifiClientSecure the only 'Client' classes which wrap the lwip implementation?

There's SimpleMDNS, UDP, NTP, and couple others I'm probably forgetting, too. FWIW WiFiClientSecure uses the WiFiClient to do the actual LWIP processing...it's operating at a higher level.

Could you elaborate about what's wrong with how the queue message structs are allocated? How would it be done better?

I apologize but I haven't really looked at my code since I wrote it a while back, but I think I did a lazy per-wrapper-function static queue entry allocation. This was because malloc during an IRQ in FreeRTOS causes a panic and the Ethernet drivers all need the pbuf_xxx functions to stuff packets. The proper thing to do is allocate/free (or better yet use a preallocated pool) entries.

Would handlePackets read the entire packet into a non-pbuf and marshal that over to the Lwip Task for processing?
or would marshaling over an invokeHandlePackets event be possible / preferable to reduce the extra allocation of buffers?

I think it's a little different from those. The _irq would disable the IRQ source momentarily (i.e. disable the GPIO IRQ for that pin, but not actually handle it) and just do an TaskNotifyFromISR to wake up the individual Ethernet task. Not actually read out and processing of packets, just notifying the full-fat Task to do something. That task would use normal wrapped LWIP calls to stuff the packet (basically handlePackets + GPIO IRQ mask re-enable). The Ethernet driver task in this case is just like any other LWIP utilizing class like WebServer.

sendFrame seems to happen 'in-line'. This seems like it has to be this way otherwise what process does the sending?

LWIP calls the sendFrame from the netif callback structure and when done properly it's called from w/in the LWIP task so it's all good since LWIP is not running from an IRQ, ever., in the "proper" configuration.

Is the cyw43 driver async_context_freertos under the hood somehow?

This runs into the limitations of running the SDK as a precompiled lib. Right now, there is only always an async_context_threadsafe_background task and the SDK really doesn't like having both versions defined (CMake #errors IIRC). That also requres the FreeRTOS config to be specified in the SDK at library build time. It's probably doable to make that available, but not something I got anywhere with when I tried.

That's why I think implementing this with one of the self-contained wired Ethernet drivers would be a best first step. The ENC 10(!)Mbit interface, for example, is a single file of ~700 lines (30% comments) and doesn't rely on any SDK bits.

[jeffreyabecker]

My response is a bit out of order so quoting a lot...

Is any arduino-pico project always going to use FreeRTOS?

Not sure I grok this in all fullness, but I know there are some users of it already (check the closed bugs 😆 ). This specific core itself, probably not. A fork with only FreeRTOS might be the way fwd, where things like the peripherals don't busy wait, etc.

I ask because that's my understanding of how Arduino on ESP32 works. The internals of [AsyncTcp] (https://github.com/ESP32Async/AsyncTCP/blob/main/src/AsyncTCP.cpp) seem to rely exclusively on FreeRTOS structures to do the queue & marshalling. Maybe that's just the case for esp32 stuff in general but it seems like a solid plan to simplify networking code.

[earlephilhower]

Sure, you can and should use FreeRTOS data structures to manage things. But while the entire ESP32 WiFi non OSS core and LWIP are already built around and require FreeRTOS, here things work without an OS and using interrupts to make user apps simpler.

So what they do, or MeNoDev's AsyncTCP does, isn't really relevant. Once LWIP and Ethernet are return to fully utilize FreeRTOS then something like AsyncTCP should almost "just work".

[jeffreyabecker]

I guess my bigger concern if we're saying that this has to support the no-os case is that it'll actually need two implementations. one for freertos, and another which handles the no-os case somehow. For now I'll just focus on freertos and make an effort to abstract away what I can -- once i'm done with my little web server rabbit hole 🤣

[earlephilhower]

I've deleted the lwip1 branch.

Over the past week I've been beating #3063 into something that works with FreeRTOS for LWIP. It's at the stage where the local drivers (i.e. wired Ethernet and the non CYW43 WiFi) now seem to be running reliably with TCP work on both cores. Still lots to do and bugs, but it incorporates everything I did in that branch and lots more.