-sPROXY_TO_PTHREAD hang in Node.js (test asani.test_pthread_dylink_basics is flaky)

[juj]

emcc test/hello_world.c -o a.js -pthread -sPROXY_TO_PTHREAD=1 -fsanitize=address

And then stress test the resulting a.js using the following Python script:

stress.py

import subprocess
import multiprocessing
import sys

COMMAND = ['node', 'a.js']

def worker(stop_flag):
    while not stop_flag.is_set():
        try:
            result = subprocess.run(COMMAND, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60)
            output = result.stdout.decode(errors="ignore")
            if result.stderr:
                output += result.stderr.decode(errors="ignore")
            if result.returncode != 0 or "hello, world" not in output:
                stop_flag.set()
                print(f"Command failed with exit code {result.returncode}. Output:\n{output}")
        except Exception as e:
            stop_flag.set()
            print(f"Error running command: {e}")
            output = e.stdout.decode(errors="ignore")
            if e.stderr:
              output += e.stderr.decode(errors="ignore")
            print(f"Command failed in exception. Output:\n{output}")

def main():
    stop_flag = multiprocessing.Event()
    procs = []
    for _ in range(multiprocessing.cpu_count()):
        p = multiprocessing.Process(target=worker, args=(stop_flag,), daemon=True)
        p.start()
        procs.append(p)

    for p in procs:
        p.join()

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")  # more portable across platforms
    main()

After a random amount of time, a few minutes, or half an hour, the above will fail with

Error running command: Command '['node', 'a.js']' timed out after 60 seconds
Command failed in exception. Output:
hello, world!

What has happened is that the node program has hung after the end of the program. I.e. the program executed correctly, but is failing to shut down.

What makes attempting to debug this difficult is that the hang does not occur all that often (although it does repeatedly happen on my CI). Adding enough console.log()s in the application JS code results in the test case top stop failing, as some kind of timing is disturbed.

As a random test, if I do this:

diff --git a/src/lib/libpthread.js b/src/lib/libpthread.js
index 44d75892c..70a7939b2 100644
--- a/src/lib/libpthread.js
+++ b/src/lib/libpthread.js
@@ -511,7 +511,9 @@ var LibraryPThread = {
 #if PTHREADS_DEBUG
     dbg(`terminateWorker: ${worker.workerID}`);
 #endif
-    worker.terminate();
+    setTimeout(() => {
+        worker.terminate();
+    }, 5000);
     // terminate() can be asynchronous, so in theory the worker can continue
     // to run for some amount of time after termination.  However from our POV
     // the worker now dead and we don't want to hear from it again, so we stub

then the hang does no longer occur, and the page survives a three hour stress test.

To reproduce the hang, it is necessary to build with asan enabled. Simply building with

emcc test/hello_world.c -o a.js -pthread -sPROXY_TO_PTHREAD=1

does not reproduce a hang in the stress test. It is unclear whether there is something fundamentally related to asan that causes the race condition and the hang; or if it is just a side effect that asan changes timings so that the hang becomes more apparent.

This hang occurs e.g. in test asani.test_pthread_dylink_basics on my CI, every 1-3 days. E.g. http://clbri.com:8010/api/v2/logs/53959/raw_inline . Though there is nothing fundamental to dynamic linking that causes the hang: the above hello world test case does not utilize dynamic linking.

[juj]

What I am struggling to understand in this scenario is why delaying worker.terminate() with a 5 second timeout helps avoid the hang... because when printing out the output in the hanging run, when the hang appears, the code worker.terminate(); is not even being reached to be executed as far as I can see. (or at least simple console.log()/.error() logging in terminateWorker() is not being hit)

I.e. it is as if delaying a function that is not even reached is helping avoid the hang. That does not make sense...

[juj]

The same hang also now occurred in another test wasm2js2.test_abort_on_exceptions_pthreads, which suggests that the issue is not specific to asan.

A common factor with asani and wasm2js2 suites is that they are both slower to run than native wasm suites.

In that test, also adding the setTimeout() delay to worker.terminate(); has an effect of making the hang vanish in a stress test.

So I suppose the common factor here is maybe running pthreads code (or specifically PROXY_TO_PTHREAD code) in node.js.