MacOS: Electrum always crashes on shutdown if started as detached daemon (electrum daemon -d)

[pricklypierre]

Steps from Command line on macOS (seen on macOS 15.6.1 but all newer macOS releases)

electrum daemon -d
# optionally use electrum via CLI or GUI (will crash regardless though)
electrum stop

Crash Reason and Call Stack

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000110
Exception Codes:       0x0000000000000001, 0x0000000000000110

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [50659]

VM Region Info: 0x110 is not in any region.  Bytes before following region: 4337221360
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      10284c000-102858000    [   48K] r-x/r-x SM=COW  /Volumes/VOLUME/*/SomeApp.app/Contents/MacOS/run_electrum

Application Specific Information:
*** single-threaded process forked ***
crashed on child side of fork pre-exec

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libdispatch.dylib                   0x7ff8079c3237 _dispatch_root_queue_push_override + 229
1   libdispatch.dylib                   0x7ff8079b7d2f _dispatch_dispose + 150
2   libdispatch.dylib                   0x7ff8079b7c09 dispatch_release + 74
3   IOKit                               0x7ff80b4ea70f __IOHIDManagerExtRelease + 233
4   IOKit                               0x7ff80b4affc9 _IOHIDObjectRetainCount + 193
5   CoreFoundation                      0x7ff807d168a4 _CFRelease + 506
6   hid.cpython-310-darwin.so              0x10effb4a3 hid_exit + 35
7   hid.cpython-310-darwin.so              0x10effb2bd 0x10eff0000 + 45757
8   Python                                 0x10c02f88d _PyObject_Call + 109
9   Python                                 0x10c166fb2 _PyEval_EvalFrameDefault + 26002
10  Python                                 0x10c15f04f _PyEval_Vector + 383
11  Python                                 0x10c02edb0 _PyObject_FastCallDictTstate + 96
12  Python                                 0x10c0c2a65 slot_tp_call + 197
13  Python                                 0x10c02eb14 _PyObject_MakeTpCall + 132
14  Python                                 0x10c1705a3 call_function + 371
15  Python                                 0x10c1668a1 _PyEval_EvalFrameDefault + 24193
16  Python                                 0x10c15f04f _PyEval_Vector + 383
17  Python                                 0x10c032d21 method_vectorcall + 481
18  Python                                 0x10c25aca1 atexit_callfuncs + 81
19  Python                                 0x10c1cdb64 Py_FinalizeEx + 68
20  Python                                 0x10c1cfe2d Py_Exit + 13
21  Python                                 0x10c1d5733 handle_system_exit + 35
22  Python                                 0x10c1d386a _PyErr_PrintEx + 42
23  run_electrum                           0x102850d2d 0x10284c000 + 19757
24  run_electrum                           0x102851157 0x10284c000 + 20823
25  dyld                                   0x202f70530 start + 3056

Thread 1:: com.apple.rosetta.exceptionserver
0   runtime                             0x7ff7ffdc40e4 0x7ff7ffdc0000 + 16612

Wallet File Corruption

I've been using electrum in daemon mode frequently with some scripts and have just been ignoring the crash since it occurs on quit. However, I ended up with a wallet .json file becoming corrupt, so I believe the issue is more serious than just an annoyance.

Analysis and Possible Solutions

The crash seems to be caused by HID/IOKit trying to clean-up after the fork() call made to detach the daemon. Apparently, they are not "fork safe." This discussion on issues with fork() on macOS might be helpful.

There are a few possible solutions, but I'm not familiar enough with the Electrum codebase to suggest which one to try:

• I might be wrong, but I'd have expected the fork to not lead to this crash if all of the IOHID calls happened after the fork(), so maybe there are some IOHID initialization happening from some python imports before the fork(). Delaying those until after the fork might avoid the issue.
• When Electrum is in detached daemon mode, processing of the "stop" command (actually might need to do it on all exit/quit calls, not just stop command) have Electrum use os._exit(code) instead of sys.ext(). I think the former would bypass the IOHID cleanup (it will skip Python atexit handlers) and avoid the crash.
• Change to a fork + execve model (not just fork) that starts a clean new interpreter for the daemon to run under.

I'm also curious about addressing the wallet corruption. I'm not sure how/when the wallet files are closed, but ideally that would be called to occur explicitly early in the process before Python starts is exit cleanup (where the crash occurs).

[pricklypierre]

Just a follow-up I created test scripts to see if delaying import hid until after the fork() would prevent the crash and it did. This crashes (exact same crash I'm seeing in Electrum after using electrum daemon -d):

import os
import sys
import hid

pid=os.fork()
if pid:
  print("starting daemon (PID %d)" % pid)
  sys.exit(0)
else:
  sys.stdout.flush()
  sys.stderr.flush()
  si = open(os.devnull, 'r')
  so = open(os.devnull, 'w')
  se = open(os.devnull, 'w')
  os.dup2(si.fileno(), sys.stdin.fileno())
  os.dup2(so.fileno(), sys.stdout.fileno())
  os.dup2(se.fileno(), sys.stderr.fileno())
  sys.exit(0)   # will crash on exit call

And delaying the import hid until after the fork does NOT crash:

import os
import sys

pid=os.fork()
if pid:
  print("starting daemon (PID %d)" % pid)
  sys.exit(0)
else:
  sys.stdout.flush()
  sys.stderr.flush()
  si = open(os.devnull, 'r')
  so = open(os.devnull, 'w')
  se = open(os.devnull, 'w')
  os.dup2(si.fileno(), sys.stdin.fileno())
  os.dup2(so.fileno(), sys.stdout.fileno())
  os.dup2(se.fileno(), sys.stderr.fileno())
  import hid   # delayed import -- no more crash on exit()
  sys.exit(0)

Lastly, I also tested using os._exit(0) instead of sys.exit(0) and that also avoided the crash. I like the delayed import hid approach better, if that is feasible. Changing to a fork + execve model is probably the most robust solution, but also probably the most work.

[SomberNight]

However, I ended up with a wallet .json file becoming corrupt

Could you open a separate issue about the wallet file corruption?

What version of Electrum? Was file encryption enabled? What are the symptoms of this corruption? How does the wallet file look like (don't share substantial contents) - does it e.g. look truncated / etc?

[pricklypierre]

Could you open a separate issue about the wallet file corruption?

Sure, done.