oidc_cache_mutex_lock: apr_global_mutex_lock() failed in 2.4.7-dev

[jlbooker]

Hi again,

I noticed an Apache crash this past weekend relating to mutex locks. We're running Apache 2.4.37 on RHEL 7, with a custom mod_auth_openidc v2.4.7-dev built from source last week. Apache's Mutex settings are all at their defaults and we're not using any cache related configuration directives for mod_auth_openidc.

[Sat Feb 27 03:40:02.199176 2021] [core:alert] [pid 8554] AH00050: Child 26895 returned a Fatal error... Apache is exiting!
[Sat Feb 27 03:40:02.199178 2021] [mpm_prefork:warn] [pid 8554] AH00166: Ignoring fatal error in child of previous generation (pid 26895).
<previous two lines repeat for each child process>
[Sun Feb 28 03:19:01.353778 2021] [mpm_prefork:emerg] [pid 9823] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sun Feb 28 03:19:01.353905 2021] [mpm_prefork:emerg] [pid 21320] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sun Feb 28 03:19:01.360542 2021] [mpm_prefork:emerg] [pid 21322] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sun Feb 28 03:19:01.363465 2021] [mpm_prefork:emerg] [pid 21319] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sun Feb 28 03:19:01.365471 2021] [core:alert] [pid 8554] AH00050: Child 21320 returned a Fatal error... Apache is exiting!
[Sun Feb 28 03:19:01.365500 2021] [:emerg] [pid 8554] AH02818: MPM run failed, exiting
[Sun Feb 28 03:19:01.366321 2021] [mpm_prefork:emerg] [pid 21323] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sun Feb 28 03:19:01.370027 2021] [mpm_prefork:emerg] [pid 22864] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sun Feb 28 03:19:01.377289 2021] [auth_openidc:error] [pid 8554] oidc_cache_mutex_lock: apr_global_mutex_lock() failed:  (22)
[Sun Feb 28 03:19:01.377328 2021] [auth_openidc:error] [pid 8554] oidc_cache_mutex_unlock: apr_global_mutex_unlock() failed:  (22)
[Sun Feb 28 03:19:01.377832 2021] [auth_openidc:info] [pid 8554] mod_auth_openidc-2.4.7-dev - shutdown

[zandbelt]

to me the logs suggest that there's a system error (i.e. resources exhausted or similar) that results in Apache unable to lock mutexes, not specific to mod_auth_openidc

[wellsbj2]

Hi Jeremy, did you get to the bottom of this please? Got a similar issue... Cheers, Brad

[jlbooker]

If I recall, there was a regression related to the 'graceful' Apache restarts not feeing up the mutexes that were in use. Our log rotation scripts used a nightly graceful restart in order to swap out the log files, so after several days there was a backlog of stuck/used mutexes and Linux would refuse to open another. Restarting the system or clearing all the mutexes and then restarting Apache worked to fix this. I think we eventually upgraded to a newer version of this module (memory is a little fuzzy because it's been a while). The ipcs Linux command can help you list/view the used semaphores/mutexes. There's an associated command to remove them all (ipcrm?). Have a look at #458.

[wellsbj2]

Okies, thank you for the quick reply, I'll take a look at #458.

Thanks again!
Brad