Blocking loop.subprocess_shell() and loop.subprocess_exec()

[soonum]

• uvloop version: 0.9.0
• Python version: 3.6.1
• Platform: UNIX
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: n.a.

Hi there,

I'm facing the same problem encountered in CPython issue #414. Since the patch (see issue #428) is not upstreamed in asyncio yet, I was wondering if there is any progess on this in uvloop.
I've tested that behavior with the 0.9.0 version and it seems to be still blocking.

@1st1, have you planned to fix this? Do you need help at some point? (I was working with @Martiusweb on the codebase which led to those issues)

[1st1]

@1st1, have you planned to fix this?

Yes. Will take a look in a week or two. If you have time to make a PR I'll review it.

[1st1]

Actually, uvloop can block only if you supply a preexec_fn argument to loop.subprocess_shell() or loop.subprocess_exec(). Could you please share your test?

[jlaine]

@soonum could you please share your test as requested by @1st1 ?

[soonum]

Sorry I wasn't following up this old issue.
In fact my use case is that one, I provide a preexec_fn argument to these methods and as a result, the loop is blocking.

[pvizeli]

We have a component in Home Assistant; they call ffmpeg with a subprocess (async). They also end sometimes (less and more) in blocking and not working event loop :( I can't provide any test code because it's not all time simple to reproduce it in a laboratory situation.

The point is, we don't use preexec_fn and end in a blocking loop :(

[pvizeli]

The problem is forking. It looks like the current implementation doesn't work on a heavy load of the main process.

[Martiusweb]

IIRC, the call to fork() can be slow because of the internals of the memory management (I think it's in O(n) with the number of memory pages of the forked process). Usually the workaround is to use vfork() when knowing that exec() will be called right after, but I don't think that this option is safe in Python. I think that the problem is more about a blocking IPC used to synchronize the parent and the child to transmit failures in preexec_fn. I made a PR that got lost for cpython with vanilla asyncio years ago for this problem: python/asyncio#428 Cheers Le jeu. 25 avr. 2019 à 21:46, Pascal Vizeli <notifications@github.com> a écrit :

…

The problem is forking. It looks like the current implementation doesn't work on a heavy load of the main process. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABPWGGLH3YZJGF6VZRNQE3PSIDAVANCNFSM4EFO3FLQ> .

-- Martin <http://www.martiusweb.net> Richard www.martiusweb.net

[pvizeli]

The point is, if we use subprocess inside executors, it work without issues. Also call the normal subprocess inside loop work with attach a streamreader to output. But the subprocess from uvloop hungs

[nicx]

any news to this? the HA community is waiting for a solution wistfully ;)

[nicx]

Hm very sad that its not going on here :( I would love using the HA stream component...
@1st1 any plan about this?

[1st1]

@1st1 any plan about this?

I still haven't received any clear answer to #118 (comment).

I know that uvloop is blocking when preexec_fn is used, but almost nobody uses it. Other than that, as @Martiusweb explained in his comment #118 (comment) it should not block.

@pvizeli

But the subprocess from uvloop hungs

Does uvloop behave differently from asyncio in your case?