Execute jobs via php-fpm rather than forking

[ebernhardson]

After a discussion with a co-worker i decided to write up a quick hack which executes jobs over php5-fpm instead of forking. This provides all the same protections as forking in regards to failures while reducing job execution overhead and providing a more natural php execution environment(vs fork). The Resque_Worker::fork method was adjusted to always return false when running the fastcgi tests.

This is just a POC, not intended for usage outside benchmarking but i think it shows promise.

https://gist.github.com/4253706

Initial benchmarks on an intel i5-2500k (4 cores no HT), ubuntu 12.04, php 5.4.8

WITH FORK:

COUNT=1 INTERVAL=0.01 QUEUE=default APP_INCLUDE=examples/worker_include.php php vendor/.../resque.php
1 Worker, no arguments
Completed 2000 jobs in 9.704s ~4.8ms per job per core
Completed 2000 jobs in 9.960s
Completed 2000 jobs in 10.014s
Completed 2000 jobs in 10.022s
Completed 2000 jobs in 10.000s
Completed 2000 jobs in 9.992s
Completed 2000 jobs in 10.310s
Completed 2000 jobs in 9.991s
Completed 2000 jobs in 10.002s
Completed 2000 jobs in 10.016s

COUNT=4 INTERVAL=0.01 QUEUE=default APP_INCLUDE=examples/worker_include.php php vendor/.../resque.php
4 Workers, no arguments, ~6.4ms per job per core
Completed 2000 jobs in 3.235s
Completed 2000 jobs in 3.250s
Completed 2000 jobs in 3.252s
Completed 2000 jobs in 3.247s
Completed 2000 jobs in 3.278s
Completed 2000 jobs in 3.282s
Completed 2000 jobs in 3.298s
Completed 2000 jobs in 3.292s
Completed 2000 jobs in 3.297s
Completed 2000 jobs in 3.304s

WITH FASTCGI:

COUNT=1 INTERVAL=0.01 QUEUE=default APP_INCLUDE=examples/worker_fcgi_include.php php vendor/.../resque.php
1 worker, no arguments
Completed 2000 jobs in 3.082s ~1.5ms per job per core
Completed 2000 jobs in 3.098s
Completed 2000 jobs in 3.102s
Completed 2000 jobs in 3.084s
Completed 2000 jobs in 3.093s
Completed 2000 jobs in 3.081s
Completed 2000 jobs in 3.111s
Completed 2000 jobs in 3.098s
Completed 2000 jobs in 3.101s
Completed 2000 jobs in 3.079s

COUNT=4 INTERVAL=0.01 QUEUE=default APP_INCLUDE=examples/worker_fcgi_include.php php vendor/.../resque.php
4 workers, no arguments
Completed 2000 jobs in 0.982s ~1.9ms per job per core
Completed 2000 jobs in 0.981s
Completed 2000 jobs in 0.988s
Completed 2000 jobs in 0.988s
Completed 2000 jobs in 0.989s
Completed 2000 jobs in 0.995s
Completed 2000 jobs in 0.987s
Completed 2000 jobs in 0.996s
Completed 2000 jobs in 0.983s
Completed 2000 jobs in 0.986s

To summarize:

FORK one core: 4.8ms/job/core four cores: 6.4ms/job/core
FASTCGI one core: 1.5ms/job/core four cores: 1.9ms/job/core

Naturally this is a synthetic benchmark with the actual job doing nothing. It is intended only to measure the amount of overhead involved in starting resque jobs. Additionally this was run over the loopback interface, as the provided FCGIClient class does not support unix sockets. It is unknown at this point how much overhead there is between tcp and unix sockets for our usage.

Thoughts? Would there be any interest in flushing out to a proper implementation?

EDIT: fixed bad math, 4 * 0.982/2000 makes for 1.9ms/job/core on 4 core fastcgi

[chrisboulton]

Ok, this is seriously cool. 😎

I'd definitely be interested in having this built in, and it might help others such as @salimane who have been trying to work with the overhead of a forking model. It will also improve the error handling/catching for failed jobs as you'll be able to get the actual error message for a job that fatal errors out and would normally leave you with a "DirtyExit" exception.

It'd be super cool to get this working via sockets at some stage as well.

I can sort of see there being different strategies for job execution - Resque_JobStrategy_Fork, Resque_JobStrategy_Fastcgi, and you can configure one at runtime. Thoughts?

[salimane]

hi
@ebernhardson @chrisboulton this is interesting. one quick problem I could see is long running jobs over fastcgi. May be something could be done to adjust for this. But as long as this is space and resource efficient than fork, I'm all for it.
Thanks

[ebernhardson]

Job strategies sound like a good approach to this. I took a stab at refactoring the current code into a strategy pattern, let me know what you think. I'm not sure that perform is the right name for the method on Resque_JobStrategy_Interface, suggestions welcome.

I also have some worry about Resque_Worker::killChild, in that the InProcess strategy has no way to kill the job(when triggered from SIGUSR1), and closing the fpm socket in the Fastcgi strategy wont necessarily stop fpm from finishing the job either. Perhaps a few notes in the documentation is all that is required though.

ebernhardson/php-resque@36f5353

@salimane
Long running tasks are certainly a valid concern. We can always set max_execution_time to 0 from the worker code so it matches the CLI environment, but running some specific testing will be worth the time invested. The time and space efficiency really need to be measured to have any certainty. My intuition tells me it wont be as clear cut as the initial benchmarks show, and will depend on the use case.

[salimane]

@ebernhardson

I quickly went through the changes at ebernhardson/php-resque@36f5353 . If I understand this correctly, this proposal comes down to instead of running the jobs within a child of the worker process, run the jobs directly in the worker process. is this so ?

[ebernhardson]

@salimane The changes in the fork dont represent fcgi yet. The first step here was just to see if @chrisboulton is ok with the direction of the refactor before i start plugging in and writing tests for fastcgi. The linked refactor passes all the original tests and should have exactly the same functionality as the current php-resque code. With the fastcgi in all jobs will be executed by the php-fpm manager, similar to how nginx would run run webpages through php-fpm. Connecting and running scripts through fpm is demonstrated from the gist in the first post.

[salimane]

@ebernhardson ok got it :)

[danhunsaker]

So... How's this coming along? I see PR #81, but it doesn't have any recent updates, either...