How to use event stream trough multiple worker

[enzofrnt]

HI, how can I set up eventstream to work across multiple worker using redis ?
For the moment I made my own consumer that I start as a thread… That maybe not the best way to do it…

Thanks you.

[enzofrnt]

I have tried to send_event from another Django process using the Redis channel layer but sse does not dispay anything.

[jkarneges]

I have not tried this but what you describe sounds fine, and is why I included your sentence about this in the readme.

Are you sure send_event is actually getting called in the other process? If it is, I can't see how it would act any different than calling it from a view in that other process.

[enzofrnt]

I have not tried this but what you describe sounds fine, and is why I included your sentence about this in the readme.

Are you sure send_event is actually getting called in the other process? If it is, I can't see how it would act any different than calling it from a view in that other process.

I don’t think you fully understood what I meant. When I send an event with Django EventStream via send_event from a process other than the main one, the main process does not receive it and does not send it to the view associated with EventStream.

Currently, I have to send messages to Redis, which are then retrieved by a consumer in the main process. This consumer then sends the events using send_event to send the event to the client.

What I mean is: is there a way to use send_event in another process such that the event is sent to the client, regardless of which Django process is managing the SSE view or sending the event?

I see that Grip may be a solution, but I’m not sure. Maybe this functionality needs to be implemented?

[jkarneges]

Ah, yes it would need to be implemented. Currently GRIP is the only way, but a Redis option would be nice. Something that works locally without anything extra (unix socket?) could be nice too for local development.

[enzofrnt]

I don’t know what Unix sockets are, but that might be an option. I was thinking that using Redis is a great idea.

[enzofrnt]

I just learned what Unix sockets are, and they seem like a pretty great option. I don’t know if I have the required competence to develop something like that, but I would try to help.

[enzofrnt]

Where and how we can start work on this ?

[enzofrnt]

Hi @jkarneges,

Here is an example of a working implementation across processes using Redis. It still needs some cleaning up and fixing, and we also need to add a Unix socket version.

If you want to try it, you can start a Django shell using python manage.py shell and send messages from this shell using the send_event method.

This configuration is required at the moment to make it work:

EVENTSTREAM_USE_REDIS = True  # Enable or disable Redis Pub/Sub
EVENTSTREAM_REDIS_HOST = 'localhost'
EVENTSTREAM_REDIS_PORT = 6379
EVENTSTREAM_REDIS_DB = 0

If we implement this and use a Unix socket as the default solution with Redis as an optional choice, will the GRIP component still be necessary? I'm not sure.

[enzofrnt]

I'm currently trying to delete the thread for Redis listening.

[jkarneges]

Nice, that looks like it works the way I'd expect.

GRIP is still a useful alternative that we should keep. It enables scaling connections & messages separately from scaling the server app. It also allows the server app to run as a cloud function/lambda.

[enzofrnt]

Ok.
Do you know how we can delete thread for Redis and only use asynchronous task ? Without blocking the execution of Django process ?

[jkarneges]

It looks like the redis python client supports async, so maybe it is possible, but I've never used it. Otherwise a daemon thread makes sense to me.

[enzofrnt]

Isn't too "heavy" for this ?