-
-
Notifications
You must be signed in to change notification settings - Fork 296
Open
Description
When starting a Django RQ worker, a read_response() in redis/connection.py almost immediately fails causing an exception, before it continues and the RQ worker seems to work fine:
{"@timestamp":"2025-05-16T11:06:49.289Z","log":{"level":"INFO"},"message":"Worker rq:worker:<hash> started with PID 7, version 2.3.2"}
{"@timestamp":"2025-05-16T11:06:49.289Z","log":{"level":"INFO"},"message":"Subscribing to channel rq:pubsub:<hash>"}
{"@timestamp":"2025-05-16T11:06:49.294Z","log":{"level":"WARNING"},"message":"Pubsub thread exiting on No permissions to access a channel"}
{"@timestamp":"2025-05-16T11:06:49.559Z","log":{"level":"INFO"},"message":"*** Listening on default, email, highpriority..."}
{"@timestamp":"2025-05-16T11:06:49.564Z","log":{"level":"INFO"},"message":"Cleaning registries for queue: default"}
Exception in thread Thread-1:
Traceback (most recent call last):
(...)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1268, in run
self.exception_handler(e, pubsub, self)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1264, in run
pubsub.get_message(ignore_subscribe_messages=True, timeout=sleep_time)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1126, in get_message
response = self.parse_response(block=(timeout is None), timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 937, in parse_response
response = self._execute(conn, try_read)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 913, in _execute
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 74, in call_with_retry
return do()
^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 914, in <lambda>
lambda: command(*args, **kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 935, in try_read
return conn.read_response(disconnect_on_error=False, push_request=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/sentinel.py", line 68, in read_response
return super().read_response(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 666, in read_response
raise response
redis.exceptions.NoPermissionError: No permissions to access a channel
{"@timestamp":"2025-05-16T11:06:49.883Z","log":{"level":"INFO"},"message":"Cleaning registries for queue: email"}
{"@timestamp":"2025-05-16T11:06:50.083Z","log":{"level":"INFO"},"message":"Cleaning registries for queue: highpriority"}
This is using django-rq 3.0.1, rq 2.3.2, redis-py 6.0.0 with Django 4.2 LTS against a Redis 7.x cluster with Redis Sentinel, but we have seen this exception at least since early February 2025 and it is reproducible with versions of django-rq/rq/redis-py going back at least a year.
To rule out Redis Sentinel I have also tested without it, and get a similar exception:
{"@timestamp":"2025-05-16T14:04:25.511Z","log":{"level":"INFO"},"message":"Worker rq:worker:<hash> started with PID 7, version 2.3.2"}
{"@timestamp":"2025-05-16T14:04:25.511Z","log":{"level":"INFO"},"message":"Subscribing to channel rq:pubsub:<hash>"}
{"@timestamp":"2025-05-16T14:04:25.514Z","log":{"level":"WARNING"},"message":"Pubsub thread exiting on No permissions to access a channel"}
{"@timestamp":"2025-05-16T14:04:25.710Z","log":{"level":"INFO"},"message":"*** Listening on default, email, highpriority..."}
Exception in thread Thread-1:
Traceback (most recent call last):
(...)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1268, in run
self.exception_handler(e, pubsub, self)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1264, in run
pubsub.get_message(ignore_subscribe_messages=True, timeout=sleep_time)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1126, in get_message
response = self.parse_response(block=(timeout is None), timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 937, in parse_response
response = self._execute(conn, try_read)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 913, in _execute
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 74, in call_with_retry
return do()
^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 914, in <lambda>
lambda: command(*args, **kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 935, in try_read
return conn.read_response(disconnect_on_error=False, push_request=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 666, in read_response
raise response
redis.exceptions.NoPermissionError: No permissions to access a channel
Looks like the warning itself was introduced in rq 2.0 with rq/rq#2132, but testing now with versions predating that change just gives another exception:
{"@timestamp":"2025-05-16T13:12:57.123Z","log":{"level":"INFO"},"message":"Worker rq:worker:<hash> started with PID 7, version 1.15.1"}
{"@timestamp":"2025-05-16T13:12:57.123Z","log":{"level":"INFO"},"message":"Subscribing to channel rq:pubsub:<hash>"}
{"@timestamp":"2025-05-16T13:12:57.620Z","log":{"level":"INFO"},"message":"*** Listening on default, email, highpriority..."}
{"@timestamp":"2025-05-16T13:12:57.652Z","log":{"level":"INFO"},"message":"Cleaning registries for queue: default"}
Exception in thread Thread-1:
Traceback (most recent call last):
(...)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1799, in run
pubsub.get_message(ignore_subscribe_messages=True, timeout=sleep_time)
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1690, in get_message
response = self.parse_response(block=(timeout is None), timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1542, in parse_response
response = self._execute(conn, try_read)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1518, in _execute
return conn.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1519, in <lambda>
lambda: command(*args, **kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1540, in try_read
return conn.read_response(disconnect_on_error=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: SentinelManagedConnection.read_response() got an unexpected keyword argument 'disconnect_on_error'
{"@timestamp":"2025-05-16T13:12:58.030Z","log":{"level":"INFO"},"message":"Cleaning registries for queue: email"}
{"@timestamp":"2025-05-16T13:12:58.232Z","log":{"level":"INFO"},"message":"Cleaning registries for queue: highpriority"}
Metadata
Metadata
Assignees
Labels
No labels