Degraded peformance for random stream queues on identical consumer

[christian-ehmig]

Describe the bug

This is a follow-up issue to the mailinglist conversation on:

https://groups.google.com/d/msgid/rabbitmq-users/9ae4c2aa-429f-46a6-9a2e-10b8f052bf31n%40googlegroups.com

We experience a major performance drop (streamed number of messages per second) among several identical Golang RMQ stream clients.

Example stream size: 2,299,304,706 messages
Retention: 2 days (= 172800s)
Message size 28 byte, disabled offset tracking

Client A and B are streaming "near real-time" while client C is behind several hours.
Read speed is often ok after a "restart" of the RMQ clients.

While client C is slow in this scenario, it is fast on another stream served by the same RMQ cluster with identical size.

Trying this patch and setting initial credits to 10k did not help 2d47c02

Initial speed was ok but dropped to a very low rate after some hours.

Reproduction steps

...

Expected behavior

Similar read performance among identical clients.

Additional context

No response

[Gsantomaggio]

Thank you @christian-ehmig
We released a new version (1.5.8) with an critical patch .
Can you re-execute the test with 1.5.8? thank you

[Gsantomaggio]

You can change the initial credits:

stream.NewConsumerOptions().
SetInitialCredits(10).

I would suggest testing a low value, between 2 and 10, so in case of a restart, it does not request too many chunks from the server.

[christian-ehmig]

We tried the new version - however, the "core" issue still persists. After a restart, the read rate is initially high for every consumer (3 of them). About 2 million message reads per second which is fine.

This degrades over time (12 hours) and falls back to 400-500k messages. As explained, some streams are up-to-date and have no delay but others are several hours behind. Sadly, on each of the three consumers, different streams are behind. We have no idea how to reproduce / debug this behavior :/

You see the restart here at around 17:45 yesterday. Today we are back at 500k.

[Gsantomaggio]

We will try to reproduce the issue. Can you please provide a simple client app that is representative of what you use in production that we could test / inspect?

[Gsantomaggio]

@christian-ehmig also: #416. (small improvement) This will prevent creating consumers in the same node when multiple consumers are created on the same application.

[christian-ehmig]

Thanks! So far it seems to end up in "the same" direction - fast after restart and getting slower over time. I will pick up the latest main version and restart again. I am still not sure that this is related to the Golang client/consumer. As far as I remember we had similar issues with Java stream consumers.

We'll try to provide isolated code samples. Our consumers are already modified - they just "consume" the message and discard it - there are no other things "going on". Furthermore we update a timestamp variable and a "read messages" counter. By comparing the timestamp, we know how far behind the given stream queue is.

[Gsantomaggio]

Thank you @christian-ehmig ,
I am trying to reproduce the issue.

The important thing is to know that you publish the messages, for example:

• rate
• message size

For example, with small chunks, the consumer may experience slow performance because the server must send many small chunks.

You can track the average of this value:

func(consumerContext stream.ConsumerContext, message *amqp.Message) {
     consumerContext.GetEntriesCount()) //<---- to see how many messages you have per chunk
	}

[christian-ehmig]

Hi, I provided an isolated sample how we setup our stream consumers in https://github.com/godmodelabs/rmq-golang-benchmark/tree/main.

I currently let this "app" run consuming from 3 streams from a 3-node cluster:

stream A: 2,591,399,653 total messages
stream B: 5,243,580,158 total messages
stream C: 44,030,410 total messages

initially, the read ratio is fine - around 1.8 mio messages / second, entries size around 2000

after around 20 minutes I'm down to 900 mio messages / second and entries size around 600 - 1100
The streams have data from the last two days (message size = 28 bytes). However, on each day messages were only generated between 08:00 - 22:00, so there are 10 hours with no updates "in-between". Can this cause any trouble when consuming the streams?

Let me know if you see any issues in the setup. As noted in the code, we use a new environment for each stream. When using one environment and multiplexing streams over it, we got even worse results. In our case it looks like that consumer threads randomly "get stuck" at low message consume rates and never recover from that situation.

[Gsantomaggio]

Thank you @christian-ehmig,
I will try to reproduce the setup.

we use a new environment for each stream.

The environment doesn't do anything special; it lookup the resources and then creates the connections for consumers and producers.

If you set One Connection per Consumer you have one TCP connection per consumer.

we got even worse results

That's odd.

I will let you know as soon I will have something.

[Gsantomaggio]

@christian-ehmig

We investigated the issue, and the problem could be related to the number of messages per chunk.
There is a known behaviour: small chunks can impact the consumption performance.

For example:

initially, the read ratio is fine - around 1.8 mio messages / second, entries size around 2000
after around 20 minutes I'm down to 900 mio messages / second and entries size around 600 - 1100
Th

As you can see, the performances drop with a lower number of messages.

Tests

It would be helpful if you could do some tests with our perfTest

1. Test one:

Populate a stream with a perTest full rate with only one producer:

java -jar stream-perf-test-1.4.0.jar --uris rabbitmq-stream://guest:guest@localhost:5552/%2f --consumers 0 --producers 1 --streams MyStream

After 1, consume without producers:

java -jar stream-perf-test-1.4.0.jar --uris rabbitmq-stream://guest:guest@localhost:5552/%2f --consumers 1 --producers 0 --streams MyStream 

After 2, run five instance consumers ( like 1)

2. Test two:
   Like the test one but with a limited rate like:

java -jar stream-perf-test-1.4.0.jar --uris rabbitmq-stream://guest:guest@localhost:5552/%2f --consumers 0 --producers 1 --streams MyStream --rate 5000 

3. Test three:

Mix publish full rate and limited rate: execute two performance instances:

java -jar stream-perf-test-1.4.0.jar --uris rabbitmq-stream://guest:guest@localhost:5552/%2f --consumers 0 --producers 1 --streams MyStream --rate 5000 
java -jar stream-perf-test-1.4.0.jar --uris rabbitmq-stream://guest:guest@localhost:5552/%2f --consumers 0 --producers 1 --streams MyStream

The goal is to determine how these different scenarios impact the consumers' rates.

Notes

The RabbitMQ stream achieves optimal performance when the chunks contain a lot of messages. The server sends chunks to the client, so bigger chunks mean better performance.

The Send Clients API aims to strike a balance between latency and throughput. At a low rate, it could create many chunks.

Golang has another API BatchSend , where it is up to the user to aggregate the messages.

In your case, where latency is less important than throughput, you could use BatchSend to aggregate messages and flush them every X milliseconds to avoid creating too many chunks.

[christian-ehmig]

Thanks @Gsantomaggio, will try to setup further benchmarks this week. I understand that this will be very hard to reproduce. Just to mention: The inserts are applied by Java clients - Golang clients are readers/consumers only. We create Java producers with a batch size of 1000.