High process memory in quorum queue

[annieblomgren]

Hi,

RabbitMQ version 3.8.16
Erlang 23.2.3
Environment="RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=+sbwt none +sbwtdcpu none +sbwtdio none"

Perftest version 2.11.0 with command ​bin/runjava com.rabbitmq.perf.PerfTest -u "testqq2" --size 10 --producers 5 --consumers 0 -f persistent -H $URL --auto-delete=false --queue-args x-queue-type=quorum -pmessages 400000

A quorum queue with 2M messages, no publishers nor consumers is having around 1.6GB in process memory.
After a full GC process memory goes down to around 1.4GB

The same number of messages for a classic queue sits around 300 MiB in process memory before a full GC, and ~60MiB after.

Have changed raft.wal_max_size_bytes and x-max-in-memory-bytes to low values but no difference.

Grafana dashboards before and after GC on QQ:
​https://snapshot.raintank.io/dashboard/snapshot/IphlxnoFeXsjqw6wVH0G0vYS2svJ2GFJ
​https://snapshot.raintank.io/dashboard/snapshot/PhdZV2Y2Z3oMyxbuft8f8nLs72vvcWL1

Question: what is taking up so much process memory for QQs, or is this to be expected?

[kjnilsson]

Quorum queues keep all messages in memory as well as on disk by default. Try configuring a max-in-memory length limit to make quorum queues re-read messages from it's raft log when they are consumed instead. Not this has a consumption performance penalty. https://www.rabbitmq.com/quorum-queues.html#memory-limit Cheers Karl

…

On Mon, 24 May 2021 at 16:26, annieblomgren ***@***.***> wrote: Hi, RabbitMQ version 3.8.16 Erlang 23.2.3 Environment="RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=+sbwt none +sbwtdcpu none +sbwtdio none" Perftest version 2.11.0 with command ​bin/runjava com.rabbitmq.perf.PerfTest -u "testqq2" --size 10 --producers 5 --consumers 0 -f persistent -H $URL --auto-delete=false --queue-args x-queue-type=quorum -pmessages 400000 A quorum queue with 2M messages, no publishers nor consumers is having around 1.6GB in process memory. After a full GC process memory goes down to around 1.4GB [image: Screen Shot 2021-05-24 at 17 22 02] <https://user-images.githubusercontent.com/48422873/119369633-83968580-bd08-11eb-96dc-5135ef81551a.png> [image: Screen Shot 2021-05-24 at 16 35 33] <https://user-images.githubusercontent.com/48422873/119368703-74fb9e80-bd07-11eb-8738-de15a3147452.png> [image: Screen Shot 2021-05-24 at 16 54 12] <https://user-images.githubusercontent.com/48422873/119368707-75943500-bd07-11eb-95f0-2e9a415e609f.png> The same number of messages for a classic queue sits around 300 MiB in process memory before a full GC, and ~60MiB after. Have changed raft.wal_max_size_bytes and x-max-in-memory-bytes to low values but no difference. Grafana dashboards before and after GC on QQ: ​ https://snapshot.raintank.io/dashboard/snapshot/IphlxnoFeXsjqw6wVH0G0vYS2svJ2GFJ ​ https://snapshot.raintank.io/dashboard/snapshot/PhdZV2Y2Z3oMyxbuft8f8nLs72vvcWL1 *Question: what is taking up so much process memory for QQs, or is this to be expected?* — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3063>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFDECNBE2I5P72R2EQLTPJVYRANCNFSM45NN54EQ> .

-- *Karl Nilsson*

[johanrhodin]

OK I tried with max-in-memory-length and it did free up some process memory but not much (from ~1.6GB to ~1.4GB).

[kjnilsson]

If you applied the policy to a queue with messages already in memory as you can see it will still keep them in memory until they are consumed. Try the test starting with an empty queue and the policy in place.

…

On Mon, 24 May 2021 at 17:21, Johan Rhodin ***@***.***> wrote: OK I tried with max-in-memory-length and it did free up some process memory but not much (from ~1.6GB to ~1.4GB). [image: rmq_process_memory_max-in-memory-length] <https://user-images.githubusercontent.com/460369/119376721-e729a000-bc81-11eb-9510-1272f10d97c0.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3063 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFDTEEYKK4MK434L2VTTPJ4HLANCNFSM45NN54EQ> .

-- *Karl Nilsson*

[johanrhodin]

OK, that brought down process memory to 478MiB 👍.

Is it possible to reduce it even further? As a comparison a (deprecated) mirrored classic queue with the same 2M messages and 1 replica is using ~2.5MiB process memory.

And for cluster where the user has already loaded in the message before applying the policy, is there a way to tell RabbitMQ to to page out those message (similar to changing vm_memory_high_watermark_paging_ratio for classic queues).

[kjnilsson]

I've done a bit of research today and come up with a couple of ways to reduce QQ memory use by ~50-60% for a queue with 2M messages and x-max-in-memory-length = 0. That does still leave ~200MiB in this case.

[johanrhodin]

This is what the garbage_collection_info looks like:

{garbage_collection_info,[{old_heap_block_size,55185655},
                          {heap_block_size,7427328},
                          {mbuf_size,50640},
                          {recent_size,36970},
                          {stack_size,11},
                          {old_heap_size,44517067},
                          {heap_size,2660543},
                          {bin_vheap_size,0},
                          {bin_vheap_block_size,46422},
                          {bin_old_vheap_size,36},
                          {bin_old_vheap_block_size,46422}]}

Attaching the full recon:info and recon:get_state, too.
recon_info.txt
recon_get_state.gz

[lhoguin]

Most of that memory can be GC @kjnilsson there's a lot of room for improvement. Everything in "old" would get to 0 on a full GC and that's 80%+ of the memory usage. Either QQ is creating a lot of garbage or it is not doing GC often enough (or both). I'm happy to look into it after I am done with the new CQ index, if you want.

[kjnilsson]

that doesn't sound quite right as I did a test with 2M messages + a full GC and it remained at ~470MiB

[lhoguin]

Right I skipped a step in my analysis, "old" would get to 0 but "new" would increase after the full GC. In their case I would expect about 120MB to be freed (and not 400+ as I initially thought), but those would probably be used again almost immediately, or kept around as unused memory for future small GCs. I don't know about your case though. I would try with fullsweep_after set to 0 to see what happens.

Clearly though the biggest win to be had is to reduce the size of the state so I would go with your initial idea above first.