RabbitMQ with MQTT - memory usage growing over time

[kbudde]

Hello,

I've followed the tickets and discussions quite carefully over the last monthes and I've seen many improvements so far but still it looks like the memory in use is growing over time until we reach the high watermark.

I'm looking for help what needs to be optimized or maybe there is still a bug left?

Environment
Ubuntu 18.04 with 4 CPUs, 16GB Memory
RabbitMQ Version: 3.8.14 # RabbitMQ was installed using apt.
Erlang Version: 23.3.4

We have around 11k devices connecting with mqtt protocol but not more than around 5k at the same time.
We have a "predictable workload". It's higher during office hours (highlighted in grey) and lower than <500 on weekends. In green you can see average memory usage per hour. (Peak memory usage is higher)

I've just checked the current memory details

and the values from today and 21 days ago with a modified "erlang memory-Allocators" dashboard

I see it increasing but I don't know why.

At the moment we restart the service every month and it's always the same. It starts low and grows over time.
It would be very nice if someone could suggest next steps.
I suspect we have to tune the memory settings, but it's a production system; so I would be happy about some hints from people with more erlang experience.
https://www.rabbitmq.com/runtime.html#allocators

[lhoguin]

You have a large increase in the ets_alloc. This suggests that some data might be accumulating in an ets table or fragmentation.

eheap_alloc only has a small increase. I don't know if that's "normal" or not.

Based on the information here it's difficult to know. Knowing how much is allocated and how much is used, per allocator type, would help. You can also run rabbitmq-diagnostics observer, go to the ets table page (E + Enter), sort by memory (m + Enter) and provide a screenshot of what you see.

[lhoguin]

OK. I don't think this is fragmentation in any case. Would be good to see current ets tables, see what size the big table currently has. Probably not nearly as big considering your numbers.

Would be good to keep an eye on how fast memory grows to see if doubling the scrape interval helped.

Not sure what else to suggest at this point.

[kbudde]

[michaelklishin]

Nothing around those tables particularly stands out. Without a way to reproduce your workload — specifically how/when connections are opened and closed — we can only guess whether the mqtt_node table is actually large or expected to be of this size.

See Dealing with High Connection Churn if you have connection churn. It can be that the MQTT client ID tracker still leaks the tiny records it uses. It can also be that your have a ton of connections in TIME_WAIT that seem open to the runtime and thus RabbitMQ, which means
that data cannot (yet) be deleted by the MQTT plugin.

[kbudde]

hi @michaelklishin, I don't think the issue is high connection churn here.
At peak time we have normally around around one connect/disconnect per second. Time_waits are not going up.

But I suspect it might be related to client ID.
The client ID is not stable at our side. It's partly random; so the number of different client IDs is theoretically not limited. If the mqtt plugin keeps statistics on them, I suspect it might be normal to see this grow. Is there anything I can tune here? Or maybe something I can check?

I will try to reproduce the issue on our test system.

[kbudde]

Just as a small update I tried to reproduce the issue but without success so far.
I'm unsure how to proceed.

[kbudde]

Hi,

I've noticed something else which might be related to the unusual memory consumption.

According to the RabbitMQ-Overview Dashboard: We have currently around 30 msg/s delivered and no messages are waiting; so all queues are nearly empty.
But the metrics for "messages published" and "messages routed to queues" are showing around 200 times higher message rates.

[michaelklishin]

This is more likely to be an issue with Prometheus metrics or their aggregation than anything related to MQTT.