You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The way telemetry core is organized right now makes it consume a lot of RAM after restart.
We have ~12000 nodes on the network right now and restart causes telemetry backend to consume over 40G of RAM before settling down a while later at under just 10G with occasional spikes to 20G.
Also huge CPU usage is observed during startup.
Some backpressure or easing for inrush requests is needed during startup or it is risking ending up in crash loop.
The text was updated successfully, but these errors were encountered:
The way telemetry core is organized right now makes it consume a lot of RAM after restart.
We have ~12000 nodes on the network right now and restart causes telemetry backend to consume over 40G of RAM before settling down a while later at under just 10G with occasional spikes to 20G.
Also huge CPU usage is observed during startup.
Some backpressure or easing for inrush requests is needed during startup or it is risking ending up in crash loop.
The text was updated successfully, but these errors were encountered: