Analyze Solr heap dump

[maxkadel]

What maintenance needs to be done?

Follow the instructions in Heap Dump section of pul_solr README to see why the heap is filling up.

Level of urgency

• High
• Moderate
• Low

Why is this maintenance needed?

The heap on production Solr is periodically getting high, forcing it to restart and impacting performance.

Acceptance criteria

• Run the task described
• Document findings
• Create any follow-up tickets stemming from findings

[sandbergja]

In reviewing heap usage on staging, the heap is not increasing to a concerning amount. Therefore, we decided not to prioritize this analysis for now, but can revisit it when we want to investigate further.

[sandbergja]

We did some preliminary exploration of the heap dump, with the goal of understanding why our heap fills up so fast. We noticed that 89.59% of the heap is solr's cache, including deprecated classes like solr.search.LRUCache (which has been replaced with a more memory-efficient solr.search.CaffeineCache). We came up with a plan to test CaffeineCache. There are other cache settings which bear further investigation.

[sandbergja]

pulibrary/pul_solr#180 also seems like it would help us to make better use of our heap. Some problems that might get in the way of trying out the changes iteratively/scientifically:

• it requires a full reindex to see the effects
• our current solr load test plan doesn't change the sort order or the fq

[sandbergja]

One more ticket that seems like it would help with our heap usage, with thanks to @maxkadel for identifying the issue: pulibrary/pul_solr#406

Closing this ticket now, so we can focus on the tickets regarding making better use of the cache.