You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Enable docValues for sort fields like author_sort, title_sort, for the catalog
Level of urgency
High
Moderate
Low
Why is this maintenance needed?
Sort fields are being created at runtime and stored in the cache. We could create these during indexing and potentially save memory and improve performance by creating them during indexing
Acceptance criteria
The cache size is reduced
This change does not significantly increase the time it takes to index
Run both read and write solr load tests and make sure there are no major regressions
If this is successful create issues to extend this to other applications
Each replica's field cache for the catalog is ~ 3 GB, so that's 12 GB Heap.
author_sort is 21 MB, title_sort is ~ 160 MB
If we enable docValues on author_sort/title_sort etc, it'll use those at index time instead of populating that huge FieldCache
Figgy's field cache is also 2.8 GB
Er, or not? It looks like these metrics are showing other core fields
The text was updated successfully, but these errors were encountered:
What maintenance needs to be done?
Enable docValues for sort fields like author_sort, title_sort, for the catalog
Level of urgency
Why is this maintenance needed?
Sort fields are being created at runtime and stored in the cache. We could create these during indexing and potentially save memory and improve performance by creating them during indexing
Acceptance criteria
Implementation notes, if any
docValues documentation: https://lucene.apache.org/solr/guide/7_7/docvalues.html
@tpendragon did some research into this, saying:
Some things I'm reading (https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html and https://risdenk.github.io/2018/10/21/apache-solr-out-of-memory-symptoms-and-solutions.html) seems to imply that making sure our fields have docvalues enabled will make the heap much smaller.
Each replica's field cache for the catalog is ~ 3 GB, so that's 12 GB Heap.
author_sort is 21 MB, title_sort is ~ 160 MB
If we enable docValues on author_sort/title_sort etc, it'll use those at index time instead of populating that huge FieldCache
Figgy's field cache is also 2.8 GB
Er, or not? It looks like these metrics are showing other core fields
The text was updated successfully, but these errors were encountered: