perf(query) Memoize the part of the logical plan tree traversal for reduced memory allocation and faster planning #1874
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request checklist
Current behavior :
Though there is no expected change in the behavior for the end users, the memory allocation for methods
getPushdownShards
,isTargetSchemaChanging
andQueryUtils.makeAllKeyValueCombos
is high. This high memory allocation can be reduced by memoizing the responses which are repeatedly called by recursing the logical plan treeWith no caching enabled the JMH tests for materialize give following results
As seen above,
getPushdownShards
,isTargetSchemaChanging
show up as two hotspots for memory allocation profiling.New behavior :
After memoizing the responses, the hotspots are no longer visible. The planner's materialize is 2-3X improvement in materialize times and then two calls don't show up in allocation hotspots anymore.