[SPARK-32444][SQL] Infer filters from DPP #29243

wangyum · 2020-07-26T10:28:40Z

What changes were proposed in this pull request?

This pr add support infer filters from DPP.

For this test suite.
Table fact_stats is partitioned by store_id and table code_stats is partitioned by store_id.
DPP add a new predicate: fact_stats.store_id IN dynamicpruning#2723.
We can infer code_stats.store_id IN dynamicpruning#2723 base on t1.store_id = t2.store_id.

Why are the changes needed?

Improve query performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

SparkQA · 2020-07-26T12:42:22Z

Test build #126568 has finished for PR 29243 at commit 69be17b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-26T19:28:14Z

Test build #126574 has finished for PR 29243 at commit f001b27.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-27T15:55:52Z

Test build #126641 has finished for PR 29243 at commit bcc81be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

bart-samwel · 2020-08-19T09:58:38Z

The partition pruning already prunes all partitions that don't satisfy that predicate, so all remaining rows will satisfy that predicate, right? Where is the performance gain then?

wangyum · 2020-08-19T10:33:11Z

Before this pr. We can only prune on fact_stats.store_id column:

== Physical Plan ==
*(3) Project [store_id#2705, code#2706, product_id#2708L]
+- *(3) BroadcastHashJoin [cast(store_id#2705 as bigint)], [store_id#2709L], Inner, BuildRight, false
   :- *(3) Project [store_id#2705, code#2706]
   :  +- *(3) BroadcastHashJoin [store_id#2705], [store_id#2707], Inner, BuildRight, false
   :     :- *(3) ColumnarToRow
   :     :  +- FileScan parquet default.fact_stats[store_id#2705] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [isnotnull(store_id#2705), dynamicpruningexpression(cast(store_id#2705 as bigint) IN subquery#2716)], PushedFilters: [], ReadSchema: struct<>
   :     :        +- Subquery subquery#2716, [id=#245]
   :     :           +- *(2) HashAggregate(keys=[store_id#2709L#2715L], functions=[])
   :     :              +- Exchange hashpartitioning(store_id#2709L#2715L, 5), true, [id=#241]
   :     :                 +- *(1) HashAggregate(keys=[store_id#2709L AS store_id#2709L#2715L], functions=[])
   :     :                    +- *(1) Filter ((isnotnull(product_id#2708L) AND (product_id#2708L < 3)) AND isnotnull(store_id#2709L))
   :     :                       +- *(1) ColumnarToRow
   :     :                          +- FileScan parquet default.product[product_id#2708L,store_id#2709L] Batched: true, DataFilters: [isnotnull(product_id#2708L), (product_id#2708L < 3), isnotnull(store_id#2709L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(product_id), LessThan(product_id,3), IsNotNull(store_id)], ReadSchema: struct<product_id:bigint,store_id:bigint>
   :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[1, int, true] as bigint)),false), [id=#275]
   :        +- *(1) ColumnarToRow
   :           +- FileScan parquet default.code_stats[code#2706,store_id#2707] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [isnotnull(store_id#2707)], PushedFilters: [], ReadSchema: struct<code:int>
   +- BroadcastExchange HashedRelationBroadcastMode(List(input[1, bigint, false]),false), [id=#283]
      +- *(2) Filter ((isnotnull(product_id#2708L) AND (product_id#2708L < 3)) AND isnotnull(store_id#2709L))
         +- *(2) ColumnarToRow
            +- FileScan parquet default.product[product_id#2708L,store_id#2709L] Batched: true, DataFilters: [isnotnull(product_id#2708L), (product_id#2708L < 3), isnotnull(store_id#2709L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(product_id), LessThan(product_id,3), IsNotNull(store_id)], ReadSchema: struct<product_id:bigint,store_id:bigint>

After this pr. We can also prune on code_stats.store_id column:

== Physical Plan ==
*(3) Project [store_id#2705, code#2706, product_id#2708L]
+- *(3) BroadcastHashJoin [cast(store_id#2705 as bigint)], [store_id#2709L], Inner, BuildRight, false
   :- *(3) Project [store_id#2705, code#2706]
   :  +- *(3) BroadcastHashJoin [store_id#2705], [store_id#2707], Inner, BuildRight, false
   :     :- *(3) ColumnarToRow
   :     :  +- FileScan parquet default.fact_stats[store_id#2705] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [isnotnull(store_id#2705), dynamicpruningexpression(cast(store_id#2705 as bigint) IN subquery#2716)], PushedFilters: [], ReadSchema: struct<>
   :     :        +- Subquery subquery#2716, [id=#250]
   :     :           +- *(2) HashAggregate(keys=[store_id#2709L#2715L], functions=[])
   :     :              +- Exchange hashpartitioning(store_id#2709L#2715L, 5), true, [id=#246]
   :     :                 +- *(1) HashAggregate(keys=[store_id#2709L AS store_id#2709L#2715L], functions=[])
   :     :                    +- *(1) Filter ((isnotnull(product_id#2708L) AND (product_id#2708L < 3)) AND isnotnull(store_id#2709L))
   :     :                       +- *(1) ColumnarToRow
   :     :                          +- FileScan parquet default.product[product_id#2708L,store_id#2709L] Batched: true, DataFilters: [isnotnull(product_id#2708L), (product_id#2708L < 3), isnotnull(store_id#2709L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(product_id), LessThan(product_id,3), IsNotNull(store_id)], ReadSchema: struct<product_id:bigint,store_id:bigint>
   :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[1, int, true] as bigint)),false), [id=#309]
   :        +- *(1) ColumnarToRow
   :           +- FileScan parquet default.code_stats[code#2706,store_id#2707] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [isnotnull(store_id#2707), dynamicpruningexpression(cast(store_id#2707 as bigint) IN subquery#2718)], PushedFilters: [], ReadSchema: struct<code:int>
   :                 +- Subquery subquery#2718, [id=#279]
   :                    +- *(2) HashAggregate(keys=[store_id#2709L#2717L], functions=[])
   :                       +- Exchange hashpartitioning(store_id#2709L#2717L, 5), true, [id=#275]
   :                          +- *(1) HashAggregate(keys=[store_id#2709L AS store_id#2709L#2717L], functions=[])
   :                             +- *(1) Filter ((isnotnull(product_id#2708L) AND (product_id#2708L < 3)) AND isnotnull(store_id#2709L))
   :                                +- *(1) ColumnarToRow
   :                                   +- FileScan parquet default.product[product_id#2708L,store_id#2709L] Batched: true, DataFilters: [isnotnull(product_id#2708L), (product_id#2708L < 3), isnotnull(store_id#2709L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(product_id), LessThan(product_id,3), IsNotNull(store_id)], ReadSchema: struct<product_id:bigint,store_id:bigint>
   +- BroadcastExchange HashedRelationBroadcastMode(List(input[1, bigint, false]),false), [id=#317]
      +- *(2) Filter ((isnotnull(product_id#2708L) AND (product_id#2708L < 3)) AND isnotnull(store_id#2709L))
         +- *(2) ColumnarToRow
            +- FileScan parquet default.product[product_id#2708L,store_id#2709L] Batched: true, DataFilters: [isnotnull(product_id#2708L), (product_id#2708L < 3), isnotnull(store_id#2709L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-27227/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(product_id), LessThan(product_id,3), IsNotNull(store_id)], ReadSchema: struct<product_id:bigint,store_id:bigint>

bart-samwel · 2020-08-19T11:28:55Z

Ah, that makes sense. Thanks for the example!

maryannxue · 2020-08-20T22:03:25Z

We don't just apply DPP to any column, right? Shouldn't we check that the inferred filter is applied on partition columns?

In that sense we should fix the DPP rule for it to be apply the same dimension table filter on partition columns from different tables. In other words, the DPP rule itself should generate a filter on fact_stats.store_id and another on code_stats.store_id.

…n column

wangyum · 2020-08-21T08:19:42Z

Yes. We should only infer the filter to partition column.

The DPP rule itself could not generate a filter on code_stats.store_id because it should be after PushDownPredicates.

I add logic to CleanupDynamicPruningFilters to remove all filters with DynamicPruning that are not filtered on the partition column.

SparkQA · 2020-08-21T10:11:21Z

Test build #127734 has finished for PR 29243 at commit 98f7275.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maryannxue · 2020-08-21T14:37:02Z

I add logic to CleanupDynamicPruningFilters to remove all filters with DynamicPruning that are not filtered on the partition column.

It is better now.
Still, the pruningHasBenefit flag carried over from the original DPP filter might not be correct for the new filter.

…HANGE_REUSE_ENABLED.key} is disabled

sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/explain.txt

SparkQA · 2020-08-24T12:11:39Z

Test build #127835 has finished for PR 29243 at commit c436bc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2020-08-24T13:00:53Z

I have fixed pruningHasBenefit for the new filter, and this change will infer more filters for subquery.

# Conflicts: # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q10.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q10/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q56.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q56/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q60.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q60/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q69.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q69/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q93.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q93/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q10a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q10a/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35a/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q5a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q5a/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a.sf100/explain.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/explain.txt # sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala

SparkQA · 2020-08-28T07:05:02Z

Test build #127987 has finished for PR 29243 at commit b5bb9a2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2020-08-28T07:22:56Z

This change is similar to #22778.

wangyum · 2020-08-28T07:23:06Z

retest this please

SparkQA · 2020-08-28T12:37:25Z

Test build #127990 has finished for PR 29243 at commit b5bb9a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2020-12-07T00:47:54Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

# Conflicts: # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q10.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q10/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q69.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q69/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q93.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q93/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q94/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q10a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q10a/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35a/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q5a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q5a/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a.sf100/simplified.txt # sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/simplified.txt # sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala

SparkQA · 2021-02-12T07:37:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39700/

SparkQA · 2021-02-12T08:39:37Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39700/

SparkQA · 2021-02-12T09:28:52Z

Test build #135118 has finished for PR 29243 at commit b5bb9a2.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2021-02-12T16:21:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39706/

SparkQA · 2021-02-12T16:51:55Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39706/

SparkQA · 2021-02-12T18:27:44Z

Test build #135125 has finished for PR 29243 at commit 9e9a633.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-13T09:41:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39717/

SparkQA · 2021-02-13T11:17:30Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39717/

SparkQA · 2021-02-13T11:25:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39718/

SparkQA · 2021-02-13T11:29:48Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39718/

SparkQA · 2021-02-13T13:24:19Z

Test build #135136 has finished for PR 29243 at commit ba71ead.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-13T15:06:05Z

Test build #135137 has finished for PR 29243 at commit c63b162.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Infer filters from DPP

69be17b

probot-autolabeler bot added the SQL label Jul 26, 2020

Change strategy to Once

f001b27

Fix Once strategy's idempotence is broken for batch Infer Filters

bcc81be

Remove any Filters with DynamicPruning that didn't filter on partitio…

98f7275

…n column

Merge remote-tracking branch 'upstream/master' into SPARK-32444

baa7796

wangyum added 2 commits August 24, 2020 14:37

Remove no benefit inferred DPP on partition column when ${SQLConf.EXC…

4901af4

…HANGE_REUSE_ENABLED.key} is disabled

Update PlanStabilitySuite

c436bc4

wangyum commented Aug 24, 2020

View reviewed changes

sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/explain.txt Outdated Show resolved Hide resolved

github-actions bot added the Stale label Dec 7, 2020

github-actions bot closed this Dec 8, 2020

wangyum reopened this Feb 12, 2021

wangyum removed the Stale label Feb 12, 2021

Infer filters from DPP

9e9a633

fix

ba71ead

fix

c63b162

wangyum closed this Mar 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32444][SQL] Infer filters from DPP #29243

[SPARK-32444][SQL] Infer filters from DPP #29243

wangyum commented Jul 26, 2020 •

edited

Loading

SparkQA commented Jul 26, 2020

SparkQA commented Jul 26, 2020

SparkQA commented Jul 27, 2020

bart-samwel commented Aug 19, 2020

wangyum commented Aug 19, 2020

bart-samwel commented Aug 19, 2020

maryannxue commented Aug 20, 2020

wangyum commented Aug 21, 2020

SparkQA commented Aug 21, 2020

maryannxue commented Aug 21, 2020

SparkQA commented Aug 24, 2020

wangyum commented Aug 24, 2020

SparkQA commented Aug 28, 2020

wangyum commented Aug 28, 2020

wangyum commented Aug 28, 2020

SparkQA commented Aug 28, 2020

github-actions bot commented Dec 7, 2020

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

[SPARK-32444][SQL] Infer filters from DPP #29243

[SPARK-32444][SQL] Infer filters from DPP #29243

Conversation

wangyum commented Jul 26, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jul 26, 2020

SparkQA commented Jul 26, 2020

SparkQA commented Jul 27, 2020

bart-samwel commented Aug 19, 2020

wangyum commented Aug 19, 2020

bart-samwel commented Aug 19, 2020

maryannxue commented Aug 20, 2020

wangyum commented Aug 21, 2020

SparkQA commented Aug 21, 2020

maryannxue commented Aug 21, 2020

SparkQA commented Aug 24, 2020

wangyum commented Aug 24, 2020

SparkQA commented Aug 28, 2020

wangyum commented Aug 28, 2020

wangyum commented Aug 28, 2020

SparkQA commented Aug 28, 2020

github-actions bot commented Dec 7, 2020

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 12, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

SparkQA commented Feb 13, 2021

wangyum commented Jul 26, 2020 •

edited

Loading