Datafusion questions

[alexwilcoxson-rel]

Hey there, I was wondering about the Datafusion TableProvider impl.

I noticed it does not implement supports_filter_pushdown and thus scan doesn't get filters provided for pruning. Is this intentional or just an oversight as Datafusion has been updated and this change just hasn't been implemented yet?

Also regarding the pruning logic:

            let pruning_predicate = PruningPredicate::try_new(predicate, schema.clone())?;
            let files_to_prune = pruning_predicate.prune(self)?;
            self.get_state()
                .files()
                .iter()
                .zip(files_to_prune.into_iter())
                .for_each(|(action, prune_file)| {
                    if !prune_file {
                        let part = partitioned_file_from_action(action, &schema);
                        file_groups
                            .entry(part.partition_values.clone())
                            .or_default()
                            .push(part);
                    };
                });

if !prune_file seems to have the opposite intended effect. Files with statistics that match the filter are returned as true in files_to_prune. Thus files that do not match the predicate are kept.

I can open an issue but just wanted to check first via discussion in case I'm missing something obvious as I'm pretty new to Datafusion and this Rust crate.

Thanks!

[roeap]

A quick scan of the docs for pruning predicate makes me think you are correct - so yes, please file an issue :).