feat: StructArray compare #1889
Conversation
| } | ||
|
|
||
| Ok(Some( | ||
| BoolArray::new(bool_builder.finish(), lhs.dtype().nullability()).into_array(), |
There was a problem hiding this comment.
Not sure what's the correct option here for nullability, the default make_comparator is null < non-null, and it doesn't return null values, so maybe we should or here with the lhs/rhs validity arrays?
|
This is super inefficient.... we recursively canonicalize everything into Arrow. Why are we implementing this at all? |
|
If we do need to, you should run compare over each field independently, assemble the compare results into a struct array, and then pass that into Arrow. You can do that using StructArrayTrait too so no requirement on the RHS encoding. And then there's probably short-circuiting that's possible |
|
We're implementing this because currently falling back to arrow panics. |
|
the underlying impl here also short-circuits, I'm not sure why this is "super inefficient" when compares to the current (buggy) implementation. |
|
I actually think your suggestion doesn't allow for short-circuiting because I have to rebuild a struct array (with what structure?). |
|
closing in favor of #1912 |
closes #1888.