Skip to content

Commit

Permalink
feat: specialized IntoCanonical for DictArray utf8/binary (#1146)
Browse files Browse the repository at this point in the history
Fixes #1041 

Change the IntoCanonical implementation for DictArray to do
canonicalize-then-take for stringy things, take-then-canonicalize for
everything else.

It is always the case for strings that canonicalize-then-take will be
faster, regardless of the compression of the values array.

Overrides #1136
  • Loading branch information
a10y authored Oct 28, 2024
1 parent 83a371c commit 8467f64
Showing 1 changed file with 12 additions and 2 deletions.
14 changes: 12 additions & 2 deletions encodings/dict/src/array.rs
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,18 @@ impl ArrayTrait for DictArray {}

impl IntoCanonical for DictArray {
fn into_canonical(self) -> VortexResult<Canonical> {
let canonical_values: Array = self.values().into_canonical()?.into();
take(canonical_values, self.codes())?.into_canonical()
match self.dtype() {
// NOTE: Utf8 and Binary will decompress into VarBinViewArray, which requires a full
// decompression to construct the views child array.
// For this case, it is *always* faster to decompress the values first and then create
// copies of the view pointers.
DType::Utf8(_) | DType::Binary(_) => {
let canonical_values: Array = self.values().into_canonical()?.into();
take(canonical_values, self.codes())?.into_canonical()
}
// Non-string case: take and then canonicalize
_ => take(self.values(), self.codes())?.into_canonical(),
}
}
}

Expand Down

0 comments on commit 8467f64

Please sign in to comment.