Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cutover to Vortex Layouts #1899

Merged
merged 74 commits into from
Jan 14, 2025
Merged
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
53599ef
DataFusion Layouts
gatesn Jan 7, 2025
e01cedc
DataFusion Layouts
gatesn Jan 7, 2025
3c9f285
DataFusion Layouts
gatesn Jan 7, 2025
6f4b711
DataFusion Layouts
gatesn Jan 7, 2025
f1e11d1
DataFusion Layouts
gatesn Jan 7, 2025
0d0b59e
add hash to vortex-expr and therefore vortex-scalar
joseph-isaacs Jan 9, 2025
b2453a4
use dyn-hash
joseph-isaacs Jan 9, 2025
e8b7d94
clippy
joseph-isaacs Jan 9, 2025
4314b62
fix
joseph-isaacs Jan 9, 2025
f122076
derive hash impl PartialEq
joseph-isaacs Jan 9, 2025
f91f062
wip
joseph-isaacs Jan 9, 2025
ca5253e
wip
joseph-isaacs Jan 9, 2025
aff1c14
wip
joseph-isaacs Jan 9, 2025
e1211bc
wip
joseph-isaacs Jan 9, 2025
e917d73
wip
joseph-isaacs Jan 9, 2025
9d47a02
Merge branch 'develop' into ji/rescope-exprs
joseph-isaacs Jan 9, 2025
0e54bb6
rescope exprs
joseph-isaacs Jan 10, 2025
912387e
Merge branch 'develop' into ji/rescope-exprs
joseph-isaacs Jan 10, 2025
c379aec
no dt
joseph-isaacs Jan 10, 2025
b4fa31e
update
joseph-isaacs Jan 10, 2025
fa0c6ac
update
joseph-isaacs Jan 10, 2025
2a0e675
update
joseph-isaacs Jan 10, 2025
2ce862a
update
joseph-isaacs Jan 10, 2025
7789787
update
joseph-isaacs Jan 10, 2025
abf15be
update
joseph-isaacs Jan 10, 2025
8ee6e0b
no add
joseph-isaacs Jan 10, 2025
21e1eea
merge
gatesn Jan 10, 2025
42c1373
merge
gatesn Jan 10, 2025
47e54ba
wip expr proj
joseph-isaacs Jan 10, 2025
116301e
wip
joseph-isaacs Jan 10, 2025
c4d7480
fix
joseph-isaacs Jan 10, 2025
12f11b5
fix
joseph-isaacs Jan 10, 2025
5853125
cleanup
joseph-isaacs Jan 10, 2025
29b6fec
todo
joseph-isaacs Jan 10, 2025
3bda2c3
I/O driver
gatesn Jan 10, 2025
71cc555
I/O driver
gatesn Jan 10, 2025
545108d
I/O driver
gatesn Jan 10, 2025
625aae7
Better buffer debug
gatesn Jan 10, 2025
fc71762
Better buffer debug
gatesn Jan 10, 2025
a6f274c
IntoArrow
gatesn Jan 10, 2025
ce6c023
Merge
gatesn Jan 11, 2025
eec354b
Fix conflicts
gatesn Jan 11, 2025
7af8780
Remove dtype index
gatesn Jan 11, 2025
d00a6ab
merge develop
gatesn Jan 11, 2025
fa0d5ce
Struct Layout
gatesn Jan 11, 2025
043a936
Merge branch 'develop' into ngates/datafusion-layouts
gatesn Jan 11, 2025
9c0ed0e
Struct Layout
gatesn Jan 11, 2025
b52817b
merge
gatesn Jan 13, 2025
97ca0bb
merge
gatesn Jan 13, 2025
84b276f
Fixes
gatesn Jan 13, 2025
48befca
Fixes
gatesn Jan 13, 2025
6d3ab6e
Cleanup Layouts
gatesn Jan 13, 2025
1f58672
Cleanup Layouts
gatesn Jan 13, 2025
49f79a4
Cleanup Layouts
gatesn Jan 13, 2025
e725890
Cleanup Layouts
gatesn Jan 13, 2025
619ffa8
Support opening Vortex files without I/O
gatesn Jan 13, 2025
442495c
Support opening Vortex files without I/O
gatesn Jan 13, 2025
52811f1
Dumb coalescing
gatesn Jan 13, 2025
bfcb2b1
Dumb coalescing
gatesn Jan 13, 2025
eb8b025
Dumb coalescing
gatesn Jan 13, 2025
c44ae3b
Dumb coalescing
gatesn Jan 13, 2025
588a4dc
Dumb coalescing
gatesn Jan 13, 2025
67e97f7
Dumb coalescing
gatesn Jan 13, 2025
7c336e0
Merge branch 'develop' into ngates/datafusion-layouts
joseph-isaacs Jan 14, 2025
94b38d6
ignore inconvertible df-exprs and run typed simplify
joseph-isaacs Jan 14, 2025
0e90cd4
Segment Cache
gatesn Jan 14, 2025
7f28796
Fix layouts (#1936)
joseph-isaacs Jan 14, 2025
9b40764
merge
gatesn Jan 14, 2025
e01cb19
Merge branch 'ngates/datafusion-layouts' of github.com:spiraldb/vorte…
gatesn Jan 14, 2025
abd97f7
merge
gatesn Jan 14, 2025
1ccc5ab
merge
gatesn Jan 14, 2025
e78905e
merge
gatesn Jan 14, 2025
ca8cb16
merge
gatesn Jan 14, 2025
ba2127e
merge
gatesn Jan 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Struct Layout
  • Loading branch information
gatesn committed Jan 11, 2025
commit fa0d5ce25c306f25d8961072078d3e49d357d1b7
10 changes: 8 additions & 2 deletions vortex-expr/src/column.rs
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ use std::sync::Arc;

use vortex_array::{ArrayDType, ArrayData};
use vortex_dtype::Field;
use vortex_error::{vortex_err, VortexResult};
use vortex_error::{vortex_err, VortexExpect, VortexResult};

use crate::{ExprRef, VortexExpr};

@@ -70,7 +70,13 @@ impl VortexExpr for Column {
)
})?
.maybe_null_field(&self.field)
.ok_or_else(|| vortex_err!("Array doesn't contain child array {}", self.field))
.ok_or_else(|| {
vortex_err!(
"Array doesn't contain column {}: {:?}",
self.field,
batch.dtype().as_struct().vortex_expect("struct")
)
})
}

fn children(&self) -> Vec<&ExprRef> {
8 changes: 7 additions & 1 deletion vortex-file/src/v2/strategy.rs
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
use vortex_dtype::DType;
use vortex_error::VortexResult;
use vortex_layout::layouts::chunked::writer::{ChunkedLayoutOptions, ChunkedLayoutWriter};
use vortex_layout::layouts::struct_::writer::StructLayoutWriter;
use vortex_layout::strategies::{LayoutStrategy, LayoutWriter, LayoutWriterExt};

/// The default Vortex file layout strategy.
@@ -12,6 +13,11 @@ pub struct VortexLayoutStrategy;

impl LayoutStrategy for VortexLayoutStrategy {
fn new_writer(&self, dtype: &DType) -> VortexResult<Box<dyn LayoutWriter>> {
Ok(ChunkedLayoutWriter::new(dtype, ChunkedLayoutOptions::default()).boxed())
if dtype.is_struct() {
// Recurse each struct field to use this same strategy
Ok(StructLayoutWriter::try_new_with_factory(dtype, VortexLayoutStrategy)?.boxed())
} else {
Ok(ChunkedLayoutWriter::new(dtype, ChunkedLayoutOptions::default()).boxed())
}
}
}