-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add shards
to valid_encodings
to enable sharded Zarr writing
#9948
base: main
Are you sure you want to change the base?
Conversation
Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient. |
Thanks @jacobbieker for opening a PR. Do you have time to add a roundtrip test for this? This test would be a good test to emulate: xarray/xarray/tests/test_backends.py Lines 2486 to 2494 in 6bea715
|
Yep! I think I've added one that works for that, and updated the Zarr V3 loading to include the Edit: Ah it seems to cause some issues with threads on 3.12 potentially? |
I seem to remember fixing this in the Dask test suite by manually cleaning up thread resources. This may help: https://github.com/dask/dask/blob/e04734b4d8959ba259801f2e2a490cb4ee8d891f/dask/tests/test_distributed.py#L338-L358 |
@jhamman The dask distributed zarr integration test is also failing on main. Would be great to have a fix applied. |
See #9967 |
@jhamman yeah, this change in #9967 fixes the tests and the failing ones pass. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this @jacobbieker !
No problem! Excited to use this a lot! I am not able to merge this PR, is it possible you can? Its saying I'm not authorized. |
We'll let @dcherian do a final review and/or merge. |
Adds
shards
to the list ofvalid_encodings
in the zarr backend, so that sharded Zarr V3s can be written.shards
not invalid_encodings
#9947