Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using a forward slash in a node names of a DataTree causes a RecursionError #9978

Open
5 tasks done
dawiedotcom opened this issue Jan 23, 2025 · 2 comments
Open
5 tasks done
Assignees
Labels
bug topic-DataTree Related to the implementation of a DataTree class

Comments

@dawiedotcom
Copy link

What happened?

When creating an xarray.DataTree object, passing the optional children parameter with / in a key of the dictionary produces an infinite recursion loop.

It appears that a ValueError: node names cannot contain forward slashes is raised, but the program then gets stuck in a mutual recursive loop between

child._set_parent(new_parent=self, child_name=name) # xarray/core/treenode.py line 333

and

self._attach(new_parent, child_name) # xarray/core/treenode.py line 115

This happens before the ValueError can be handled by the call to xarray.DataTree.

What did you expect to happen?

Either of the following:

  • Just raise ValueError("node names cannot contain forward slashes"), without the recursion issue.
  • Even better: using / in the keys of the children parameter produces nested DataTree objects. This was be behaviour in xarray-contrib/datatree.

Minimal Complete Verifiable Example

from xarray import DataTree

# Wrapping this in try, except handles the RecursionError, not the ValueError
dt = DataTree(children={'a/b': DataTree()})

MVCE confirmation

  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 182, in children
    child._set_parent(new_parent=self, child_name=name)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 159, in _attach
    self._post_attach(parent, child_name)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 723, in _post_attach
    _validate_name(name)  # is this check redundant?
    ~~~~~~~~~~~~~~^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 676, in _validate_name
    raise ValueError("node names cannot contain forward slashes")
ValueError: node names cannot contain forward slashes

During handling of the above exception, another exception occurred:

... many repetitions of 'ValueError: node names cannot contain forward slashes' and it's stack trace

Traceback (most recent call last):
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 182, in children
    child._set_parent(new_parent=self, child_name=name)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 152, in _attach
    self._pre_attach(parent, child_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 528, in _pre_attach
    check_alignment(path, node_ds, parent_ds, self.children)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 149, in check_alignment
    align(node_ds, parent_ds, join="exact", copy=False)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 883, in align
    aligner.align()
    ~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 572, in align
    self.find_matching_indexes()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 254, in find_matching_indexes
    obj_indexes, obj_index_vars = self._normalize_indexes(obj.xindexes)
                                  ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/alignment.py", line 215, in _normalize_indexes
    for idx, index_vars in Indexes(xr_indexes, xr_variables).group_by_index():
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File ".venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 1624, in group_by_index
    for i, index in self._id_index.items():
                    ^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 1515, in _id_index
    self.__id_index = {id(idx): idx for idx in self.get_unique()}
                                               ~~~~~~~~~~~~~~~^^
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "xarray_datatree_recursion_loop.py", line 2, in <module>
    DataTree(children={'a/b': DataTree(name='leaf')})
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 506, in __init__
    super().__init__(name=name, children=children)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 691, in __init__
    super().__init__(children=children)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 87, in __init__
    self.children = {name: child.copy() for name, child in children.items()}
    ^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/common.py", line 333, in __setattr__
    object.__setattr__(self, name, value)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 187, in children
    self.children = old_children
    ^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/common.py", line 333, in __setattr__
    object.__setattr__(self, name, value)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 187, in children
    self.children = old_children
    ^^^^^^^^^^^^^

... many repetitions jumping between the previous two calls

  File ".venv/lib/python3.13/site-packages/xarray/core/common.py", line 333, in __setattr__
    object.__setattr__(self, name, value)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 187, in children
    self.children = old_children
    ^^^^^^^^^^^^^

  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 152, in _attach
    self._pre_attach(parent, child_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 521, in _pre_attach
    if name in parent.dataset.variables:
               ^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 614, in dataset
    return self._to_dataset_view(rebuild_dims=True, inherit=True)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 553, in _to_dataset_view
    coord_vars = self._coord_variables if inherit else self._node_coord_variables
                 ^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/datatree.py", line 541, in _coord_variables
    *(p._node_coord_variables_with_index for p in self.parents),
                                                  ^^^^^^^^^^^^
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 338, in parents
    return tuple(self._iter_parents())
  File ".venv/lib/python3.13/site-packages/xarray/core/treenode.py", line 305, in _iter_parents
    node: Tree | None = self.parent
                        ^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

Anything else we need to know?

I tested this on the latest main branch in a new conda environment, but it appears the issue is present in every version of xarray since the introduction of DataTree.

Environment

INSTALLED VERSIONS

commit: None
python: 3.13.1 (main, Dec 4 2024, 18:05:56) [GCC 14.2.1 20240910]
python-bits: 64
OS: Linux
OS-release: 6.12.9-arch1-1
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development

xarray: 2025.1.1
pandas: 2.2.3
numpy: 2.2.1
scipy: 1.15.1
netCDF4: 1.7.2
pydap: None
h5netcdf: None
h5py: None
zarr: None
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.10.0
cartopy: None
seaborn: None
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: None
pip: None
conda: None
pytest: 8.3.4
mypy: None
IPython: None
sphinx: None

@dawiedotcom dawiedotcom added bug needs triage Issue that has not been reviewed by xarray team member labels Jan 23, 2025
Copy link

welcome bot commented Jan 23, 2025

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

@TomNicholas TomNicholas added topic-DataTree Related to the implementation of a DataTree class and removed needs triage Issue that has not been reviewed by xarray team member labels Jan 23, 2025
@TomNicholas
Copy link
Member

Thank you for reporting!

Just a note for myself that this is related to #9441, #9490, and #9477

@TomNicholas TomNicholas self-assigned this Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug topic-DataTree Related to the implementation of a DataTree class
Projects
None yet
Development

No branches or pull requests

2 participants