Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Figure out why coordinates are having an extra dimension added #1

Closed
mattjbr123 opened this issue Oct 1, 2024 · 7 comments · Fixed by #35
Closed

Figure out why coordinates are having an extra dimension added #1

mattjbr123 opened this issue Oct 1, 2024 · 7 comments · Fixed by #35
Assignees

Comments

@mattjbr123
Copy link
Collaborator

          Successfully run with parallelisation with this environment. 

Though I've now noticed that the pipeline has added a time dimension to the variables that we converted from data variables to coordinate variables in the preprocessing function (so the variables x_bnds, y_bnds and crs)

_Originally posted by @mattjbr123 in #9

@mattjbr123
Copy link
Collaborator Author

Spinning it off into it's own issue given it's a fairly minor problem at this stage

@mattjbr123 mattjbr123 transferred this issue from NERC-CEH/object_store_tutorial Oct 7, 2024
@mattjbr123
Copy link
Collaborator Author

The error will likely be in the pangeo functions that wrap around Beam, so that's probably the place to start...

@mattjbr123
Copy link
Collaborator Author

Following the advice here to find out which step of the pipeline the extra dimensions get added, as a starting point.

Annoyingly, a Beam print function can only be placed at the end of a Beam pipeline, (though my hunch is there is some more fancy syntax that allows for pipeline branching and works around this), so it's a case of one-by-one putting the print statement after each function and analysing the output until we see the extra dimensions added.

mattjbr123 added a commit that referenced this issue Nov 21, 2024
@mattjbr123
Copy link
Collaborator Author

mattjbr123 commented Nov 21, 2024

Looks like it's to do with the ConsolidateDimensionCoordinates() function, which I guess isn't massively surprising... Will confirm and look into exactly what it's doing next.

@mattjbr123
Copy link
Collaborator Author

Something I may have forgotten - is this a bug that only occurs when running in parallel?
Need to check.

@mattjbr123
Copy link
Collaborator Author

Now I'm not so sure it's occuring with either. I'm currently unabled to reproduce the error, even with an identical setup AFAICT. So 🤞 this was a spurious bug or typo somewhere that has now resolved itself. I'm trying out a few scenarios to be sure.

@mattjbr123
Copy link
Collaborator Author

Seems to be miraculously resolved. I'll close this issue with a view to keeping an eye on things and reopening if it reoccurs.

mattjbr123 added a commit that referenced this issue Dec 10, 2024
@mattjbr123 mattjbr123 linked a pull request Dec 12, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging a pull request may close this issue.

1 participant