Remove the max_steps_gap parameter and missing timestamp functionality in select_time_slice()
#107
Labels
bug
Something isn't working
Comments
|
you can see if your suggestion will work by making the fork repository and than commit your suggested changes on that if it work you can open a PR @dfulu |
pvprajwal
added a commit
to pvprajwal/ocf-data-sampler
that referenced
this issue
Jan 9, 2025
# Pull Request ## Description This PR removes `max_steps_gap` and `fill_selection` parameters and `_sel_fillinterp()` and `_sel_fillnan()` as they are no longer required for training. Fixes openclimatefix#107 ## Checklist: - [.] My code follows [OCF's coding style guidelines](https://github.com/openclimatefix/.github/blob/main/coding_style.md) - [.] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [.] I have checked my code and corrected any misspellings
pvprajwal
added a commit
to pvprajwal/ocf-data-sampler
that referenced
this issue
Jan 9, 2025
# Pull Request ## Description This PR removes `max_steps_gap` and `fill_selection` parameters and `_sel_fillinterp()` and `_sel_fillnan()` as they are no longer required for training. Fixes openclimatefix#107 ## Checklist: - [.] My code follows [OCF's coding style guidelines](https://github.com/openclimatefix/.github/blob/main/coding_style.md) - [.] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [.] I have checked my code and corrected any misspellings
|
Hi, @dfulu is this issue still relevant? If yes, i'd like to take a look and possibly resolve it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In the
slice_datasets_by_time()function, the call toselect_time_slice()used for the satellite data usesmax_steps_gap=2- see here.The value of
max_steps_gapis entirely ignored in this function. This parameter is a carry over from ocf_datapipes and the slicing logic which uses this setting has not been ported across yet. Quite possibly we will never want to use this functionality since this was intended for use only in a production setting where data might be missing. We do this in the app now instead. During training, since we filter to t0 times where no satellite data may be missing, our current usage would never hit this anyway. We do not have the functionality in our t0 filtering to allow any missing time steps, so we should never need this.I suggest we:
_sel_fillinterp()function which it was intended to be used with._sel_fillnan()function and thefill_selectionparameter. I don't think we use these or will ever use these for training.The text was updated successfully, but these errors were encountered: