Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NumpyBatch and BatchKey considerations #21

Open
dfulu opened this issue Aug 13, 2024 · 2 comments
Open

NumpyBatch and BatchKey considerations #21

dfulu opened this issue Aug 13, 2024 · 2 comments

Comments

@dfulu
Copy link
Member

dfulu commented Aug 13, 2024

We should think about our current way of using NumpyBatch and BatchKeys.

The NumpyBatch works well with torch dataloader to automatically convert the samples to torch tensors and thus feed the model. However it is harder to probe when we want to check if the data is as expected.

Batchkeys have been quite inflexible in the past and for this reason are not used in summation model and are hard to add to or subtract from without corrupting alrerady saved batches

We should investigate if there is a better, but still simple way of packaging batches of data

@dfulu dfulu added enhancement New feature or request discussion labels Aug 13, 2024
@peterdudfield peterdudfield removed the enhancement New feature or request label Sep 4, 2024
@peterdudfield
Copy link
Contributor

peterdudfield commented Sep 4, 2024

Yea, I wonder if we could still use Batchkeys, but in the Batch we use the values of them, not the Key. This i think means we can change the BatchKeys and we wont have those iterative problems. I think it would be changing
batch[BatchKey.gsp] =x. to batch[BatchKey.gsp.name], or something like that.

On a slightly different note something fun that we could do too. Having a Sample/Batch object could be useful. Here we add function like save, load and plot to help standardise things

@peterdudfield
Copy link
Contributor

Now moved into here for GSP, Satellite, and NWP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants