Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-compute means and standard deviation of satellite data, ignoring 'rectangle of zeros' and ignoring v2/eumetsat_zarr_2020_02.zarr #31

Open
JackKelly opened this issue Nov 23, 2021 · 3 comments · May be fixed by #35
Open

Re-compute means and standard deviation of satellite data, ignoring 'rectangle of zeros' and ignoring v2/eumetsat_zarr_2020_02.zarr #31

JackKelly opened this issue Nov 23, 2021 · 3 comments · May be fixed by #35
Assignees
@JackKelly
Labels
enhancement New feature or request

Comments

@JackKelly
Copy link
Member

JackKelly commented Nov 23, 2021 •
edited
Loading

Detailed Description

Issue #30 documents an issue in non-HRV Zarr data whereby there's a 'rectangle of zeros'. We should re-compute the means and standard deviations, ignoring northern latitudes which often have a 'rectangle of zeros'.

openclimatefix/nowcasting_dataset#484 documents an issue whereby v2/eumetsat_zarr_2020_02.zarr is full of noise from -30,000 to 30,000. We should ignore v2/eumetsat_zarr_2020_02.zarr when computing the means and stds.

Possible Implementation

Maybe create a simple script in Satip which computes the means and standard deviations (using dask?)
@JackKelly JackKelly added the enhancement New feature or request label Nov 23, 2021
@JackKelly JackKelly moved this to Todo in Nowcasting Nov 23, 2021
@JackKelly JackKelly changed the title Re-compute means and standard deviation of satellite data, ignoring 'rectangle of zeros' Re-compute means and standard deviation of satellite data, ignoring 'rectangle of zeros' and ignoring v2/eumetsat_zarr_2020_02.zarr Nov 23, 2021
@JackKelly JackKelly self-assigned this Nov 23, 2021
@JackKelly
Copy link
Member Author

JackKelly commented Nov 23, 2021 •
edited
Loading

I'll try to do this later this afternoon using means = ds_from_zarr['stacked_eumetsat_data'].mean(dim=['time', 'x', 'y']).compute()

@JackKelly JackKelly moved this from Todo to In Progress in Nowcasting Nov 23, 2021
@JackKelly
Copy link
Member Author

OK, I've written a little script to get the means and mins...

Just ignoring the eumetsat_zarr_2020_02.zarr (and not ignoring the "rectangle of zeros") the new stats are:

Coordinates:
  * variable  (variable) <U6 'IR_016' 'IR_039' 'IR_087' ... 'WV_062' 'WV_073'
Data variables:
    mean      (variable) float64 141.9 800.7 701.6 733.8 ... 116.1 601.8 517.8
    std       (variable) float64 157.3 200.6 181.3 183.3 ... 135.5 177.6 159.4,)

which, reassuringly, look identical to the existing means and standard deviations!

I'll also compute the means and standard deviations ignoring the "rectangle of zeros".

I'll update this when my script runs again and prints all the values out!

This was referenced Nov 23, 2021
Closed
Draft
@JackKelly
Copy link
Member Author

The "rectangle of zeros" starts at 1,037,047 meters (OSGB "northing") and extends north. So need to ignore everything north of this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JackKelly JackKelly

Labels
enhancement New feature or request
Projects
No open projects
Nowcasting
Status: In Progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Create script to compute stats openclimatefix/Satip
1 participant
@JackKelly