Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for MTG Satellite processing #138

Open
jacobbieker opened this issue Dec 4, 2024 · 1 comment
Open

Add support for MTG Satellite processing #138

jacobbieker opened this issue Dec 4, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@jacobbieker
Copy link
Member

There is the new satellite from EUMETSAT for the 0 degree service that is now operational. It has a faster refresh (10min vs 15) and more channels (16 vs 12) and higher spatial resolution, so would be good to include.

Detailed Description

The MTG satellite is now fully operational, providing 16 spectral channels, with much higher resolution data than the MSG currently being used. The MTG will also eventually replace the MSG for the 0 degree service sometime next year (timeline is 6 months after it is operational).

Context

Higher resolution, more frequent data (every 10 minutes) can help with forecasting. It would also be nice to be able to have it in the GCP archive to make it more easily accessible.

Possible Implementation

It is already available in the data store from September to now, both the high resolution data, and the nominal resolution data, in the Data Store.
https://data.eumetsat.int/data/map/EO:EUM:DAT:0665
https://data.eumetsat.int/data/map/EO:EUM:DAT:0662

@jacobbieker jacobbieker added the enhancement New feature or request label Dec 4, 2024
@jacobbieker jacobbieker self-assigned this Dec 4, 2024
@jacobbieker
Copy link
Member Author

Few updates here:

Once pydata/xarray#9948 is merged, we could then write sharded Zarrs, which should cut down the massive number of chunks on disk we have with the current satellite imagery (and would be worse with MTG)

Using xbitinfo to get how many bits we need to keep 99.9% of the information in the imagery, we could save half the storage by going from float32 to float16.

High resolution 1km
<xarray.Dataset> Size: 28B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    ir_105    (inflevel) int64 8B 9
    ir_38     (inflevel) int64 8B 8

High resolution 500m
<xarray.Dataset> Size: 28B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    nir_22    (inflevel) int64 8B 12
    vis_06    (inflevel) int64 8B 15

Low resolution 1km
<xarray.Dataset> Size: 76B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    nir_13    (inflevel) int64 8B 12
    nir_16    (inflevel) int64 8B 3
    nir_22    (inflevel) int64 8B 12
    vis_04    (inflevel) int64 8B 17
    vis_05    (inflevel) int64 8B 4
    vis_06    (inflevel) int64 8B 4
    vis_08    (inflevel) int64 8B 4
    vis_09    (inflevel) int64 8B 15

Low resolution 2km
<xarray.Dataset> Size: 76B
Dimensions:   (inflevel: 1)
Coordinates:
    dim       <U1 4B 'x'
  * inflevel  (inflevel) float64 8B 0.999
Data variables:
    ir_105    (inflevel) int64 8B 9
    ir_123    (inflevel) int64 8B 9
    ir_133    (inflevel) int64 8B 9
    ir_38     (inflevel) int64 8B 8
    ir_87     (inflevel) int64 8B 9
    ir_97     (inflevel) int64 8B 9
    wv_63     (inflevel) int64 8B 9
    wv_73     (inflevel) int64 8B 10

For these, we could either keep float32 and use BitRound to reduce storage costs by making it more compressible, or change to float16, and potentially also use BitRound. The one issue is the ones that are quite close to or above 16 bits of info, the high resolution 500m visible channel, and the vis_09 and vis_04 channels in the low resolution as well.

@devsjc Any thoughts on what might be better approach? Wouldn't want to get rid of info, could also keep only some of the variables in float32, and the rest in float16 I guess

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant