ElasticsearchProviderQueryMapping

Overview

pygeoapi's Elasticsearch provider provides the capability to offer up ES indexes via OGC API - Features and OGC API - Records. This wiki page explains how OGC API queries are translated into their equivalent ES queries.

Technical Details

• code is at https://github.com/geopython/pygeoapi/blob/master/pygeoapi/provider/elasticsearch_.py
• support is for ES 7+
• the basic setup for ES to pygeoapi integration is having an ES index of GeoJSON documents, and explicitly setting the geometry object as a ES geo_shape type in the index mappings
• a custom provider can be written to work with alternative ES setups, which would then serialize results as GeoJSON for pygeoapi upstream callers
• a useful litmus test is running ogrinfo directly against a given ES index. If the ES index is compatible with GDAL/OGR, it will be compatible with pygeoapi's ES provider
• a given ES index requires the document _id to be consistent with an equivalent property (can be named anything) in the underlying GeoJSON document's properties object (this allows pygeoapi to do id queries with consistency
• for OGC API - Records, we exclude certain internal fields which are not displayed to the user
• limit and startindex parameters from the OGC API endpoint are not passed in the ES request payload, but as parameters to the Python Elasticsearch API
• track_total_hits is a (boolean) parameter on whether to show actual counts (defaults to a maximum of 10000)
• there may be better ways to formulate these ES queries

Requests

items endpoint with no query parameters

URL: http://localhost:5000/collections/records/items?f=json

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": []
        }
    },
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}

items endpoint with bbox parameter

URL: http://localhost:5000/collections/records/items?f=json&bbox=-142,42,-52,84

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "geo_shape": {
                        "geometry": {
                            "shape": {
                                "type": "envelope",
                                "coordinates": [
                                    [
                                        -142.0,
                                        84.0
                                    ],
                                    [
                                        -52.0,
                                        42.0
                                    ]
                                ]
                            },
                            "relation": "intersects"
                        }
                    }
                }
            ]
        }
    },
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}

items endpoint with bbox and property parameters

URL: http://localhost:5000/collections/records/items?f=json&bbox=-142,42,-52,84&title=foo

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "geo_shape": {
                        "geometry": {
                            "shape": {
                                "type": "envelope",
                                "coordinates": [
                                    [
                                        -142.0,
                                        84.0
                                    ],
                                    [
                                        -52.0,
                                        42.0
                                    ]
                                ]
                            },
                            "relation": "intersects"
                        }
                    }
                },
                {
                    "match": {
                        "properties.title": "foo"
                    }
                }
            ]
        }
    },
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}

items endpoint with bbox and q parameters

URL: http://localhost:5000/collections/records/items?f=json&bbox=-142,42,-52,84&q=foo

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "geo_shape": {
                        "geometry": {
                            "shape": {
                                "type": "envelope",
                                "coordinates": [
                                    [
                                        -142.0,
                                        84.0
                                    ],
                                    [
                                        -52.0,
                                        42.0
                                    ]
                                ]
                            },
                            "relation": "intersects"
                        }
                    }
                }
            ],
            "must": {
                "query_string": {
                    "query": "foo"
                }
            }
        }
    },
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}

items endpoint with q and datetime parameters

URL: http://localhost:5000/collections/records/items?f=json&q=metar&datetime=2020-09-11T00:00:00Z/2020-09-12T00:00:00Z

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "properties.datetime": {
                            "gte": "2020-09-11T00:00:00Z",
                            "lte": "2020-09-12T00:00:00Z"
                        }
                    }
                }
            ],
            "must": {
                "query_string": {
                    "query": "metar"
                }
            }
        }
    },
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}

items endpoint with bbox, datetime, and q parameters

URL: http://localhost:5000/collections/records/items?f=json&bbox=-152,42,-52,84&q=metar&datetime=2020-09-11T00:00:00Z/2020-09-12T00:00:00Z

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "geo_shape": {
                        "geometry": {
                            "shape": {
                                "type": "envelope",
                                "coordinates": [
                                    [
                                        -152.0,
                                        84.0
                                    ],
                                    [
                                        -52.0,
                                        42.0
                                    ]
                                ]
                            },
                            "relation": "intersects"
                        }
                    }
                },
                {
                    "range": {
                        "properties.datetime": {
                            "gte": "2020-09-11T00:00:00Z",
                            "lte": "2020-09-12T00:00:00Z"
                        }
                    }
                }
            ],
            "must": {
                "query_string": {
                    "query": "metar"
                }
            }
        }
    },
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}

items endpoint with and q and sortby parameters

URL: http://localhost:5000/collections/records/items?f=json&q=metar&sortby=title

ES request:

{
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [],
            "must": {
                "query_string": {
                    "query": "metar"
                }
            }
        }
    },
    "sort": [
        {
            "properties.title.raw": {
                "order": "desc"
            }
        }
    ],
    "_source": {
        "excludes": [
            "properties._raw_metadata"
        ]
    }
}