How to fully index OpenMetadata into ElasticSearch

[harshach]

OpenMetadata uses MySQL or Postgres to store metadata from various sources, and it also uses ElasticSearch or OpenSearch to index the metadata to provide Data Discovery.
OpenMetadata uses the EventPublisher interface to publish metadata changes to ElasticSearch. For example, a user adding a tag to a Table or updating the description of a table generates a Metadata Event which ElasticSearchEventPublisher will later consume
However, there may be cases where users want to run full-index to update the ElasticSearch, this Show and Tell will go through all the details of how OpenMetadata updates ElasticSearch/OpenSearch and how one can do full re-index of OpenMetadata into ElasticSearch.

[harshach]

How OpenMetadata uses ElasticSearch/OpenSearch for data discovery

The above architecture shows how OpenMetadata uses ElasticSearch or OpenSearch.

1. Ingestion uses REST APIs to create or update metadata entities such as Tables, Pipelines, Dashboards, etc..
2. As they are getting ingested into OpenMetadata, it inserts or updates the data in MySQL/Postgres. While also generates a Metadata Change Event. For more details about the Metadata Change Event please look into this schema
3. The ChangeEvents are delivered into EventPubsliher interface which will have implementations for ElasticSearch, Slack etc..
4. In this case ElasticSearchEventPublisher takes the change event updates or creates a new document in ElasticSearch

This method helps us keep the ElasticSearch consistent with Ingestion or User changes that are happening in OpenMetadata.

However, in some cases, we may need to run a full reindex of OpenMetadata. Examples such the ElasticSearch cluster went down and lost data or there are network issues and the uptime of ElasticSearch being compromised.

Fully re-index ElasticSearch

There may be cases where you would like to fully re-index ElasticSearch. OpenMetadata uses MySQL or Postgres to store the metadata and this metadata can be accessed through our REST APIs.
OpenMetadata ships with an ElasticSearch connector that will allow users to fully index the Metadata contents into Elasticsearch.

Install OpenMetadata ElasticSearch Connector

Run the below commands in your terminal to install the ElasticSearch connector

mkdir -p openmetadata; cd openmetadata

python3 -m venv env

source env/bin/activate

pip3 install --upgrade pip setuptools

pip3 install 'openmetadata-ingestion[elasticsearch]'

Create a configuration file using the below YAML template

---
source:
  type: metadata_elasticsearch
  serviceName: openMetadata
  serviceConnection:
    config:
      type: MetadataES
      includeTables: 'true'
      includeUsers: 'true'
      includeTopics: 'true'
      includeDashboards: 'true'
      limitRecords: 10
  sourceConfig:
    config: {}
sink:
  type: elasticsearch
  config:
    index_tables: 'true'
    index_topics: 'true'
    index_dashboards: 'true'
    es_host: localhost
    es_port: 9200
workflowConfig:
  openMetadataServerConfig:
    hostPort: http://localhost:8585/api
    authProvider: no-auth

copy the above contents into a file elasticsearch.yaml

Run the ElasticSearch connector

metadata ingest -c elasticsearch.yaml 

My ElasticSearch/OpenSearch uses https and username/password

In case of secure ElasticSearch/OpenSearch we provide option to configure those in the yaml file

---
source:
  type: metadata_elasticsearch
  serviceName: openMetadata
  serviceConnection:
    config:
      type: MetadataES
  sourceConfig:
    config: {}
sink:
  type: elasticsearch
  config:
    es_host: localhost
    es_port: 9200
    es_username: 'username'
    es_password: 'password'
    use_ssl: true
workflowConfig:
  openMetadataServerConfig:
    hostPort: http://localhost:8585/api
    authProvider: no-auth

In the above config under sink we have the following configurations.

1. es_username
2. es_password
3. use_ssl
   Please use these configs to index into a secure elasticsearch/opensearch

Please let us know if you have any issues with this tutorial in our slack

[pmbrull]

Here you can find the reference of all supported configurations for the ES Sink.