Lineage Data Not Getting Ingested

[Prajwal214]

I’m using Open Metadata version 1.4.1, and I haven’t been able to get any lineage for the database service. I triggered a search reindex and re-ran the metadata and lineage ingestion. I have verified that all the required permissions are available.

[Prajwal214]

To accurately capture and maintain lineage information, please verify and provide the following details:

1. Identify Lineage-Creating Queries:

   • Indicate if there are any Create Table As Select (CTAS), MERGE, JOIN, or other queries that could create lineage. These queries are typically recorded in the query history table.

2. Specify Involved Tables:

   • State whether the query involves source, target, and intermediate tables. This helps us understand the data flow.

3. Submit the SQL Query:

   • Provide the exact SQL query for which you expect lineage information. This allows us to trace the data movement accurately.

4. Adjust Query Duration:

   • If possible, increase the query duration interval. The default interval for capturing lineage data is set to 1 day. Extending this duration can help gather comprehensive lineage data.

5. Parsing Query Error:

   • If you encounter "error parsing query" logs, you can add the query directly with the dialect here. We continuously improve the parsing coverage in each release.

6. Missing Lineage Information in Dashboards and Tables:

   • If you don't see lineage in Dashboards < > Tables, review the database service name configuration in the ingestion and ensure the tables exist in OM.

Pipeline Lineage

Pipeline lineage is only supported as an edge and not as a node. Below is an example of how it will be supported as an edge between two nodes:

• If a pipeline is used as an edge, you'll get the pipeline's latest status.
• It will also show the SQL with the transformation and a description.
• More details on the pipeline itself can be found at the pipeline object.

Refer to the Documentation Lineage Workflow