Do I need to use Airflow?

[giacomochiarella]

We have already a pipeline scheduler (Prefect) in our infrastructure. We would like to use it instead of Airflow included in the boundle. Is it possible? Is there any documentation about the ingestion itself without mentioning Airflow?

Thanks!

[Sachin-chaurasiya]

Hello @giacomochiarella , Thanks for reaching out.
We have a doc for prefect integration here https://docs.open-metadata.org/openmetadata/integrations/prefect., this may help.

[giacomochiarella]

@Sachin-chaurasiya thank you for pointing it out! I will definitely dig into it

[giacomochiarella]

@Sachin-chaurasiya thank you for pointing it out once again! I've tried to setup a Prefect flow but seems to me there are some imports which I do not have after installed openmetadata-ingestion package. In prefect_openmetadata.flows there is the following import

from metadata.utils.connections import (
    get_connection,
    test_connection,
)

do you have any idea where does this come from? What package and version? They do not mention here what version of openmetadata-ingestion they are using. I've installed 0.13.2.6, I've installed openmetadata-ingestion, openmetadata-ingestion-core, openmetadata-ingestion[postgres] but none gave me above-mentioned import. Any idea?

[Sachin-chaurasiya]

Maybe @pmbrull can help you here.

[giacomochiarella]

@pmbrull any feedback could be helpful please

[giacomochiarella]

I've found another not found import

from metadata.generated.schema.entity.services.ingestionPipelines.ingestionPipeline import (
    PipelineState,
)

in metadata.ingestion.api.workflow