README.md

Modern Data Platform Cookbooks - v1.17.1

Inhere we are documenting cookbooks on how to use the platform:

• Airflow

  • Schedule and Run Simple Python Application with Airflow - 1.16.0

• Trino (Formerly Presto SQL)

  • Trino, Spark and Delta Lake (Spark 2.4.7 & Delta Lake 0.6.1) - 1.11.0
  • Trino, Spark and Delta Lake (Spark 3.0.1 & Delta Lake 0.7.0) - 1.11.0
  • Querying S3 data (MinIO) using MinIO - 1.11.0
  • Querying Azure Data Lake Storage Gen2 data (ADLS) from Trino - 1.15.0
  • Querying data in Postgresql from Trino - 1.11.0
  • Querying data in Kafka from Trino (formerly PrestoSQL) - 1.14.0
  • Querying HDFS data using Trino - 1.11.0
  • Trino Security - 1.16.0

• MinIO

  • Serving a static Web application from MinIO

• MQTT

  • Using Confluent MQTT Proxy
  • Using HiveMQ with Kafka Extensions - 1.12.0

• Spark

  • Run Java Spark Application using spark-submit
  • Run Java Spark Application using Docker
  • Run Scala Spark Application using spark-submit
  • Run Scala Spark Application using Docker
  • Run Python Spark Application using spark-submit
  • Run Python Spark Application using Docker
  • Spark and Hive Metastore - 1.15.0
  • Spark with internal S3 (using on minIO)
  • Spark with external S3
  • Spark with PostgreSQL - 1.15.0

• Delta Lake Table Format

  • Spark with Delta Lake - 1.16.0

• Iceberg Table Format

  • Spark with Iceberg - 1.16.0

• Hadoop HDFS

  • Querying HDFS data using Presto
  • Using HDFS data with Spark Data Frame

• Livy

  • Submit Spark Application over Livy

• Apache NiFi

  • NiFi ExecuteScript Processor with Python - 1.16.0
  • NiFi Registry with Git Flow Persistence Provider - 1.16.0

• StreamSets Data Collector

  • Consume a binary file and send it as Kafka message
  • Using Dev Simulator Origin to simulate streaming data - 1.12.0
  • Loading StreamSets Pipeline(s) upon start of container - 1.14.0

• StreamSets DataOps Platform

  • Creating a self-managed StreamSets DataOps Environment using Platys - 1.14.0

• StreamSets Transformer

  • Using StreamSets Transformer to transform CSV to Parquet & Delta Lake - 1.16.0

• Kafka

  • Simulated Multi-DC Setup on one machine - 1.14.0
  • Automate management of Kafka topics using Jikkou - 1.17.0
  • Azure Event Hub as external Kafka - 1.16.0
  • SASL/SCRAM Authentication with Zookeeper - 1.17.0
  • SASL/SCRAM Authentication with KRaft - 1.17.0
  • SASL/PLAIN Authentication - 1.17.0

• Confluent Enterprise Platform

  • Using Confluent Enterprise Tiered Storage - 1.13.0

• ksqlDB

  • Connecting through ksqlDB CLI
  • Custom UDF and ksqlDB
  • Handle Serialization Errors in ksqlDB

• Kafka Connect

  • Using additional Kafka Connect Connector
  • Using a Kafka Connect Connector not in Confluent Hub - 1.14.0

• Apicurio Registry

  • Apicurio Registry with SQL Storage (PostgreSQL) - 1.14.0

• Oracle RDBMS

  • Using private (Trivadis) Oracle EE image - 1.13.0
  • Using public Oracle XE image - 1.16.0

• Neo4J

  • Working with Neo4J - 1.15.0
  • Neo4J and yFiles graphs for Jupyter - 1.16.0

• Tipboard

  • Working with Tipboard and Kafka - 1.14.0

• Architecture Decision Records (ADR)

  • Creating and visualizing ADRs with log4brains - 1.12.0

• Jupyter

  • Using Jupyter notebook with Spark and Avro - 1.16.0
  • Using JupyterHub - 1.16.0

• Dataverse

  • Dataverse with Minio (S3) storage - 1.17.0

• MLflow

  • Using MLflow from Jupyter - 1.16.0

• Docker Logging

  • Collecting Docker Logs with Loki - 1.17.0