Feedback

  Suggest improvement

  Edit page

  View page source

Load and Export

You have a variety of options to connect and integrate with 3rd-party ETL applications, mostly using CrateDB’s PostgreSQL interface.

This documentation section lists corresponding ETL applications and frameworks which can be used together with CrateDB, and outlines how to use them optimally.

Apache Airflow / Astronomer

A set of starter tutorials.

• Automating the import of Parquet files with Apache Airflow

• Updating stock market data automatically with CrateDB and Apache Airflow

• Automating stock data collection and storage with CrateDB and Apache Airflow

A set of elaborated tutorials, including blueprint implementations.

• Automating export of CrateDB data to S3 using Apache Airflow

• Implementing a data retention policy in CrateDB using Apache Airflow

• CrateDB and Apache Airflow: Building a data ingestion pipeline

• Building a hot and cold storage data retention policy in CrateDB with Apache Airflow

Tutorials and resources about configuring the managed variants, Astro and CrateDB Cloud.

• ETL with Astro and CrateDB Cloud in 30min - fully up in the cloud

• ETL pipeline using Apache Airflow with CrateDB (Source)

• Run an ETL pipeline with CrateDB and data quality checks

Apache Flink

• Data Ingestion using Kafka and Kafka Connect

• Build a data ingestion pipeline using Kafka, Flink, and CrateDB

• Community Day: Stream processing with Apache Flink and CrateDB

• Executable stack: Apache Kafka, Apache Flink, and CrateDB

Apache Kafka

• Data Ingestion using Kafka and Kafka Connect

• Executable stack: Apache Kafka, Apache Flink, and CrateDB

• Tutorial: Replicating data to CrateDB with Debezium and Kafka

Azure Functions

• Data Enrichment using IoT Hubs, Azure Functions and CrateDB

dbt

• Using dbt with CrateDB

Debezium

• Tutorial: Replicating data to CrateDB with Debezium and Kafka

• Webinar: How to replicate data from other databases to CrateDB with Debezium and Kafka

Kestra

• Setting up data pipelines with CrateDB and Kestra

MongoDB

• Import data from MongoDB

MySQL

• Import data from MySQL

Node-RED

• Ingesting MQTT messages into CrateDB using Node-RED

• Automating recurrent CrateDB queries using Node-RED

Singer / Meltano

• meltano-target-cratedb

• meltano-tap-cratedb

• Examples about working with CrateDB and Meltano

🚧 Please note these adapters are a work in progress. 🚧

SQL Server Integration Services

A demo project which uses SSIS and ODBC to read and write data from CrateDB:

• Using SQL Server Integration Services with CrateDB

StreamSets

• Data Stream Pipelines with CrateDB and StreamSets Data Collector