Skip to content
Login
Get CrateDB
Login
Get CrateDB
Home Resources Videos How to update stock market data automatically with cratedb and airflow

How to update stock market data automatically with CrateDB and Airflow

In this video, I show how to update stock market data automatically with CrateDB and Airflow.
First, I show how to run CrateDB with Docker and create a table to store the financial data. Then, to orchestrate the process of regular data updates, I will make an Airflow project and establish the connection to CrateDB. Once I set up the Airflow project, I show how to write workflow tasks and dependencies in Python as nodes and edges of an Airflow DAG. With the ready DAG, I execute the tasks in the Airflow UI and show the data in CrateDB.
Follow this tutorial in written form at https://community.crate.io/t/how-to-update-stock-market-data-automatically-with-cratedb-and-airflow/1304


Chapters:
Overview - 00:28
Run CrateDB and create a table - 01:38
Install Astronomer and initialize Airflow project - 03:40
What is a DAG? - 07:15
Import operators and modules - 08:11
Download task - 09:24
Prepare data task - 11:21
Insert data task - 12:08
Set task dependencies - 12:58
Put DAG together - 14:06
Execute DAG in Airflow UI - 14:50
Check out data in CrateDB - 16:03

 

Useful links:

Presenter: Rafaela Sant'ana, Customer Engineering Crate.io

Subscribe to the Crate.io Newsletter now