Short Note on Apache Airflow in My MBP Docker

Running Airflow in Docker - Airflow Documentation

After I tried docker airflow on my macbook pro, It turned out that the worker can not run. I found out that default docker desktop memory is set to 2GB. After I change it to 6GB, It run smoothly.

You can set memory in docker desktop dashboard to support Celery Executer.

If you don’t want to allocate more memory to docker desktop, you can change from Celery Executer to Sequential Executer and remove worker and flower because sequential executer not using it. You can change it in your docker-compose.yml from apache airflow official link above. The sequential executer is enough to try out on your machine.

Change AIRFLOW_CORE_EXECUTER in docker-compose.yml

Example on Github

GitHub - krisadas/airflow-sequential: Example yml file for Apache Airflow with Sequential Executer

Create 4 Folders in current path. (logs, dags, plugins, postgres-db-volume)

mkdir ./dags ./logs ./plugins ./postgres-db-volume

Run docker-compose airflow-init

docker-compose up airflow-init

Run docker-compose up

docker-compose up

Check airflow at http://localhost:8080

Default login is airflow / airflow

For DAG Example, You can follow official airflow example.

Tutorial - Airflow Documentation