Short Note on Apache Airflow in My MBP Docker

After I tried docker airflow on my macbook pro, It turned out that the worker can not run. I found out that default docker desktop memory is set to 2GB. After I change it to 6GB, It run smoothly.

You can set memory in docker desktop dashboard to support Celery Executer.

If you don’t want to allocate more memory to docker desktop, you can change from Celery Executer to Sequential Executer and remove worker and flower because sequential executer not using it. You can change it in your docker-compose.yml from apache airflow official link above. The sequential executer is enough to try out on your machine.

Change AIRFLOW_CORE_EXECUTER in docker-compose.yml

Example on Github

Create 4 Folders in current path. (logs, dags, plugins, postgres-db-volume)

mkdir ./dags ./logs ./plugins ./postgres-db-volume

Run docker-compose airflow-init

docker-compose up airflow-init

Run docker-compose up

docker-compose up

Check airflow at http://localhost:8080

Default login is airflow / airflow

For DAG Example, You can follow official airflow example.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Art Krisada

Art Krisada

Never stop learning, because life never stop teaching.