writing / / 1 min read
Airflow on one EC2 box
Notes from running Apache Airflow with LocalExecutor and a Postgres metadata database on a single EC2 instance to schedule PySpark jobs.
I needed something to schedule existing PySpark ETL jobs, and a managed Airflow was more than the job called for. So: one EC2 instance, Airflow with LocalExecutor, and PostgreSQL as the metadata database.
Why LocalExecutor
LocalExecutor runs tasks as processes on the same machine, so there's no broker or worker fleet to run. The catch is that it needs a real database for metadata, not SQLite, which is why Postgres is in the picture from day one.
Keeping it alive
The webserver and scheduler each run as a systemd service, so they start on boot and restart when they die. That is most of the "operations" story on a single box.
What this doesn't give you
One machine is one failure domain, and every task shares its CPU and memory. It's a fine start for existing jobs. It's not where you stop if the workload grows.