work / 2026 / In progress
DataFlow
A metadata-driven ETL framework that compiles YAML pipeline definitions into DuckDB, Polars or Spark jobs.
What it is
DataFlow is a small ETL framework. You describe a pipeline in YAML; it compiles that into an intermediate representation, then into an execution plan, and runs it on whichever engine fits the data: DuckDB, Polars or Spark.
Why
Most pipelines I see don't need a cluster, and the ones that do shouldn't need a rewrite. The idea is to keep the pipeline definition the same and let the engine change underneath it.
How it works
YAML → IR → ExecutionPlan → engine. The engine is picked by data size in bytes rather than row count.