Apache Airflow - challenges (2024)

Airflow has been the go to tool for workflow and scheduling , There are various tools in Big Data space, like are prefect, Dagster, Airflow, Azkaban , Apache Dolphin scheduler etc. How ever Airflow and Dagster has been on top of adoption.

Limitations of Airflow

• Airflow is highly complex and Non-Intuitive :- Airflows "pipeline/workflow as code" is quite powerful, how ever it violates the fundamental principle - "separation of concerns" - it requires decent python programmer to create jobs as tasks in DAG , sequence it as workflow , add more workflows for any logistic works like compaction for production ready data, handle the failure scenario and test , debug the DAG paths. It can get extremely complex to debug in production. My point is why would i need to have my workflow as code ? It can be configuration, i see no reason for some one to code the workflow and debugging in production is counter intuitive.
• Apache airflow ( open source version ) doesn't support multi tenancy out of box :- This is quite fundamental for any project to go to production, logical segregation of resources is very practical requirement to adoption, how ever this can be mitigated with k8s as deployment strategy but there are rough edges.
• Adoption & Steep learning curve - As airflow requires Data engineer to know its scheduler, executor & workers, Airflow's framework is exhaustive , which means need to spend significant amount of time understanding this framework, API . secondly Airflow has tons of configuration parameters, while it can be seen as flexibility, it is time consuming..
• Airflow doesnt come with Pipeline versioning :- Airflow doesnt store workflow metatdata changes, which means there is no way you could track workflow history, rollback pipelines as and when required - can be tricky in production
• Debugging with Airflow is nightmare :- Its easy to code workflows and keep things moving , how ever as and when the flows get complex, debugging also becomes extremely difficult with tons of inter dependency both north and south. pipelines can easily get spegetti as it can have issue can arise any where. for example operators and templates are building blocks of workflow thats defined declaratively in DAG, it can also be interchanged, operators can execute code as well as orchestrate workflows, this can further complicate debugging. Executors can be either local or remote, Local is some time consuming but it can be straight forward but cant say same about remote executors. RCA can be daunting with complex environment such as these.

• Data lineage is still rudimentary :- Tracking and monitoring is important in any pipeline. According to Airflows documentation , it clearly calls out as "experimental - subjected to change" - which means as and when your pipe line evolves, you need to re check DAG , perhaps even rework. Airflow doesn't collect native information about the Job, owners or repo's, its herculean effort to understand what gets affected downstream due to what ever changes that might have happened in upstream.
• Airflow may not trigger dot on time :- Airflow monitors all tasks, DAGs, and triggers once the dependencies are complete, by design Airflow DAG will run at the end of schedule_interval. Airflow operates in UTC by default.

You wrote DAG , scheduled it to run lets say every other hour, starting from 10 AM today, every thing is pretty neat till now, you check back at 11 AM and you find that your DAG has triggered, however log indicates there was one run at 10 AM, dont be surprised, if i say this is expected behavior, because,Airflow will run DAG at the end of the scheduling_interval, which means 10 AM DAG gets trigged by 11AM.

• Tasks are bottleneck- There are multiple reasons why tasks can get bottleneck. one of the reasons is due to too many concurrent DAGs or with too many concurrent Tasks with in DAGs. This an be tuned in aifrlow.cfg by increasing the higher number of tasks in parrallelism. if you are planning to execute more DAGs, you should increase variable max_active_runs_per_dg and for concurrent tasks increase the default limit of airflow_celery_worker_concurrency.

So Airflow is a beast on its own, its flexible but its counter-intuitive for adoption and IMO in 2023, its not worth coding your DAGs or workflows when you have better alternatives like Apache Dolphin Scheduler , which offers DAG creationby simple DND, can scale easily without too much of tuning, multi tenancy is available out of box.