Description:
As institutions integrate an increasing number of data sources, ETL pipelines in Azure Data Factory (ADF) often become complex, repetitive, and difficult to maintain. Many teams face challenges such as managing numerous datasets, creating separate pipelines for each process, relying on hard-coded configurations, and struggling to scale efficiently.
This session presents a practical approach to designing scalable and reusable ETL frameworks in ADF. It explores how to simplify pipeline architecture by reducing duplication, leveraging parameterization, and using dynamic datasets to move away from rigid designs. The session also highlights how a metadata-driven approach can support flexible and scalable solutions as data environments evolve.
Through real-world patterns and examples, attendees will see how these techniques improve maintainability and efficiency while supporting long-term growth in modern data platforms.
About this event:
Presenters:
- Sahar Pani , Data Report Specialist, University Of Illinois-Gies College of Business
Track:
Reroute to Efficiency — Streamline processes at warp speed to reduce complexity and improve operations.
Experience Needed:
Intermediate
Learning Outcome:
By the end of this session, participants will understand key challenges in managing ETL pipelines in Azure Data Factory and learn how to design scalable, reusable frameworks. They will explore the use of parameterization, dynamic datasets, and metadata-driven approaches to reduce duplication, improve maintainability, and support evolving data integration needs.
Maximum Capacity:
50 or less
Additional Keywords:
Azure Data Factory (ADF),ETL Frameworks, Dynamic Datasets, Parameterization, Maintainability, Pipeline, Optimization