Our Azure Data Engineering Journey: Pipelines, Databases, and Lessons Learned

Modern data platforms depend on reliable pipelines, thoughtfully designed databases, and effective collaboration between data engineers. In this session, we will share our practical experience building and supporting an Azure data platform using Azure Data Factory and Azure SQL Managed Instance. We will explain how Azure Data Factory orchestrates data extraction, loading, transformation, scheduling, and monitoring, while Azure SQL Managed Instance supports structured storage, database processing, security, and performance. Using a representative workflow, we will follow data from its source through staging, validation, transformation, and production tables. We will discuss parameterized pipelines, incremental loading, stored procedures, error handling, audit logging, and recovery strategies. We will also share challenges we encountered, solutions that worked, and lessons that improved reliability and maintainability. Attendees will leave with practical ideas for connecting pipelines and databases, clarifying responsibilities, troubleshooting failures, and developing dependable Azure data workflows that support reporting, analytics, and other institutional data needs.

About this event:

Presenters:

• Sahar Pani, Data Engineer, Gies College of Business- University of Illinois
• Michael Nelson, Data Engineer, Gies College of Business- University of Illinois

Track:

Core Stats — Your infrastructure isn’t flashy, but neither is dying because you dumped Vitality. Sessions on foundational IT services, security, reliability, and core competencies.

Experience Needed:

Intermediate

Learning Outcome:

By the end of this session, attendees will be able to describe how Azure Data Factory and Azure SQL Managed Instance support an end-to-end data workflow. They will identify practical approaches for building parameterized pipelines, organizing staging and production database layers, implementing incremental data loads, monitoring executions, and responding to failures. Attendees will also gain insights from real implementation challenges and lessons that they can apply when designing or improving their own Azure data solutions.

Additional Keywords:

Azure Data Factory, Azure SQL Managed Instance, Data Engineering, Data Pipelines, ETL/ELT, Database Design, Workflow Automation, Data Integration, Monitoring and Reliability, Error Handling