Mr. Teja Krishna Kota

AI-Enhanced ETL Frameworks for Scalable Cloud Data Warehousing

Abstract:

As data volumes surge across industries, traditional ETL (Extract, Transform, Load) pipelines struggle with scalability, latency, and operational overhead. This talk presents a next-generation approach to ETL design, incorporating Generative AI techniques and intelligent orchestration to optimize performance, automate schema evolution, and enhance data reliability.

Drawing from real-world implementations in the healthcare domain, including enterprise-scale cloud migrations and secure pipeline integrations, this session will explore how AI-powered modules can dynamically adapt transformation logic, identify data quality anomalies, and reduce manual intervention. The talk will cover architecture patterns involving Snowflake, Databricks, Python-based ETL, and AI agents that assist in rule generation, metadata inference, and auto-documentation.

By blending applied intelligence with robust engineering principles, this talk aims to provide a roadmap for organizations looking to modernize their data infrastructure with smart, self-optimizing ETL frameworks. Attendees will gain actionable insights into designing ETL workflows that are not only scalable but also intelligent, secure, and resilient.