Pankaj Dureja

AI-Driven Evolution of Data Pipelines into Self-Orchestrating Platforms for Intelligent Decision-Making

Abstract:

Data pipelines have traditionally been designed as structured processes focused on moving and transforming data from one system to another. While effective in stable environments, these approaches often struggle to keep up with rapidly changing data landscapes and the growing demand for real-time insights.

In this talk, Pankaj Dureja shares a practical perspective on how data pipelines are evolving into intelligent, self-orchestrating platforms. Instead of relying solely on predefined workflows, these systems are designed to adapt based on data behavior, operational conditions, and business priorities. By incorporating elements of artificial intelligence, pipelines can move beyond execution to become responsive systems that adjust dynamically, detect inconsistencies, and improve their performance over time.

The discussion draws on real-world implementations to illustrate how modern data platforms can support faster and more reliable decision-making by reducing manual intervention and enabling more meaningful insights. It also highlights how this shift is redefining the role of data pipelines—from backend processes to active enablers of business value.

The session concludes with a forward-looking perspective on how organizations can transition toward AI-driven data ecosystems, where pipelines continuously learn, adapt, and play a central role in intelligent decision-making.

Profile:

Pankaj Dureja is a Data Engineering Manager at EOG Resources with close to two decades of experience in building and scaling data platforms that support critical business functions. He has worked across multiple industries, including healthcare, travel, retail, and oil and gas, developing a strong foundation in transforming data into actionable insights.

His work focuses on designing efficient and reliable data workflows, modernizing legacy systems, and enabling organizations to better leverage their data. He is particularly interested in how traditional data engineering practices can evolve with the adoption of artificial intelligence to create more adaptive and intelligent systems.

Pankaj has authored several research publications in the areas of data pipelines, workflow automation, and domain-specific analytics. His approach combines hands-on implementation with forward-thinking ideas, aiming to bridge the gap between data engineering and intelligent decision systems.

Through his work, he continues to explore how data platforms can become more responsive, scalable, and aligned with business needs, helping organizations unlock greater value from their data.