Sushil Kumar Tiwari

Architecting Resilient Data Pipelines: Best Practices for Scalable ETL in Multi-Cloud Environments.

Abstract:

As organizations adopt multi-cloud strategies to enhance agility and minimize vendor lock-in, the need for resilient and scalable data pipelines has become a critical engineering priority. This session presents architectural principles and best practices for designing fault-tolerant, high-performance ETL (Extract, Transform, Load) systems across heterogeneous environments such as AWS, Azure, and Google Cloud.

Drawing from experimental evaluations and real-world implementation patterns, the session outlines a layered architecture combining event-driven orchestration, containerized microservices, and metadata-driven automation. It highlights how unified orchestration frameworks, intelligent workload distribution, and self-healing mechanisms contribute to operational resilience and efficiency in distributed infrastructures.

The discussion covers methodologies for cross-cloud replication, idempotent ETL design, and adaptive resource allocation that ensure continuity under failures and dynamic workloads. It also examines how AI-assisted optimization and anomaly detection can improve workload scheduling, failure prediction, and cost governance in large-scale data ecosystems.

Attendees will gain a practical understanding of how modular design, metadata-driven orchestration, and AI-based automation enable consistent performance, maintain data quality, and support compliance in multi-cloud environments. The session concludes by exploring emerging trends in autonomous pipeline governance, AI-driven orchestration, and sustainable data engineering practices that are shaping the future of cloud-agnostic data systems.

Profile:

Sushil Kumar Tiwari is an accomplished ETL/BI Cloud Engineer with over 16 years of experience in data warehousing, business intelligence, and cloud-based data integration solutions. He has a strong technical background in designing, developing, and optimizing enterprise ETL frameworks using Informatica PowerCenter, Informatica Intelligent Data Management Cloud (IDMC), Snowflake, and AWS services.

Sushil brings deep expertise in data warehousing concepts, dimensional modeling, and OLAP, along with hands-on proficiency in SQL, PL/SQL, Python, and Shell scripting. He has successfully optimized complex ETL workflows and managed large-scale data platforms across industries such as banking, insurance, and financial services. His technical skill set includes Snowflake, AWS Redshift, DynamoDB, Oracle, and analytics tools like Tableau and AWS QuickSight.

Throughout his career, Sushil has led and contributed to multiple enterprise-level initiatives, including data migration, integration, and validation projects for global organizations such as Harvard Business School, Asurion (USA), Emirates NBD (Dubai), and Genpact (India). His leadership experience includes overseeing end-to-end data warehouse migrations, performance tuning, test strategy development, and mentoring teams within Agile and Scrum frameworks.

Sushil holds a Master of Computer Applications in Computer Science from Rajasthan University and a Bachelor’s degree in Mathematics from Ajmer University. He is a Snowflake SnowPro Core certified professional and an AWS Certified Associate Architect, and has received multiple Spot Awards and Employee of the Month recognitions for his contributions.