Sushil Kumar Tiwari

Architecting Resilient Data Pipelines: Best Practices for Scalable ETL in Multi-Cloud Environments.

Abstract:

As organizations adopt multi-cloud strategies to enhance agility and minimize vendor lock-in, the need for resilient and scalable data pipelines has become a critical engineering priority. This session presents architectural principles and best practices for designing fault-tolerant, high-performance ETL (Extract, Transform, Load) systems across heterogeneous environments such as AWS, Azure, and Google Cloud.
Drawing from experimental evaluations and real-world implementation patterns, the session outlines a layered architecture combining event-driven orchestration, containerized microservices, and metadata-driven automation. It highlights how unified orchestration frameworks, intelligent workload distribution, and self-healing mechanisms contribute to operational resilience and efficiency in distributed infrastructures.
The discussion covers methodologies for cross-cloud replication, idempotent ETL design, and adaptive resource allocation that ensure continuity under failures and dynamic workloads. It also examines how AI-assisted optimization and anomaly detection can improve workload scheduling, failure prediction, and cost governance in large-scale data ecosystems.
Attendees will gain a practical understanding of how modular design, metadata-driven orchestration, and AI-based automation enable consistent performance, maintain data quality, and support compliance in multi-cloud environments. The session concludes by exploring emerging trends in autonomous pipeline governance, AI-driven orchestration, and sustainable data engineering practices that are shaping the future of cloud-agnostic data systems.

Profile:

Sushil Kumar Tiwari is an experienced ETL/BI Cloud Engineer with over 16 years of expertise in data warehousing, business intelligence, and cloud-based ETL solutions. His technical background spans the design, development, and testing of data integration frameworks using Informatica PowerCenter, IDMC (Informatica Cloud Services), Snowflake, and AWS services.
Sushil has extensive experience working with data warehousing, dimensional modeling, and OLAP concepts, along with hands-on proficiency in SQL, PL/SQL, Python, and Shell scripting. He is skilled in optimizing complex ETL workflows and managing large-scale data environments across various industries, including banking, insurance, and financial services. His technical toolkit includes platforms such as AWS Redshift, Snowflake, DynamoDB, and Oracle, as well as analytics tools like Tableau and AWS QuickSight.
Throughout his career, Sushil has led multiple enterprise-level projects, including data migration, integration, and validation initiatives for clients such as Harvard Business School, Asurion (USA), Emirates NBD (Dubai), and Genpact (India). His leadership roles have involved overseeing end-to-end data warehouse migrations, performance tuning, test strategy design, and team mentoring within Agile/Scrum frameworks.
Sushil holds a Master of Computer Applications (Computer Science) from Rajasthan University, Jaipur, and a Bachelor of Science (Mathematics) from Ajmer University, Ajmer. He is certified as a Snowflake SnowPro Core and AWS Certified Associate Architect. Over the years, he has been recognized for his contributions with multiple Spot Awards and an Employee of the Month Award at Synechron and Wipro.