Mr. Somnath Banerjee

Building Scalable and Cost-Effective Data Pipelines in the Cloud

Abstract:

In today’s data-driven world, organizations are rapidly transitioning to cloud-based architectures to handle large-scale data processing efficiently and cost-effectively. This talk explores the key principles, best practices, and technologies for designing scalable, resilient, and cost-optimized data pipelines in the cloud.

We will discuss batch vs. streaming pipelines, the selection of cloud-native ETL tools, and the role of serverless computing in optimizing performance. Additionally, we will cover cost-saving strategies, such as efficient data storage, compute optimization, and autoscaling, along with security and compliance considerations. Real-world case studies will illustrate how organizations leverage cloud platforms like AWS, Azure, and Google Cloud to manage large-scale, real-time, and batch data processing.

Attendees will gain actionable insights on reducing costs, improving scalability, and leveraging emerging trends like AI-driven automation, data mesh architectures, and edge computing. Whether you are a data engineer, architect, or business leader, this session will equip you with the knowledge to build high-performance, cost-efficient cloud data pipelines that drive business success.