Mr. ASHOK GADI PARTHI

Seamless Data Integration on Google Cloud: Strategies for Scalable and Real-Time Workflows

Abstract:

Seamless data integration is the backbone of modern cloud architectures, ensuring efficient, scalable, and real-time data movement across distributed systems. This session delves into practical strategies for data integration on Google Cloud, focusing on key services such as Cloud Storage (GCS), BigQuery, Pub/Sub, Dataflow, Spanner, and Redis. Attendees will gain insights into designing high-performance pipelines that support both batch and real-time processing, optimizing for latency, cost, and scalability. Additionally, we will explore a real-world implementation of Change Data Capture (CDC) from Spanner to BigQuery using Dataflow, showcasing techniques to ensure fault tolerance, consistency, and efficiency in streaming data pipelines. The talk will provide best practices, lessons learned, and actionable insights to help engineers architect resilient and scalable data workflows on GCP, empowering businesses to drive real-time analytics and data-driven decision-making.