Pushkar Devanahalli Gopalakrishna

The Housing Intelligence Engine: Large-Scale Real Estate Analytics with Apache Data Sketches.

Abstract - Real estate data contains an enormous number of entities and events: hundreds of millions of properties, historical sales, mortgages, ownership changes, foreclosures, valuations, and other property-level records. At this scale, seemingly simple analytical questions such as "How many unique properties were sold after a foreclosure?" or "How much overlap exists between investor-owned properties and recently sold properties?" can require expensive scans, joins, distinct aggregations, and large data shuffles. This talk presents a practical architecture for building a large-scale real estate analytics engine using Apache Data Sketches and a commercially available property dataset such as ATTOM. We use property and transaction records as the foundation for a system that continuously produces compact, mergeable sketches representing different populations of properties. The system uses HyperLogLog (HLL) sketches to efficiently estimate cardinalities and Theta sketches to perform approximate set operations, including unions, intersections, and differences. For example, an analyst can ask how many properties were sold in Seattle, how many of those were investor-owned, or how many properties experienced both a foreclosure and subsequent sale - all without repeatedly scanning hundreds of millions of raw records. The talk walks through the architecture from bulk data ingestion and distributed processing through sketch generation, hierarchical aggregation, storage, and query execution. We will demonstrate how sketches can be partitioned by geography, time, property characteristics, and transaction attributes, then merged at query time to answer previously expensive analytical questions. We will also explore the practical trade-offs involved in using approximate data structures in production, including accuracy, error bounds, sketch size, mergeability, time-window management, data corrections, reprocessing, and determining when exact computation is still required. The goal is not simply to explain how HLL and Theta work, but to show how Data Sketches can become a practical building block for a Big Data analytics platform capable of processing hundreds of millions of real-world property records efficiently.

Brief Profile - I'm a Staff Software Engineer at Snap Inc., where I lead publisher-side ads infrastructure serving 400+ million daily active users. Previously, I worked at Cruise (GM's autonomous vehicle subsidiary) and spent nine years at Amazon building production distributed systems, including AWS IoT Events, a real-time stateful stream processing service handling millions of IoT messages per second. With 15+ years of experience spanning cloud computing, distributed systems, IoT infrastructure, and ML systems at production scale, I've had the chance to build across large tech companies, a high-growth autonomous vehicle startup, and hyperscale cloud infrastructure - giving me a firsthand view of how AI-assisted development is reshaping engineering teams at every stage.