Data Observability for Petabyte-Scale Game Telemetry

2K Games is an American video game publisher and one of the world's leading publishers of interactive entertainment. The company has a diverse portfolio of games created for PC, consoles, and mobile platforms, with blockbuster franchises including NBA 2K, Borderlands, and many others. 

When these games launch, they generate massive volumes of telemetry data— remote information collected from player movement, in-game actions, purchasing behaviors, and interactions with game elements or other players. Every click, move, and purchase inside a title generates data that powers decisions across marketing, LiveOps, and game development. This data helps identify and solve issues, optimize game mechanics, and enhance player experience.

With terabytes to petabytes of new data streaming every few minutes, visibility and accuracy become critical. 2K’s data engineering team then turned to Soda Cloud to gain continuous observability into these high-volume pipelines. Sid Srivastava, 2K's Director of Data Governance, Data Quality, and MLOps, was the strategic leader driving this transformation. 

Currently, with Soda’s metric monitoring and anomaly detection, 2K has a single dashboard that tracks data quality automatically across every title. Engineers can compare quality trends between game versions, carry over thresholds from previous releases, and ensure that each new dataset meets expected performance standards. 

The challenge: volume, velocity, and the visibility gap

Game telemetry is essential for game development in today's gaming landscape. It provides stakeholders with insights into player interaction, game performance, and user engagement: 

• Engagement data tracks session duration, level completion rates, and player progression through the game. 

• In-game analytics provide heat maps showing where players are dying most frequently, indicating areas that might need patches or difficulty adjustments. 

• Studios receive detailed maps showing player activity patterns, allowing them to respond quickly to issues. 

• Marketing pipeline data adds another layer, tracking campaign performance, offer effectiveness, and player acquisition costs.

For marketing use cases, 2K processes terabytes of data daily. For LiveOps scenarios, where games are live and being continuously monitored and patched, data volumes approach petabyte scale during peak periods. Refresh rates are equally demanding, with near real-time data arriving in five to fifteen-minute intervals. This high-velocity data collection is particularly intense during the first sixty days after a game launch, when player engagement is highest and marketing decisions are most critical.

Adding to the complexity, different 2K games run on different platforms and have their own tech stack. Some operate on Databricks, while others use Snowflake. Real-time streaming happens via Kafka and Confluent, with on-the-fly transformations using Spark. Multiple downstream consumers depend on this data, from Tableau dashboards to marketing systems to studio analytics tools, each with their own requirements and expectations for data quality and timeliness.

Therefore, 2K’s challenge was to scale observability over these massive amounts of telemetry data so it could be safely used to power critical business functions.

Before implementing Soda, 2K's data team relied on an internally built anomaly detection tool. However, this tool generated a high volume of false positives, which created severe alert fatigue and eroded trust in monitoring. Also, the cascading failures and inconsistencies could impact downstream marketing and operational decisions.

There was also no unified data quality framework, and testing was inconsistent across the organization. Individual engineers created ad hoc tests based on their own judgment, or in some cases, skipped testing entirely. There was no visibility into what checks existed across projects. And, perhaps most critically, marketing teams were often the first to discover data issues.

To close this visibility gap and catch problems before data got to its consumers, 2K needed a solution that could:

• Continuously monitor high-volume pipelines across Databricks and Snowflake.

• Detect and surface anomalies before they reached downstream teams.

• Standardize data quality checks across hundreds of datasets and game titles.

• Scale to handle new game releases without adding manual overhead.

The solution: observability at the speed of play

Rather than building complex testing frameworks that require constant maintenance, 2K wanted an observability layer that would: 

• automatically detect anomalies using ML-powered pattern recognition, 

• provide a single source of truth for data quality status, 

• alert engineers proactively before business impact occurred, and 

• scale effortlessly as new games and datasets were added. 

This approach represented a fundamental shift from the manual, reactive model they had been using to an automated, proactive model that could keep pace with the velocity and volume of their gaming data.

To solve these challenges, 2K Games implemented Soda Cloud in mid-2025. Soda proved to be powerful yet simple to implement. The metrics monitoring tool could handle 2K's massive data volumes with minimal overhead, process them efficiently, handle refresh cycles, learn from historical patterns to reduce false positives, and provide AI-powered anomaly detection that adapted to each game's unique data patterns.

2K previously used Soda Core. The migration to Soda Cloud took roughly 60 days (from POC to full rollout). The transition was smooth because teams were already familiar with Soda’s open-source syntax. The upgrade was driven by the need for advanced features like anomaly detection, metrics monitoring, and executive visibility via dashboards and alerting. 

By monitoring data in near real time, Soda now helps the data team detect and fix anomalies long before they affect downstream teams. Current monitoring includes: presence checks to confirm that data arrives as expected, null checks to ensure critical fields are populated, freshness checks to confirm 5-15 minute refreshes, and volume anomaly detection to flag unexpected row count changes.

Record-Level Anomaly Detection

Soda AI’s proprietary Record-Level Anomaly Detection (RAD) provides instant, broad coverage across every column, row, and segment — without the need to create a single check. After datasets are onboarded, the built-in backfilling and backtesting analyze historical data in real time to reveal patterns and trends. 

Soda AI’s record-level anomaly detection analyzes 1B rows in 64 seconds, learning from feedback to improve precision and reduce false positives over time.

The algorithm then develops a thorough understanding of what “normal” looks like, being able to flag unusual records with high precision, and alert the right person automatically. Also, when users mark results as expected or unexpected, each piece of feedback helps the algorithm learn, refine its predictions, and adapt over time.

At 2K, when Soda detects an issue, alerts are generated and classified as P0 or P1 based on impact. Issues are logged in the engineering backlog with defined SLAs. Engineering addresses root causes and adds new checks if needed to prevent recurrence.

Automated checks result in big time savings, catching issues such as data not loading correctly or having redundancy, which helps prevent a lot of backfill.

The impact: from data quality management to full data observability

The transformation at 2K was operational and cultural. Before Soda, marketing teams discovered issues first, engineering responded reactively, multiple failure points accumulated, and alert fatigue meant real issues could be dismissed. After Soda, engineering has full observability, marketing receives only verified data, issues are caught before propagation, and intelligent monitoring reduces false positives.

2K Games achieved the balance: near real-time data delivery with high confidence in accuracy. Soda Cloud dashboard became a source of truth, offering real-time quality status across all games, alert routing integrated into engineering workflows, and executive-level visibility into quality trends.

“I think a lot of the positives that we saw were mostly along the lines of not seeing downstream impacts, essentially. And that was a big win for us, in terms of preventing multiple layers of failures and trying to catch those issues upfront.” – Sid Srivastava, Director of Data Governance, Quality, and MLOps at 2K

Looking ahead: governance maturity and MLOps

2K's governance team was being formalized during the Soda implementation, and the tool became a catalyst for structuring their approach. Building on this success, 2K is now expanding Soda’s observability capabilities into ML pipelines and automated reporting.

Looking ahead, 2K’s plans include:

• ML Observability: The company is building an MLOps team and sees alignment with Soda's roadmap. With the NannyML acquisition, Soda is expanding into ML observability.

• Automation of Reports: The team is building custom Databricks-based dashboards using Soda's API to provide game-specific views, executive reporting, and segmented views where each team sees only their relevant checks.

• Data Contracts: Data contracts are a key goal. They plan to leverage this concept more deeply as their governance framework matures. 

Get in touch

Schedule a demo with the Soda team to find out how much you could optimize your data quality strategy across your entire data ecosystem.