Data management has been around for decades (DAMA DM-BOK), but it only truly became a priority after the 2008 financial crisis. Regulators demanded large banks to provide documentation on how they calculated their financial & risk metrics.

These regulations (like BCBS239) also outlined the need to appoint data owners, implement approval processes around changes to calculation logic, and add quality controls at various levels in the data value chain. This marked the advent of a new role: the Chief Data Officer (CDO). Gartner later labelled the initial CDO’s and their focus on data management for regulatory compliance the CDO v1.

As the economy started soaring again, CDO’s applied some of their learnings in regulatory compliance to the other critical data processes in their organization, with many other industries quickly following their lead. This proved to be valuable as new regulations, like GDPR & CCPA, required similar data processes to be implemented (like the right to be forgotten).

Competitive CDOs also started shifting their focus to the offensive aspects of their data strategy. They revamped their data stack (powered by the cloud), and made it easier to experiment and build data products and services.

The next big challenge for CDOs will be all about operationally managing digital products & services that are in production. As part of their target operating model, modern CDOs are building strategies to identify problems that require attention across 4 areas: infrastructure, data, models and applications.

Infrastructure monitoring is there to ensure data workloads happen fast, and have failover capabilities.

Data monitoring is there to make sure all the necessary data is there, on-time and fit-for-use.

Model monitoring to help understand when models are degrading and need to be retrained.

Application monitoring to make sure the application that serves the data product is snappy and works without hiccups.

At the center of this innovation is the data platform team (data engineers) and the CDO organization (target operating model). They need to work as a team to identify, prioritize and resolve data issues.

Data engineers, for example, will be much more involved in helping monitor data and pipelines so that they, as well as analysts and SMEs can spot issues before they cause any damage. The CDO organization will be there to help define the target operating model, and decide on which data issues are most critical to resolve first.

As organizations are automating with data, data monitoring and operations will become crucial as flaws in data create risks (reputational, operational) as well as costs (cleaning, patching and backfilling data).

At Soda, we're passionate about data in production. We help data engineers and CDOs monitor datasets and send meaningful alerts to the relevant teams so they can take action.

Get in touch if you want to learn more!