Data Must be Owned by Business

The story repeats itself. When I talk to companies about their data initiatives, I notice how quickly they jump aboard the latest data hype train. Perhaps with the goal of becoming 'data-driven', implementing 'data mesh' or 'democratizing data', only to receive disappointing results.

We tend to believe that having a Project Manager responsible for a data product is the ultimate goal. The truth is that having a reliable dashboard with many sales graphs adds no value on its own. What we end up having is a situation in which significant resources are allocated to establishing a data initiative, but there is still a clear lack of genuine investment from key business stakeholders.

This reality makes successfully tracking performance difficult and frequently leads to the monitoring of technical metrics such as 'time taken to resolve sales issues' or, more concerning, 'time required for data to be available for the support team'. Any technical solution without a clear business aim, defined targets, and observable metrics risks becoming useless over time. This frequently results in instances in which businesses look back on their investments with regret, creating a sense of 'we wasted so many resources on these data buzzwords'.

Don't believe me? Look at this Reddit post with all the 'hot takes' from hundreds of data engineers: What's your controversial DE opinion? My disgruntled Data Engineer colleagues feel the same way.

Even in an ideal scenario, where your pipelines are pristine, dashboards run fast, requirements are clearly defined, and data is both clean and normalized, success doesn't seem to be guaranteed. As I see it, many organizations prioritize the idea of being data-driven over a genuine commitment to thoroughly analyzing the data and interpreting the numbers in a meaningful way. Then, this frequently results in a superficial engagement with metrics, leading to insights that fail to say anything other than the obvious and, many times, even overriding existing executive intuitions and objectives.

Most of the time, the best you can hope for is that a chart you created works to divert a middle manager's focus away from excessive meddling, rather than using the data to berate some sales and support teams for failing to meet arbitrary, non-analytical targets.

Additionally, the concept of positive business impact is commonly used to reinforce a stakeholder's decision that coincidentally (or not) aligns with the data. I refer to it as 'data gut feeling confirmation driven'.

It doesn't matter who gets the data; being a good DE means getting it to the person who can have the greatest impact on revenue and/or costs. That means you and your team must prioritize projects with the potential for significant impact. The engineering portion should be easy.

To drive the point home, we can consider product ownership in Software Engineering. A product or service is worthless unless it generates tangible, measurable business value. No one creates an authentication service, an e-commerce site, or a complex backend API service simply because everyone else does. These are always driven by a desire to increase revenue. The same applies to data engineering and data products.