Data mesh has the potential to solve a big data problem. The book, Data Mesh: Delivering Data-Driven Value at Scale, authored by Zhamak Dehghani, the creator of data mesh, is a guide for practitioners, architects, technical leaders, and decision makers on their journey from traditional big data architecture to distributed and multidimensional approach to analytical data management.
In this episode, Zhamak and Jesse talk about the book and unravel some of the complex and critiqued areas of data mesh, and shed light on how to succeed with data mesh.
This episode is part of a trilogy of conversations with Zhamak and Jesse, as they delve into the different sides of data mesh. Before you dive in, do make sure you’ve listened to their first and second conversations.
You're listening to the Soda Podcast, and this is the Data Dream Team series. Coming up on the next episode is part two of our in depth conversation with Zhamak Dehghani. Zhamak needs little introduction as the founder of the data mesh - a paradigm shift to how we manage data, and author of the critically acclaimed book, data mesh, delivering data driven value at scale. In this episode, we're continuing on from part one. Listeners, if you haven't listened to part one, stop, rewind and hear Jesse and Zhamak talk through the reality, the dream, and the execution as they bring us up to speed on what's been happening since they met in season one. Now in part two, Zhamak and Jesse are going to delve into her book in which she guides practitioners, architects, technical leaders, and decision makers on their journey from traditional big data architecture to distributed and multidimensional approach to analytical data management. Let's pick up where they left off.
So we're going to, in this podcast, go through some of the points that we either disagree on, or I think what will probably happen is it will be a reflection or perhaps a refinement of that that I didn't understand quite correctly. But that said, once again, this episode is not an introduction to data mesh. When I originally pitched this episode to Zhamak, it was, Hey, let's have a conversation on the finer points of this. And this is what I think is missing out there... I don't think Zhamak wanted to do another introduction to data mesh. She wants to have this conversation of let's get deep and let's go deep.
Yes, yes, yes. I'm excited. I really enjoyed the conversation last time, and I enjoyed actually listening to the conversation last time, because when you listen to both people talking at the same time, it's a very different experience and I enjoyed it. So I'm looking forward to it. Let's do it. I'm ready.
So, where we left our intrepid hero last season, was at the end of 2021. You were still in the middle of writing the book and navigating through Covid. Where are you now?
Yes. So I was still hoping to have the book published by the end of 2021, which was wishful thinking. So from then to now, I finished the book. I published the book, so that's behind me. I took my time off after taking almost three months off ThoughtWorks, my day job. Right this moment, I'm in the studio having a conversation with you, but during my time off I guess I'm reflecting on where we've come from with regard to data mesh.
I think I can't escape from it. I thought that at this point in time, data mesh is behind me and I've moved on to the next thing. But I think it has become my calling, because I see there's still a big gap between the reality, the dream, and the execution. So where I am right now is daydreaming. A lot of daydreaming about the future, and closing that gap between the vision, the dream, the aspirations, and the execution, the reality, the path to get there.
It's funny you say that, because I've talked to some of the people at ThoughtWorks who are 10 years out from their book, 20 years out from their book, and they said, Oh, I wrote the book so I'd never have to talk about it again. And 10 years on, 20 years on, they're still talking about it. So we'll see if that's a level of success you get.
Yes, I've been in fact warned to be careful how I be recognized and kind of remembered, I suppose. To not become this - what's the expression, single pony show? I can't remember, there's an expression - that data mesh doesn't become my world. And there are still a lot of problems in technology to be solved. And I suppose I have the privilege of this great vantage point to work with a lot of clients globally and find those patterns and find those gaps.
But I must say, I'm not there yet. I feel there is a great potential for us to innovate and bring this idea to life. Clearly, the industry has spoken that there is a pain point, there is a problem, and the solution has resonated with them. So I don't think I'll be able to escape the gravity of data mesh and the solution space or problem space around it for quite a while. Hopefully, in 10 or 20 years, hopefully I've moved on by then, but at least for the next, I can see for the next 2, 3, 5 years, that that's going to be the focus.
Well, I don't think you'll ever be a one trick pony, if that's the expression you're trying to think of, because I looked at your resume. There's all kinds of things, but there's this weird thing that happens in show business and books, and that is, everybody remembers that thing that you are known for rather than, well, here's what I want to be. Here's what I want to be known for.
So now let's talk a little bit about the book itself. My quote for the book is, and it's in the book itself, so you're kind enough to choose it, is: "data mesh is one of those concepts where we ask ourselves why we aren't doing this all along. This book will be our guide." And those of you who haven't read the book, or maybe you've been hearing all about Zhamak or data mesh, and you just haven't dug into it.
One of the things I think you did the best job I've seen in a book in a while was your diagrams. I think you did not just, here's a visualization, there I did it, put something together. I think you put a great deal of effort into your diagrams. And I could see that because I make diagrams for a living, too. So you could see, oh, somebody put a lot of effort into making sure that they labeled that right and that it's clear. And so I think you did two great ones. There were two examples that I saw, 1-2 and on page nine and 1-3 on page 10, that I thought really made it pretty quickly to be able to show that.
Yes. It's the interplay of the relationship of the four principles.
Yes. So once again, we're talking about figure 1-2, Four Principles of Data Mesh and Their Interplay. I thought this did a really good job of breaking down something that's pretty complex. You could have spent a lot of time, maybe 10 pages, explaining this diagram, or you can say, here is what I mean in a diagram and saved a lot of not just trees but conceptual overhead of doing this. So you did a great job with that one. The other one that I saw was Data Mesh At A Glance, and that was Operating Models to Data Mesh Principles in figure 1-3. It broke down here we have our data products, our business domains, our self-service. So great job with those.
Oh, thank you.
If nothing else, buy it for those diagrams, because did I assume that correctly, you spent a lot of time on those diagrams?
Yes. So I always try to convey a concept in different ways. And for me, I'm a visual learner. And I guess I'm a conceptual thinker and to be able to conceptualize complex or demonstrate complex ideas in figure and shapes and their relationship, I think it's really necessary. So usually, when I think about a complex topic, I think about how I can depict this? And I have this giant whiteboard in my office that I start drawing different ways of drawing things. The first diagram that you mentioned, I had probably multiple versions of that to show the interplay of why we have these four principles and why all of these are necessary, not just one or two. And finally landed on the one that I drew.
And somewhat, there's a selfish, I guess, reason behind these diagrams, that they help me. The process of drawing them helps me unpack the complexity, unpack the blind spots, and actually understand it myself better, to clarify it better. So I'm glad that they resonated.
One of the things that I think, since both of us are authors, there's a certain amount of joy we get from hearing people actually reading our stuff. It is genuinely good to hear that all that time we spent, it is a significant amount of time to write a book.
Yes. Reading reflections, criticism, or constructive feedback, it's the most wonderful, I think, feedback of the work you put in. There's a particular gentleman, he writes on LinkedIn every Friday his reflections on the book. And I feel, oh, I feel heard when he publishes his reflections. And some of his work and some of his words raises a lot of good questions and conversations.
So yeah, I highly recommend that if people are reading it, share back what you think and what's missing or where the conversation needs to go. One of my intentions behind the book was elevating the conversation beyond what it is and really moving on into how to make data mesh accessible to a wide range of organizations, not the ones that have a lot of money to spend on the technology.
So another thing to know as you're heading into this book - and I say that as somebody who's spent a lot of time on this, written my own book on a very similar subject - this is a really meaty book. It's gonna take you a decent amount of time to digest it. You're going to have to reread things. It's not for the faint of heart. That said, we won't go through every single piece, but I agree with the vast majority of the book.
Let's talk about execution. Coming back to your diagrams, you did a great diagram in figure 6-4, Macro Drivers for the Creation of Data Mesh. And this is kind of, I think, your metric for who is data mesh applicable to? To that end, I don't think I've ever seen anybody ask you this question: Is data mesh for everyone?
Definitely not. There is a complexity, inherent complexity in building distributed systems and even more complexity building a data-oriented, distributed system or ecosystem. So unless your business has an equal inherent complexity, I wouldn't pick data mesh as a starting point.
I would say it's the same advice as I would've given, you know, 10 years ago, seven years ago, people wanting to do microservices. If your business is small, if the number of diverse sources and teams where the data originates from, the diversity of use cases for your data are small, perhaps a monolithic, centralized solution of a lake or a warehouse with that long pipeline, value stream pipeline that we talked about, perhaps that's okay. That's okay to get started with.
If you are an organization that has the complexity, but you are waiting to buy a solution, again, this is not a good place to start. This is not a good point in time to adopt data mesh, because you won't buy it. And so I would again say it's for the organizations that have technology at their core. You know, they're investing in building, integrating, and have technical teams, long-standing technical teams that have the ability to build data products, share data products, utilize those data products in their applications. Sure, data mesh is for you. But if you don't, and you just wanna buy and integrate, now is not the right time.
And I think it's really important to - and I think this will actually help with the whole perception of data mesh being this total silver bullet - it's you saying these sorts of things. Hey, this isn't for everybody. This is for complexity. This is for high volatility. So for people who are listening to this, in figure 6-4, it's great expectations of data, it's great divide of data, it's scale, it's business complexity and volatility, it's discord between data investments and returns.
And so when Zhamak is just talking about the complexity, this is what I see in my consulting business as well. If you tried out Hadoop - perhaps, let's make it technical for people - if you tried out Hadoop for a small data problem, it's massive overkill. In a very similar way, if you tried out data mesh for a low-complexity problem, things are working relatively well, hey, maybe this will actually cause the inverse to it, where you will explode out some things that don't need to be exploded out.
And when I talk about exploding, you explode them in terms of complexity, in terms of numbers of technologies, in terms of numbers of people you need. And all of a sudden, the bureaucracy is expanding to meet the needs of the expanding bureaucracy. You haven't actually improved your returns. You are simply bigger, with more things, with more moving parts, with no value behind it. And that's really one of my worries there.
Completely agree with you. And I would say if you have the complexity, if you have the ability to execute on a kind of technically heavy infrastructure and foundation, but yet your organization is not ready to make data part of every team's agenda, again, you're gonna over-index on technology and solutioning and really not solve the people and team problem.
I see a lot of courageous leaders that, you know, have a top down mandate for embedding data-driven solutions in many use cases, many teams, empowering the people with regard to education. I think that's a wonderful start and that's a necessary element of organizational readiness for adopting data mesh.
Because data mesh is not supposed to be a complex solution, bounded within the walls of a data team. It's supposed to be part of this cross-functional, domain-oriented ecosystem of teams that have aligned themselves with business and bring data as a capability, as a concern, as an outcome, getting value from data as an outcome, from those teams and to those teams. So organizationally, leaders need to go beyond outsourcing data to the data team and the CDO for them to be ready to really adopt data mesh.
So that brings me to perhaps one of the first things that I disagree with you on. You talk about this execution being, companies should bring their own compute, but I believe that we should both have compute being provided and, if necessary, bring your own compute. Could you talk more about why you talked about that?
So bring your compute. You've got to refresh my memory. Is that from the infrastructure point of view, as in the teams bringing…?
Infrastructure point of view, yeah.
Yeah. So I think from an infrastructure point of view, there is an ideal space, ideal point that you wanna get to, and there are steps along the way, right? And there's a spectrum of infrastructure. I think the ideal space that you want to get to is where business domain or cross-functional domain teams are just focusing on the business problem at hand, right? The necessary elements of software and data that are directly impacting the outcome.
So, let's put it in an example from the book. If you are the team that is producing the personalized playlist, music playlist or podcast playlist, for all of your customers, all you want to focus on is tuning the machine learning or classification model that's ultimately generating those playlists, focusing on those playlists and modeling of those. So you're focusing on modeling, writing code, modeling your both machine learning model, modeling the data that you're using, and gathering, connecting to the data sources that really feeds that model, right? So that's the part you focus on.
All of the infrastructure that enables you, infrastructure that creates, that leads to creation, maintenance of the playlist as a data product, infrastructure that runs that machine learning model, the compute that runs the transformation engine for aggregating data from different sources, from the listener and play events and all of these other sources, that compute and that infrastructure ideally is provided self-serve to you. And you shouldn't worry about it, right? You just should worry about what is essential for providing this curated playlist.
So I think that's the ideal stage that we want to get to, that self-serve infrastructure that as a developer, as a data developer, as a playlist developer, allows me to just focus on building this playlist, right, what's unique to that domain. And part of that compute, part of that storage, part of that transformation engine, there's a lot of pieces, right?
So I think we're probably on the same page from the infrastructure perspective, but the reality is that organizations collect technical debt, they have competing infrastructure solutions and competing platform solutions and what platform fits them all probably is a difficult, ideal state to get to. And it, we need a healthy marketplace of tools and technologies within the organization that enables these domain teams.
And it might be that in some cases that particular domain had evolved their own, I don't know, compute engine or, you know, workflow management engine, or whatever it is that they use. And they become very much integral to what they do, so they don't wanna give it up and change their code to use something else. So there might be somewhere on that journey of harmonizing the infrastructure or they want the diversity and some of the tools that they need are not provided. So they may need to bring in their own pieces of infrastructure into that self-serve infrastructure.
But hopefully, if there is a healthy marketplace, kind of an open source marketplace platform provided by these teams, almost always the best solution wins, right, the solution, the compute engine, let's say, that is flexible, it's easy to use, have good documentation, is self-serve, and hopefully domain teams get to the point that they don't have to worry about the infrastructure. Just worry about domain-specific functionality. Is that the part of the compute that you were referring to, when you refer as to compute?
Yeah. So that actually brings me to perhaps the other manifestation of this, and that is: How do these downstream teams choose the right technologies? And I think that's really, so you mentioned the whole infrastructure, and infrastructure's a whole different thing. And then I think at the very end of that, you start saying, okay, now expose this with the right technology. And I think that's where I've experienced this a lot. Those downstream teams are not the ones that are very good at choosing that right technology.
Yeah. The self-serve platform of data mesh is a product, is probably multiple products, right? And having a platform product owner, which is one of the roles that I talk about briefly in the book, is really important, because you're right that the end users that are using, they're focusing on curating the playlist, perhaps are not the people to be in charge of running a cluster of, I don't know, Spark or whatever compute they're using, or even choosing what could be the best technology right now.
That's not their area of focus, their subject matter expertise. Their subject matter expertise is something else. So then really it's the job of the platform people or infrastructure people to act as a product owner, a product manager, and talk to those teams and really understand their needs and provide the technology that they need.
And that's a really hard problem, because if you imagine your platform as this one kind of centralized surface of tools, and there's just one team that takes care of this big surface area of technology, that team in itself becomes a bottleneck. So then you have to start breaking apart that team into their infrastructure domain-oriented teams, as in, you have a domain within that platform team that is focusing on just, perhaps, the developer experience of building and managing or managing the life cycle of data products.
You have a team that is focusing on security and encryption, and so on and so on. And I have given some examples of that in the book to remove that new bottleneck that you've created by considering platform as a product for internal best developer experience. But I do, in short, agree with you that the domain teams, infrastructure is not their area of expertise. And it shouldn't be, for good reason. Their time and effort and expertise should be spent on solving that data-driven business problem.
And I think that's my definition of data product in a data mesh situation or motif, and that is, it's data exposed with several different technologies in the way that the consumers of that consume it the best way possible, where you're not forcing them. I know in the past or even now, there're companies that are trying to figure out - how do we make it so that the user doesn't even know what they're querying from and that sort of thing, and make some sort of intuition about that.
I'm not sure if that's going to be very possible in this sort of situation, but I think it does take a little bit of education of your business users: Hey, when you do this sort of query, this is the manifestation of that data product to do, to use. And we may get there at some point where it's gonna be automatic, but I just don't think that it will ever be a hundred percent. I think it's a smart person that has to deal with that.
Yeah. So data products, as you said, they're first and foremost encapsulating access to data that is modeled around a particular business domain for a diverse set of analytical processing use cases. So first and foremost is a domain oriented construct, is a logical, business-focused construct.
And then secondary, it has interfaces for access that are designed for analytical workloads, whether it is a SQL-based query for generating insights a report on top of it, or it is file-based access or, and maybe hopefully a completely new model of access that pushes competition and processing into the data and just gather the insights that we're looking for. Like APIs, for example, to give you information about the shape and distribution of the data without really seeing the data. Without really querying directly what the data is, you can query insights from the data.
So these are all modes of access to that data. And I think it's a very still fairly nascent space in terms of evolution, this multimodal analytical, natively designed for analytical workloads set of APIs. If you build a data product today, probably those APIs look like SQL end point or a file end point or a park-a-file [?] end point or an event endpoint or something along those lines. But I don't think that really leads to the vision of the mesh. I think we need to elevate above that so that you can push more sophisticated processing into the data product, and just get the results that you care about rather than move the data.
Because a lot of the data sharing APIs today are still centered around data movement, data movement, like give me parts of the data so that I can process it. I can process it, right? Not that you, the data product, the source can process it. So, I'm excited. I think that it's gonna be an interesting space if we set the bar to say, if you move fast, you know, if you fast forward and these multi-model data sharing APIs that you talked about can directly solve, let's say it's training a machine learning model without moving data into a feature store, right?
I can train a machine learning model distributedly on a mesh of data products, knowing where the features of the data products, the sources for my features that I'm training machine learning model are, and I can just express my training distributedly and the data products will just train my machine learning model in distributed fashion. I think we've hit an objective, which is about not keeping moving data around and centralizing data.
It was probably six years ago, seven years ago when I started talking about real-time training of models and people looked at me funny, and I think that's where we're going. If we're doing this right, we will be doing our model training in real time. And we will just have some kind of cut over that we do of dropping a new model, test it out, make sure it's ready for production, and put it in.
Yeah, we do it real time or we do it distributedly.
Yes. So when going deeper into these people problems, I think it's worth pointing out, data mesh won't inherently fix certain data people problems. So as I personally look at data engineering teams, for example, they're not doing Agile correctly most of the time. They're saying they're doing Agile. They may be doing a daily scrum, but they're not dealing directly with their customers like they should be.
They took all the fun parts, all the things that they wanted to do, and that whole, "oh, and by the way, in order for all this to work, you need to involve your customers." So we have these sorts of issues that I think that will continue with them of "I want to do the fun parts of data mesh, but I'm not going to do those harder parts."
I also think that data mesh is going to be so people intensive, I worry as well that we'll have companies going in with these under-resourced - resource being number of people - will have this thing that needs 20 people, let's say, and they're trying to gut it out with 10 people and it is just going to not end well, because I think data mesh requires even more people executing even better. Is that your take as well?
Yeah. I agree with your point around people just picking up rituals. It's really easy to do an Agile ritual. We do a standup, and we have this backlog, and as you said, we have these screens, we've got Agile. And not really read the Agile manifesto and see what is behind these rituals. So we just copy the rituals and we assume the outcome comes from the other end.
And I think data mesh would become, is turning into that, right? We use the language of, oh, I've got data products and I've got domains. I mean, I see tools from actual cloud providers that now expose that language. Like, these are your domains and these are your data products. And here you go, you've got a data mesh. So I agree that if you don't deeply look into the motivation and the values that we try to nurture in organizations and really inherently change not just the language and not just the ritual, but the belief system, the cultures behind that belief system, and the values, yeah, we will use new words, but we won't see the impact.
In terms of your second part of your question around the number of people, I don't necessarily think you need more people, but you need to, I guess, repurpose or change the roles and responsibilities and what the existing people do. So I think if you say that I am data-driven today, and I have a lot of data that I've wrangled and put into the lake and I've got a team responsible for it, and the output that I tend to generate is the same output of this with the same number of people, I just wanna reconfigure that, you probably don't need to add more people. You just redistribute these people somewhere else. The output remains the same, maybe the velocity of that output changes.
But if you say, actually I've got a ton of new sources of my data, a ton of new use cases that I want to enable, then yes, I agree that you probably need more people to satisfy this new demand around data, but that's not necessarily because you're doing data mesh, but because you're actually changing one of the main parameters of the output that you are trying to produce. So it’s a combination of more people doing data, but more of the existing technologists that you have doing data work, not bringing data people into the organization, but really empowering, educating, and repurposing some of the outcomes that existing people are working on.
Well, that brings me to a question or perhaps a question and a statement. My experience is, let's take a vanilla software engineer. And I've taught a lot of vanilla software engineers, usual enterprise people, and teaching them anything about data distributed systems is just, it's not that easy. It's not that it isn't just easy, it's also going to take time, but it's also a whole issue of, they just don't care. This isn't their thing.
And so when I talk about more people, I don't think you'll be able to take some of those people, move them laterally, or move them to a different position. I simply foresee that you'll have to be hiring that new person that can actually do it.
Yeah. You said something key in there, that they don't care. So if your software engineers don't care about data, you shouldn't do data mesh. What does that tell us? So if a software - and I have been a software engineer, and honestly, when I was a software engineer, I didn't care about data, because I cared about, let's say, I'm building a service around onboarding new listeners, right? I care about the uptime of that service. I care about the APIs for that service to make sure that the transactionality, I can bring this user on board, send the emails, run the workflow for onboarding of the listener.
That's all working in one nice transaction. And the data behind it, the state of the yes, the profile of the user and their address and contacts and so on is underneath in the guts of some database, and that's the last worry that I have, right? I worry about the execution, competition, and the APIs, the surface APIs of that service.
So naturally, I don't care about data, but what data mesh tries to do is to say that software engineers should care and think about how they can do a wonderful onboarding of a listener using data. So, is there a way that we can make that onboarding process more intelligent, smarter, more effective using the data? And if that becomes my concern, now I am a data-driven part of the organization. I'm making decisions. Now I care about the data because I care about, oh, where in the process I'm losing the user? I need to collect metrics and data and analyze it to understand where users get bored and they fall off the registration process. What are the profiles of people that we're onboarding so that we can target that profile? So, unless that software engineer has an inherent need for that data.
Okay. Zhamak, could you read that excerpt for us?
“In my mind, future generalists will be able to work with data and create and share data through data products, or use them for future engineering and machine learning training when the model has already been developed by specialist data scientists. Essentially, they use AI as a service.
As of now, the majority of work requires specialization and requires a large amount of effort to gain expertise over a long period of time. This inhibits the entry of generalist technologies and has led to a scarcity of data specialists.”
So you talk there about these non-specialists. How do you think non-specialists will be able to start working with data mesh?
So I guess I go back to two points. Today, a generalist developer must know continuous delivery. I mean, continuous delivery is a key part of how we build software, and if we don't do it, we build software that is very hard to maintain, right? So it's one of our basic tools. Or conversely, another example, building APIs, building services is, it's a job that a lot of the generalist developers do. I think working with data, developing data APIs, modeling data for analytical use cases, embedding analytical use cases and applications has to become a key part of app development.
It has to become an essential ingredient of how we build applications. How do we bring the generalist to just kind of cross that I suppose, chasm? I would challenge them with the right problem. Like, I go back to the answer that I gave a little while ago. I know how software developers think, I've been one myself. We love to solve problems and we love to get to the outcome. Like you said, we wanna have a solution in production. And a lot of us love continuous improvement. There is always a better way to do things. So I would just pose the right question and the right problem to my generalist teams, to guide them to think about delivering software very differently.
For example, let's think, take another example: Let's say if you're a retailer and you're building a logistic application that wants to find the timing, the route to deliver products from your warehouse to your stores. You have a choice of building a rules engine or using a rules engine and write a bunch of if and else rules, if this is time of the day, or if it's from this warehouse to that shop to, you know, take these actions, or you have a choice of solving that as an optimization problem: What is the optimized, local optimizable global optimized points for, you know, approach for routing with given different parameters?
And then, hence, that becomes a data-driven way of solving the same problem that perhaps is more adaptable and solves a problem that you didn't even envisage because it looks for patterns in the data. So then let's say, if we give the developers the right problem, then there is an element of education and then there is an element of self-serve technology, right? So I think for software engineers of the future, they need to learn about at least application of machine learning, application of analytics in their domains.
And that has to be, you know, I think that's a non-negotiable part of an application. We can't build applications without using data anymore, not just storing data as a state, but using the data. So maybe the next generation software engineers will get that education in their university as part of the basic, you know, basic programming skills.
That's a scenario I've been thinking about, too, as much as I talked about a disturbing trend I've been seeing of zero database training, I don't even know SQL, maybe we're going to see this flip side of people saying, oh, actually, that was foundational. You need to know this. And on top of your continuous integration, now you need to know how do you expose your data correctly? It's more, maybe it's something as simple as here's how you construct a JSON payload that goes in.
Yes. And for developers to be able to do their work again, we've got to also remove the friction from the tooling. The tooling we have today is very much targeted to that specialized end of the spectrum and not a developer that's coming out of the bootcamp. Right! So I think the tooling needs to change to use primitives and language and vocabulary that is really centered around the concept of the data products or application of the data and data as you talk about. Right? So I think the level of abstraction needs to increase to make it so easy for developers to have that fast, rapid feedback in using data in solving their problems. And as you've said, the education.
And now we get to the point, I think, where you and I disagree, but will be interesting to check that. There's a quote you have in the book, "A monolithic or centralized technology solution leads to centralized points of control in teams." And I disagree on this. I disagree that a centralized technology, whether that's, maybe it's Spark, maybe it's something like that, or back in the day Hadoop, I don't see that that's a centralized point of control.
I think that that was a way for people with some of their power trips, their previous power trips to continue to exert control, where if there was a DPA, that DPA was worried that they were going to get fired the next day. And so they would exert the same level of control, the same sort of plans over this and use data governance as a way of saying no. And so what I think is, I still believe that we need centralized teams.
We need a centralized data engineering team, and I still believe that we will need monolithic systems, not just because they can do some things, general purpose, but it's also about an economy of scale. That economy of scale would extend all the way to license prices, where if different teams are using different technologies, you won't get an economy of scale there. Skills of people - people will no longer be able to use another team's thing because they had too much leeway or even the hardware usage. What say you?
I would say I completely agree with the economy scale and I would say you want to optimize, you know, in the areas that are really not essential to your business, right? So you wanna optimize on infrastructure in terms of the scale economy. And yes, that leads to a centralized team that's providing assured infrastructure like Spark so that, you know, other people don't have to worry about it, right? So I agree with that.
The only thing I would say is that even those solutions, even platform teams, even non-business critical infrastructure teams, have to work on an open interface kind of policy so that we don't end up with that accumulation of power that we can't change vendors anymore. We're stuck with this one technology and everything falls apart. So there is a cost that we incur by saving money by using one technology for everyone, right? One team, one technology, self-serve, great. And then we would pay the cost later on. We pay the cost through locking and not being able to shift and not being able to move to the next generation technology.
So my, I guess, addition to that is, centralized technology is great as long as it provides you open kind of standard inter-fixes, that you can replace them with something else and you don't pay the very hefty price later on.
I think that that's key and I don't think that's as well appreciated as it should be. Going deeper into the technology side, for example, I think that Kafka's API and binary protocol will outlive the technology itself. It provided a, probably a standard API that we're going to be talking about in 10 years. It may not be Kafka on the back end, but there may be that same programmatic API, perhaps even binary API there.
Now, there is one part that I don't think you covered in the book, and that is, how do you handle joined or derivative data products? Put a different way is, you have team A creating a data product, and we have this other team down the line that says, well, I need a join of that. Who in a data mesh is responsible for that?
Yeah. I refer to it in a couple of places. I refer to them as aggregate data products, and then I immediately say, be very careful, do not have many of these aggregated products, probably you don't need them. So I would say, yeah, joins and kind of forming these you know, aggregate data products is a necessary part of the mesh.
But we’ve got to think about, is this aggregate data product that we're generating correlating data from different sources, has some inherent business logic and business value in it, or is this just for I don't know, simplicity and, is it is an optimization of technology or is there a real inherent business, core business problem being solved by this aggregate?
If it's the case of this correlation and aggregation inherently having business rules, complex transformation that you don't want to replicate in five other places you just want to do it once, and there is a value that you can directly derive from that aggregate, that's a data product in its own right, and it needs to have a team to look after it. And the choice of where that team lives or who that team is, it could be a team that is aligned with one of those key sources or the outcomes of the sources, or a team that is more aligned with the outcome that we are trying to get, or the value that we get from that data product, right? So it could be any of us or a completely new team we set up.
So let's say, you know, as an example, we want to have some sort of an insight around customers. So information about a customer comes from many different touchpoints that the customer has. But maybe customer attrition or customer acquisition, it's a new insight that joins and aggregates this data, but it adds a layer of sophistication and complexity, right? A computation that we are now layering on top of that simple aggregate. So maybe there is a customer acquisition team or a customer attraction team or attrition team that is looking after that data product.
But if this is just a matter of, well, we're just joining customer from order and joining customer from payment, and we give a more holistic customer, I would say, let's be careful with those, because we're just adding more things that we have to maintain long term that don't really add much value. And push those joins to the knowns on the edge of the mesh. So, because federated kind of join and query is an inherent functionality of your mesh, you can run it from anywhere. You can run your joins from even the reporting system that is finally reporting, generating that report.
If that's a simple join and it's just a matter of optimizing the times that the join runs at the time that the join takes to run, I would say push that still to the nodes and perhaps optimize when the join runs on the edge node so that you have some sort of an async join happening before the report refreshes or something along those lines. And be cautious with this over-ambitious kind of, you know, aggregates. Does that, do you think that makes sense to you?
I would love to say yes, but my experience says, doing a join, even a two-way join, an end-way join, is just so complex. So if you look at, let's say a Spark cluster and they're doing a join, that's the majority of what your network traffic, your disk traffic is doing. So if you start to say, okay, now let's do that n number of times, let's do that same join twice, three times, four times, five times. Boy, that makes me scared because I'm thinking, yeah, that's a lot of process that's a . . .
So why don't we talk about ideal, and then we’ll think about what we have today. I would say, ideally, we want to get technology to the level that an end team and the end organizational impact shouldn't be optimized around the performance of the network and the disk. I think I should put a team up to generate any data product, because there is value, there's business, complexity and value in that, not because I'm saving traffic.
But I completely agree with you that technology we have today is optimizing for disk and network, right, and computation. So here we go. We have another, I guess, a space for innovation that the technologies are smart enough to realize that, oh, I've done this join three times and there's a cache. And you know, that level of centralized infrastructure that sits underneath it can do all the magic and the hard, solve the hard problems so we don't push that machine optimization to the human organization.
There’s another section of your book, let’s have you read that, I think it’s really relevant here.
"Data mesh by default embraces the fact that organization complexity leads to multi-platform, multi-hosting environments. To support this model, data mesh offers an approach to platform architecture that composes services with standard interfaces. So data can be shared across different services and hosting environments."
So my argument here is, I don't think data mesh actually solves a multi-plane data platform issue, it just moves it elsewhere, kind of to what we were just talking about is, it doesn't solve a join. A lot of smart people have been trying to figure out how to do joins faster. It kicks the can further. Do you think that this multi-plane, multi-platform kicks the can further?
I think it forces, I don't know if it kicks the can further, but it definitely is a forcing function for the infrastructure providers to solve the problem. Right? So today, the infrastructure folks, let's say cloud providers that provide a long list of data infrastructure tooling are incentivized, those tools are designed around the incentive of, bring your data to me. Bring your data to my cloud, I host it, and I will give you all these wonderful tools that you can use to get value from your data.
So data mesh challenges that incentive, it says, well, can we live in a world, that data can remain in control of the organizational unit that is, you know, most apt in managing that data, they're in the best position to manage that data,and yet be able to get value access to that data, get value from that data.
And very quickly, if you extend that logic, you get to this point that that organizational unit is not under the control of one cloud, it’s actually, it's two organizations with trust boundaries, with two different infrastructure underneath it, right? So then it becomes a forcing function for technology providers to solve a new problem, to solve the problem of data sharing across hosting environments, right? They don't have to solve that problem today, and there's no forcing function for them to even think about that problem.
Well, we just see it with Google Dataplex, right? They say, okay, if data mesh is real, now they elevate their product saying, actually, your data can be anywhere, and I give you a control plane. Of course, it's controlled by Google, but I give you a control plan for accessing that data. So it is, maybe you can describe it as pushing the can somewhere else, but it is a forcing function for us to reimagine and redesign infrastructure to solve a new problem that we didn't have to solve necessarily before.
So I think somebody sitting here would be thinking Zhamak is either from the future or lives in the future, but I live in the now. And what do you say to that person?
I say you are very right. According to Strengthsfinder 2.0, I am a futurist. <laughs> It is one of my strengths, I suppose, according to them. But living in the now, I think let's, you know, let's, we don't have to solve the global scale data mesh problem today. We have to solve an enterprise or even a domain within an enterprise scale problem. And let's look at the technology we have today, let's find the ones that work better, let's keep pushing your vendors.
And if you're a vendor, let's keep reimagining your products or your product strategy to bridge the gap between this future and the present. And I think with present day, the data mesh implementations are fairly primitive. They don't have all of the sophisticated distributed machine learning training models. No, they don't. And you're right, they do have aggregated data products that try to optimize for the performance of the join and create organizational structures that unfortunately formed because of that, and that's just a stepping stone toward that future. But keep an eye on the future, but take small steps, think big, move fast with what we have today and incrementally get to that point.
I would add on to that try to think in the, in this future sense and try not to paint yourself into a corner.
Yeah.
There's a thing I've been seeing a lot lately and thinking about a lot lately, and that is the two-year tenures of people in either companies or even in vendors, for example. It really is a huge disservice, because you get people who don't see the end result of what they're trying to do. So I think this is a pretty important key.
You've worked at ThoughtWorks for 10 years, so you haven't seen one cycle, you've seen several cycles, and you can see, getting that experience really gives you a different view. You and I were talking before about - since both of us are in consulting fields, something like this is pretty much impossible to do if you worked at a single company.
For example, the Spotify model - the Spotify model worked at Spotify. For a while. But once we start to say, well, will it work here? And will it work here? You can't really get that data. You basically have to be a consultant like us and see these trends across the world. I work with companies all over the world. You work with companies all over the world, and we see this isn't a trend relegated to San Francisco Bay Area, nor is it in Germany, in manufacturing. You see these things that are all over the world, and it's only once you start to see this, the forest through the trees, then you can start to say, oh, that thing, there's a thing here. And we need to put more time and effort into this.
I completely agree with you. I think the human brain is a pattern-matching machine. And depending on where the purview and the scope of the exposure of that human brain, we find patterns within that scope. An engineer in an organization of course finds patterns in that organization. A consultant that works globally and goes from one industry to another industry and one client space to another client space starts seeing patterns that emerge at a much more macro level.
And if you're a connector, connectedness is my other superpower, apparently, according to Strengthsfinder. So if you can kind of connect the dots, you will see the patterns that emerge. And then we have to go back to those engineering teams and product companies to really solve different parts of that pattern, right, that has emerged.
My first half of my career was focused on R & D deep tech products. I have, you know, contributed a bunch of patterns of design, networking algorithms and networking protocols. So it was very much a product-focused life, which gave me appreciation for deep technology from, bottom up from hardware. I worked on hardware up the stack and the complexity that exists there and the problems you have to solve. But the second half of my career was a consultant for the last 10 years. And that gave me appreciation for problem-solving at a very different scale, at an organizational kind of human ecosystem scale.
So let's reflect on what you've done, now that the book is out. What's your favorite section or part of the book?
Probably, if I put myself in the shoes of the audience, where they can get most value would be just the first part of the book, which is really going into the why, or the first two parts of, the why and the how.
Personally - so that's from the audience's perspective, as in, just read the Prologue, <laughs>, you'll get the gist of what, or the first chapter. I'm proud of being able to convey a very complex idea with an example, which I owe to Martin Fowler, my mentor, one of the reviewers of the book. He really pushed me to think about describing a complex idea with a simple example, to get people excited to read more, to want to read more. So I think that's probably my favorite part of the book, and I rewrote it multiple times.
From the, kind of the engineer at heart perspective, which is a selfish perspective, I think my probably favorite part of the book is the architecture part, when I go into the details of the architecture and technology.
Oh, it's so hard, Jesse. Now you make me think about all of the parts of the book, and I can't really pick one, because I don't want to say, oh, people's side, organization, of course, I'm really happy about that. And I hope people read that. It's a hard question. Can I pass?
There's no pass.
There’s no pass?
We put you in the hot seat for a reason.
Oh!
You said bring it on, and I brought it and there you are.
Hmm. Yeah, maybe I'll go back to the first answer that I gave. The example, just the example. This Daff, Inc. Company, don't you think it's a wonderful company. I actually saw a colleague of mine tweeted and said, are they hiring? You know, I use this hypothetical company Daff, Inc. So I think that company, wherever that company appears in the text, is a favorite part of the book, and I want to go work there.
Okay. Well, you can start that company. So one thing I heard, as you were talking through your writing style, I found it interesting, a few things. I wrote mine basically sequentially, but you wrote your first part at the end. Why did you do it that way?
I didn't intend to, I had these wonderful reviewers that, as I said, Martin Fowler was a prolific writer and he's an, he's an amazing writer. I mean, I have been encouraging Martin to write a book about writing technical content. He really challenged me, because I did the same thing, I had this logically sequenced structure when I started: Why, what, how, how organizationally, how technically.
But he really challenged me to think about how to start the book so that it really captures the essence of data mesh. If somebody is not interested, they get the essence, and also paints a picture of what it is in a way that hopefully makes everyone who opens the book interested to read the rest of the book. So that's why I ended up rewriting the beginning of the book again and again at the end. But I think it was a result of their challenge.
Well, the book's the better for it, but I don't envy the work that has to go into that.
Yes.
You think you're done and, oh, by the way, let's go back and do that again. It was too, it wasn't fun enough the first time.
Yes. I think as a writer, I definitely, if I give one advice it would be, pick the best authors out there as your early book reviewers, if you have the opportunity, because you learn so much. I mean, I learned so much in the process and there’s still, like, so much to learn. I mean, I'm a novice for sure. But I came out of it a completely different writer.
So tell me, do you feel accomplished?
No, hardly, hardly. I think it's just the beginning.
What's the biggest lesson you learned about yourself, and what has this taught you about yourself and your belief system?
Yeah, I think writing the book was very interesting for me. I learned the personality or the state of mind that I was carrying with me when I wrote the first chapters by rereading what I wrote. I don't know exactly what I discovered, but at least I discovered the state of mind that I was in, and some of it I liked and I kept and some of it I absolutely hated and left behind. I realized, for example, when I was writing, I was very much attached to the past and writing a lot about the past and the problems of the past, and I think I left a quote in the book that says, "Today's problems are the results of solutions of the past" or something along those lines, "The solutions of the past leads to the problems of today."
So I think I was very much for a long time stuck in that past and criticizing the past and pointing out the problems of the past. And I was reminded by my wonderful reviewers that I'm generally an optimist and a futurist. So let's just stop focusing on that, and actually let's focus on the future and the solution and get off this critical high horse that I was sitting on.
So I think, I don't have one, this concrete answer, but I would say it's fascinating when you read what you write in this state of flow, state of, you know, just stream of consciousness coming out and put those words on the paper and then come back, and can I read them and say, "Oh, I don't really like that person who wrote those things, you know, she's judgemental!" And kind of change that. So I definitely learned about my state of being.
It's great that you talk about flow. I don't think I've seen many tech authors talk about flow, but when you get into a state of flow, it is just so beautiful. And then you go and you try to recreate that same situation. You think, well, why am I not in the state of flow immediately? And maybe that's part of the burnout that I experienced.
I completely agree. If I had sat there today to rewrite that work, I wouldn't be able to, it was just a fascinating, wonderful space that I was in for this year, year and a half. And I worked through the holidays. Like I, you know, I was visiting my mother, my sister in Australia last year during the holiday. And I would spend a couple of hours with my mom or an hour with my mom, and then I would go up and I'd just keep writing for a few hours. And yeah, it's just your whole world, everything that surrounds you is about that book. So every signal that you hear, everything you hear and read, just kind of channels itself into the writing and kind of influences your writing.
And it, it is just a, it is just a fascinating space to be. I thoroughly enjoyed it. Even though, as you said, very painful long hours, no holiday. But it's a super creative space, and a learning space. Like, writing is just the best way to learn about yourself, about your subject matter and everything around that subject matter is, you know, and if you're a learner you would really enjoy, I think, writing.
Well, Zhamak, that brings us to the end of our second talk. I hope people really enjoyed this. I appreciate how openly you share. And I love the passion that you bring it with. Thank you so much for being here.
Thank you, Jesse. Thank you for the honest conversation and wonderful questions that you posed. I enjoyed it.
Wow. Thank you, Zhamak, thank you, Jesse. As Zhamak said, we're talking about data mesh and that's its own thing. Zhamak has planted the seeds, but the forest in the trees will grow by other people and other people's work. And so it's over to all of you - data practitioners, architects, leaders, and decision makers to take data, mesh the concept, the book, and make it yours. We'll be keeping our eye on Zhamak and what she gets up to next. It's bound to be something phenomenal that will help us find a common ground to solve the problems that many are facing when it comes to good data. Because we know when that happens, that change is really possible.
To hear more great conversations on building data teams, visit DreamTeam.Soda.io and listen to your peers and professionals that can support you as you build your data dream team. We'll meet you back here soon at the Soda Podcast.
Listen to our first conversation with Zhamak from 2021, Data for Everyone, or her second conversation with Jesse, Reflections.
