5 operational challenges in adopting data mesh architecture
Lessons learned to overcome operational challenges along the data mesh journey and empower organizations to access, analyze and operationalize data at scale, successfully.
In our previous blog, we looked at the common technical challenges facing organizations that have embarked on the process of adopting data mesh architecture, and how to address them. Now let’s look at the top five operational challenges, which if not properly negotiated, can be equally detrimental to success.
1. Failure to automatically update data catalogs with every release
As previously discussed, data product discoverability is part of DATSIS and a key element of data meshes. Most data meshes employ a data catalog or other ad hoc mechanisms to make their data products discoverable. A data catalog can be used as an inventory of data products in a data mesh, most often using metadata to help support data discovery and governance.
Any mechanism used for discoverability must be kept up to date to protect the usefulness of the data mesh. Out-of-date documentation is often more damaging than no documentation at all. For this reason, we recommend that data meshes employ a docs-as-code scheme, where updating the data catalog is part of the code review checklist for every pull request. With each merged pull request, updated metadata enters the DevOps pipeline and automatically updates the data catalog. Depending on the data catalog, it may be updated directly via an API, pulling JSON files from an S3 bucket, or using other methods.
2. Treat your data lake team like heroes
Most companies struggle with scaling their data analytics operations because they insist on using the right tool for every job – even to jobs that don’t need or respond best to these tools For example, a solution where ETL pipelines pump data into a data lake has a finite capacity to deliver value. In many ways, such a system is monolithic and does not scale well. A data lake excels at ad hoc queries and computationally intensive operations, but the centralized nature of the lake can make it hard to include pipelines from every domain in the company. On the other hand, a data mesh can be built to include an almost arbitrary number of domains. The drawback, however, is that very computationally intensive operations can be time-consuming. Use the right architecture to solve the right problems, and recognize the value of your data engineers. They play an important role. If they don’t feel valuable, or if they feel their jobs are threatened by a data mesh, they will act against it - even though the architectures are complimentary. The solution is to use the right architecture to solve the right problems and recognize the value of your data engineers.
3. Data mesh requires extra work by domain teams
In a data mesh, domain teams maintain ownership of their data and create data products that expose that data to the rest of the company. If an engineering team handles the data mesh work, their capacity for other engineering work will decrease – at least at the beginning.
However, the alternative is often a tightly coupled data pipeline, which is inherently fragile since data changes at the application level can result in erroneous data being fed into data lakes and subsequent defects in reports produced by data engineers. In addition, troubleshooting these defects is time-consuming and frustrating. For example, when the source of the defect is found to be something like a change in a field type or the way a particular field is used, there can be a lot of friction between the engineering and data teams.
4. Shadow data analytics and mandates
Many companies suffer when their current data architecture and data team have reached their scale limits. Growing delays when adding new ETL pipelines, generating new reports and running ad hoc queries can lead to a situation where consumers simply resort to their own sources of data – something known as “shadow data analytics”.
Such a phenomenon is understandable. And yet, while people need data to do their jobs, the practice often bypasses any sort of data governance or data security and tends to result in erroneous use of data.
When faced with shadow data analytics, one approach is to mandate that all staff use the sanctioned data analytics system – Mandates might be effective Over time however, mandates exact a toll in employee autonomy and initiative. A better approach is to build a comprehensive data architecture (often including data lakes) that is accurate and responsive to consumer needs to the extent they won’t be tempted by shadow analytics at all. Such a system has the following characteristics:
- Follows DATSIS principles and the “independently deployable” rule as described in our previous blog (Problem #1)
- Enforces SLAs for minimum performance requirements for all data products
- Employs a transparent process to request, schedule and implement new data products
5. Failure to accurately evolve data products
As a company evolves, so does its data – often in unpredictable ways. Changes typically fall into two categories:
- Changes in company domain structure
- Changes in the structure and nature of the data itself within each domain
Data meshes should be built to adapt to these changes. Adding domains to a data mesh is simple: first add data products, and then ensure they are discoverable in a data catalog or similar product. Finally, build dashboards or other display resources as necessary.
Removing data products occurs less frequently, is slightly more difficult and is usually done manually. If another data product consumed the removed data product, it needs to be examined. For example, does it still make sense to expose the consuming data product to users? And how are consumers of that data product notified about changes or complete removals of data products? The answers will be different for each company and should be considered carefully.
Pursuing technological fit
A big advantage of the data mesh architecture is its scalability and ability to start small and grow as demand grows. Early mistakes tend to be small mistakes and teams learn through experience how to manage increased demand for data while avoiding the political and technical pitfalls inherent in providing actionable data to business users. Indeed, a technological fit is one of the major considerations for any organization’s efforts for adopting and implementing a data mesh architecture.
To mesh or not to mesh?
Data lakes and meshes are excellent solutions for different problems. To be able to embrace data mesh successfully, it’s important to understand the organizational data needs and accordingly restructure data platforms, redefine the roles of data domain owners and overhaul structures to make data product ownership feasible.
