The “Datadogs” of tomorrow. How the Data Quality, Monitoring & Observability wave is building up.
Yes, Data Quality is a thing. Finally.
In this article, you’ll read about
- Last decade’s impressive rise of SaaS-based monitoring of cloud applications,
- Why Data Quality, Monitoring & Observability SaaS is following in these footsteps,
- The new “Datadogs” are coming: a wave of Data Quality & Monitoring start-ups is building up,
- How these start-ups tackle the market from different angles
Are you interested in curated, high-quality content around MLOps?
Follow me on LinkedIn to stay up-to-date.> Do you have feedback or would like to chat? Drop me a message!
Have you heard of Datadog?
The chances are high that you’ve worked with Datadog or, at least, you’ve seen the purple logo with the friendly dog carrying a graph. Datadog was founded in 2010 with the goal to reduce friction between application developers and system administrators. They set out to build a cloud infrastructure monitoring service (SaaS), including dashboarding and alerting.
With the explosion in cloud adoption throughout the last decade, they found themselves in a golden market and provided much needed monitoring for all the apps that were now deployed to cloud infrastructure. The rocket launched:
- 2010/2011: $1–2 million (seed round)
- 2012: $6.2 million (Series A)
- 2014: $15 million (Series B)
- 2015: $31 million (Series C)
- 2016: $94.5 million (Series D; ~$700 million valuation)
- 2019: $648 million (IPO; ~$8 billion valuation)
- 2022: Valuation of ~$50 billion at time of writing
It’s a true tech success story. Things just matched up: Great market, great timing, a great product that solves a tremendous pain. Of course, Datadog is a prominent example, but by far not the only SaaS-based cloud monitoring solution that took off during that time (have you heard of Dynatrace?).
What is big, hot and always worse than expected? Data!
Big Data is real, inarguably. First, it was Data Warehouses helping us tame it, then Data Lakes, then Data Lakehouses, now Data Mesh is the new hot thing. We went from centralization and preparation of data (DWH) to centralization and pushing preparation more towards the end user (DL) and covered all shades of grey in-between (DLH).
With the advent of the Data Mesh, we’ve seen a strong push towards domain-specific decentralization of data, its ownership and architecture. Data is considered a (co-)product that teams are responsible for and provide to end users and other teams. For example, a team that is responsible for the chat bot (product A) of a large e-commerce website is also responsible for appropriately handling and managing the associated data (product B) and to make it available to management via dashboards and other teams in processed form.
Naturally, the additional responsibility for treating data as a product poses a challenge for teams that are usually not specialized in Data Engineering. This means that many teams need to somehow get the job done even without the appropriate skills in place. This contributes to three trends we can observe at present:
- Rising diversity of responsibilities: teams are becoming increasingly cross-functional.
- The job market for Data Engineers has been heating up significantly — we’re seeing an extreme shortage of qualified candidates.
- Tools are becoming more user-friendly and low-code to enable data-savvy, but less tech-savvy roles such as Data Analysts take over responsibilities.
It’s not surprising that one of the key challenges that (not only, but even more so) cross-functional product teams are facing when dealing with the additional responsibility of offering their data as a product to end users as well as other teams is ensuring continuous high levels of data quality. Being confident about your data means that you need to continuously validate the expectations you have about it. Ideally, teams do so by (1) implementing various types of Data Quality Tests, ranging from simple uniqueness checks to assessments of drift in distributions. Resulting metrics are then pushed to an (2) alerting and monitoring system.
What’s new about this? Why now?
Data has been a big topic for quite a while now. So why is Data Quality Testing and Data Monitoring/Observability becoming really hot just now? There are 3 key reasons:
- First, the obvious one: We’re in 2022. By now almost every noteworthy company has realized that data is an asset that potentially holds large value.
- Which is why organisations have been investing tremendous amounts in setting up their data infrastructure for storage, transformation and analytics. The result is explosive growth of companies that provide these capabilities, such as Tableau (snatched up by Salesforce), Snowflake ($71 billion valuation at time of writing) or DataBricks ($38 billion valuation in 08/2021).
- This is the point where an organisation feels fit to reap the benefits of their data and the hard work that it put into making it properly usable. They get DBT ($4 billion valuation in 02/2022) to lower the entry-barrier for writing data pipelines to deal with the shortage of Data Engineers, they build the first dashboards and develop the first Machine Learning models at scale.
“Why do these numbers not add up?”
“Why is my ML model suddenly spitting out complete nonsense predictions?”
Yikes.
Now that data has actually arrived at the centre of the company, quality issues are bubbling up. At the same time, critical business decisions are made based on dashboards and models that run on data with known and unknown quality shortcomings. In addition, thanks to decentralized approaches to data ownership such as the Data Mesh, these quality issues are not with a single team or two, but they’re bubbling up across tens or hundreds of teams whose data sets have deep, deep spaghetti-like up- and downstream dependencies. Even worse, many of these teams might not even have an expert Data Engineer who can help them cut through the complexity and lift their pipelines to the next level.
Many organisations are exactly at this stage. This is why Data Quality & Monitoring/Observability is becoming hot right now. Some market research agencies expect this market to grow by a CAGR of ~20% until 2029, totalling several thousand billion USD globally.
The wave is building up. Get to know the new Datadogs!
Naturally, some careful observers and industry veterans have anticipated these problems and have founded start-ups that aim to help out there. Also, some world-renowned VCs such as YCombinator have made lots of early bets in this market (ref. the Airtable below). Let’s have a closer look at the “new Datadogs”. Since there’re some good, deep comparisons of the technical aspects of some of these tools, we’ll focus more on the high-level.
The following Airtable (better view via the direct link) provides an overview of some of the most notable potential rising stars in the field, including details such as funding as well as a very brief comment on their focus (no guarantee on correctness & completeness!).
Looking at these seven companies, besides technical aspects, we can observe four highly interesting trends:
Low-code & No-code
In the spirit of DBT, some start-ups in the field are focusing on making Data Quality testing easily accessible without requiring the deep knowledge of Data Engineers by offering low-code ways of defining tests (e.g. re_data, Soda, Metaplane (even no-code!), Elementary). Given the trends laid out above around decentralization of data and skill shortage, this is a very promising direction. Others keep the focus on Data Engineers (e.g. Superconductive/great_expectations, Databand).
Automated Profiling & Anomaly Detection
Setting up your own Data Quality Tests can be quite a lot of work. Also, your test coverage will hardly ever be complete. Some solution providers are battling this with automated profiling and running unsupervised as well as semi-supervised Anomaly Detection algorithms over your (meta-) data (e.g. Anomalo, Metaplane, Soda). Reliable automagical detection of issues in your Data without explicit rule-settings can be considered the Holy Grail of Data Quality Testing. Note that the “real” Holy Grail has never been found…
Two-pillar SaaS business model
Roughly half of the listed start-ups are following an Open Source/Commercial split (e.g. Soda, re_data, Databand, soon also Elementary and Superconductive/great_expectations). The Data Quality Testing part is developed as open source whereas the Monitoring/Observability part is marketed as SaaS platform. This model is especially interesting as OSS has proven to be an effective distribution channel to get a foot into the doors of potential commercial customers.
From Data Monitoring to Observability
The market is shifting its terminology from “Data Monitoring” to “Data Observability”. The latter has a broader meaning as it goes beyond “simple” monitoring. Besides low-code Data Quality Testing and Monitoring of generated metrics, some of the start-ups provide in-depth lineage information (e.g. Metaplane, re_data, Elementary). This allows for assessing the full depth of data issues and understanding potential up- and downstream problems. Moreover, some of the platforms support data issue tracking and aim to enhance collaboration when fixing data issues (e.g. Soda, perhaps soon also Elementary).
The Data Quality & Monitoring/Observability market has been gaining a lot of traction. It’s an exciting field to be in. Naturally, not all the start-ups that have been popping up will make it. However, the market has recognized the overarching problem that needs to be solved. A strong indicator for this is that large VCs have made bets on multiple companies in the same market.
We can be confident that sufferers of data quality issues have a lot to look forward to. There’s light on the horizon!
> Do you have feedback or would like to chat? Drop me a message!
Have you read my last article on why vertical prototyping is incredibly important to get value of out Machine Learning projects and how MLOps helps achieve this at scale? Have a look!
