I was preparing a year-end review session with my team at Saivo and went through all my notes from my conversations with startup founders and investors throughout 2023. Here are the trends that distilled from all those interactions.
Despite the excitement around Generative AI, many startups and growing companies are still grappling with the same fundamental needs in analytics:
LLMs are pushing self-service to new heights. We can now ask a question in natural language, and an AI-powered apps create the query for us, run the query against our database, return the resulting values of that query, and can even create nice visualizations. I am seeing many players in the BI space building a conversational UI/UX, from new and modern ones such as Zenlytic to bigger ones such as Thoughspot.
If we only use LLMs to write queries against a badly designed data schema, or if we don’t provide the right context, these models will generate inaccurate and inconsistent results. AI needs to be paired with the following pieces:
If you want an intro to the Semantic Layer, I strongly recommend this series of articles by David Jayatillake.
In 2023, at Saivo we extensively tested Semantic Layer implementations like Cube’s and dbt’s. Our preference is increasingly towards dbt-MetricFlow.
As we implemented the dbt Semantic Layer in many projects, we saw the huge potential (in autonomy, flexibility, and scalability in analytics), and we were very hopeful about a quick adoption in the market.
Yet, when I talked to data leaders, even if they agreed that the centralized source of truth for metrics is very powerful, the concept of the Semantic Layer was unknown to most of them. We are at the Early Adopters phase in the S-curve. However, for the Semantic Layer to gain broader adoption, two things need to happen:
In my discussions, about 6 out of 10 companies using a BI tool are on Power BI. About 3 out of 10 use other major BIs (e.g., Tableau or Looker), and the remaining 10% use lower-end or modern platforms such as Data Studio or Metabase.
Even if new entrants in the market offer flashy features, customers have invested so much in creating the assets on the BI (charts, dashboards,…) and training their colleagues to adopt the tool, this makes the switching cost incredibly high. The only driver to switch is cutting costs — more on that in point 5 below.
2023 has been a difficult year for fundraising, especially in the growth stage. This has led to tighter budgets, especially in data initiatives which are a second-order need for the business.
When I speak with decision-makers, conversations often echo sentiments like, “This is not a priority for us” or “We need to bring the costs of our data infrastructure down”. They don’t need vitamins, they need painkillers, understandably so.
The macroeconomic indicators projected for 2024, such as interest rates staying almost flat, don’t suggest significant optimism for a substantial upturn in the venture market. Consequently, I expect continued focus on unit economics and profitability, which means a strong focus on ROI for everything related to data. Will this scenario lead to consolidation in the market, as many people say? Maybe, but I hold a nuanced view on this — More on that in point 6. In either case, vendors and service providers need to be ready to demonstrate tangible and high value to businesses.
Interestingly though, the current climate may have a counter-effect. Startups that want to raise have to show strong metrics, maintain meticulous reporting, and illustrate their unit economics with great clarity and accuracy. When I chat with partners at VC firms, a recurring theme starts to emerge: “We need our portfolio companies to be super diligent and effective in their reporting”. In this situation, data becomes a painkiller…
As Jim Barksdale famously said, “There are only two ways to make money in business: One is to bundle; the other is unbundle.”
The question in the data industry is: where do greater opportunities lie? Is it in building “all in one” data platforms (as Microsoft is trying to do with Fabric) or in fostering “best of breed” yet narrowly focused SaaS solutions? Many data leaders have offered insightful perspectives on this debate. I recommend this great article by David Jayatillake.
From my conversations throughout 2023 with startups and small organizations, I see a preference for more integrated, bundled solutions. This trend resonates with the economic realities discussed in point 5 — bear markets and budget constraints.
While I am certainly not qualified to make predictions on the market dynamics, my opinion is that the modern data stack needs to be simplified, especially for small organizations. If you ask me, I would advocate for a leaner data stack like the following:
You don’t need more than that.
Earlier in 2023, GitHub CEO said that “AI is lowering the cost of building great software”. I foresee this impact in the data space as well. The cost of creating data assets (metrics, charts, dashboards,…) will be decreasing rapidly with Gen AI and there will be an increasing number of data artifacts within organizations. This is undoubtedly beneficial, but it’s not without its challenges.
Imagine the following scenario: More people creating new metrics (some of them already existed), creating dashboards (that are rarely consumed),… This situation may lead to a certain degree of chaos in the organization. So, what’s the solution? Like it or not, it seems inevitable that governance will become necessary. This idea might seem contrarian with the ethos of agile, ‘move fast and break things’ startups. However, in my conversations in 2023, there is a growing call for this need, “We need to bring more control and better collaboration in our data analytics”.
Yet, we need to approach data governance more efficiently, without burdening startups with costly subscriptions for specialized tools. The solution, I believe, lies in harnessing the very technology that’s driving this problem — AI itself.
Looking ahead to 2024, it’s clear we’re all in for a challenging year in the data industry, especially for startups. We’ll need to be flexible, think creatively, and keep pushing forward. I’m really interested to see how we all, from small startups to established companies, tackle these challenges and keep growing in the data world.