Everyone is working on an “AI strategy” these days and BI is no different. The dream is obvious - imagine asking your BI analyst any question and getting an accurate and actionable answer that completely transforms your business.
Consider a fairly common BI request and how it escalates: What was our revenue last quarter? How does it compare it to the prior quarter? What about year over year? But these are all simply pulling some metrics and not offering anything insightful. What you really want to do is understand what this means and what to do next. You want to have an actual conversation with your analyst. Imagine being able to ask why revenue was lower than you expected. And then digging in to understand whether it was due to anomalies in certain regions or due to a different product mix. Now imagine combining this line of questioning with the ability to do web research. You’d be able to compare your numbers to public company peers and see how you compare. Beyond peers you can also look at how your suppliers or partners are doing and see if there’s any correlation with these broader factors. What if you didn’t even need to have this conversation at all and instead this analyst system would surface these insights automatically. Moving to this on demand model would completely change the way modern BI works - you get rid of all the stale reports and dashboards that no one looks at and instead surface insightful and actionable data. It’s no wonder everyone’s excited about the possibilities.
Unfortunately we’re far away. I barely trust the queries I write and I’m expected to trust some magical AI? There’s so much nuance and complexity in our businesses that we’re much better off investing in data quality before trying to layer on these AI systems. “Garbage in, garbage out” as they say. The current generation of AI tools are great when you can quickly get a sense of the accuracy of the response. It’s great as a chat interface when you’re trying to change a flight or receive some customer support. In those cases you’re able to have a proper conversation and iterate towards a desired outcome. It’s completely different when you ask for revenue for last quarter and get a single number, let’s say $10M. Is that right? Why wasn’t it $11M? Without already knowing the correct value it’s tough to know if the response is right. And this only gets worse with more complex measures. Revenue should be easy to calculate; churn and retention, on the other hand, have multiple definitions and good luck having the AI figure out the right one.
BI companies understand this and are trying to find the right way of incorporating AI into their products. One example is Seek. Their approach is to use AI to generate the query but then have humans sign off on it. Clearly that’s suboptimal since you’re both delaying a response and also interrupting the data and analytics engineer workflow so there’s also a mechanism to automatically approve the generated query based on a confidence score. Another example is Zenlytic. Their approach is to have the metric definitions be defined outside of the AI but then use AI tools to generate queries using these definitions. It’s a clever idea and taps into this need to have well defined metrics yet take advantage of AI-powered chat tools.
The above are two examples but there are countless others trying to find a way to incorporate AI into BI. We’re still in the early stages and it’s obvious that as AIs get more and more powerful they’ll be able to take on more of the BI burden. To take advantage of these trends companies need to invest in data quality and make sure their data is clean, well documented, and well modeled. That way they can take advantage of these tools before their competition. Vendor data we receive tends to be much higher quality than our own. Consider Stripe: they have a mature data model and understand the type and structure of reports their customers need. They invested a ton of effort in cleaning their data to make sure there are no surprises when customers pull reports. It’s this vendor data that can take advantage of the current generation of AI tools. At the same time given this data is so well structured, documented, and organized do you need AI at all?