Part 2 of 2: Solving the Small Returns Problem
Continued thoughts from “Solving the Big Data, Small Returns Problem in 2020 — Part 1 of 2: The Big Data Problem.”
Businesses know they need to collect data because it holds immense value. You often hear someone casually state, “Well, as I was saying, it’s simple. Just unlock the power of data to grow your business.” But it’s far from simple. It is extremely difficult. All of it. Especially all of the stuff that happens in between.
There’s palpable frustration and bewilderment in the market as executives greenlight millions of dollars of investment on cloud storage, log stores, and analytics platforms, yet don’t quite get the returns they were expecting. Artificial intelligence, machine learning, and deep learning aren’t close to the general-purpose miracle solution most think they are. They require highly skilled people who are removed from the day-to-day business the data is meant to help.
The not-so-secret is that whenever you want to do something of value with your data today, 80 percent of the initiative’s resources will go towards finding, connecting, onboarding, and preparing the data. Then about 18 percent of work time is used to craft SQL like queries, build dashboards, and in more sophisticated cases make training data to get basic machine learning algorithms working, (which requires extensive data science skills). That leaves about 2 percent for someone to actually make use of the data and spend time discovering something of note. This is why most don’t end up making use of their data. Out of the box monitoring and analysis tools leave them with a basic dashboard when they dream to do so much more. So it’s easy to see how returns are not aligning with investments.
The data warehouses and data lakes haven’t disappeared. Something a lot of people don’t understand is that we are still using technology that is 10 years old to build the future. There hasn’t been as much evolution in infrastructure as people think. We still live in the relational database and professional services world built by Oracle. And this world is built by extremely technical people for technical people.
Think about leading products today. The product fits a need, but to use the product you have to hire a whole team of technical people along with it to implement it. Or if you’re looking to migrate and move a lot of data in your log store, you’re hiring lots of professional services because you can’t hire in-house to manage the project. You buy 600 licenses for a data analytics product “anyone in your organization can use” but only three people can really use it. You buy 10 terabytes of log store capacity and only use half a terabyte. After months of getting your data in good shape to analyze it, you get maybe 5 dashboards out if it. That are already outdated. Maybe that equals out to $2 million per dashboard, not counting opportunity cost.
All technicians are humans, but not all humans are technicians. If we are going to build a new world, we should build it from the ground up and from a human-centric angle. We need to flip the model of thinking from data and tech-first to use case first. To make this possible, we finally need to get around to answering three basic, yet wildly complicated questions: What data do we have? Where is it? And how do we get value from it?
We’ve learned that having more data doesn’t equate to having better insights. So we need to collect data specific to our questions. We still have to work with legacy architectures and infrastructures that have been cobbled together over time, and data in various forms from different sources that were never designed to work together in harmony. So we need to be meticulous about where we keep our data and how we organize it, so that it’s visible and accessible.
And as far as getting value from data, we need to put the human element back into analytics. The analytics will only be as good as the person that asks challenging questions. That person should not have to have a technical background to do so. We need to enable those who are curious and want to explore their data freely and adventurously. Arming people with the tools they can actually use will be the catalysts that move analytics from iterative decisions to innovative breakthroughs.
To get there, there’s a lot of very hard work that must happen in between it all.
