How many layer-cake data flow, architecture and process flow diagrams have you seen? In almost every one, data is invisible — except in a cylinder (or trash can) or as an arrow. What’s in the arrow? And where are all the arrows going? All of these diagrams are devised from a technical point of view, not a practical one. They don’t address the one fundamental question:
What’s the point of all this data?
That’s one of those questions that has one simple answer and a million more complicated ones, but the simple one is:
To inform decision-making.
What is missing from virtually every diagram I’ve ever seen is: how data informs decision-making. One would assume that after all the transfers, cleansing, integration, modeling, optimization, scale-up/scale-out, some miracle happens and analysts are rewarded with knowledge like an oyster exuding pearls. But that isn’t what happens.
When the triple threat of Big Data/Cloud/AI hit like a tsunami, attention to how data informs decision-making was sidelined. Instead, all eyes were on ‘data scientists’ who with their programming, statistical and subject matter skills combined were the new data alchemists. The problem was the domain of their investigations was very different from what others in the organizations were doing. In fact, the ‘old’ way, using data warehouses and Business Intelligence actually became derided as old-school. It wasn’t even discussed. After all, it didn’t deal with the scale of big data, it didn’t involve mysterious practices like Machine Learning or Recurrent Neural Networks. And existing analysts were excluded from the party because they didn’t know Python, or R or SQL.
No, they used tools that didn’t even require coding or scripting. Business Intelligence (BI) is a terrible term, because it isn’t always about business, and intelligence is a term that implies something no technology has. Yet when analysts in organizations like businesses, government and NGO’s, for example, need to understand what has happened, what is happening and what they should do about it, they turn to tools grouped under the heading of BI. There is a vast array of types and capabilities under that heading.
Which brings me to Shadow IT. From the report, How much is Shadow IT costing you?:
Some recent trends highlight the financial and security implications of Shadow IT:
- Gartner’s studies found that between 30 and 40% of IT spending in large enterprises goes to Shadow IT while Everest Group puts it closer to 50%.
- Core found Shadow IT has exploded by 59% due to Covid-19, with 54% of IT teams considering themselves ‘significantly more at risk’ of a data breach.
- NCSC found 21% of organizations experienced cyber events due to a non-sanctioned IT resource.
- NCSC also found 60% of organizations don’t include shadow IT in their threat assessment.
But is Shadow IT always a bad thing? When you consider the need to proliferate BI tools throughout an organization, perhaps a surge in business user adoption is a good problem to have. From a BI point of view, then, ‘Shadow IT’ actually facilitates the transfer of information and decision-making.
Now, if you want to earn the disdain of a big data poser, just mention ‘BI’. To a big data person, BI represents everything that is wrong with computing. What they won’t mention, because they don’t understand it, is the true magic of BI. It’s how people think. They will tell you it can’t scale. It relied on prebuilt structures of only aggregated data. That BI ‘cubes’ have to be rebuilt from scratch for even minor changes. Actually, all of these things are true with previous generations of BI, but it has all changed because, ironically, cloud computing changed it. Using cloud technologies, new ways of providing BI eliminate all of the old drawbacks. The emergence of cloud-native data warehouses put the emphasis right back on BI.
And here is a dirty little secret — what do you think data scientists use to manage the results of their runs? BI.
Arranging things by attributes/dimensions/hierarchies and treating them as navigation points and filters, BI instantly calculates numbers (aka metrics). Using BI, one can arrange data in multidimensional arrays with just point-and-click. No code. Just click. And the real magic happens when you can make subsequent analyses based on the previous ones and follow a trail, that you can navigate forward or backward, drilling into detail, rolling up, pivoting and other BI kinds of navigation. Try that in Java. Or SQL.
My take
Business Intelligence is a fungible term. When I was first introduced to it in 1991, the need for this type of functionality was still met by mainframe Decision Support tools like Express, SAS, Comshare and a smattering of otters. But as client/server computing made its way into the data center, four independent vendors jumped into the breach — Cognos, Business Objects, Arbor Software and Microstrategy. Only the latter still exists as an independent company. The first three were absorbed by IBM, SAP and Oracle, respectively.
There are now dozens of BI companies, but one fundamental shift happened in the 21st century. Analytical tools with a strong visualization orientation emerged (not at the same time): Tableau (now part of Salesforce), QlikSense, PowerBI and Domo. “The rapidly evolving digital competencies in new business models can have an excellent impact on the global market,” states Fortune Business Insights, in a report, titled Business Intelligence Market Size, Share & COVID-19 Impact Analysis. BI market size is expected to reach $39.35 billion USD by 2027, exhibiting a CAGR of 8.5% during the forecast period.
So in summary, those busy in big data and data science have missed the point. They don’t really understand what people do for a living. If you want to use data to run your business you need interactive, navigable data tools that work the way you think. I’ll get deeper into that in my next article.