Big data is not new. It’s been around for decades and it has been big for quite some time. So how did big data get so sexy all of a sudden. Well, like many things, it did not get so interesting because of the traditional data professionals who have historically been collecting and using data. It got interesting when someone started applying data techniques in new endeavors. Endeavors which no-one had paid any attention to. Think of Moneyball and the story of Billy Beane using numbers and analytics, to produce above average performance with a below-average budget and lack of star power. It got a lot of people’s attention. Among others, one of them must have been Nate Silver, who not only developed a model for Major League Baseball but went on to accurately predict the US election results with almost 100% accuracy in 2008 and 2012.
These kinds of events put data modeling on the stage as a necessary tool in decision making. But what has really catapulted big data to become the Queen of the Prom are a number of underlying trends that have multiplied its importance.
1. Proliferation of computing devices
We can credit Apple and the iPhone with the rapid penetration of the smartphone. And the distribution of computing power has skyrocketed with the ARM processor being designed into all sorts of internet-of-things. So intelligent, interactive and mobile computing is growing at a staggering rate as new applications continue to be created.
2. Development of new processing power techniques
We can credit Google for publishing papers on how they processed data, and we can credit the release of Hadoop as an open source system in 2006. The availability of these new techniques and tools made it possible to process large amounts of data relatively quickly and economically.
3. Fast-changing markets
The pace of change in the last decade has increased to a whole new level. We are seeing large amounts of technological disruption that is constantly changing our behaviors. From how we read books to how we organize meetings are all in flux.
And it is the third characteristic that now compounds the importance of big data. With all this change occurring, it becomes vital to determine where to put our efforts. If you have useful and actionable data, in real time, you can get an edge. So it now becomes important to put in place data processes that monitor and measure what is happening. And to allow use to predict where things are headed, as well as better react to our predictions. If things were happening at a slower pace, there would be much more time for a human to allow the big picture to fill in. But with the pace of change today, we have to start applying more data modeling or we will miss the next turn.
From a sales and marketing perspective, the focus becomes on determining our product-market fit on a continual basis. What fits today may no longer fit tomorrow. The use of big data amounts to keeping many ears to the ground, in a host of different contexts. This will help us win new customers faster and adjust to new market conditions as they evolve.