I was struck recently by two different perspectives on Big Data momentum. Computing Research just published their 2015 Big Data Review in which they found continued momentum for Big Data projects. A significantly higher number of their survey respondents in 2015 are using Big Data projects for operational results. In a contrasting view, Gartner found that only 26% of the respondents were running or even experimenting with Hadoop.
If you dig a little deeper into the Computing study, you’ll see that it’s speaking about a wider range of Big Data options than just Hadoop. The study mentions that 29% of the respondents are at least considering using Hadoop specifically, up from 15% last year. So the two studies are closer than they look at first glance, yet the tone is strikingly different.
One possible explanation is that the Big Data movement is much bigger than Hadoop and it’s easier to be optimistic about a movement than a technology. But even so, I’d tend towards the optimistic view of Hadoop. If you look at the other technologies being considered for Big Data, analytics tools and databases (including NoSQL databases) are driving tremendous interest, with over 40% of the Computing Research participants evaluating new options. And the Hadoop community has done a tremendous amount of work to turn Hadoop into a general purpose Big Data platform.
You don’t have to look very far for examples. Apache Spark is now bundled in mainstream distributions to provide fast in-memory processing, while Pivotal (a member of the Open Data Platform along with WANdisco) has contributed Greenplum and HAWQ to the open source effort.
To sum up, the need for ‘Big Data’ is not in dispute, but the technology platforms that underpin Big Data are evolving rapidly. Hadoop’s open nature and evolution from a processing framework to a platform are points in its favor.