NoSQL/Big Data Consolidation
One of my annual predictions at the Pacific Northwest Business Intelligence Summit was that there will be some NoSQL/big data consolidation. Some companies will be acquired, some will go out of business, and a few independents may merge. Some of these moves will be “soft” on all involved (the companies themselves and their clients) and some will be “hard”.
There are simply too many companies in the space (already) with dubious distinctive value propositions – derivatives of HDFS, scale-out commodity machines with high scalability and failover, use of MapReduce, fit for the cloud, open source model, etc.
The market itself is going to be huge. I’m in no disagreement directionally with IDC, who states the market for Big Data technology and services will reach $16.9 billion by 2015. This number cannot be reached without serious uptake from the midmarket as well as the enterprise market.
I am in disagreement with those who say that big data implementations thus far have largely avoided all the problems inherent in new technology – unfocused, untargeted, technology focused instead of business focused, etc. Those issues are mostly there so far and not avoided. Sorry.
There is increasing recognition of the need to focus these projects. While some have been abandoned or stalled out at the pilot process, others are refocusing and finding the budget to go forward. Ultimately ROI or transformative organizational change need to be delivered with these projects. The vendors that support this type of change will be the winners. I believe that many of the winners are present in the market now, but many of them are only being incubated, either within or outside of the NoSQL leaders today.
I delved into this subject of the big data space with Peter Goldmacher, Senior Analyst with Cowen and Company and co-author of “Quick Take: Hadoop World Observations: One Step Close to Mainstream Adoption.”
In that report, in making the case for legacy replacement, Peter talked about open source being free and that “Better technology at a lower price is a bad combination for the legacy providers.” While true, I still see an ultimate place for both – technically as well as practically. I also believe the large vendors have yet to make their ultimate plays into NoSQL. If done well, they can do quite well, but the pressure is on again in those executive boardrooms.
Who will make the right decisions and who will make the Sybase decision to turn away being SAP’s primary partner because they didn’t want to develop row-level locking?
Peter addressed the practical issues head on when he listed the “good news for the incumbents:
1. Incumbents have dominant account control
2. IT departments are slow to change
3. Contracts with existing vendors are nearly impossible to get out from underneath”
It reminded me of the movie “Who Killed the Electric Car?” where this same type of confluence of factors ended up being the culprit.
I thought some of the other points in the report were spot on, like “Entire categories like ETL, integration and data warehouses exist solely to make up for shortcomings in old technology and IT deployment decisions.” I have said for years that data warehouses exist due to shortcomings in the operational systems and that many functions, master data management most prominently, are picking up for those shortcomings in the operational environment and beginning to shave around the edges of the data warehouse’s value proposition.
Change is clearly afoot in information management. As Peter’s report says “The market won’t support 50+ Big Data infrastructure companies.”