The middle of May saw me road tripping to not just one, but two Enterprise Search Summits to tell some war stories in doing search at scale. I attended my first ESS back in New York in 2010, and I was curious to see how the conversation had changed in the intervening three years. Would we still be talking about the same things three years later?
So first the bad news…
The Search industry as weve known it for the past 15 years is dead. There used to be many large pure search platform vendors like Autonomy, Endeca, FAST, and theyve all been driven into the arms of larger companies. These deep pocketed companies, and their deep pocketed customers, made Enterprise Search a niche cozy business to be in for years. There was a lot of conversation around “What Next” for the market. And some mourning of the commoditization of search, as evinced by the rise of both open source search engines like Solr and ElasticSearch, as well as the various small companies (BasisTech, SmartLogic, Raytion) building products that extend search engines, but arent search engines of their own.
Martin White puts it, Enterprise Search is only (only!) a billion dollar business, compared to Business Intelligence at $14 billion, or the ERP market at $45 billion or so. And since Sharepoint 2013, with Microsofts acquisition of FAST, has turned itself into the dominant enterprise search engine/CMS solution, this redirects much of that billion dollars of revenue to Microsoft. So this sounds like a stagnating small market right where wed all better learn Sharepoint right?
Not so fast!
Stephen Arnold had something different to say in his very memorable talk on Big Data vs Search. The most interesting slide, which I wish I could find a original of to share, was of the word “Search” with the word “Big Data” splatted all over it. Conveying the idea that there was this massive buzzword coming that was going to steam roll everything in its way, and we all better get ready for its arrival because its going to change everything. He did write about some of his observations of how the search market is going to have to adapt to this.
However, I think he has the impact of the “splat” backwards. I think that Big Data is finally going to let the search community deliver what it has been promising for so many years. I like to talk about the impact of Machine Learning and Natural Language Processing in building smart search experiences. The search engine can move beyond being a statistical token matching tool to an actual discovery interface. One of the attendees commented that “That sounds like an Autonomy pitch from 2004″, and yet that pitch is now becoming a reality, at a cost that any enterprise can afford, not just the deep pocketed ones. Big Data is going to let us build the search engines that weve promised our users so many years, and its re-invigorating the search market.
So heres the good news for the search market:
Once you move beyond the definition of our market as the narrowly defined “Enterprise Search” to the more general “Discovery for Enterprises”, our market size explodes. Gartner in their 2012 Magic Quadrant for Business Intelligence and Analytics Platforms report led off with the quote:
The dominant theme of the market in 2012 was that data discovery became a mainstream BI and analytic architecture. The market also saw increased activity in real time, content and predictive analytics.
And search engines are the dominant platform for delievering discovery applications.
As more evidence of the value of search, have you noticed that all the major Big Data vendors have been integrating search into their offerings? Cloudera came out this week with Cloudera Search that puts search on Hadoop. LucidWorks, DataStax, and Elasticsearch all have offerings that marry Big Data and Search together.
So lets all get beyond Search meaning a query box and 10 blue links on a web page, and lets embrace the new world of Discovery that Big Data enables! And next year, instead of meeting up at the Enterprise Search Summit, lets attend the Discovery Summit for Enterprises.