Doug Turnbull
Category: Uncategorized
I’m happy to be taking the search relevance mantra to the CIO crowd in my article in CIO Review Magazine (p65). In the article, I highlight that search is how everyday users expect to interact with Big Data. So much of the focus in Big Data has been analytics and I think this is unfortunate. Everyday human beings want to interact with data in human terms – not through SQL, map-reduce, or dashboards. They want to type in a few search terms and get back results relevant to them and their goals. Nevertheless, the current mantra from data science and “Big Data” startups is “analytics! analytics! analytics!”. The emphasis is on selling the data structure dujour that can solve all your analytics problems in a single silver-bullet NoSQL Hadoopy platform.
Compared to this latest trend in “NoSQL analytics”, it’s interesting how long-lived Lucene has been. Lucene-based search like Elasticsearch and Solr has been quietly crept into all sorts of applications. Both for analytics and search. Recently, it seems the trend even in search circles has been heavily focused on analytics, with compelling features like Elasticsearch’s aggregations, and columnar-like DocValues gaining a lot of visibility in this area.
The quieter but perhaps more important story, however, is how the search bar has crept into nearly every application. Look at your phone. So many applications have search at their core. OpenTable and Yelp for finding restaurants near me. Spotify and iTunes for music. Youtube for video. We don’t even think about how often we’re using search. We’re not conscious of how each of these are unique and diverse user experiences over data focussed on search. We just search! We certainly, however, think about when it doesn’t return results we want!
So as you develop your applications, and you think about what “Big Data” actually means, ask yourself should a focus on relevant search be part of the solution? Are users going to want to see data or talk to data? Ask for things in natural-language terms? If so, think about how tuning the relevance of your search results might be a better place to spend your smart folk’s time! It could certainly pay off.
Anyway, I’d love to hear feedback on the article. If you’d like to discuss, please contact me!