Top 7 Mistakes Organizations Make With Search

Doug TurnbullJune 29, 2016

This article reflects on lessons learned from our book Relevant Search. Readers of this article can use discount code oscrelsearch to get 39% off your purchase! Buy Now

After much sweat and tears our book Relevant Search is out! Relevant Search reflects the wisdom we’ve acquired over the years helping many clients improve search. I thought it would be an appropriate time to recap where many organizations get stuck with search

vectors are fun

Simple search for "coffee" on costco returns espresso machines and coffee-colored furniture. Would you revisit this site after this experience?

1. Millions for SEO. Not a dime for in-app/on-site search

Sure placing highly on Google is important. Yet when users end up on your site, you need to keep helping them. One of the most common interactions users try is through your site search. Most implement a canned site search that dissapoints users, not one customized to their needs. This teaches users that there’s nothing special about your site. Sure they found you on Google, but will they remember to come back to you? Or are you just yet another landing page? A bump on the road that doesn’t stand out from the Google noise?

The best SEO is building such an amazing experience that users remember to type your URL into the address bar.

2. Thinking of search as a technical problem

Search developers rarely know what “relevant” means for your application. A search application built for a hospital purchasing department looks nothing like a search application built for finding travel accomidations. Only non-technical managers, domain experts, and those familiar with users can define search requirements. Sure interesting technical problems abound in search, information retrieval, and natural language processing. But these solutions are meaningless wastes of money without understanding the user and business goals behind search. The bottom line, you need more than developers and data scientists. You need colleagues that understand your users, the domain, and the business to improve search.

3. Failing to study users

Have you heard of the HPPO? The “Highest Paid Person’s Opinion”? Instead of studying real users, the CEO comes along and types a pet query. The results are terrible. The search team tucks tail and tries to solve the problem. It turns out though that the CEO was executing searches no user ever runs.

Getting information on how your users search is hard. It takes constant work from analysts, domain experts, and technologists. Some applications require heavy understanding of analytics. Others applications are very domain specific and require traditional usability studies. Regardless of the work required, the payoff is enormous. Getting into your users heads, understanding what they intend with their searches, understanding how they search is the key to optimizing relevance for your needs.

4. Failing to iterate quickly and measure

Search, and especially relevance work, is highly iterative. You need to validate your relevance investments on a regular basis. Many organizations, however, find it painful to deploy regular tweaks to their relevance solution. They want to “wait” until perfection is achieved. Instead we advocate tweaking search against validation criteria using Test Driven Relevancy approaches. Upon deploying you need to validate these changes are helping: preferably A/B testing against key performance indicators that indicate how search is impacting your business and users. These indicators have been identified and validated by studying your users (and knowing what you hope to get out of search). Build yourself enough slack to fail fast and roll changes back if you need to!

5. Trusting black-box, silver-bullet products

The search marketplace is unfortunately replete with black-box solutions that promise to psychically scan your users brains and give them exactly what they want. One common pattern is to track how users click on results and change search ranking accordingly. This can actually make search worse when done poorly. If you hear a marketing person utter the words “machine learning” and “big data” without a technical person to explain the approach, run the other way!

Beyond the marketing noise, there is a lot of buzz around the idea of Learning to Rank these days. Learning to Rank is the science behind tracking user behavior and optimizing relevance. But the first step in doing learning to rank is to be really good at understanding your user’s behavior. Studying your users is non-trivial and takes work (as we said above).

Before getting deep into machine learning, work on simpler and more fundamental approaches to relevance. How does your search engine’s ranking function work? Is it prioritizing the factors that a general user sees as important? Ninety percent of a good search solution can be obtained just by learning how to better use your search engine.

6. Discounting guidance & perceived relevance

Good search is not just great relevance ranking. Users frequently need guidance to support the experience. This encompasses a number of user experience features: spell checking, typo tolerance, autocomplete. Good autocomplete guides users to search terms you feel confident will yield good results for them. Spellchecking and typo tolerance help to avoid scenarios where users get 0 results. Faceted browsing is another common form of in-search navigation. Often letting users guide themselves through facets and filters is far simpler than trying to solve really hard relevance problems.

Perhaps the most overlooked feature is highlighting. Highlighting points out to users why results matched the query with highlighted research. Yet many forget to use highlighting. The result is many times the “relevance” problems are really perceived relevance problems. Users type search terms but can’t figure out why a result matched. Perhaps only the title is shown. So they assume that the result is not relevant. Showing highlighted keywords in context helps them understand how the search engine is working. It lets the refine their queries carefully so they can explore and find.

7. Limitting your imagination to traditional search

vectors are fun

Various relevance-driven components in Stackoverflow's UI

There’s more you can do to bring users to content they need. The future is relevance-driven applications. Recommendations, conversational user experience, voice search to name a few.

A good example of this is stackexchange. You can see on the image on the right some examples. Stackexchange puts search and recommendations in quite a bit of the application.

  • Enter a question, Stackoverflow tries to find duplicates
  • Close a question, Stackoverflow suggests candidate duplicates
  • Suggest meta posts to active users (shown on right)
  • Suggest jobs relevant to the current user (shown on right)
  • Link questions related to this question (shown on right)

Fundamentally it’s still all about the idea of relevance: bringing users to content they need. If you can master this, you’ll build personalized experiences that augment and go beyond search to answer user’s questions often before they even think to ask them!

The future is information retrieval

Everyone is on the machine learning bandwagon these days. And with good reason: the cloud is commoditizing compuiting similar to how fossil-fuels commoditized mechanical labor. Machine learning will revolutionize many industries.

But machine learning for what exactly? I’d argue that what many are trying to do is to make applications smarter for you. Information retrieval, the science behind search and recommendations, is exactly about aligning users information needs to content. It’s getting you the right vacation home. It’s finding the speakers that will make your convertible sound amazing with the top down. It’s about suggesting the ideal “scenic route” through the country based on your ideal mix of mountain views and winery stops.

Sure Information Retrieval has been around in search for decades. It’s been in recommendation systems for years. But Information retrieval is in every emerging form factor: conversational UX and voice search. It’s in your Apple watch and Google Inbox when it decides what notifications are relevant for you. It’s in the doctor’s chart as she makes treatment recommendations for you. There’s a “relevance” problem behind each of these use cases, and THAT is where the smart applications will differentiate themselves from the boring point-and-click applications of today.

Everyone is lost thinking about AI and skynet. But Information Retrieval, not AI, is the next decade. Are you ready to harness it? If you want a leg up on getting to the smarter future, read Relevant Search. It will give you the fundamentals of how to prioritize the right user signals in relevance-driven applications.

Get the experts!

We’re the Information Retrieval experts. Our focus is building relevance-driven applications of tomorrow. If you need a better search solution, more relevant search results, or are interested in building recommendation-driven solutions, be sure to get in touch and we’d love to chat.




More blog articles:


Let's do a project together!

We provide tailored search, discovery and analytics solutions using Solr and Elasticsearch. Learn more about our service offerings