Search is a big deal — how else could we navigate the digital age where information overload is the status quo? But despite search being omnipresent, it can still seem like a black box. Although there is a lot of deep technical knowledge involved in search, this is not a prerequisite for being successful at understanding how your search is performing.
At OSC, our mission is to empower search teams; we want to give organizations the understanding of what’s happening inside of their black box. The first step on this journey of mastering their search is always the same: learn how to measure search. These are 5 things we recommend to measure search well:
1. Record searches
The ability to log all of your user’s searches is built into modern search engines, but is usually turned off by default because logging everything can slow response time at scale. However this data is the best way to learn what your users are looking for, so it’s worth sampling. Whether it’s detecting a seasonal shift in products or identifying popular queries for auto-completion, this has one of the highest ROIs in search optimization.
This answer shows how to do this by configuring Elasticsearch’s Slow log and the equivalent Solr config parameter is slowQueryThresholdMillis.
2. Track zero result searches
The first thing you look for in the query logs is what searches failed to produce results. Nothing is more frustrating than looking for something you can’t find, and one of the best ways to make your users mad is to give them zero results. Keeping a count on the number ofs earches that end in this frustration is a simple metric to start monitoring search.
This idea comes straight from the book Search Analytics for YourSite which is a must read if you are interested in getting the most of your search data.
3. Establish a search report
Seeing is believing and, until seeing search data becomes part of your organization’s habits, it can be easy to ignore. This report doesn’t have to be a custom workflow—it could be Google Analytics—but the point is to socialize the information within your teams. Regularly sharing are port will go a long way to building a shared understanding of the areas to improve in for better search experience.
This blog post walks you through how to set this up for your site’s search with GoogleAnalytics.
4. Get direct user feedback
Feedback is critical to making sure you search is on the right track.Net promoter score has been massively popular for measuring brand loyalty because it’s fast and easy for people to complete.Consider doing something like this for your search results: a quick pop-up with 3 options, smile / neutral / frown, asking how happy they were with their search experience. Going straight to the source and asking the people who matter most, your users, takes the guesswork out of measuring your search.
Being feedback blind is always a bad idea, but we see it a lot when it comes to search. Read this post by OSC’s Doug Turnbull for more discussion about this problem.
5. Build solid judgments
Judgments tell you how good a result is for a query. They can come directly from human raters or be extracted from site interaction analytics. Regardless of the source, judgments are the measuring stick for search, so we want to make sure this process is dependable. Invest in doing this right and so you can have confidence that the changes you make to search are good ones.
OSC maintains Quepid, a free open-source search relevance tool bench that’s designed to collect, manage and track your document judgments through a web interface.
We see search strategy as the same as business strategy, because it has an impact on every area of your business—it can’t be siloed to just the search team. Improving search requires cross-department collaboration paired with meaningful ways to measure performance. I hope this post has given you a few ways to get started measuring search today so you can have better search tomorrow. If you have any questions about applying these measurement ideas, reach out to us. We are always happy to help organizations maximize their search skills!