Charlie Hull
Category: Relevancy
Several conversations with clients I’ve had recently have made me think about the issue of control when it comes to search quality improvement: specifically, how much control a search team needs to make a real difference to business value.
How much control and access a search team has to the actual search engine will depend on several factors, some of which are interdependent, including:
- is the search engine part of another system, e.g. an ecommerce package or content management system (like SAP Hybris or Drupal)
- is the search engine hosted internally on on-premise servers, on the cloud or as a fully managed service
- does a separate team in the organisation manages the search engine
- what processes, systems, firewalls, access points etc. exist to manage any access to the search engine
- what is the process for pre-production testing
- how long will it take for any changes to search configuration to be seen in production
Let’s look at a couple of examples, both based on real-world situations:
Search is managed by a partner
In company A, the Solr search engine cluster is managed by the IT team at a third-party partner organisation, who make sure it is kept up to date, has sufficient hardware resources and maintains high availability (HA). However to maintain their service level agreement (SLA), this organisation will only accept changes to the search configuration after significant pre-production testing, via a process of change orders & approvals which can take several weeks. If the search team want to update even a single synonym they need to supply a new file to be uploaded to the Solr cluster, wait for this to be tested and then to be deployed.
Search is managed by internal IT as part of another platform
In company B, a separate in-house IT team manages an Elasticsearch cluster on-premise – but actually, they’re managing an ecommerce package that bundles Elasticsearch to provide search capability. The ecommerce package is a slightly older version than is current, which includes an even older version of Elasticsearch. The IT team aren’t even very sure what Elasticsearch is or does. If our search team want to add a plugin to Elasticsearch (perhaps OSC’s open source Learning to Rank enhancement) they will have to consider how practical it is to back-port it to this earlier version of Elasticsearch. Also, the way Elasticsearch queries are built is controlled by the ecommerce package, so this must be taken into consideration – did the creators of the package understand relevance engineering? Maybe not, considering what crazy JSON is being generated!
Ready to improve search
Our search team, perhaps fresh from attending OSC’s Think Like a Relevance Engineer training, are keen to start improving relevance – they want to use Solr’s admin interface to check how source data is being analyzed, or start crafting slightly different Elasticsearch queries using an OR operator rather than the current AND operator. They’ve already set up some processes for measuring search relevance so they can test the impact of their changes. Having prioritized search issues in terms of business impact, they’re ready to start building better search! The request to change a configuration file is sent…
…and then, nothing happens, at least not quickly.
A priority mismatch
The problem here is that the search team and whoever is managing the search engine configuration don’t have the same priorities. An IT team tasked with managing a high-availability, performant search engine service will care about uptime, latency and stability, not necessarily search relevance. Any configuration changes could affect this – perhaps a new type of query will need more memory, or cause garbage collection issues. To prevent anyone inadvertantly causing performance or stability problems (or even just deleting an index by mistake) any access to the search engine must be locked down, blocked and firewalled. No-one wants to be woken up in the middle of the night because a newly-minted relevancy engineer decided to run some crazy experiments! (For the cynical amongst us I should also mention that the longer it takes to approve something, the more time could be charged by a partner with a support contract, although this isn’t always the case).
So, we can end up with a situation where the IT team actively resist change, and for good reason. If changes are allowed they have to be made carefully, with appropriate testing before approval is given. For a search team attempting to make swift improvements to search quality, this is a problem.
How to gain more control of search
Be inclusive
The IT team, tasked with ‘keeping the lights on’, are also part of your wider search team, even if they work for a different organisation or group. How they manage the search engine will directly affect how quickly you can improve search quality. Bring them into the fold: invite them to your meetings, explain your new-found focus on search relevance, find a way for them to attend relevancy training, create a sense of shared responsibility. If they understand better why you need access or a quick configuration update and how this can drive business objectives than they will be more able and likely to help. Be clear about what you need – and also, what you don’t.
Consider change
If it is at all feasible, consider how your search engine could be brought closer to your search team: perhaps on-premise hosting is a better idea than an outsourced service, if this outsourcing is actively preventing search improvement. Even if you don’t carry through, considering this option might help an intransigent partner come up with ideas to help improve the situation – no-one wants to lose a support contract!
Introduce relevance testing
Your current pre-production tests may focus on speed, uptime and perhaps a few ‘smoketests’ – maybe firing a few thousand common queries at the search engine to see what happens. You may not have relevance testing as part of this suite. Perhaps you can use Quepid to create some test cases and manual relevance judgements for some of your most important queries, and fold these tests into your pipeline using the Agent Q scripts?
Work around the problem
Changing search engine configuration files isn’t the only way to improve search quality. Improving the quality of source data will also help: make sure you have content creation guidelines and that the content team also understand how search works – for example, keyword stuffing is as bad an idea for internal search as it is for SEO.
Putting tools directly into the hands of your customer-facing team is a great way to improve responsiveness to search issues. Querqy is a query pre-processor that lets you embed business rules into search by rewriting queries. Combined with SMUI, a graphical front end, it is a powerful way to add a searchandizing capability (find out how this can work from our Pete the Product Owner blog series). Note that to implement Querqy and SMUI you will need to work with your IT team – they work as a plugin for Solr or Elasticsearch and need to be installed correctly and have suitable access. However, changing Querqy rules is a lot less disruptive and risky than making changes to Solr or Elasticsearch configuration files.
Don’t fall at the first hurdle
If your search team doesn’t have some control of search engine configuration then your search quality improvement project may fall at the first hurdle. You need the ability to make changes quickly and the support across your organisation to make this possible. Make sure your management understand what’s at stake!
We help organisations across the world understand how to build better search – get in touch if you want to discuss your project.
Image from Car Vectors by Vecteezy