In a previous blog, I wrote about how important it is to consider how to train your staff in search skills so that everyone can work effectively on the shared problem of search tuning. Unfortunately most of the training offerings available are designed for technical staff — developers or architects — and these go into considerable depth about code, algorithms, implementation and scalability. Fascinating stuff if (like me) you come from a technical background — but perhaps not so useful for the other members of your organization; however, the content team or marketing team still need to know the basics of how search works.
So how do you give your team the background knowledge they need to be effective? Let’s go back to the beginning. Before we had search engines that ran on computers, we had other ways of searching for information. Visiting your local library and talking to the staff there might have been your first step — and if you were looking for something specific they might consult an index, probably held on paper cards. The concept of the index is the foundation of all modern search engines. Another great way to think about this is the index at the back of a book — it lets you look up, using certain keywords or phrases (called ‘terms’ in search engine parlance), where in the book the information you want is held. This can be a vital concept to explain to the team and it can be done without using any software.
The paper-based search engine
One exercise I’ve successfully used in the past is to build a single-document index entirely on paper with a client’s team — taking a single piece of content (e.g. a product page from the client’s own website), describing the concepts of text extraction, fields, analysis, stemming etc., and working with the team to decide which terms should end up in the index. Here’s an example of this process. We can see the original text, then the fields containing various terms (some of them stemmed) and then a ‘terms list’ that would be part of the index, showing in brackets the count of how many times a particular term appears in various documents:
This can then lead to a discussion of another important part of the search engine — constructing the query from what the user types. In the example above, one might ask the team to think about how to answer the user’s question, ‘do you sell an oak door,’ how this question is analyzed, stemmed etc. in a similar way to produce a query, and then how we apply this query to the index to decide whether the document on the left is a relevant result.
Parts of a search engine
To identify where the index fits into the larger picture, an overview of the various parts of a search engine is useful. Here is a diagram of a generic search engine I created for the book Searching the Enterprise:
This overview can be used to explain how the various processes that create the index work, and how users interact with this index. Again, there is no need to show software code at this point.
Not quite Google…
So after your introduction to the basic principles of search, your team might carry out further research on the web — but it’s not quite as simple as typing in ‘how does a search engine work’.
One of the first issues you will face is that confusion is rife between the web search offered by Google, Bing and others and the site search or internal search you’re concerned with. Look for courses or books on search engines and there is plenty of information on SEO (making sure your website appears high in Google’s results). Many people assume that internal or site search engines work the same way as web search engines and the truth is…well, sort of. However learning about the mysteries of SEO won’t be very useful for your team.
Non-technical (or not so technical) information on internal and site search is available, but it can be surprisingly hard to find among the technical detail and buzzword-heavy vendor marketing. The Search Network publishes independent reports aimed at business people and Search Insights 2020 contains some very useful glossaries and reading lists. Morville & Callender’s Search Patterns book is a standard reference on search user interfaces. Daniel Tunkelang’s series of blogs are easy to read and cover many aspects of search engines including query understanding. Martin White has written many books on search and information management and his site has many useful resources. Designing the Search Experience by Tyler Tate and Tony Russell-Rose focuses on user experience and talks about search as a journey.
Lastly, encouraging your technical team to present what they do to the rest of the organisation (and to external groups such as Meetups) can be an effective way to share knowledge, as well as helping to build up essential communication skills. You should aim to develop a shared understanding amongst your entire team of the basics of search technology and how you use it in your organisation.
If you’d like help with educating your wider team about search, do get in touch.