Getting good search results is hard; maintaining good relevancy is even harder. Fixing one problem can easily create many others. Without good tools to measure the impact of relevancy changes, there’s no way to know if the “fix” that you’ve developed will cause relevancy problems with other queries. Ideally, much like we have unit tests for code to detect when bugs are introduced, we would like to create ways to measure changes in relevancy.
This is exactly what was done at OSC. They have developed a series of tools and practices that allow them to work with content experts to define metrics for search quality. Once defined, they can instantly measure the impact of modifying their relevancy strategy, allowing them to iterate quickly on very difficult relevancy problems.
Look forward to seeing you there! Be sure to RSVP.