Measuring Search & AI

A full day workshop led by the OpenSource Connections team

AI promises to revolutionise how we access information – but how can we measure the benefits?

In this workshop we will explore the space of evaluation strategies for search engines. This will include the role of user judgments, both explicit and implicit, evaluation metrics that can be computed with those judgments, and the general process for continuous experimentation that yields continuous improvement.

But wait, where’s my AI? you ask. We will explore the use of AI techniques, including Large Language Models (LLMs) to generate judgments for us automatically. We will finally explore employing all of the things we have learned to the problem of evaluating AI-assisted search, and other AI-enabled activities.

Get the practical tools and skills you need to succeed in leading your next AI-based Search project.

Prerequisites

• What do you need: A curious mind, a basic understanding of the experimental process, and a desire to understand how to pick and choose across all the enabling technologies that are available for search and search adjacent activities.
• Who should attend: Developers, product managers, relevance engineers, and everyone else who wonders about how to tell if the magic box is actually magic.

Your trainer

David Fisher has over 30 years of experience designing, building and implementing information extraction and information retrieval systems. As a principal software engineer on the open source Lemur Project, while working in the Center for Intelligent Information Retrieval, he has developed custom applications and components for numerous search engines, including Indri, Galago, and Tsidy. His primary focus has been on efficient indexing structures and complex retrieval model implementations.

At OpenSource Connections, David has worked on Generative AI applications for clients in the financial analytics sector, vector search for image content and helped create several self-led training courses on search and machine learning.

In addition to his engineering experience, David has been a lecturer at the University of Massachusetts Amherst Manning College of Information and Computer Sciences for close to 20 years. He has taught undergraduate Software Engineering, undergraduate Search Engines, and developed and taught the masters level Applied Information Retrieval course.