Haystack LIVE! : A brief tale of search experiments

Haystack LIVE! our regular online gathering returns with a talk from the team at Digitec Galaxus, the biggest online retailer in Switzerland:

A/B-test is a powerful tool for evaluating if a new feature improves users’ search experience. In this presentation, we introduce learnings generated in past experiments. First, we show how combining A/B-tests in a more general experimental process with Bayesian Optimization, and Offline Evaluation can speed up our learning curve for optimizing features’ weights. We present a real case example where a better weight for a feature is found with Bayesian Optimization, validated with Offline Evaluation, and the uplift for CTR is estimated in an A/B-test. Using the same real case, we show the intuition behind Bayesian Optimization in the one-dimensional case. Further, we discuss the challenges we are currently facing when trying to optimize several weights simultaneously (e.g., convergence issues, reaching suboptimal solutions). Next, we introduce an A/B-test where new suggestion algorithms are Pareto optima calculated in an Offline Evaluation (e.g., Offline metrics: Precision, NDCG). session.

Presented by Abel Camacho, Simon Gubler and Joel Widmer.

Get Notified about Upcoming Events