Background
The collaboration between Deutsche Bahn (DB) and OpenSource Connections (OSC) began in late 2022 with a TLRE Solr training session for members of DB’s search team. This initial engagement set the stage for a deeper exploration of DB’s search capabilities.
Following the training, OSC conducted a comprehensive Discovery phase, which uncovered a critical insight: even within DB’s search team, there were notable knowledge gaps concerning their flagship product, DB Suche (DB Search).
OSC’s Discovery process leverages a maturity matrix and a detailed questionnaire in combination with expert analysis. This approach not only identified these gaps but also led to three actionable recommendations:
- Establish an offline experimentation environment to evaluate search result quality.
- Develop context-sensitive search result pages to improve the user experience and result comprehension.
- Implement a systematic search management framework to address underperforming queries.
By presenting these findings in a clear and actionable format, OSC effectively secured management support for implementing the recommended improvements.
The Accelerator Implementation Project
The Accelerator aimed to empower DB’s search team to optimize their platform’s relevance, ranking, and usability. Here’s how OSC delivered:
1. Offline Experimentation Setup
OSC established a robust offline testing environment, enabling DB to evaluate search result quality systematically:
- Tooling: Setup of Quepid and a dedicated testing search engine instance for offline experimentation.
- Representative Query Sets: Used explicit judgements and implicit judgements based on user behavior data for evaluation.
- Initial Experiments: These provided insights into strengths and weaknesses in matching and ranking.
- Ranking Optimization: Adjusted ranking factors such as length normalization and boosting of specific document types to refine scoring.
- Baseline Configuration: Created a structured setup of search fields and field weights for consistency.
2. Process and Tooling Improvements
The project introduced tools and processes to streamline search management, ensuring long-term usability:
- Structured handling of synonyms, curated hits, and redirects to enhance user information access.
- Contribution to the open-source query rewriting library Querqy for improved search functionality.
- Enhanced tracking to fill data gaps, capturing implicit judgements and identifying user intentions.
3. Enablement and Empowerment
Through six months of intensive collaboration, OSC empowered DB’s search team with:
- Established tools and processes to drive continuous optimization of search result quality.
- Metrics and judgement systems for objective measurement of search result quality.
- A solid foundation for future enhancements, including AI-driven relevance optimization and modernization of the search stack.
4. Strategic Foundations
As part of the Accelerator, OSC collaborated with DB on long-term planning:
- Contributed to a strategy paper on “AI and Search,” guiding the future direction of DB’s search platform.
- Defined proof of concepts for evaluating AI technologies in search.
Results
The six-month Accelerator transformed DB’s approach to search optimization, yielding measurable improvements in both tooling and process maturity. It also set the stage for ongoing innovation, including the planned integration of AI technologies.
Conclusion
OSC’s partnership with Deutsche Bahn demonstrates the tangible impact of expert-led search result quality optimization. Through 8,560 explicit human judgments and over 1.57 million implicit judgments, OSC guided a structured, iterative experimentation process—conducting seven experiments with multiple refinements. As a result, Deutsche Bahn achieved a measurable improvement in search quality, with DCG@10 increasing by 5.4% (from 17.64 to 18.60). By addressing foundational gaps, implementing advanced tools, and fostering a culture of continuous improvement, OSC helped DB establish a sustainable and data-driven approach to search result optimization.
“Without your systematic approach and your expertise, we would not have managed this in two years. The Accelerator has lived up to its name.”
Julia Beck, Product Owner
For more information about OSC’s Accelerator program and how it can empower your search team, contact us today.
