Eric Pugh
Category: OpenSearch
Background
On April 24, 2024 I co-presented at Haystack with Stavros Macrakis from the AWS OpenSearch team on the User Behavior Insights work that we’ve been doing to establish a standard for tracking user behavior that is specifically optimized for the needs of Search teams. The video and slides will be published soon and I’ll update this blog post with the links as they come – but we also gave a talk in Berlin a couple of weeks later at OpenSearchCon EU and the video of that talk is available now:
Why we need UBI for tracking user behavior
Tracking what users do with their search results is hard. In the projects we’ve seen, we’ve often had to develop custom code and/or patch together partial solutions using Google Analytics, Adobe Experience Cloud or indeed whatever the client already has available for website analytics, if anything. OpenSource Connections has been collaborating with the OpenSearch team to develop User Behaviour Insights as a better way – collecting (ethically) the data we need and sending it back into the search engine itself for analysis and also as a source of data for machine learning techniques like Learning to Rank. It’s a hugely exciting project that will make a large impact on the world of search optimisation.
We’ve been using Chorus, an open source reference implementation for tunable, measurable e-commerce search, as our test platform (learn more about Chorus in this talk).
Stavros and I took advantage of spending a few days with 300+ of our closest friends and colleagues in the Search Relevance space, at first Haystack US, and then two weeks later at OpenSearchCon EU, to gather feedback on what we had presented at the conference, and in the spirit of open source I wanted to share what we learned. Please take this as a request to reach out to either of us via Relevance Slack or the Open Search Slack channel and provide your own feedback!
What we learned at the conferences
Structured Data is a Foundation for Successful Analytics
People really loved the idea of a standard schema for tracking user behaviour in response to search queries. The number of questions that we got after the talk spoke to a desire for better solutions than what most of us work with.
The schema is the most critical part of establishing the new User Behavior Insights standard. We need to strike a careful balance between a flexible “anything goes” schema and a super rigid (and therefore too limited) schema. Instituting an extremely flexible schema where almost anything can be captured will require the establishment of some sort of complex meta-schema on top! You can do it, and the folks at Snowplow give an example, but it does add a lot of complexity. On the other hand, if we go with an extremely structured schema, you’ll know exactly what data is supported by UBI but we may find ourselves boxed in. We were asked a lot of questions about “how extensible is the data that UBI can track”. This has reinforced that we need to articulate our schema as a set of “must” attributes, some “should” attributes, and then plenty of extension points where additional data can be indexed to fit the needs of individual builders.
We have written and documented a schema, more properly referred to as a specification. To support the goal of a machine readable schema that can be used to verify the communications between components in UBI, we introduced a JSON Schema compliant schema for UBI. This Schema is published here and the 1.0 release of the schema will be published via Github very soon. It’s exciting to be able to validate query and event data against the underlying UBI schema files to confirm interoperability.
To Be a Session or Not To Be a Session?
Originally we had in our schema the idea of the front end passing a client_id representing the browser/app that is being tracked, a query_id that represents the query, and a session_id. In conversations though, it feels like the concept of a session is very overloaded, and honestly, can the front end really determine what is a session? Isn’t it really something that you determine on the backend using your own criteria? A session, which really probably should be a “user journey”, is a hard thing to define, and expecting a front end to manage it is unlikely. (The only place I have ever seen a user journey tracked by the front end is when the work is very structured, with a beginning and an end that is built into the workflow, like when a patent examiner looks for prior art). I suspect we shouldn’t even touch the concept of session in our schema. You want it? Fine, pass it in as your own attribute!
In the aforementioned event schema, we have added client_id to the specification. We have decided not to add explicit user_id or session_id properties to the the events, though you can add them as additional properties to the event_attributes field.
The Open Source Way Implies An Agnostic Schema
Our audience told us to keep the schema agnostic of how OpenSearch works. Sure, we are doing an implementation for OpenSearch, but we want other engines to adopt the UBI schema, to set a standard across the search tuning industry. The formalized UBI schema has no OpenSearch specific details. To validate that the schema is independent, we spiked out an implementation for Apache Solr as well!
Are All Actions Queries?
Infinite Scroll and Pagination are both activities that could generate a new query_id because you are changing your search query by adding depth parameters, and yet conceptually these activities aren’t new queries. Just the same query with a different view on the result set. So keep the query_id the same. However, pick a facet, and that definitely generates a new query_id. This is a place where we need the front end client to be somewhat aware of when a query changes and when it doesn’t.
A community member who leads analytics for his search teams reached out to learn more about UBI, and one of the things we decided is to do a bit more gap analysis between some teams’ existing implementations of tracking user behavior and the data collected and what UBI has defined. This also highlighted we need to think about the evolution of the UBI schema beyond 1.0, and what that governance structure looks like.
Metadata as a Long Distance Relationship
Lastly, we heard from a number of folks that we definitely need in our initial version to be able to track metadata associated with events that describe the context of the original search, but without sending it all the way to the client, storing it there, and then passing it back in as an event. For example, maybe you want to track the margin on your products but don’t want to expose that data in the browser. We need a way to shortcut the circle of data processing and keep that data internal to the server side, but still marry it up with subsequent events. This doesn’t exist in our Chorus reference implementation yet but will soon!
What’s next with UBI
There are a few immediate things:
- If you have followed along with the Chorus for OpenSearch reference implementation, it is built on OpenSearch 2.13 and an early MVP of the UBI plugin. We’re updating Chorus to use OpenSearch 2.14.
- We are anticipating that in the first week of June we’ll have the official repository for the OpenSearch specific UBI code created as part of the OpenSearch project’s Github account. Once that is done we’ll cut the official OpenSearch 2.14 version of the plugin.
- Along with that release process, we’ll create the 1.0 release of the UBI spec in our own UBI repository. That will give us a line in the sand for where we are starting from, and then we’ll bend our thinking towards what gaps need to be filled with an eye to a 1.1 version of the spec sometime over the summer.
- We’re then aiming for some improvements and hardening for the OpenSearch 2.15 release in the second half of June. If we need to revise the UBI spec for any obvious gaps, we can do it then as well.
- The Solr version of the UBI plugin is out there, but since Solr operates on, well, it’s own timeframe, we’ll see when that lands!.
In the fall we’re going to start thinking more about how to use the data that we are gathering. Think calculating implicit judgements, automated query intent, balancing lexical and semantic search via hybrid search, plus starting to ship some more dashboards for typical questions we ask of our search, like “what are underperforming queries”? or “Do I have documents that never seem to be clicked on?” There are lots of ideas floating around!
Remember, if you have feedback or ideas, please reach out to either myself or Stavros via Relevance Slack or the Open Search Slack channel. We’re building UBI for everyone!