Doug Turnbull
Category: Elasticsearch
As I’ve written about extensively, I strongly believe that search engines are the future of recommendation systems. The biggest reason being cost: today’s search engines require extensive amounts of expertise and skill to build – usually from scratch. Open source search engines like Solr or Elasticsearch, however, increasingly have capabilities that simplify most forms of user behavior pattern recognition – on top of already existing robust ranking and relevance capabilities.
Today we’re showing off what we mean. We’ve taken one promising product in this arena, Elastic Graph, and applied it to a classic recommendations data set: movielens. Movielens has years of explicit user ratings of movies. It serves as one of the canonical data sets for performing recommendations.
You can see the source code here, and set up your own demo. One thing you’ll notice is the number of settings we’ve exposed: different options for tuning the behavior of Elastic Graph. This really just scratches the surface. Elasticsearch is a ranking machine: customizing the relevance of search and recommendations is where it excels.
After you play with the demo, follow along here if you’re curious how it all works.
How does Elastic Graph deliver tremendous recommendations? Sampling & Significance
This demo implements what’s known as user-to-user recommendations. The documents in the index are users themselves, and the movies they’ve liked. In other words, something like
{ "liked_movies": ["1234", "582", "581"]}In the demo, you indicate a few movies you like. Elastic Graph uses search to find users that also like those movies. It then tells you what movies are statistically significant in that set of users. In other words, it recommends to you movies people similar to you also enjoy.
How exactly does this work? This is a combination of three processes, enumerated below:
1. Crafting a query that searches for similar users.
The first job is to create a query that will find users like you. In the case of our demo, we craft a query that simply ORs together the movie ids you like into a big search. Something like
{ "match": { "liked_movies": ["1234", "521"] }}The most relevant results for this query those users most similar to you (they like many of the movies that you do). This being search, focussed on relevance, those users bubble to the top of the results, followed by increasingly less similar users, down to users who only enjoy one movie that you also enjoy.
Of course, this query may not be the best way to do this similarity, but hey, good news is this is Elasticsearch. You have an extensive Query DSL to use to tweak and tune how similar users are identified. There’s even a whole book for that ;).
2. Limiting to N most similar, instead of all results
With a query like the one above, there’s a chance you’ll get a large number of results. Remember you’re searching users. Like I said, many of these users will match only one movie, others will match many of your liked movies. How do we prioritize users that like many of the same movies you do, and ignore the others?
Well luckily, users are ordered by relevance. The users most similar to you come to the top. Those just barely similar at the bottom (with those not at all similar not even being included in the results).
It’s a good idea to have a cutoff somewhere that eliminates less similar users. Then you use only incorporate these top N most similar users when formulating recommendations. This is precisely what something like the sampling aggregration does and also what Elastic Graph does. When we give Elastic Graph a query like the one above, we specify the “sample_size”. Here’s our proto-graph request, adding a section for a sample_size. (The variable query below simply being the Query DSL query from above.)
{ "query": query, "controls": { "sample_size": 500 } }I hope you’re starting to see why search makes a tremendous graph exploration platform: it focuses on significant and relevant connections using simple, effective, and highly customizable relevance ranking.
3. Statistically significant co-occurrences between you and similar users
Now that we’ve limited to the top 500, what’s next? Well we need to examine the movies they’ve watched and tell you about the ones you haven’t seen yet. Those are your recommendations!
At first blush, the solution seems obvious. Count up the total number of movies in this set. Show you the ones you don’t know about yet: ordered by that count. If “Forrest Gump” occurs 20 times and “Rambo III” occurs 15, then that’s how we should recommend movies to you.
But there’s actually a big problem with this. In any random sampling of users, looking at the raw counts, we’re probably going to get counts that roughly correspond to what’s globally popular. It’s like presidential polling: if we poll 1000 random voters, we sort of assume that the results will roughly be in line with the overall voting preference.
We actually don’t want a sense of what’s globally popular in this result set: we want a sense of what’s oddly popular to users like you. For example, hypothetically speaking, 80% of sci fi geeks may like Star Trek. The general movie viewing population may only enjoy Star Trek 10% of the time. If you’re like me, you fall in with the sci fi geeks. Which means your recommendations will show you what’s much more interesting to these users when compared to the global population of users.
How exactly does Elasticsearch score this? Well there’s a number of methods, but I would recommend reading either my blog article or the documentation on significant terms aggregration.
More importantly, how to we ask Elastic Graph to perform significance calculation for the results of a search query? To do that, we introduce the “vertices” in our request below, indicating the field from the search result to use for the significance calculation. Looking at the complete picture of what this request will accomplish:
query is issued to Elasticsearch, users similar to you are return ranked by relevanceThe 500 most similar users are takenRaw counts (basically a facet) are gathered for the movies in liked_movies, we might note that Forrest Gump occurs 10 times, Star Trek 8 times, and so onThe local raw count is compared to the global preference for these movies to measure their significance. We might score Star Trek as very significant (10.5) and Forrest Gump as less significant (1.5)
{ "query": query, "vertices": [ { "field": "liked_movies", "exclude": likedIds } ], "controls": { "sample_size": 500 },}You’ll also note the exclude line in vertices. This simply excludes movies you like from the significance calculation. Users similar to you like these movies as well, so there’s no point in running them through the significance calculation. Instead, this lets us focus on movies you haven’t seen, but users like you really like.
Tuning Relevance
In our demo, we expose a few extra ways to seed the recommendations graph above. This gets at another core strength of Elasticsearch. You can give Elasticsearch more features of your users (more about what they like, age, demographics, etc).
With movielens, we can extract some information about each user’s preferences and explicitly include it in the seed query
These include (as pictured above):
- Identifying genres you seem to like, and highlighting users that also like those genres
- Identifying release years you like (specifically broken into half-decades)
- Using the description text from each movie you like and running a “more like this” on movies they like
Incorporating them is as simple as extending the query used above. Perhaps if we included genres you seem to like, we could find similar users with the query:
{ "bool": { "should": [ {"match": { "liked_movies": ["1234", "521"] }}, {"match": { "liked_genres": ["science fiction"]}} ] }}Some of these seem to work better than others (the description text often provides oddly compelling results). We encourage you to experiment and let you know. If you want to see how the query is changing with each setting, check out where the query is built in the source code.
Finding connections
This is graph afterall! One of the fascinating capabilities of Elastic Graph as a recsys platform is its ability to look beyond the current set of recommendations into a secondary tier. For example, if Elastic Graph recommends Rocky to you, Elastic Graph will tell you what’s interesting for Rocky. By default it will tell you this globally: perhaps showing you Rambo, Rocky, Turner and Hootch, Stop or My Mom Will Shoot, and other popular Stallone movies.
But what’s interesting to me is you can guide the exploration contextually: scoping it to items that you likely consider relevant. To do this, we can scope using the query above, perhaps only keeping the liked_movies clause to construct a guidingQuery as shown below.
In the demo we give you the option to toggle global/local graph exploration. Expanding on the graph query above, this becomes
{ "query": query, "vertices": [ { "field": "liked_movies", "exclude": likedIds } ], "controls": { "sample_size": 500 }, "connections": { "query": guidingQuery, "vertices": [ { "field": "liked_movies", "exclude": likedIds } ] }}We introduce the “connections” section. Everything under “vertices” is the same as above. For each primary recommendation, show me other statistically significant liked_movies but don’t show me what I like. Elastic Graph will come back detailing vertices of additional length and how they’re connected to our recommendations (vertices of depth 0).
The more interesting line is “guidingQuery.” This scopes the exploration only to plausible recommendations for me. The end result is for each movie I might be interested in watching, I can see instantly other movies I plausibly would enjoy that sort of “go down that trail” of interest. For example:
There’s a couple of interesting observations here
- The Fighter might not be a direct recommendation to me, but it seems that its “close to me” in the graph – friends of friends seem to like it
- I can imagine helping users with a conundrum of choice. I often see a movie I’m vaguely interested in: seeing what else is down that rabbit hole can really help – focus what I want to watch instead of faffing around with high-level recommendations that cover my broad tastes
What else can you do?
One thing we point out in Relevant Search, is that you choose open source search engines because they’re a framework, not a solution. That means that with great responsibility comes great power. You could apply the functionality here to implement additional capabilities. Some I can imagine include:
- Personalized Search – use Elastic Graph’s rankings to reweight search results for a user.
- Social Search – can we reweight search according to direct social relationships?
- Taste Profiles – bucketing recommendations and search results by specific tastes, genres, or other notable properties
I’d love to hear if you have any neat ideas as well. Don’t hesitate to get in touch.
The future is easier-to-build recsys
I’ve been impressed with how open source search has transformed traditionally hard problems into far simpler endeavours. These days, without having incredible depth in search or analytics, you can get a solution up and running relatively quickly. Decades ago, search was the purview of extremely sophisticated organizations. Lucene-based search (Solr/Elasticsearch) have laid waste to those days. The same thing is happening in analytics. Organizations seem to be replacing tools like Splunk with Elasticsearch/Logstash/Kibana at neck-break speed, resulting in far simpler tools that can be more easily customized by developers.
Recommendation systems are next. If you’re contemplating a recommendation system, you probably think you need to beef up staffing and infrastructure. You’re imagining expensive proprietary tools or an extensive custom development effort. Before you see anything compelling it will be months.
This simple demo was thrown together in 10s of hours with Elastic Graph. It can always be improved, for sure – just like any search application can be improved. More importantly, with Elasticsearch those improvements can come iteratively. With the Elasticsearch Query DSL, relevance tuning, and aggregrations you can make constant tweaks and deep alterations to how Elasticsearch behaves.
Another way to say this is Elastic Graph democratizes recommendations just like Elasticsearch has democratized search and analytics. If you’re a medium-sized organization contemplating a straight-forward recommendation system without the fuss and expense, this is a great place to invest your $$.
If you’re interested in discussing further, I’d be happy to chat about our expertise building recommendations with Elasticsearch. I hope you’ll get in touch if you’d like to discuss how we can help evaluate whether Elastic Graph is a good fit for building a recommendation system for you. If you’d like me to discuss this in-depth with your team, I speak for free at your company’s knowledge sharing events/lunch and learns!