Blog

How to make Quepid talk to your .NET Search API

Atita Arora
Category: Ecommerce

By Atita Arora (OpenSource Connections), Bego Terzimustafic & Mark Johnson (Emerson Ecologics)

In the e-commerce business, there are several factors directly or indirectly affecting your success. A smooth UI and irresistible offers help – but there is one thing that takes users’ experience and the profits to a different level, and that is helping users find a suitable product/service they need, with minimum effort. 

The idea is wrapped in a concept called Search Relevance and tuning it can be hard if you are not equipped with the right tools. Relevance is defined by different formulae for different businesses and results are often are enriched and treated before being rendered to the users through the search API.

At OSC, we always advise our clients to invest in experiment driven offline labs to simulate real-life search problems:

  • Increasing the experiment rate correlates with better search
  • Offline testing is faster & cheaper than an online A/B test
  • The lab creates repeatable measurements of search quality
Search API
Search Quality: A Business-Friendly Perspective – Peter Fries, Haystack 2018

In the search sandbox with Quepid

We recommend using Quepid for this purpose as it helps you set up and improve a repeatable and reliable process of experimentation that the whole team can understand. Using these test-driven techniques and working with content experts and marketing teams, you can efficiently define and track search correctness. This solid foundation helps prevent you sliding backwards and can fully leverage the power of search. Quepid is also part of the Chorus stack, a reference implementation for building powerful, accurate and measurable e-commerce search (read our blog series on Pete, the e-commerce search product manager).

Quepid itself is a sandbox for changing the Solr or Elasticsearch query and seeing the effects on search results. You have the ability to tweak and tune your search engine configuration. In this blog we are intending to resolve a specific business problem where these ‘tuning parameters’ are controlled by a Search API. This API wraps up significant business logic, and while we want to reduce the amount of manipulation from a relevance perspective, it still is going to shape the queries.   

A common approach is to introduce to the API a /quepid end point that wraps the existing business logic of search and then returns the response in the native Solr JSON format. This endpoint then appears to Quepid to behave just like the standard Solr API (here’s the documentation on what Quepid expects). We also would like to add some additional parameters that give Quepid additional visibility into what is happening. Here’s how all this fits together:

Our test system is running on Microsoft .NET so the following code samples are in C#.

Approach

  1. Add a HTTP GET /quepid end point to the API that looks like Solr.
  2. Wrap existing business logic in the search API, adding the echoParams and debug=true parameters being sent to Solr.
  3. On the response, translate back to a Solr response.   Notice the interesting formatting around JSONP, if the parameter json.wrf is provided then the JSON gets wrapped in some extra text.
  4. Add an API key which ensures /quepid is identified and has basic security

Solution

To add the extra parameters (json.wrf , echoParams , debug) to the existing search API we leveraged IDictionary from .NET as an optional parameter to the API. We added IDictionary<string, IEnumerable<string>> urlSolrRequest .

This ensures we can blend the API changes required for our Quepid endpoint with existing search API calls without having to worry about backward compatibility as the dictionary is processed only when it is available to the request API:

if(urlSolrRequest != null)
{
   var flattenedParams = urlSolrRequest
       .SelectMany(p => p.Value.Select(q => new {
       key = p.Key,
       value = q
       })
   );
   foreach (var solrRequestParam in flattenedParams)
   {
       request.AddQueryParameter(solrRequestParam.key, solrRequestParam.value);
   }
}

And then we processed the search request as usual:

var response = await _solrClient.ExecuteTaskAsync<SolrResponse<T>>(request, token);

The final request would look like this (our API is available on port 43668) :

http://localhost:43668/quepid?
q=<<query_from_quepid>>&
apiKey=<<custom_api_key_for_quepid>>&
fl=<<list_of_displayed_fields>>&
wt=json&debug=true&debug.explain.structured=true&hl=false&rows=10&indent=true&echoParams=all

Challenges 

Request parameter with a dot (json.wrf)

This one was probably the first hurdle we ran into and we resolved it by adding & annotating json.wrf as a JsonProperty:

public class SolrRequest
{
   public string defType { get; set; }
   public string qf { get; set; }
   public string bf { get; set; }
   public int ps { get; set; }
   public string fq { get; set; }
   public bool forceElevation { get; set; }
   public string elevateIds { get; set; }
   public string echoParams { get; set; }
   public bool? debug { get; set; }
   [JsonProperty("json.wrf")]
   public string jsonWrf { get; set; } = null;
}

The SolrRequest class above is the representation of the various additional parameters you can include in your Request Object.

Response format with search API key and json.wrf parameter

The Quepid documentation tells us that the API response from Quepid is expected in a particular format wrapped with a json.wrf parameter value, hence we switched our approach and created a new response model :

public class SolrSearchResult : SearchResult
{
   public string Body { get; set; }
   public string ContentType { get; set; }
}

We need to return the response from the Search API as a plain string, which will include the wrapper using the json.wrf parameter value. If we don’t do this, Quepid will generate an Invalid JSON Response error.

Query parameter naming

The query parameter at the client is called SearchTerm while Quepid expects and requests the query in a parameter q like in Solr, so to counter this issue we explicitly mapped the query, which was received as additional params (as Dictionary above) list and was mapped to the client query parameter “SearchTerm”.

Search API Security

To ensure we had a basic level of security implemented we added a secure key in the config and processed the request only when the same secure key was passed along with the request:

if (apiKey.Equals(solrSettings.QuepidApiKey))
{
    // Process the Authenticated request
}

Images not part of the search API response

Product images have an essential role in the e-commerce business and thus also when gathering judgements. Quepid can show images along with search results to help judges assess relevance. Our search API was not returning image information in the way Quepid expects, so to resolve this we processed the API response and conjugated the image URL along with the response body. The Library JObject came to our rescue here:

private JObject addImageUrlToBody(JObject body,SolrSearchResult solrSearchResult)
{
   foreach (var doc in  body["response"]["docs"].Children())
   {
       int id = doc["id"].Value<int>();
       var thumbImageUrl = solrSearchResult.
           docList.FirstOrDefault(i => i.ProductId == id)?.
           Items.FirstOrDefault()?.imageUri;
       doc["thumbImage"] = thumbImageUrl;
   }
   return body;    
}

We return the augmented response as : 

return Content(solrSearchResult.Body, solrSearchResult.ContentType);

Conclusion

This has been a fun exercise that taught us a lot about how to connect Quepid to a custom search API. Along the way we reported and resolved three issues with Quepid:

We hope this helps someone with similar needs – Nevertheless, we would love to be asked any questions or for clarifications or suggestions. Reach out to us on Relevance Slack!

If you need help using tools like this to improve relevance for your e-commerce site, get in touch.


Image by Fishing Net Vectors by Vecteezy