Opening up search is an ethical imperative

As search professionals, we’re often lost in our everyday minutiae. We don’t appreciate the impact of our work on everyday lives.

The data shows that when users encounter non-trivial amounts of content, they search. Just check out this report from Cludo, where they point to a 75% increase in their customer’s site-search traffic in 2019. They conclude:

The rapid growth in on-site search usage is likely a result of an ever-increasing volume of online content that needs to be parsed through …

… site visitors desire immediate results, as opposed to navigating several levels deep into a website

Users increasingly skip everything else, and go straight to search. Yet compare building search functionality to regular software development. A 12-year-old can learn to code a basic, yet satisfying game. But anything non-trivial with search? Where the student could ‘grok’ what they built? No luck. Instead you’ll need a crack team of PhDs, senior engineers, and domain experts. And good luck explaining ‘why’ the search engine does anything!

This is a shocking state of affairs given search’s ubiquitous impact on human well-being. And no I don’t think I’m overreaching. Search might mean a doctor diagnosing a patient with tricky symptoms. Bad search results might have life or death consequences. E-Commerce isn’t about buying pointless frivolities. It’s increasingly society’s economic glue. We no longer call on someone in sales to describe our needs verbally. Instead we request via the e-commerce search bar. Add job search, dating search, enterprise search, food delivery, grocery, legal, real estate, and so on, and you get a picture where search is indeed eating the world. What human activity will exist that won’t involve a search bar?

Why hasn’t the search practitioner community progressed?

Practical search know-how languishes. Search hasn’t had the network effect to develop best practices like general software dev. Yet it’s important enough that a handful of organizations do invest deeply in it. It’s a big enough business opportunity that Google can invest billions in an entire closed internal ecosystem. The rest of us soldier on in our siloed organizations, unable to collaborate or learn beyond our immediate colleagues.

Sure, open source search engines like Solr and Elasticsearch have long existed. But even here, organizations only collaborate on core infrastructure. They jealously guard crucial relevance and search app development know-how. Only recently have tendrils of a broad-based practitioner search community come into life via conferences like Haystack and MICES as well as community reports like Search Insights.

Part of the problem is visual. Product managers see that a new app has a different UI layout. It will therefore need customization to get right. Two search apps don’t look that different. There’s almost always a text entry box and a set of results, regardless of the domain. Amazon and Nike’s search might look similar enough. It’s not obvious that Amazon’s full search experience (relevance ranking, UI, the full package) differs a great deal from Nikes. Amazon sells ads in search; with a huge inventory, caring about supplier and marketplace needs when ranking. Nike has different concerns, focusing on brand loyalty and its specific fitness domain.

Many treat search as a utility. A problem that should be well-solved by now. They think, well I can just plug in a UI element that will handle my search, and move on. They imagine engineers toiling away somewhere behind the scenes to make obscure optimizations to make the search better. They don’t see that domain-specific qualities of search requires extensive customization to work in the natural language and jargon native to your product and user base.

Instead of search as the side-act of a GUI application, users want the opposite! They see GUI elements as a supporting role in a search application. Search is the main event!

Democratizing Search is Good For Everyone, Not Just Search People.

Search’s ubiquity makes it a big blind-spot for many organizations. Search requires deep expertise to do well, in a way that normal application development does not. If we, as humans, are increasingly dependent on search for everything from medicine to commerce to employment, isn’t search’s closed-off, barely functioning practitioner community a huge problem for everybody?

Yes. This sad state of affairs means that savvy vendors and even internal search teams will use search’s mystical nature to play on our fears. They’ll lay out complex buzzwordy machine learning terminology during a sales cycle or to win organizational stature. It’s easy to gain leverage when there’s valuable, hard won expertise you don’t have. When search seems so hard, and so few of us really understand it, how can we evaluate all the so-called solutions in front of us?

If, as a company, we’re hide-bound to a solution we don’t understand and can’t manage, then our users might suffer. If, further, we never talk to people outside our company about search, then everybody’s users suffer – which includes the entire economy and indeed society!

As users are reaching more-and-more for search, supporting the community collectively helps ensure positive outcomes for society as a whole. We’ll collectively help doctors find the right diagnosis for a suffering patient; support a purchasing agent find the right parts for an airplane they’re manufacturing; uplift lawyers seeking to hold the powerful accountable by helping them find solid legal precedent for their arguments.

There was a time that software development carried the same mystical connotations as search does today. Eventually we reached a critical mass where this could not be tolerated anymore. Open Source in the early 2000s was one part of this. This helped break down silos of expertise. Allowing sharing not just code: but software engineering know-how, techniques, and expertise.

We must do the same for search. Without it, we’re stuck in a stagnant loop, all solving the same problems. We won’t be able to make progress as medium-sized companies. We will deprive the next generation of search engineers and search product managers from being able to deliver value to their customers and create more sophisticated capabilities for society.

And on those everyday people using search: they deserve say in this too! We have an ethical obligation to be transparent in how we use their data in search. To delete data when asked. To be clear in what data is gathered, how and if it’s tied to real users, and who it’s shared with. It also means taking care in what we optimize for. Many search or similar applications have tremendous power. We must take sacred the user’s humanity, so we don’t send them into misinformation echo-chambers or confuse ads with organic search results.

Finally, in the end, don’t these users too deserve to understand how search works, in the same way we ought to be overcoming other kinds technical illiteracy? Growing the know-how of practitioners helps here, by building on a snowball effect of expertise.

It’s part of OpenSource Connection’s mission to help with these problems. To teach and help others work on search. It’s why I wrote Relevant Search and am contributing to AI Powered Search. It’s why we created the search relevance conference, Haystack (CFP ongoing!). We can’t do it alone, we need commitment from other organizations to teach and grow. To not hoard knowledge, but to break it out of siloes, and share at conferences like Haystack, at local meetups, writing, and other venues.

How your company can help!

How can you help in this endeavor? One way is to be a visible, healthy part of the search community. To clearly be a center of gravity anchoring that community. Each of these deserves it’s own blog post to do it justice!

  • Do open source right: The search world needs open source tooling. Doing open source right means learning how to engage the projects you’re a part of, not just consuming. It means organizational accountability around building and maintaining healthy open source, not just throwing software over the fence for visibility. It means evangelism, documentation, and building traction for a project – not just coding. This isn’t easy, it takes work and skill.
  • Be radically open: When your team shares via talks/blogs, don’t do it in half measures. Don’t hold back. The best content is detailed, deep, and shares our embarrassing mistakes. Tell the full, unedited story of your project’s evolution. When you demonstrate your openness and show the true challenges, potential hires will want to come help! Folks will feel be empowered to tell their stories in the same way. You’ll reinforce a standard for openness we sorely need to democratize this stuff.
  • Create search talent: I’m asked all the time how to hire search talent. The truth is, the best orgs build search talent. They create an apprenticeship pipeline through tools like pairing, training, and small projects. If you can do this right, the side effect is more people in the market to do the work, which is inherently democratizing.
  • Attract and support senior talent from the community: When you’re a radically open organization, doing open source right, with your own engine for generating great talent – senior search folks will notice. They’ll see you’re working on hard problems and they’ll seek you out. Senior staff are the most engaged in the community, by giving them a place to thrive, you support the community. You also link your company to that community by bringing the most senior members into your fold. But senior folks will only do this if they feel you’re supporting their career needs to grow and support open source!
  • Vendors: get your customers together: A proprietary search engine company? Much like Lucidworks has done with Activate, bring folks together that use your product. This isn’t all about open source. Organizations have good reason to reach for a search product. If you’re open, honest, and transparent, and move away from buzzword salad and talk about real problems your customers will love you. Let your customers talk to you and each other, they’ll get a clear view on how best to use your product and beyond to more general search skills & know-how. Besides, you are in a position with your marketing budget, to actually HAVE a search conference!
  • Big Cos: get your search teams together: Do you have many search product teams? We have seen more than one company put together a successful internal search conference. Your teams probably work in similar domains. They have tool, knowhow, and expertise to share with each other. Without an internal conference like this, they won’t know who to call on for help, or whether another team is solving similar problems. Internally, by broadening the expertise, you contribute to the broader community’s base of knowledge!

In short, you can help by creating a synergy[1] between your company’s success and the community’s. When enough companies do this together, I believe it will create a runaway snowball effect in time. We will reach a point where companies struggle less to put together the pieces for a reasonable search application, with talent not locked away at Google, Academia, or a few small places. We’ll collectively all be in a better position to fulfill our mission in supporting everyday people. Eventually to the point where this transparency can help with the broader ethical and privacy concerns of our field.

If all of this interests you, we hope to see you at Haystack (CFP Feb 7th!). We push to be a radically open practitioner search conference, we hope you can make it!

[1] Consulting drinking game, take a sip!