Weather fork-cast: Cloudy with a chance of Elasticsearch

There have been some interesting moves by cloud computing providers over the last few weeks with respect to search and relevance, particularly concerning the Elasticsearch open source stack. As many of you will know, Elastic is the company formed by Elasticsearch creator Shay Banon to offer commercial licenses for his open source search engine and it also offers a hosted version, Elastic Cloud.

Giant cloud provider Amazon Web Services (AWS) also offer a hosted Elasticsearch offering, although historically it has somewhat lagged behind the latest versions of Elasticsearch and prevented various options such as the use of custom plugins. As a user you have other options of course – to run Elasticsearch yourself on a cloud stack or to use specialist providers such as OneMoreCloud’s Bonsai. It really depends on how much of the administration you want to handle, how much control over configuration you need and of course, on your budget.

Elasticsearch isn’t the only open source product Amazon provides as a hosted service, and as a reaction to this several other companies including Redis Labs have been updating their licenses to prevent what they see as a large, rapacious rival from eating their business model, for example by adding the Commons Clause. This prevents services such as Amazon from selling hosted versions of the various open source projects. Reactions to this have been mixed – some see it as an affront to the very idea of open source licensing by restricting one of the core freedoms, some see AWS as a freeloader happily making vast sums from the efforts of others and agree that Something Must Be Done.

In parallel, Elastic have started to intermingle their open source code with commercially-licensed (although available in source code form) modules that provide features such as security restrictions. This hasn’t been universally popular either – the standard distribution of Elasticsearch now contains commercially licensed code and to get the pure, open source only version you need to download a different package. Despite protestations from Shay Banon and others that their hearts remain open source, this move makes it difficult for anyone embedding Elasticsearch into another product or hosting it to operate without breaking the license terms, which would seem like a protectionist move by Elastic.

Amazon’s reaction has been to effectively fork the open source project by releasing their own, purely open source version of Elasticsearch, folding in contributions from other projects (there’s always been a parallel ecosystem of plugins and enhancements for Elasticsearch that cover some of the functionality of Elastic’s commercial add-ons). So, Amazon now have a way to continue hosting Elasticsearch without stepping on Elastic’s toes and they also get some great PR as supporters of open source – but we should not lose sight of the fact that this is also a blatantly commercial move against Elastic’s protectionism.

Not to be outdone, Elastic recently announced a more friendly collaboration with Google Cloud, who of course are a major AWS competitor. This is Elastic’s own hosted service running on Google’s infrastructure, and it seems they’re not the only company playing nice.

For the Elasticsearch user, a richer ecosystem cannot be a bad thing – it gives us more choice when choosing a suitable technology stack. However although Amazon have promised to contribute any improvements back to the main Elasticsearch project, it will be Elastic who decide whether to accept these (as they control all contributions) and thus whether the two projects will gradually grow apart. We can’t be sure yet what will happen, and so the future is still a little cloudy.

If you need strategic advice on open source search or relevance don’t hesitate to contact us, and we hope to meet some of you at our Haystack conference in a few weeks.