Charlie Hull
Category: Big Data
Let’s turn the clock back a few years, perhaps to 2012 or so, when I wrote a somewhat snarky blog about Big Data, then the buzzword du jour. Big Data was a poorly defined marketing term – some would say ‘Big Data is anything that doesn’t fit in Excel’ – used to sell anything from servers to software to consulting. Underneath all the buzz, however, there were some seismic changes in the IT sector happening, with large companies realising they needed a strategy to build the skills and capabilities to deal with data that was, well, a lot bigger than it had been a few years before and was only going to grow in size. We saw the rise of open source platforms like Hadoop, large scale machine learning becoming possible and there was quite a lot of M&A (mergers and acquisitions) activity as giant IT companies looked to bring in expertise in this new area.
Avoid failure when choosing your strategy
I can see a parallel today with AI, which currently makes up at least 50% of the activity on my LinkedIn feed (and about 5% of the sense). It’s a buzzword, sure, but there are exciting and transformative possibilities and every company now needs an AI strategy. It’s less clear how to make this happen, especially if it’s a new area for your business to move into and/or you operate at significant scale.
If you’re a large enterprise, you’re now faced with some hard choices about whether to invest in new, possibly unproven technologies or wait until the vendors you’ve relied on can add AI features to their platforms. If you’re a vendor you need to add these features, fast, before you get left behind by a more adaptable and speedy AI-first startup. If you want to build your own solution from open source parts, we’re not even sure how to define open source for AI yet.
Meanwhile, some experts and thought leaders are rapidly backfilling their expertise, scratching out words like ‘crypto’ and hastily writing ‘AI’ on top. Journalists are luckily able to use AI image generation software to create yet another beautiful android to illustrate their AI articles, and VCs are frantically throwing money at anything AI related just to make sure they’re not missing out on future riches. It’s a crazy time – but it will eventually settle down.
Reduce the pain of making it work
At the moment we’re in the phase of AI where everyone has a prototype or a demo (we’ve built some ourselves) – but the next, harder, phase will be to deploy these as reliable, production-ready systems to solve real customer problems at a sensible cost (both financially and environmentally). In short, making AI work in practice. Learning how to measure the quality of results from AI-powered systems will be absolutely key, so you can sure they are actually solving your business problems (we’re already working on this at OSC using Quepid). There are many other challenges around creating, tuning, deploying and operating ML models, especially at scale. Some companies will need to ask themselves hard questions such as ‘are we ready for AI?’ (which we’re already addressing as part of our Discovery engagements).
Also, Data is going to be both Big and Important – you can’t generate the right AI-powered answer without reliable and easy-to-access ground truth – so choosing the right platform will be key. Not all existing databases will be able to add AI functionality successfully, but then again not all new AI products will scale or their vendors be able to support every customer need. We’ll undoubtedly soon see a winnowing of solutions, companies and providers.
Help is available
So how do you find the right people to help build amazing AI solutions? Well, we can learn from the days of Big Data: you need to find those who have been working with these technologies even before some of them were labeled as AI: machine learning, neural networks, natural language processing, data extraction and transformation, query processing, all at scale. Which all sounds very much like the areas we deal with when building search engines, just like in the Big Data era! It’s no coincidence that Hadoop, one of the key Big Data technologies, was created by Doug Cutting who also created the Lucene search engine library, or that many of the aforementioned M&A activities involved IT giants buying smaller search vendors. Search skills turned out to be very applicable to Big Data problems.
Perhaps you can join us at Haystack Europe on 20th & 21st September, in person in Berlin or online, where you can hear from us and many leading search experts on AI-related topics.
So I’ll conclude with a simple message: if you need help building AI, ask a search expert. We’ve been here before.