Quepid Migrates from Flask to Rails

Something has been cooking

We recently worked on moving Quepid from a Python backend to a Ruby backend, something Joel Spolsky would frown upon:

“They did it by making the single worst strategic mistake that any software company can make:

They decided to rewrite the code from scratch.”

– Joel Spolsky, referring to the Netscape rewrite that supposedly led to their doom.

source: http://www.joelonsoftware.com/articles/fog0000000069.html

So why did we go on and do something so “dangerous”, and arguably “stupid”? The answer to that requires a little bit of history.

Why we built Quepid

OSC’s mission is to help clients build the best search experience tailored for their customers and users. And just like any builder, you are as good as the tools you have at your disposal.

To be honest, the tools available for a search engineer are not so great. One of the nagging problems commonly faced is debugging why the search engine returned a specific set of results which did not match the expectations. So Splainer was born. Splainer helped developers understand what was driving the results and why.

Search, though, is not just a developer problem. Search is a conversation between multiple parties. So although Splainer’s addition to the tool belt helped make the job easier for developers, something else was needed to help the conversation between the developers and the domain experts. The domain experts are the ones who know if the results set returned by the search engine. They are the ones who have the knowledge and expertise to judge the results of the search, so how can we bring that knowledge to the process and workflow that the developers follow? That’s where Quepid comes in.

Tech Stack

Splainer was built with AngularJS, and the reason behind it is very simple. Splainer needed to be visual, not a command line tool, in order to make it easy to decipher cryptic debugging messages coming from Solr (and later Elasticsearch). The browser is the obvious choice for that. Splainer also needed to be able to communicate to clients’ search engine regardless of whatever security is employed that might block requests to the servers. So with an in browser app, as long as the user is running it from within the firewall (or VPN), the app can then access the search engine, without being blocked (which wouldn’t be the case if the app is running on servers outside of the firewall). So AngularJS made sense for that, specially since Splainer did not save any data, in fact it did not have a database at all. It was purely a front-end, single page application that ran in the browser. And that worked very well.

Quepid needed to leverage the power of Splainer, and add the missing pieces that were needed to integrate the domain experts into the flow. So it needed to have an AngularJS component, to be compatible with Splainer, as well as something to keep track of the judgments made by the users. So a backend with a connection to a database are now required.

Since the majority of the application runs in one screen, the case screen, and that is heavily controlled by Angular, Quepid needed a backend that was light and could work with Angular in a nice way. It did not need a fully fledged MVC framework, but a light weight API. And since Quepid at the time was mostly used internally at OSC, and primarily by its creator, Doug, he picked the language he was most familiar with, which is Python, and Flask which provided a good starting point for a light weight backend.

So to recap, these were the primary languages, frameworks, and dependencies for Quepid:

• Javascript
  • npm
  • bower
  • Angular JS
• Python
  • Flask
• MySql

And that worked, for a while…

Evolution of Quepid

Not long after its creation, people started showing interest in Quepid. Which made us happy. But also meant that whatever used to work in a hacky way for OSC might not cut it as an enterprise solution where people pay money to use the service.

Quepid needed more features, like [Organizations] (/blog/2015/08/25/quepid-gets-organized/) to share cases with other members. And as the feature set grew, so did the complexity of the application. Until we reached a point where the current stack was no longer conducive to developers being productive and fast.

During development, we hit a number of road blocks that had us say something like “This would be easier if we were using Rails”, or “Rails takes care of this for us”.

After a number of these, we decided it was time. It was time to say thank you to Flask, you served us well, but our journey together ends here.

There were a bunch of cases where the tools in Flask weren’t as mature as those in Rails (eg. managing migrations). But mostly, the thing that probably convinced us the most that we should make the switch is the integration between Angular and the backend. With Flask, we almost always had to duplicate the code between the Python side and the Javascript side. Services in Angular had to mimic behavior in the Python APIs and services. Whereas on the Rails side, there are Ruby gems that exist to bridge between the two parts of the application.

And as one very wise man once said:

“No code is more optimal than no code. No code is faster than no code. No code is more maintainable than no code.”

– Aaron Patterson

So we jumped on the opportunity to be able to delete code!

The key to a migration like this is to pick a scope, and stick to it. We did not do a full rewrite (I can hear Joel give a sigh of relief here). We had a functioning application that people use and have become accustomed to. Our goal is to make the codebase more maintainable and easier to add new features to. So the scope we picked was to replace the backend, and only the backend. That is very important. It defined what the task is, and limited the possibility to venture on a long and never ending rewrite.

There were many instances when the thought “Ouuhh, let me refactor that” would come to mind, but one must resist.

The truth about code is that it might never be good enough. There will always be ways to improve it. But the budget and time constraints make that a tricky proposition. And the reality is that if we keep it up to the developer to control his desires, the dark side will eventually prevail. Which is why setting the scope ahead of time was key to making this process a successful one.

What helped the transition from the Flask backend to the Rails backend is that we had previously done a reorganization of the code to make the APIs RESTful (as much as possible). We also had good test coverage that touched most of the aspects of the API. So, to a certain extent, replacing the APIs would be a smooth transition since not much from the Angular side, which is still the biggest piece of the app, had to change.

The stack now became the following:

• Javascript
  • npm
  • bower
  • AngularJS
• Ruby
  • Ruby on Rails
• MySql

Although there are still redundant pieces (eg. asset management in Rails and Bower), and parts that can be improved and cleaned up, this switch provides a solid basis for improving the codebase in the future.

What’s in it for me?

“Great, good job guys. But what’s in it for me?” you may ask. Well, that’s a great question.

Rewriting an application is not easy. It requires a lot of effort and time, and if you ask Joel, it’s the single worst strategic mistake a dev team can make. So why did we do it? We did it so that we can better serve our customers.

There are lots of things we want to do with Quepid. Lots of cool new features we want to add. Things that we hope you will love, things that will make you want to use Quepid more and more. We are excited about these features because we are all users of Quepid at OSC. We believe that Quepid is the best tool for relevancy tuning out there, and we depend on it as much as our clients do.

So in order for Quepid to be the best it can be we need to make sure that developers can maintain the application, and fix any bugs that occur at a fast pace. And when it’s time to add something new, it should be a joy to work on the Quepid code base, not scary.

So although you may not notice anything different in the app at the moment, we think that you will enjoy the progress that is coming in the near future. We hope you are as excited about what’s to come as we are.

Happy Tuning!