Doug Turnbull
Category: Uncategorized
We’ve been using Redis extensively for Quepid, our search relevancy collaboration canvas. Redis is a convenient and efficient way to interact with in-memory data structures. In fact, we’ve been using it as our primary database.
… and called it SubRedis!11
One thing that’s stuck out as problematic for us is isolating and segmenting parts of our application within Redis. I want to give each part of Quepids backend a safe and reliable Redis sandbox, without running 10 instances of Redis on my box. For example, if the user database does a flushdb, I want it to be only resetting the set of users. Similarly, I don’t want to worry that keys within the user database will clash with keys with the query-ratings database. Isolation is a good thing!
Introducing SubRedis
So what I’ve done is create Subredis – a simple Python keyspace wrapper around Redis. In short, SubRedis lets you do this:
r = redis.from_url("redis://localhost:6379")sub = SubRedis("keyspacename", r) #My very own Redis!sub.set("foo", "bar")sub.get("foo")sub.flushdb()Each SubRedis plays entirely in its own keyspace. Here for example, “flushdb” will only ever flush keys in “keyspacename” and the key “foo” will exist only here in this keyspace.
Redis has traditionally been a very flat key-value store. However, SubRedis allows us to create hierarchy within Redis. For example, to get the user’s set of queries for Quepid, we can easily do:
queryRedis = SubRedis(userId + "-queries", redis)And viola, each user has their own Redis instance ready to go and wholly owned by them. Moreover, theres no reason a SubRedis cannot in turn aggregate a SubRedis itself, achieving arbitrary levels of hiererachy:
queryRedis = SubRedis(userId + "-queries", redis)ratingsRedis = SubRedis(queryId + "-ratings", queryRedis)Pretty powerful stuff!
More Details
Subredis is a simple wrapper, all it does is prepend the name “keyspacename” to keys it receives. So a call in redis-cli to “keys *” before flushdb above would yield:
keyspacename_fooSubRedis implements most of the methods of redispy’s StrictRedis with a few exceptions where it may not be possible to intercept the keys. Every data structure is supported. Transactions are supported via pipelines. We currently don’t support many of the admin methods for safety sake (ie bgsave etc) and Lua scripting is not feasible with this approach. As SubRedis uses the “_” delimiter, its not encouraged to create key names with “_” in them.
Nevertheless, this has been extremely convenient when developing Quepid. Each component can trust in having its own isolated Redis instance, separated from other components.
So if you’re a Redis user, give SubRedis a try! Simply pip install subredis and let me know what you think (and if you find bugs!). I hope to blog about the fun Python metaprogramming that made this relatively simple to perform in the future.
And if you’re one of those people still wondering what data structure is right for your job, let us know! It may be Redis, or it could be any number of other things we specialize in (Cassandra, Solr, Hadoop, etc).