Real-time Doctest Checking In Vim

Vim is a terrific editor.

The one thing that kills me though is “the cycle”–the cycle of going from Vim to the console to try out what I just wrote, then back to Vim back to fix what i just wrote, then back to the console to try again. Most annoyingly this cycle is usually for little things that many modern IDEs can catch in real-time like every programming languages version of the proverbial forgotten semicolon. In Python, for example, I often find myself forgetting colons after blocks, or somehow screwing up indentation.

Anything I can do to catch the stupid stuff early and avoid wasting precious minutes in fix-try mode is invaluable. This is where the Vim plugin Syntastic really shines. Syntastic runs any kind of lint or compiler against your code when vim writes the file. It then gives you a nice bright arrow on the left saying HEY STUPID. Which is great. You see instantly where you goofed.

For Python, the lint/error checking tool that syntastic uses by default is Flake8. Flake8 combines error and Python style checking (against PEP8) into one tool. It tells you about the errors you definitely need to fix (missing colons and what-not) but also nags you about keeping your code consistently styled based on PEP8. Together, Flake8, Syntastic, and Vim give you a pretty amazing set of tools for doing rapid Python development. My mistakes are caught instantly and fixed right away and I spend less time in the fix-try cycle between Vim and the console.

Theres one ingredient though that seems to be screaming to be added to this killer combo – integration of Python doctests. If youre unfamiliar, doctests provide a very low barrier to entry method of developing basic tests around a Python class or function. In Python its pretty typical to develop a class and head to the interactive shell to try out your new creation. By playing around with the most obvious cases in the Python shell, you can see very quickly whether or not your code basically works. Doctests capture your experience in the shell by building some basic tests as usage examples in the class or functions docstring:

 1 2 3 4 5 6 7 8 9101112131415161718
class Adder(object):    """    Doctests examples, showing usage as if     Adder were used in the console:    >>> a=Adder(2,3)    >>> a.add()    5    >>> b=Adder(3,3)    >>> b.add()    5    """    def __init__(self, x, y):        self.x = x        self.y = y    def add(self):        print self.x + self.y

In the example above, theres 2 doctests for the Adder class. Everything after »> is treated as if its entered into the Python shell. Resulting output (returned or in this case printed) in the Python shellwould get output on the next line. The first doctest (adding 2 and 3) will pass. The latter, however, should fail (3+3 != 6).

Doctests fit nicely with rapid prototyping and experimentation that Python is often ideal for. They cant and shouldnt replace hundreds of detailed unittests, but they are great when you want to get something working reasonably well fast.

What would make them even more powerful is if we could integrate them with the Vim/Syntastic/Flake8 combo. Ideally we could seamlessly integrate doctest failures with the other Syntastic errors so test failures / syntax errors / style issues get all treated equally in the Syntastic/Vim interface. In fact thats exactly what Ive done! In my experimental Flake8 fork Ive added a fairly rudimentary first iteration prototype of running doctests and showing their failures by outputing them in Flake8:

Ive done this through using Pythons doctest module. Right now the code is fairly preliminary, using the doctest unittest API to run tests and then parse the output with some regexs. If youre curious how to work with this API, you can check out this file which has the meat of the doctest running for a specified path. As this code depends on parsing output, Im not certain how well it will work outside of the Python 2.7 line that Im using. If I have time and this continues to seem useful, Ill be switching to the advanced doctest API which should be more portable.

Ive installed this first iteration locally as a fun thing to try out. Of course it seems like an awesome idea, but Im curious to see how well it works in practice. Is the seamless integration with other Syntastic errors appropriate? Or do we need a more test-specific Vim plugin? Do testing errors need to be more verbose? Or should one really switch to a more debug-centric outside of an editor mode to get the deep information about a test failure? Would this kind of approach be appropriate for more in-depth traditional unit test frameworks? Will the tests take such a long time to run that vims write performance will plummet?

If youre curious, I encourage you to install Syntastic then try out my Flake8 fork (install via distutils) and let me know what you think (and more importantly what bugs you encounter). You may find you prefer it over the vanilla version if you use a lot of doctests. You may find Im nuts. Either way Id love to hear from you on what you think.