At MichiganLabs, we believe that unit testing is a vital part of developing any application. Tests help document the expected behavior of your code, give instant feedback on mistakes while refactoring, and help prevent you from writing sphagetti code (since it would be difficult to test). So how do you know when you have written enough tests? This is where coverage measurement tools come in.

Coverage tools will track what parts of your code are executed by your test cases and give you a report of what remains untested. These tools exist for virtually every language. The one I use for Python is called coverage.py. However, these tools sometimes have nuances that could trick you into thinking you have a higher amount of coverage than you really do.

Here’s a sample of some code I wrote using the Flask-RESTful framework:

class UserResource(Resource):
 @marshal_with(user_fields)
 def get(self, user_id=None, username=None):
 if user_id:
 user = User.get_by_id(user_id)
 elif username:
 user = User.get_by_username(username)
 if not user:
 abort(404)
 return user

I also wrote some test cases, which do something like this:

GET /users/1, assert response == 200
# Verify that I can retrieve a valid user by username
GET /users/joshfriend, assert response == 200
# Verify that I can retrieve a valid user by user ID
GET /users/9001, assert response == 404
# Verify that trying to retrieve a nonexistent user returns 404

I even made sure I was executing every line of code by using coverage.py:

$ coverage run --source app/ -m py.test tests/
$ coverage report --show-missing

And the result looked OK. 100% coverage! Right?…

tests/api/test_user_api.py ........................
Name Stmts Miss Cover Missing
------------------------------------------------------------
app/api/user 88 0 100%

… Not really. Looking back at the code, I realized I forgot something important about Python. The if statement that tests to see if a user ID was provided does not check explicitly for None. Thus, a user ID of 0 will be ignored (, None and empty containers are considered False to an if statement). Let’s add another test case to see what happens:

GET /users/0, assert response == 200

Since the user_id provided is ignored and no username was supplied, the user variable is never assigned. The test case will throw an exception:

UnboundLocalError: local variable 'user' referenced before assignment

Fortunately, coverage.py has a convenient way of looking for this kind of gap in test coverage by recording the branches traversed in the code instead of only recording which lines were executed:

$ coverage run --branch --source app/ -m py.test tests/
tests/api/test_user_api.py ........................
$ coverage report --show-missing
Name Stmts Miss Branch BrMiss Cover Missing
--------------------------------------------------------------------------
app/api/user 88 0 30 1 97%
$ coverage html

Note the extra command to generate an HTML report. The console reporter only reports the number of branch statements that are missing coverage, not where they are. The HTML report is an easy way to see that info. The generated report is stored in htmlcov/index.html. For the example code above, the coverage report looks like this:

This made it obvious that I was not exercising every possible path through the code. The fix to make the new test case pass was really simple:

class UserResource(Resource):
 @marshal_with(user_fields)
 def get(self, user_id=None, username=None):
 if user_id is not None:
 user = User.get_by_id(user_id)
 else:
 user = User.get_by_username(username)
 if not user:
 abort(404)
 return user

Bear in mind that just because your coverage tool now reports 100% branch coverage, it does not mean your test coverage is comprehensive. Branch coverage monitoring is a good tool to help you get there but be sure to think about other edge cases, such as raised exceptions when calling external library functions. Multiple conditions or the use of the all() builtin inside an if statement can also be tricky since the coverage tool will only record whether or not the execution path went through the if or the else section, not which conditions of the if statement made it take that route.