The Flask Mega-Tutorial, Part VII: Unit Testing

This is the seventh article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Recap

In the previous chapters of this tutorial we were concentrating in adding functionality to our little application, a step at a time. By now we have a database enabled application that can register users, log them in and out and let them view and edit their profiles.

In this session we are not going to add any new features to our application. Instead, we are going to find ways to add robustness to the code that we have already written, and we will also create a testing framework that will help us prevent failures and regressions in the future.

Let's find a bug

I mentioned at the end of the last chapter that I have intentionally introduced a bug in the application. Let me describe what the bug is, then we will use it to see what happens to our application when it does not work as expected.

The problem in the application is that there is no effort to keep the nicknames of our users unique. The initial nickname of a user is chosen automatically by the application. If the OpenID provider provides a nickname for the user then we will just use it. If not we will use the username part of the email address as nickname. If we get two users with the same nickname then the second one will not be able to register. To make matters worse, in the profile edit form we let users change their nicknames to whatever they want, and again there is no effort to avoid name collisions.

We will address these problems later, after we analyze how the application behaves when an error occurs.

Flask debugging support

So let's see what happens when we trigger our bug.

Let's start by creating a brand new database. On Linux:

rm app.db
./db_create.py

or on Windows:

del app.db
flask/Scripts/python db_create.py

You need two OpenID accounts to reproduce this bug, ideally from different providers, so that their cookies don't make this more complicated. Follow these steps to create a nickname collision:

  • login with your first account
  • go to the edit profile page and change the nickname to 'dup'
  • logout
  • login with your second account
  • go to the edit profile page and change the nickname to 'dup'

Oops! We've got an exception from sqlalchemy. The text of the error reads:

sqlalchemy.exc.IntegrityError
IntegrityError: (IntegrityError) column nickname is not unique u'UPDATE user SET nickname=?, about_me=? WHERE user.id = ?' (u'dup', u'', 2)

What follows after the error is a stack trace of the error, and actually it is a pretty nice one, where you can go to any frame and inspect source code or even evaluate expressions right in the browser.

The error is pretty clear, we tried to insert a duplicated nickname in the database. The database model had a unique constrain on the nickname field, so this is an invalid operation.

In addition to the actual error, we have a secondary problem in our hands. If a user inadvertently causes an error in our application (this one or any other that causes an exception) it will be him or her that gets the error with the revealing error message and the stack trace, not us. While this is a fantastic feature while we are developing, it is something we definitely do not want our users to ever see.

All this time we have been running our application in debug mode. The debug mode is enabled when the application starts, by passing a debug=True argument to the run method. This is how we coded our run.py start-up script.

When we are developing the application this is convenient, but we need to make sure it is turned off when we run our application in production mode. Let's just create another starter script that runs with debugging disabled (file runp.py):

#!flask/bin/python
from app import app
app.run(debug=False)

Now restart the application with:

./runp.py

And now try again to rename the nickname on the second account to 'dup'.

This time we do not get an error. Instead, we get an HTTP error code 500, which is Internal Server Error. Not a great looking error, but at least we are not exposing any details of our application to strangers. The error 500 page is generated by Flask when debugging is off and an unhandled exception occurs.

While this is better, we are now having two new issues. First a cosmetic one: the default error 500 page is ugly. The second problem is much more important. With things as they are we would never know when and if a user experiences a failure in our application because when debugging is turned off application failures are silently dismissed. Luckily there are easy ways to address both problems.

Custom HTTP error handlers

Flask provides a mechanism for an application to install its own error pages. As an example, let's define custom error pages for the HTTP errors 404 and 500, the two most common ones. Defining pages for other errors works in the same way.

To declare a custom error handler the errorhandler decorator is used (file app/views.py):

@app.errorhandler(404)
def not_found_error(error):
    return render_template('404.html'), 404

@app.errorhandler(500)
def internal_error(error):
    db.session.rollback()
    return render_template('500.html'), 500

Not much to talk about for these, as they are almost self-explanatory. The only interesting bit is the rollback statement in the error 500 handler. This is necessary because this function will be called as a result of an exception. If the exception was triggered by a database error then the database session is going to arrive in an invalid state, so we have to roll it back in case a working session is needed for the rendering of the template for the 500 error.

Here is the template for the 404 error:

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
  <h1>File Not Found</h1>
  <p><a href="{{ url_for('index') }}">Back</a></p>
{% endblock %}

And here is the one for the 500 error:

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
  <h1>An unexpected error has occurred</h1>
  <p>The administrator has been notified. Sorry for the inconvenience!</p>
  <p><a href="{{ url_for('index') }}">Back</a></p>
{% endblock %}

Note that in both cases we continue to use our base.html layout, so that the error page has the look and feel of the application.

Sending errors via email

To address our second problem we are going to configure two reporting mechanisms for application errors. The first of them is to have the application send us an email each time an error occurs.

Before we get into this let's configure an email server and an administrator list in our application (file config.py):

# mail server settings
MAIL_SERVER = 'localhost'
MAIL_PORT = 25
MAIL_USERNAME = None
MAIL_PASSWORD = None

# administrator list
ADMINS = ['you@example.com']

Of course it will be up to you to change these to what makes sense.

Flask uses the regular Python logging module, so setting up an email when there is an exception is pretty easy (file app/__init__.py):

from config import basedir, ADMINS, MAIL_SERVER, MAIL_PORT, MAIL_USERNAME, MAIL_PASSWORD

if not app.debug:
    import logging
    from logging.handlers import SMTPHandler
    credentials = None
    if MAIL_USERNAME or MAIL_PASSWORD:
        credentials = (MAIL_USERNAME, MAIL_PASSWORD)
    mail_handler = SMTPHandler((MAIL_SERVER, MAIL_PORT), 'no-reply@' + MAIL_SERVER, ADMINS, 'microblog failure', credentials)
    mail_handler.setLevel(logging.ERROR)
    app.logger.addHandler(mail_handler)

Note that we are only enabling the emails when we run without debugging.

Testing this on a development PC that does not have an email server is easy, thanks to Python's SMTP debugging server. Just open a new console window (command prompt for Windows users) and run the following to start a fake email server:

python -m smtpd -n -c DebuggingServer localhost:25

When this is running, the emails sent by the application will be received and displayed in the console window.

Logging to a file

Receiving errors via email is nice, but sometimes this isn't enough. There are some failure conditions that do not end in an exception and aren't a major problem, yet we may want to keep track of them in a log in case we need to do some debugging.

For this reason, we are also going to maintain a log file for the application.

Enabling file logging is similar to the email logging (file app/__init__.py):

if not app.debug:
    import logging
    from logging.handlers import RotatingFileHandler
    file_handler = RotatingFileHandler('tmp/microblog.log', 'a', 1 * 1024 * 1024, 10)
    file_handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'))
    app.logger.setLevel(logging.INFO)
    file_handler.setLevel(logging.INFO)
    app.logger.addHandler(file_handler)
    app.logger.info('microblog startup')

The log file will go to our tmp directory, with name microblog.log. We are using the RotatingFileHandler so that there is a limit to the amount of logs that are generated. In this case we are limiting the size of a log file to one megabyte, and we will keep the last ten log files as backups.

The logging.Formatter class provides custom formatting for the log messages. Since these messages are going to a file, we want them to have as much information as possible, so we write a timestamp, the logging level and the file and line number where the message originated in addition to the log message and the stack trace.

To make the logging more useful, we are lowering the logging level, both in the app logger and the file logger handler, as this will give us the opportunity to write useful messages to the log without having to call them errors. As an example, we start by logging the application start up as an informational level. From now on, each time you start the application without debugging the log will record the event.

While we don't have a lot of need for a logger at this time, debugging a web server that is online and in use is very difficult. Logging messages to a file is an extremely useful tool in diagnosing and locating issues, so we are now all ready to go should we need to use this feature.

The bug fix

Let's fix our nickname duplication bug.

As discussed earlier, there are two places that are currently not handling duplicates. The first is in the after_login handler for Flask-Login. This is called when a user successfully logs in to the system and we need to create a new User instance. Here is the affected snippet of code, with the fix in it (file app/views.py):

    if user is None:
        nickname = resp.nickname
        if nickname is None or nickname == "":
            nickname = resp.email.split('@')[0]
        nickname = User.make_unique_nickname(nickname)
        user = User(nickname = nickname, email = resp.email)
        db.session.add(user)
        db.session.commit()

The way we solve the problem is by letting the User class pick a unique name for us. This is what the new make_unique_nickname method does (file app/models.py):

    class User(db.Model):
    # ...
    @staticmethod
    def make_unique_nickname(nickname):
        if User.query.filter_by(nickname=nickname).first() is None:
            return nickname
        version = 2
        while True:
            new_nickname = nickname + str(version)
            if User.query.filter_by(nickname=new_nickname).first() is None:
                break
            version += 1
        return new_nickname
    # ...

This method simply adds a counter to the requested nickname until a unique name is found. For example, if the username "miguel" exists, the method will suggest "miguel2", but if that also exists it will go to "miguel3" and so on. Note that we coded the method as a static method, since it this operation does not apply to any particular instance of the class.

The second place where we have problems with duplicate nicknames is the view function for the edit profile page. This one is a little tricker to handle, because it is the user choosing the nickname. The correct thing to do here is to not accept a duplicated nickname and let the user enter another one. We will address this by adding custom validation to the nickname form field. If the user enters an invalid nickname we'll just fail the validation for the field, and that will send the user back to the edit profile page. To add our validation we just override the form's validate method (file app/forms.py):

from app.models import User

class EditForm(Form):
    nickname = TextField('nickname', validators=[DataRequired()])
    about_me = TextAreaField('about_me', validators=[Length(min=0, max=140)])

    def __init__(self, original_nickname, *args, **kwargs):
        Form.__init__(self, *args, **kwargs)
        self.original_nickname = original_nickname

    def validate(self):
        if not Form.validate(self):
            return False
        if self.nickname.data == self.original_nickname:
            return True
        user = User.query.filter_by(nickname=self.nickname.data).first()
        if user != None:
            self.nickname.errors.append('This nickname is already in use. Please choose another one.')
            return False
        return True

The form constructor now takes a new argument original_nickname. The validate method uses it to determine if the nickname has changed or not. If it hasn't changed then it accepts it. If it has changed, then it makes sure the new nickname does not exist in the database.

Next we add the new constructor argument to the view function:

@app.route('/edit', methods=['GET', 'POST'])
@login_required
def edit():
    form = EditForm(g.user.nickname)
    # ...

To complete this change we have to enable field errors to show in our template for the form (file app/templates/edit.html):

        <td>Your nickname:</td>
        <td>
            {{ form.nickname(size=24) }}
            {% for error in form.errors.nickname %}
            <br><span style="color: red;">[{{ error }}]</span>
            {% endfor %}
        </td>

Now the bug is fixed and duplicates will be prevented... except when they are not. We still have a potential problem with concurrent access to the database by two or more threads or processes, but this will be the topic of a future article.

At this point you can try again to select a duplicated name to see how the form nicely handles the error.

Unit testing framework

To close this session on testing, let's talk about automated testing a bit.

As the application grows in size it gets more and more difficult to ensure that code changes don't break existing functionality.

The traditional approach to prevent regressions is a very good one. You write tests that exercise all the different features of the application. Each test runs a focused part and verifies that the result obtained is the expected one. The tests are executed periodically to make sure that the application works as expected. When the test coverage is large you can have confidence that modifications and additions do not affect the application in a bad way just by running the tests.

We will now build a very simple testing framework using Python's unittest module (file tests.py):

#!flask/bin/python
import os
import unittest

from config import basedir
from app import app, db
from app.models import User

class TestCase(unittest.TestCase):
    def setUp(self):
        app.config['TESTING'] = True
        app.config['WTF_CSRF_ENABLED'] = False
        app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'test.db')
        self.app = app.test_client()
        db.create_all()

    def tearDown(self):
        db.session.remove()
        db.drop_all()

    def test_avatar(self):
        u = User(nickname='john', email='john@example.com')
        avatar = u.avatar(128)
        expected = 'http://www.gravatar.com/avatar/d4c74594d841139328695756648b6bd6'
        assert avatar[0:len(expected)] == expected

    def test_make_unique_nickname(self):
        u = User(nickname='john', email='john@example.com')
        db.session.add(u)
        db.session.commit()
        nickname = User.make_unique_nickname('john')
        assert nickname != 'john'
        u = User(nickname=nickname, email='susan@example.com')
        db.session.add(u)
        db.session.commit()
        nickname2 = User.make_unique_nickname('john')
        assert nickname2 != 'john'
        assert nickname2 != nickname

if __name__ == '__main__':
    unittest.main()

Discussing the unittest module is outside the scope of this article. Let's just say that class TestCase holds our tests. The setUp and tearDown methods are special, these are run before and after each test respectively. A more complex setup could include several groups of tests each represented by a unittest.TestCase subclass, and each group then would have independent setUp and tearDown methods.

These particular setUp and tearDown methods are pretty generic. In setUp the configuration is edited a bit. For instance, we want the testing database to be different that the main database. In tearDown we just reset the database contents.

Tests are implemented as methods. A test is supposed to run some function of the application that has a known outcome, and should assert if the result is different than the expected one.

So far we have two tests in the testing framework. The first one verifies that the Gravatar avatar URLs from the previous article are generated correctly. Note how the expected avatar is hardcoded in the test and checked against the one returned by the User class.

The second test verifies the make_unique_nickname method we just wrote, also in the User class. This test is a bit more elaborate, it creates a new user and writes it to the database, then ensures the same name is not allowed as a unique name. It then creates a second user with the suggested unique name and tries one more time to request the first nickname. The expected result for this second part is to get a suggested nickname that is different from the previous two.

To run the test suite you just run the tests.py script:

./tests.py

If there are any errors, you will get a report in the console.

Final words

This ends today's discussion of debugging, errors and testing. I hope you found this article useful.

As always, if you have any comments please write below.

The code of the microblog application update with today's changes is available below for download:

Download microblog-0.7.zip.

As always, the flask virtual environment and the database are not included. See previous articles for instructions on how to generate them.

I hope to see you again in the next installment of this series.

Thank you for reading!

Miguel

63 comments

  • #1 Epsilon said :

    Thank you for making this effort. May I suggest a subject for one of you future entries? I am really interested to see how you would structure a larger Flask application. A clear explanation of when, how and why to use blueprints and split up you app into modules (and how to initialize those) would help a lot. I find these things very hard to figure out without some guidance. Digging through the Flask mailing list archives reveals other users with the same problem. I know of the existence of the large-app-how-to wiki page and I browsed through the source of several Flask applications on Github/BitBucket, but nothing beats a well guided explanation (imho). Anyway, I gained an extra RSS feed. :-)

  • #2 Miguel Grinberg said :

    @Epsilon: thanks for your comment. I personally don't find blueprints that interesting. The way I structure my applications is based on mitsuhiko's large app how to, but without the blueprints. This little microblog app is also structured like that. For really big apps I just split the views.py, models.py and forms.py into multiple files each, but I don't go too crazy on splitting things up because then maintaining dependencies between modules becomes a nightmare.

  • #3 MIquel said :

    My congratulations for this series of posts!. I'm relatively new to Flask and this is by far the best intro I've found. Apart from being quite complete and self consistent is written using a good didactic style. I'm looking forward forthcoming posts about your microblog application. ¡Enhorabuena!

  • #4 Raj said :

    Great series , I am going to follow your articles. Speaking of the login manager with Python 2.7 on my setup , had to refer to flask_login as follows from flask_login import LoginManager

  • #5 Wellington Cordeiro said :

    I had an issue early in this post on Windows 7. When I run the db_upgrade.py file, after deleting and creating the database. I have this stack trace. http://dpaste.com/788109/ I'm not quite sure how I can correct this. The database is being deleted. But appears to be still in memory or something.

  • #6 Raj said :

    Delete the database file. If not connect to db and drop the table. You can connect to SQLite db like this on unix sqlite3 mydatabase.db You get to the sqlite3 prompt >.show tables Should see the tables Then drop the table >drop table table name ;

  • #7 Miguel Grinberg said :

    @Wellington: sorry about that, I obviously missed it. I have made a small update to the db_create.py script, and also updated this page. The problem is really that db_create.py is creating the latest version of the database, when you run db_upgrade.py after that it tries to create tables that already exist in the database. If you just run db_create.py you should be fine, but make sure you update your db_create.py script from either the zip file or from the database chapter. Sorry about that, I find that no matter how much effort I put into testing and proofreading my articles there is always something I miss.

  • #8 Marcel said :

    Hi Miguel, I really appreciate the effort you put into creating this tutorial. There is a small omission in your steps today, however. Since EditForm now excepts an argument, original_nickname, views.py needs to be edited as well and the current nickname passed in when instantiating EditForm. The fix was present in your zip file but is not mentioned here.

  • #9 Miguel Grinberg said :

    @Marcel: good catch, I have now updated the article. Thanks!

  • #10 Dogukan Tufekci said :

    Hello Miguel! Thank you very much for this awesome tutorial. I believe developing an exemplary app is the best way to learn Flask. My question is in regards to 'Sending errors via email' part: I run the app with both run.py and runp.py however I dont seem to receive any emails when I activate the 'unique nickname bug'. I'd like to learn how I can send the bugs let's say using a gmail account to an admin with another gmail account. Thanks!

  • #11 Dogukan Tufekci said :

    @Miguel Bug fix is missing for download. @oid.after_login nickname = User.make_unique_nickname(nickname)

  • #12 Miguel Grinberg said :

    @Dogukan: good catch on the missing make_unique_nickname, it should now be fixed. Regarding sending emails, you may want to test sending of emails in the Python shell. You will find some code and instructions in the 'Email Support' article. If you need to send emails to other email addresses you can add all the admin emails you want in the config.py file, look for the ADMIN array there.

  • #13 George Mabley said :

    Hey Miguel. First of all, thank you for making the tutorial. It has made this whole process less daunting and very fun. I may have missed it while following along in earlier lessons, but I had to import User from app.models when adjusting the forms.py in this lesson. Also, running Windows 7 I had to import os in my tests.py. Your files revealed the missing lines, but this page could be updated for us writing the code out ourselves. Again, thank you for all the help!

  • #14 Miguel Grinberg said :

    @George: Thanks for alerting me of these little problems. Both are corrected in the text above.

  • #15 Austin Hulak said :

    Hi Miguel, Just want to stop and tell you that this tutorial is phenomenal. I've been going through almost every Flask tutorial on the web, and not a single one goes into the amount of depth/detail as yours. Kudos on that. One thing that stands out particularly is that you cover Unit testing. As a fairly novice programmer, this is something I haven't seen a lot of on the internet. It's almost as if all the tutorials out there just want to see you get something live.. but just barely. I know you have a lot of topics on deck for this blog, but if you could go into more depth on unit testing (eg how to design a proper unit test, test drive development) that would be very appreciated. Or at the least, do you know of any good resources to learn more about this? Thanks for all the tutorials!! (After this I'm gearin' up for the Arduino Tut) Cheers, Austin

  • #16 Miguel Grinberg said :

    @Austin: Thanks. There is more about unit testing in part XVI, where I implement code coverage for the unit testing framework and then use the coverage data to discover what code my unit tests are missing.

  • #17 aikah said :

    first , i love your tutorial , i'm adapting your script for a django tutorial , i'll come back at you later about it. Can i suggest you make your make_unique_nickname function recursive instead on using a while true loop ? you just need to pass the version as a param at each iteration ;) thanks again !

  • #18 Miguel Grinberg said :

    @aikah: recursive functions could use a big stack if many levels of calls occur, they can even cause a stack overflow.

  • #19 Ashwin said :

    Great tutorial! I have a similar question in mind when designing my own Flask based web app. I have my oauth consumer handlers in a separate module say oauthmodule.py. If there is an exception in one of the oauth sequences, obviously an exception will be raised. Does this translate directly to a 500 error in the Flask's controllers?

  • #20 Miguel Grinberg said :

    @Ashwin: Yes, any time your code throws an exception Flask will issue a 500 page back to the client.

  • #21 Seth said :

    Thank you so much for this series Miguel, it has been really helpful in learning the Flask framework. I am currently having issues with both the email logging and file logging. I am running Ubuntu 12.04 as my development environment and have my flask application happily running in a virtualenv. The issue is that I can only launch the mail server as sudo, and even so, emails are not being sent or received. Secondly, I am unable to use the RotatingFileHandler as I get an ImportError when attempting to run my application. Have you experienced these issues or have any suggestions as to what the issue may be?

  • #22 James said :

    Hi Miguel, Thanks a ton for this tutorial, it's a huge help! I am running into an odd issue, I'm following along using a hosted environment over at pythonanywhere.com Everything has been great so far, except when setting up my errorhandler views, as long as I am returning an error code along with the render_template, it will not render the template and instead I get the ugly 404 or 500 server errors. I can return a string fine, I can return just the render_template call, but if I render_template('whatevertemplate.html'), whatevererrorcode, I get no template and just the server error screen (404, 500, what have you). Can you shed any light on why this could be happening? Since it seems to work fine without returning an error code along with the template, I've removed them, but I'm not sure what the side effects of this is for other error handling purposes.

  • #23 Miguel Grinberg said :

    @James: can you return a string and an error code, or does that also fail?

  • #24 Vasco Correia said :

    Hello, and thanks for the tutorials. I have (I think) a problem with unit testing. The tests.py script is just taking too long. And when I interrupt the process, the stack trace shows that the execution is going into the flask/ folder created by virtualenv (http://pastebin.com/Mr2QY9rS). Is it running the tests of all the libs? Is this what's supposed to happen? Thanks in advance.

  • #25 Miguel Grinberg said :

    @Vasco: Have you checked that your code matches what I have on github? From the stack trace it appears one of the tests is in a endless loop when trying to find a unique nickname.

Leave a Comment

Note: all comments are screened before they are published. Thank you for your patience!