2012-08-01T06:50:31Z

The Flask Mega-Tutorial, Part VII: Unit Testing

(Great news! There is a new version of this tutorial!)

This is the seventh article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Recap

In the previous chapters of this tutorial we were concentrating in adding functionality to our little application, a step at a time. By now we have a database enabled application that can register users, log them in and out and let them view and edit their profiles.

In this session we are not going to add any new features to our application. Instead, we are going to find ways to add robustness to the code that we have already written, and we will also create a testing framework that will help us prevent failures and regressions in the future.

Let's find a bug

I mentioned at the end of the last chapter that I have intentionally introduced a bug in the application. Let me describe what the bug is, then we will use it to see what happens to our application when it does not work as expected.

The problem in the application is that there is no effort to keep the nicknames of our users unique. The initial nickname of a user is chosen automatically by the application. If the OpenID provider provides a nickname for the user then we will just use it. If not we will use the username part of the email address as nickname. If we get two users with the same nickname then the second one will not be able to register. To make matters worse, in the profile edit form we let users change their nicknames to whatever they want, and again there is no effort to avoid name collisions.

We will address these problems later, after we analyze how the application behaves when an error occurs.

Flask debugging support

So let's see what happens when we trigger our bug.

Let's start by creating a brand new database. On Linux:

rm app.db
./db_create.py

or on Windows:

del app.db
flask/Scripts/python db_create.py

You need two OpenID accounts to reproduce this bug, ideally from different providers, so that their cookies don't make this more complicated. Follow these steps to create a nickname collision:

  • login with your first account
  • go to the edit profile page and change the nickname to 'dup'
  • logout
  • login with your second account
  • go to the edit profile page and change the nickname to 'dup'

Oops! We've got an exception from sqlalchemy. The text of the error reads:

sqlalchemy.exc.IntegrityError
IntegrityError: (IntegrityError) column nickname is not unique u'UPDATE user SET nickname=?, about_me=? WHERE user.id = ?' (u'dup', u'', 2)

What follows after the error is a stack trace of the error, and actually it is a pretty nice one, where you can go to any frame and inspect source code or even evaluate expressions right in the browser.

The error is pretty clear, we tried to insert a duplicated nickname in the database. The database model had a unique constrain on the nickname field, so this is an invalid operation.

In addition to the actual error, we have a secondary problem in our hands. If a user inadvertently causes an error in our application (this one or any other that causes an exception) it will be him or her that gets the error with the revealing error message and the stack trace, not us. While this is a fantastic feature while we are developing, it is something we definitely do not want our users to ever see.

All this time we have been running our application in debug mode. The debug mode is enabled when the application starts, by passing a debug=True argument to the run method. This is how we coded our run.py start-up script.

When we are developing the application this is convenient, but we need to make sure it is turned off when we run our application in production mode. Let's just create another starter script that runs with debugging disabled (file runp.py):

#!flask/bin/python
from app import app
app.run(debug=False)

Now restart the application with:

./runp.py

And now try again to rename the nickname on the second account to 'dup'.

This time we do not get an error. Instead, we get an HTTP error code 500, which is Internal Server Error. Not a great looking error, but at least we are not exposing any details of our application to strangers. The error 500 page is generated by Flask when debugging is off and an unhandled exception occurs.

While this is better, we are now having two new issues. First a cosmetic one: the default error 500 page is ugly. The second problem is much more important. With things as they are we would never know when and if a user experiences a failure in our application because when debugging is turned off application failures are silently dismissed. Luckily there are easy ways to address both problems.

Custom HTTP error handlers

Flask provides a mechanism for an application to install its own error pages. As an example, let's define custom error pages for the HTTP errors 404 and 500, the two most common ones. Defining pages for other errors works in the same way.

To declare a custom error handler the errorhandler decorator is used (file app/views.py):

@app.errorhandler(404)
def not_found_error(error):
    return render_template('404.html'), 404

@app.errorhandler(500)
def internal_error(error):
    db.session.rollback()
    return render_template('500.html'), 500

Not much to talk about for these, as they are almost self-explanatory. The only interesting bit is the rollback statement in the error 500 handler. This is necessary because this function will be called as a result of an exception. If the exception was triggered by a database error then the database session is going to arrive in an invalid state, so we have to roll it back in case a working session is needed for the rendering of the template for the 500 error.

Here is the template for the 404 error:

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
  <h1>File Not Found</h1>
  <p><a href="{{ url_for('index') }}">Back</a></p>
{% endblock %}

And here is the one for the 500 error:

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
  <h1>An unexpected error has occurred</h1>
  <p>The administrator has been notified. Sorry for the inconvenience!</p>
  <p><a href="{{ url_for('index') }}">Back</a></p>
{% endblock %}

Note that in both cases we continue to use our base.html layout, so that the error page has the look and feel of the application.

Sending errors via email

To address our second problem we are going to configure two reporting mechanisms for application errors. The first of them is to have the application send us an email each time an error occurs.

Before we get into this let's configure an email server and an administrator list in our application (file config.py):

# mail server settings
MAIL_SERVER = 'localhost'
MAIL_PORT = 25
MAIL_USERNAME = None
MAIL_PASSWORD = None

# administrator list
ADMINS = ['you@example.com']

Of course it will be up to you to change these to what makes sense.

Flask uses the regular Python logging module, so setting up an email when there is an exception is pretty easy (file app/__init__.py):

from config import basedir, ADMINS, MAIL_SERVER, MAIL_PORT, MAIL_USERNAME, MAIL_PASSWORD

if not app.debug:
    import logging
    from logging.handlers import SMTPHandler
    credentials = None
    if MAIL_USERNAME or MAIL_PASSWORD:
        credentials = (MAIL_USERNAME, MAIL_PASSWORD)
    mail_handler = SMTPHandler((MAIL_SERVER, MAIL_PORT), 'no-reply@' + MAIL_SERVER, ADMINS, 'microblog failure', credentials)
    mail_handler.setLevel(logging.ERROR)
    app.logger.addHandler(mail_handler)

Note that we are only enabling the emails when we run without debugging.

Testing this on a development PC that does not have an email server is easy, thanks to Python's SMTP debugging server. Just open a new console window (command prompt for Windows users) and run the following to start a fake email server:

python -m smtpd -n -c DebuggingServer localhost:25

When this is running, the emails sent by the application will be received and displayed in the console window.

Logging to a file

Receiving errors via email is nice, but sometimes this isn't enough. There are some failure conditions that do not end in an exception and aren't a major problem, yet we may want to keep track of them in a log in case we need to do some debugging.

For this reason, we are also going to maintain a log file for the application.

Enabling file logging is similar to the email logging (file app/__init__.py):

if not app.debug:
    import logging
    from logging.handlers import RotatingFileHandler
    file_handler = RotatingFileHandler('tmp/microblog.log', 'a', 1 * 1024 * 1024, 10)
    file_handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'))
    app.logger.setLevel(logging.INFO)
    file_handler.setLevel(logging.INFO)
    app.logger.addHandler(file_handler)
    app.logger.info('microblog startup')

The log file will go to our tmp directory, with name microblog.log. We are using the RotatingFileHandler so that there is a limit to the amount of logs that are generated. In this case we are limiting the size of a log file to one megabyte, and we will keep the last ten log files as backups.

The logging.Formatter class provides custom formatting for the log messages. Since these messages are going to a file, we want them to have as much information as possible, so we write a timestamp, the logging level and the file and line number where the message originated in addition to the log message and the stack trace.

To make the logging more useful, we are lowering the logging level, both in the app logger and the file logger handler, as this will give us the opportunity to write useful messages to the log without having to call them errors. As an example, we start by logging the application start up as an informational level. From now on, each time you start the application without debugging the log will record the event.

While we don't have a lot of need for a logger at this time, debugging a web server that is online and in use is very difficult. Logging messages to a file is an extremely useful tool in diagnosing and locating issues, so we are now all ready to go should we need to use this feature.

The bug fix

Let's fix our nickname duplication bug.

As discussed earlier, there are two places that are currently not handling duplicates. The first is in the after_login handler for Flask-Login. This is called when a user successfully logs in to the system and we need to create a new User instance. Here is the affected snippet of code, with the fix in it (file app/views.py):

    if user is None:
        nickname = resp.nickname
        if nickname is None or nickname == "":
            nickname = resp.email.split('@')[0]
        nickname = User.make_unique_nickname(nickname)
        user = User(nickname = nickname, email = resp.email)
        db.session.add(user)
        db.session.commit()

The way we solve the problem is by letting the User class pick a unique name for us. This is what the new make_unique_nickname method does (file app/models.py):

    class User(db.Model):
    # ...
    @staticmethod
    def make_unique_nickname(nickname):
        if User.query.filter_by(nickname=nickname).first() is None:
            return nickname
        version = 2
        while True:
            new_nickname = nickname + str(version)
            if User.query.filter_by(nickname=new_nickname).first() is None:
                break
            version += 1
        return new_nickname
    # ...

This method simply adds a counter to the requested nickname until a unique name is found. For example, if the username "miguel" exists, the method will suggest "miguel2", but if that also exists it will go to "miguel3" and so on. Note that we coded the method as a static method, since it this operation does not apply to any particular instance of the class.

The second place where we have problems with duplicate nicknames is the view function for the edit profile page. This one is a little tricker to handle, because it is the user choosing the nickname. The correct thing to do here is to not accept a duplicated nickname and let the user enter another one. We will address this by adding custom validation to the nickname form field. If the user enters an invalid nickname we'll just fail the validation for the field, and that will send the user back to the edit profile page. To add our validation we just override the form's validate method (file app/forms.py):

from app.models import User

class EditForm(Form):
    nickname = StringField('nickname', validators=[DataRequired()])
    about_me = TextAreaField('about_me', validators=[Length(min=0, max=140)])

    def __init__(self, original_nickname, *args, **kwargs):
        Form.__init__(self, *args, **kwargs)
        self.original_nickname = original_nickname

    def validate(self):
        if not Form.validate(self):
            return False
        if self.nickname.data == self.original_nickname:
            return True
        user = User.query.filter_by(nickname=self.nickname.data).first()
        if user != None:
            self.nickname.errors.append('This nickname is already in use. Please choose another one.')
            return False
        return True

The form constructor now takes a new argument original_nickname. The validate method uses it to determine if the nickname has changed or not. If it hasn't changed then it accepts it. If it has changed, then it makes sure the new nickname does not exist in the database.

Next we add the new constructor argument to the view function:

@app.route('/edit', methods=['GET', 'POST'])
@login_required
def edit():
    form = EditForm(g.user.nickname)
    # ...

To complete this change we have to enable field errors to show in our template for the form (file app/templates/edit.html):

        <td>Your nickname:</td>
        <td>
            {{ form.nickname(size=24) }}
            {% for error in form.errors.nickname %}
            <br><span style="color: red;">[{{ error }}]</span>
            {% endfor %}
        </td>

Now the bug is fixed and duplicates will be prevented... except when they are not. We still have a potential problem with concurrent access to the database by two or more threads or processes, but this will be the topic of a future article.

At this point you can try again to select a duplicated name to see how the form nicely handles the error.

Unit testing framework

To close this session on testing, let's talk about automated testing a bit.

As the application grows in size it gets more and more difficult to ensure that code changes don't break existing functionality.

The traditional approach to prevent regressions is a very good one. You write tests that exercise all the different features of the application. Each test runs a focused part and verifies that the result obtained is the expected one. The tests are executed periodically to make sure that the application works as expected. When the test coverage is large you can have confidence that modifications and additions do not affect the application in a bad way just by running the tests.

We will now build a very simple testing framework using Python's unittest module (file tests.py):

#!flask/bin/python
import os
import unittest

from config import basedir
from app import app, db
from app.models import User

class TestCase(unittest.TestCase):
    def setUp(self):
        app.config['TESTING'] = True
        app.config['WTF_CSRF_ENABLED'] = False
        app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'test.db')
        self.app = app.test_client()
        db.create_all()

    def tearDown(self):
        db.session.remove()
        db.drop_all()

    def test_avatar(self):
        u = User(nickname='john', email='john@example.com')
        avatar = u.avatar(128)
        expected = 'http://www.gravatar.com/avatar/d4c74594d841139328695756648b6bd6'
        assert avatar[0:len(expected)] == expected

    def test_make_unique_nickname(self):
        u = User(nickname='john', email='john@example.com')
        db.session.add(u)
        db.session.commit()
        nickname = User.make_unique_nickname('john')
        assert nickname != 'john'
        u = User(nickname=nickname, email='susan@example.com')
        db.session.add(u)
        db.session.commit()
        nickname2 = User.make_unique_nickname('john')
        assert nickname2 != 'john'
        assert nickname2 != nickname

if __name__ == '__main__':
    unittest.main()

Discussing the unittest module is outside the scope of this article. Let's just say that class TestCase holds our tests. The setUp and tearDown methods are special, these are run before and after each test respectively. A more complex setup could include several groups of tests each represented by a unittest.TestCase subclass, and each group then would have independent setUp and tearDown methods.

These particular setUp and tearDown methods are pretty generic. In setUp the configuration is edited a bit. For instance, we want the testing database to be different that the main database. In tearDown we just reset the database contents.

Tests are implemented as methods. A test is supposed to run some function of the application that has a known outcome, and should assert if the result is different than the expected one.

So far we have two tests in the testing framework. The first one verifies that the Gravatar avatar URLs from the previous article are generated correctly. Note how the expected avatar is hardcoded in the test and checked against the one returned by the User class.

The second test verifies the make_unique_nickname method we just wrote, also in the User class. This test is a bit more elaborate, it creates a new user and writes it to the database, then ensures the same name is not allowed as a unique name. It then creates a second user with the suggested unique name and tries one more time to request the first nickname. The expected result for this second part is to get a suggested nickname that is different from the previous two.

To run the test suite you just run the tests.py script. On Linux or Mac:

./tests.py

And on Windows:

flask/Scripts/python tests.py

If there are any errors, you will get a report in the console.

Final words

This ends today's discussion of debugging, errors and testing. I hope you found this article useful.

As always, if you have any comments please write below.

The code of the microblog application update with today's changes is available below for download:

Download microblog-0.7.zip.

As always, the flask virtual environment and the database are not included. See previous articles for instructions on how to generate them.

I hope to see you again in the next installment of this series.

Thank you for reading!

Miguel

113 comments

  • #26 Vasco Correia said 2013-07-13T11:09:04Z

    Yeah, I had an infinite loop in the make_unique_nickname function. Thanks.

  • #27 drew said 2013-08-02T04:31:46Z

    Miguel, can you describe how the error logging we set up here compares to a service like New Relic? Do the services overlap? Would it be beneficial to use both?

    I'm very excited over you new book.

  • #28 Miguel Grinberg said 2013-08-03T01:49:52Z

    @drew: New Relic does a lot more than logging, they look at system, application and database performance and give you a nice web interface to see the results over time. Even if you had that capturing logs can be useful for debugging. Services like New Relic may also save logs for you, that isn't clear from a brief look at their site. But you still need to have your app output errors and stack traces to the log.

  • #29 dowlf said 2013-10-09T23:16:46Z

    Miguel:

    First, I can't thank you enough for your tutorials. A month ago I had no python or flask skills at all, and now my site is coming along nicely, thanks to your guidance. Currently I am struggling through trying to make it look decent, there is so much to learn, so little time!

    I wanted to give a tip to others that might be working through this tutorial on how to make the logging email detailed above work with gmail (I for one have configured sendmail for the last time!)

    I followed the code laid out in the following link:

    http://mynthon.net/howto/-/python/python%20-%20logging.SMTPHandler-how-to-use-gmail-smtp-server.txt

    It involves extending the logging.handlers.SMTPHandler class and overriding the SMTPHandler.emit() method with the proper TLS code. I put the class in a separate module to avoid too much clutter in init.py and then called that instead of the SMTPHandler that is normally called. Works like a charm. It is easily found through a web search, but I figured I would post it here to point newbies like me in the right direction.

  • #30 Jordan said 2013-10-24T07:23:22Z

    Hey Miguel, quick question: what's the point of using args and *kwargs in the EditForm and Form constructor? There shouldn't be more than one argument, the nickname, right?

    Signed, A confused Flaskee

  • #31 Miguel Grinberg said 2013-10-24T14:21:24Z

    @Jordan: this is called defensive programming. In the current version there is no need for additional arguments, but when coded in this way the function will not need to change after you add additional fields to the form.

  • #32 ToM_Korn said 2013-12-12T22:18:41Z

    hey miguel :) first of all thanks for the tutorial - it's really a great job you've done here.

    i have a small suggestion for this chapter. instead of creating two open id accounts i just made a db entry over the python shell to check if there is the same name already in use. it's much faster and worked perfect for me :)

    lovely regards ToM

  • #33 Jake said 2014-01-08T23:47:44Z

    It's probably worth noting that whenever you make one of these helper scripts you need to give it +x permission in *nix environments. Some beginners may be puzzled by that.

    I usually remember about it when [tab] autocomplete doesn't work for the script in my shell.

  • #34 Kenny Landes said 2014-01-28T23:56:06Z

    Hi Miguel, this really is an excellent tutorial. I'm new to Python and have done a couple of tuts that resulted in useless applications (for me) that made a lot of assumptions about the user experience level. I come into this as an intermediate beginner, and it not only makes sense, but it is conceivably something I would build in the future, so it's very relevant to my interests. Thanks for that!

    My question is about tests.py. When I run the test (OSX 10.9.1), Terminal goes to a new line with a dot and stays that way forever. Is this the expected outcome? I'm not getting any error messages, but it seems like I should at least get back to my command prompt. I'm going to move on, but if you can tell me if I've missed something, that would be great. I compared my code to the download package and get the same result. I also went through all 33 previous comments to see if this issue was mentioned before and don't see anything.

  • #35 Miguel Grinberg said 2014-01-30T17:00:59Z

    @Kenny: No, that is definitely not expected. If you Ctrl-C at the point it is hung do you get a stack trace? That may give you some clues. Feel free to share the stack trace, happy to look at it.

  • #36 jda2000 said 2014-01-31T06:24:44Z

    Hey Miguel,

    While I appreciate that overriding the validate method of the EditForm Class demonstrates something that in a professional career you will someday have to do, it may also be useful to point out that you can make your own validators. Here's what I tried:

    from flask.ext.wtf import Form, ValidationError ... def is_unique_or_original_nickname(form, field): if field.data == form.original_nickname: return user = User.query.filter_by(nickname = field.data).first() if user != None: raise ValidationError('This nickname is already in use. Please choose another one.') ... nickname = TextField('nickname', validators = [Required(), is_unique_or_original_nickname])

    Naturally, this still needs the constructor to save the original nickname in the form like you have it.

  • #37 jda2000 said 2014-01-31T06:37:25Z

    Hey Miguel,

    Thanks for this outstanding tutorial that I find to be excellent in many ways.

    I put "print "NOT Debugging"" after the "if not app.debug:" in app/init.py and I see "NOT Debugging" if I start it with run.py or runp.y

    This is not a big deal but I thought you might want to know.

  • #39 Miguel Grinberg said 2014-02-06T02:34:22Z

    Note that the Flask-WTF version that I use in this tutorial is fine, this is a change that they introduced in a later release.

  • #40 @miggie said 2014-02-06T12:09:34Z

    oh sry :D idk why i installed that package without number so it took the newest one maybe... here are my versions Flask==0.10.1 Flask-Login==0.2.9 Flask-SQLAlchemy==0.16 Flask-WTF==0.9.4 Flask-WhooshAlchemy==0.54a Jinja2==2.7.2 MarkupSafe==0.18 SQLAlchemy==0.7.9 Tempita==0.5.2 WTForms==1.0.5 Werkzeug==0.9.4 Whoosh==2.5.6 blinker==1.3 decorator==3.4.0 itsdangerous==0.23 sqlalchemy-migrate==0.7.2 wsgiref==0.1.2

  • #41 Joe said 2014-02-21T14:30:17Z

    The error handler functions under "Custom HTTP error handlers" are both called internal_error. I think you probably want to call the first one something like not_found_error. As an aside, I'm very surprised that Python appears to work in spite of the redefinition.

  • #42 Miguel Grinberg said 2014-02-22T06:20:42Z

    @Joe: yes, you are correct. I have made the correction in the article, I need to also update the code. Things work anyway because both handlers get mapped to the second function, which shadows the first.

  • #43 Brad said 2014-02-22T17:51:54Z

    Just a comment on an error I received:

    File "/home/bdm/work/flask/microBlog/blog/forms.py", line 18, in validate user = User.query.filter_by(nickname = self.nickname.data).first() NameError: global name 'User' is not defined

    This was fixed by adding the following to forms.py: from models import User

  • #44 sandhya nair said 2014-05-02T05:45:52Z

    I am trying to write unit test into py test. Can you help me out how these test should be converted in py test?

  • #45 Petar Mihaylov said 2014-05-02T13:40:30Z

    Hello Miguel!

    I am really enjoying this tutorial and look forward to the release of your book! I have been following so far and I have not really run into anything I was not able to figure out. And now I have a dilemma...

    I have been trying to make SMTPHandler use TLS to send the e-mails and I have been unsuccessful (I an not an expert programmer). But the real questions is, do I really care if someone intercepts an error stack from my application?

    Please help me figure out how to modify the SMTPHandler section, so it would send e-mails on port 465 instead of port 25.

    Thanks!

  • #46 Miguel Grinberg said 2014-05-03T01:28:16Z

    @Petar: check out my "flasky" project on github. This is the app that is used in my book. See file config.py, class ProductionConfig. It's all there. :)

  • #47 Petar Mihaylov said 2014-05-05T13:10:26Z

    @Miguel Thanks so much!

  • #48 Richard said 2014-05-13T14:04:55Z

    Awesome guide! It really helps me learn Python (and especially Flask). But there are several warnings in your example code.

    Change comparisons like xxxx == None with xxxx is None

    PyCharm (IDE) gives the following warning: This inspection highlights comparisons with None. That type of comparisons should always be done with 'is' or 'is not', never the equality operators.

    PEP 8: comparison to None should be 'if cond is None:'

  • #49 Miguel Grinberg said 2014-05-13T14:25:24Z

    @Richard: yes, back when I wrote this tutorial two years ago I did not make an effort to be PEP8 complaint. Note that also the example code does not run on Python 3, which wasn't supported by Flask back then. I am planning a round of fixes to update this code in the near future.

  • #50 Richard said 2014-05-13T15:12:28Z

    @Miguel: a tutorial made 2 years ago, and you still respond under the hour. You sir, deserve a medal.

Leave a Comment