The Flask Mega-Tutorial, Part XI: Email Support

(Great news! There is a new version of this tutorial!)

This is the eleventh article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

• Part I: Hello, World!
• Part II: Templates
• Part III: Web Forms
• Part IV: Database
• Part V: User Logins
• Part VI: Profile Page And Avatars
• Part VII: Unit Testing
• Part VIII: Followers, Contacts And Friends
• Part IX: Pagination
• Part X: Full Text Search
• Part XI: Email Support (this article)
• Part XII: Facelift
• Part XIII: Dates and Times
• Part XIV: I18n and L10n
• Part XV: Ajax
• Part XVI: Debugging, Testing and Profiling
• Part XVII: Deployment on Linux (even on the Raspberry Pi!)
• Part XVIII: Deployment on the Heroku Cloud

Recap

In the most recent installments of this tutorial we've been looking at improvements that mostly had to do with our database.

Today we are letting our database rest for a bit, and instead we'll look at another important function that most web applications have: the ability to send emails to its users.

In our little microblog application we are going to implement one email related function, we will send an email to a user each time he/she gets a new follower. There are several more ways in which email support can be useful, so we'll make sure we design a generic framework for sending emails that can be reused.

Configuration

Luckily for us, Flask already has an extension that handles email called Flask-Mail, and while it will not take us 100% of the way, it gets us pretty close.

Back when we looked at unit testing, we added configuration for Flask to send us an email should an error occur in the production version of our application. That same information is used for sending application related emails.

Just as a reminder, what we need is two pieces of information:

• the email server that will be used to send the emails, along with any required authentication
• the email address(es) of the admins

This is what we did in the previous article (file config.py):

# email server
MAIL_SERVER = 'your.mailserver.com'
MAIL_PORT = 25
MAIL_USERNAME = None
MAIL_PASSWORD = None

# administrator list
ADMINS = ['you@example.com']

It goes without saying that you have enter the details of an actual email server and administrator above before the application can actually send emails. We are not going to enhance the server setup to allow those that require an encrypted communication through TLS or SSL. For example, if you want the application to send emails via your gmail account you would enter the following:

# email server
MAIL_SERVER = 'smtp.googlemail.com'
MAIL_PORT = 465
MAIL_USE_TLS = False
MAIL_USE_SSL = True
MAIL_USERNAME = os.environ.get('MAIL_USERNAME')
MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')

# administrator list
ADMINS = ['your-gmail-username@gmail.com']

Note that the username and password are read from environment variables. You will need to set MAIL_USERNAME and MAIL_PASSWORD to your Gmail login credentials. Putting sensitive information in environment variables is safer than writing down the information on a source file.

We also need to initialize a Mail object, as this will be the object that will connect to the SMTP server and send the emails for us (file app/__init__.py):

from flask_mail import Mail
mail = Mail(app)

Let's send an email!

To learn how Flask-Mail works we'll just send an email from the command line. So let's fire up Python from our virtual environment and run the following:

>>> from flask_mail import Message
>>> from app import app, mail
>>> from config import ADMINS
>>> msg = Message('test subject', sender=ADMINS[0], recipients=ADMINS)
>>> msg.body = 'text body'
>>> msg.html = '<b>HTML</b> body'
>>> with app.app_context():
...     mail.send(msg)
....

The snippet of code above will send an email to the list of admins that are configured in config.py. The sender will be the first admin in the list. The email will have text and HTML versions, so depending on how your email client is setup you may see one or the other. Note that we needed to create an app_context to send the email. Recent releases of Flask-Mail require this. An application context is created automatically when a request is handled by Flask. Since we are not inside a request we have to create the context by hand just so that Flask-Mail can do its job.

Pretty, neat. Now it's time to integrate this code into our application!

A simple email framework

We will now write a helper function that sends an email. This is just a generic version of the above test. We'll put this function in a new source file that will be dedicated to our email support functions (file app/emails.py):

from flask_mail import Message
from app import mail

def send_email(subject, sender, recipients, text_body, html_body):
    msg = Message(subject, sender=sender, recipients=recipients)
    msg.body = text_body
    msg.html = html_body
    mail.send(msg)

Note that Flask-Mail support goes beyond what we are using. Bcc lists and attachments are available, for example, but we won't use them in this application.

Follower notifications

Now that we have the basic framework to send an email in place, we can write the function that sends out the follower notification (file app/emails.py):

from flask import render_template
from config import ADMINS

def follower_notification(followed, follower):
    send_email("[microblog] %s is now following you!" % follower.nickname,
               ADMINS[0],
               [followed.email],
               render_template("follower_email.txt", 
                               user=followed, follower=follower),
               render_template("follower_email.html", 
                               user=followed, follower=follower))

Do you find any surprises in here? Our old friend the render_template function is making an appearance. If you recall, we used this function to render all the HTML templates from our views. Like the HTML from our views, the bodies of email messages are an ideal candidate for using templates. As much as possible we want to keep logic separate from presentation, so emails will also go into the templates folder along with our views.

So we now need to write the templates for the text and HTML versions of our follower notification email. Here is the text version (file app/templates/follower_email.txt):

Dear {{ user.nickname }},

{{ follower.nickname }} is now a follower. Click on the following link to visit {{ follower.nickname }}'s profile page:

{{ url_for('user', nickname=follower.nickname, _external=True) }}

Regards,

The microblog admin

For the HTML version we can do a little bit better and even show the follower's avatar and profile information (file app/templates/follower_email.html):

<p>Dear {{ user.nickname }},</p>
<p><a href="{{ url_for('user', nickname=follower.nickname, _external=True) }}">{{ follower.nickname }}</a> is now a follower.</p>
<table>
    <tr valign="top">
        <td><img src="{{ follower.avatar(50) }}"></td>
        <td>
            <a href="{{ url_for('user', nickname=follower.nickname, _external=True) }}">{{ follower.nickname }}</a><br />
            {{ follower.about_me }}
        </td>
    </tr>
</table>
<p>Regards,</p>
<p>The <code>microblog</code> admin</p>

Note the _external=True argument to url_for in the above templates. By default, the url_for function generates URLs that are relative to the domain from which the current page comes from. For example, the return value from url_for("index") will be /index, while in this case we want http://localhost:5000/index. In an email there is no domain context, so we have to force fully qualified URLs that include the domain, and the _external argument is just for that.

The final step is to hook up the sending of the email with the actual view function that processes the "follow" (file app/views.py):

from .emails import follower_notification

@app.route('/follow/<nickname>')
@login_required
def follow(nickname):
    user = User.query.filter_by(nickname=nickname).first()
    # ...
    follower_notification(user, g.user)
    return redirect(url_for('user', nickname=nickname))

Now you can create two users (if you haven't yet) and make one follow the other to see how the email notification works.

So that's it? Are we done?

We could now pat ourselves in the back for a job well done and take email notifications out of our list of features yet to implement.

But if you played with the application for some time and paid attention you may have noticed that now that we have email notifications when you click the follow link it takes 2 to 3 seconds for the browser to refresh the page, whereas before it was almost instantaneous.

So what happened?

The problem is that Flask-Mail sends emails synchronously. The web server blocks while the email is being sent and only returns its response back to the browser once the email has been delivered. Can you imagine what would happen if we try to send an email to a server that is slow, or even worse, temporarily offline? Not good.

This is a terrible limitation, sending an email should be a background task that does not interfere with the web server, so let's see how we can fix this.

Asynchronous calls in Python

What we really want is for the send_email function to return immediately, while the work of sending the email is moved to a background process.

Turns out Python already has support for running asynchronous tasks, actually in more than one way. The threading and multiprocessing modules can both do this.

Starting a thread each time we need to send an email is much less resource intensive than starting a brand new process, so let's move the mail.send(msg) call into thread (file app/emails.py):

from threading import Thread
from app import app

def send_async_email(app, msg):
    with app.app_context():
        mail.send(msg)

def send_email(subject, sender, recipients, text_body, html_body):
    msg = Message(subject, sender=sender, recipients=recipients)
    msg.body = text_body
    msg.html = html_body
    thr = Thread(target=send_async_email, args=[app, msg])
    thr.start()

The send_async_email function now runs in a background thread. Because it is a separate thread, the application context required by Flask-Mail will not be automatically set for us, so the app instance is passed to the thread, and the application context is set up manually, like we did above when we sent an email from the Python console.

If you test the 'follow' function of our application now you will notice that the web browser shows the refreshed page before the email is actually sent.

So now we have asynchronous emails implemented, but what if in the future we need to implement other asynchronous functions? The procedure would be identical, but we would need to duplicate the threading code for each particular case, which is not good.

We can improve our solution by implementing a decorator. With a decorator the above code would change to this:

from .decorators import async

@async
def send_async_email(app, msg):
    with app.app_context():
        mail.send(msg)

def send_email(subject, sender, recipients, text_body, html_body):
    msg = Message(subject, sender=sender, recipients=recipients)
    msg.body = text_body
    msg.html = html_body
    send_async_email(app, msg)

Much nicer, right?

The code that allows this magic is actually pretty simple. We will put it in a new source file (file app/decorators.py):

from threading import Thread

def async(f):
    def wrapper(*args, **kwargs):
        thr = Thread(target=f, args=args, kwargs=kwargs)
        thr.start()
    return wrapper

And now that we indirectly have created a useful framework for asynchronous tasks we can say we are done!

Just as an exercise, let's consider how this solution would look using processes instead of threads. We do not want a new process started for each email that we need to send, so instead we could use the Pool class from the multiprocessing module. This class creates a specified number of processes (which are forks of the main process) and all those processes wait to receive jobs to run, given to the pool via the apply_async method. This could be an interesting approach for a busy site, but we will stay with the threads for now.

Final words

The source code for the updated microblog application is available below:

Download microblog-0.11.zip.

I've got a few requests for putting this application up on github or similar, which I think is a pretty good idea. I will be working on that in the near future. Stay tuned.

Thank you again for following me on this tutorial series. I look forward to see you on the next chapter.

Miguel