2012-12-12T07:54:49Z

The Flask Mega-Tutorial, Part XI: Email Support

(Great news! There is a new version of this tutorial!)

This is the eleventh article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Recap

In the most recent installments of this tutorial we've been looking at improvements that mostly had to do with our database.

Today we are letting our database rest for a bit, and instead we'll look at another important function that most web applications have: the ability to send emails to its users.

In our little microblog application we are going to implement one email related function, we will send an email to a user each time he/she gets a new follower. There are several more ways in which email support can be useful, so we'll make sure we design a generic framework for sending emails that can be reused.

Configuration

Luckily for us, Flask already has an extension that handles email called Flask-Mail, and while it will not take us 100% of the way, it gets us pretty close.

Back when we looked at unit testing, we added configuration for Flask to send us an email should an error occur in the production version of our application. That same information is used for sending application related emails.

Just as a reminder, what we need is two pieces of information:

  • the email server that will be used to send the emails, along with any required authentication
  • the email address(es) of the admins

This is what we did in the previous article (file config.py):

# email server
MAIL_SERVER = 'your.mailserver.com'
MAIL_PORT = 25
MAIL_USERNAME = None
MAIL_PASSWORD = None

# administrator list
ADMINS = ['you@example.com']

It goes without saying that you have enter the details of an actual email server and administrator above before the application can actually send emails. We are not going to enhance the server setup to allow those that require an encrypted communication through TLS or SSL. For example, if you want the application to send emails via your gmail account you would enter the following:

# email server
MAIL_SERVER = 'smtp.googlemail.com'
MAIL_PORT = 465
MAIL_USE_TLS = False
MAIL_USE_SSL = True
MAIL_USERNAME = os.environ.get('MAIL_USERNAME')
MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')

# administrator list
ADMINS = ['your-gmail-username@gmail.com']

Note that the username and password are read from environment variables. You will need to set MAIL_USERNAME and MAIL_PASSWORD to your Gmail login credentials. Putting sensitive information in environment variables is safer than writing down the information on a source file.

We also need to initialize a Mail object, as this will be the object that will connect to the SMTP server and send the emails for us (file app/__init__.py):

from flask_mail import Mail
mail = Mail(app)

Let's send an email!

To learn how Flask-Mail works we'll just send an email from the command line. So let's fire up Python from our virtual environment and run the following:

>>> from flask_mail import Message
>>> from app import app, mail
>>> from config import ADMINS
>>> msg = Message('test subject', sender=ADMINS[0], recipients=ADMINS)
>>> msg.body = 'text body'
>>> msg.html = '<b>HTML</b> body'
>>> with app.app_context():
...     mail.send(msg)
....

The snippet of code above will send an email to the list of admins that are configured in config.py. The sender will be the first admin in the list. The email will have text and HTML versions, so depending on how your email client is setup you may see one or the other. Note that we needed to create an app_context to send the email. Recent releases of Flask-Mail require this. An application context is created automatically when a request is handled by Flask. Since we are not inside a request we have to create the context by hand just so that Flask-Mail can do its job.

Pretty, neat. Now it's time to integrate this code into our application!

A simple email framework

We will now write a helper function that sends an email. This is just a generic version of the above test. We'll put this function in a new source file that will be dedicated to our email support functions (file app/emails.py):

from flask_mail import Message
from app import mail

def send_email(subject, sender, recipients, text_body, html_body):
    msg = Message(subject, sender=sender, recipients=recipients)
    msg.body = text_body
    msg.html = html_body
    mail.send(msg)

Note that Flask-Mail support goes beyond what we are using. Bcc lists and attachments are available, for example, but we won't use them in this application.

Follower notifications

Now that we have the basic framework to send an email in place, we can write the function that sends out the follower notification (file app/emails.py):

from flask import render_template
from config import ADMINS

def follower_notification(followed, follower):
    send_email("[microblog] %s is now following you!" % follower.nickname,
               ADMINS[0],
               [followed.email],
               render_template("follower_email.txt", 
                               user=followed, follower=follower),
               render_template("follower_email.html", 
                               user=followed, follower=follower))

Do you find any surprises in here? Our old friend the render_template function is making an appearance. If you recall, we used this function to render all the HTML templates from our views. Like the HTML from our views, the bodies of email messages are an ideal candidate for using templates. As much as possible we want to keep logic separate from presentation, so emails will also go into the templates folder along with our views.

So we now need to write the templates for the text and HTML versions of our follower notification email. Here is the text version (file app/templates/follower_email.txt):

Dear {{ user.nickname }},

{{ follower.nickname }} is now a follower. Click on the following link to visit {{ follower.nickname }}'s profile page:

{{ url_for('user', nickname=follower.nickname, _external=True) }}

Regards,

The microblog admin

For the HTML version we can do a little bit better and even show the follower's avatar and profile information (file app/templates/follower_email.html):

<p>Dear {{ user.nickname }},</p>
<p><a href="{{ url_for('user', nickname=follower.nickname, _external=True) }}">{{ follower.nickname }}</a> is now a follower.</p>
<table>
    <tr valign="top">
        <td><img src="{{ follower.avatar(50) }}"></td>
        <td>
            <a href="{{ url_for('user', nickname=follower.nickname, _external=True) }}">{{ follower.nickname }}</a><br />
            {{ follower.about_me }}
        </td>
    </tr>
</table>
<p>Regards,</p>
<p>The <code>microblog</code> admin</p>

Note the _external=True argument to url_for in the above templates. By default, the url_for function generates URLs that are relative to the domain from which the current page comes from. For example, the return value from url_for("index") will be /index, while in this case we want http://localhost:5000/index. In an email there is no domain context, so we have to force fully qualified URLs that include the domain, and the _external argument is just for that.

The final step is to hook up the sending of the email with the actual view function that processes the "follow" (file app/views.py):

from .emails import follower_notification

@app.route('/follow/<nickname>')
@login_required
def follow(nickname):
    user = User.query.filter_by(nickname=nickname).first()
    # ...
    follower_notification(user, g.user)
    return redirect(url_for('user', nickname=nickname))

Now you can create two users (if you haven't yet) and make one follow the other to see how the email notification works.

So that's it? Are we done?

We could now pat ourselves in the back for a job well done and take email notifications out of our list of features yet to implement.

But if you played with the application for some time and paid attention you may have noticed that now that we have email notifications when you click the follow link it takes 2 to 3 seconds for the browser to refresh the page, whereas before it was almost instantaneous.

So what happened?

The problem is that Flask-Mail sends emails synchronously. The web server blocks while the email is being sent and only returns its response back to the browser once the email has been delivered. Can you imagine what would happen if we try to send an email to a server that is slow, or even worse, temporarily offline? Not good.

This is a terrible limitation, sending an email should be a background task that does not interfere with the web server, so let's see how we can fix this.

Asynchronous calls in Python

What we really want is for the send_email function to return immediately, while the work of sending the email is moved to a background process.

Turns out Python already has support for running asynchronous tasks, actually in more than one way. The threading and multiprocessing modules can both do this.

Starting a thread each time we need to send an email is much less resource intensive than starting a brand new process, so let's move the mail.send(msg) call into thread (file app/emails.py):

from threading import Thread
from app import app

def send_async_email(app, msg):
    with app.app_context():
        mail.send(msg)

def send_email(subject, sender, recipients, text_body, html_body):
    msg = Message(subject, sender=sender, recipients=recipients)
    msg.body = text_body
    msg.html = html_body
    thr = Thread(target=send_async_email, args=[app, msg])
    thr.start()

The send_async_email function now runs in a background thread. Because it is a separate thread, the application context required by Flask-Mail will not be automatically set for us, so the app instance is passed to the thread, and the application context is set up manually, like we did above when we sent an email from the Python console.

If you test the 'follow' function of our application now you will notice that the web browser shows the refreshed page before the email is actually sent.

So now we have asynchronous emails implemented, but what if in the future we need to implement other asynchronous functions? The procedure would be identical, but we would need to duplicate the threading code for each particular case, which is not good.

We can improve our solution by implementing a decorator. With a decorator the above code would change to this:

from .decorators import async

@async
def send_async_email(app, msg):
    with app.app_context():
        mail.send(msg)

def send_email(subject, sender, recipients, text_body, html_body):
    msg = Message(subject, sender=sender, recipients=recipients)
    msg.body = text_body
    msg.html = html_body
    send_async_email(app, msg)

Much nicer, right?

The code that allows this magic is actually pretty simple. We will put it in a new source file (file app/decorators.py):

from threading import Thread

def async(f):
    def wrapper(*args, **kwargs):
        thr = Thread(target=f, args=args, kwargs=kwargs)
        thr.start()
    return wrapper

And now that we indirectly have created a useful framework for asynchronous tasks we can say we are done!

Just as an exercise, let's consider how this solution would look using processes instead of threads. We do not want a new process started for each email that we need to send, so instead we could use the Pool class from the multiprocessing module. This class creates a specified number of processes (which are forks of the main process) and all those processes wait to receive jobs to run, given to the pool via the apply_async method. This could be an interesting approach for a busy site, but we will stay with the threads for now.

Final words

The source code for the updated microblog application is available below:

Download microblog-0.11.zip.

I've got a few requests for putting this application up on github or similar, which I think is a pretty good idea. I will be working on that in the near future. Stay tuned.

Thank you again for following me on this tutorial series. I look forward to see you on the next chapter.

Miguel

102 comments

  • #26 Miguel Grinberg said 2013-10-30T04:35:15Z

    @Jordan: There are plenty of tasks in software development that are unpleasant. I agree with Jeff that email is one of them. If you can afford to pay someone else to do it for you then great, but in my opinion setting up a server to send emails is several orders of magnitude simpler than developing code, so we are all well capable of doing it, and doing it well.

  • #27 Gigi said 2013-11-24T22:26:27Z

    Best tutorial ever! I noticed a little typo: "payed attention" should be "paid attention".

  • #28 Marcus Darden said 2013-12-15T18:13:01Z

    Great work!

    In the third code block of "Follower Notifcations", the url_for should use single quotes around the endpoint function.

  • #29 Miguel Grinberg said 2013-12-15T18:58:57Z

    @Marcus: good catch. No matter how careful I try to be there's always little things that slip. Thanks!

  • #30 Ale Borba said 2013-12-31T01:11:38Z

    Hi Miguel,

    The RuntimeError is not happening on the command line, but in the Thread function send_async_email():

    Take a look:

    Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 808, in bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 761, in run self.target(*self.args, **self.kwargs) File "/home/alexandre/Projetos/microblog/app/emails.py", line 11, in send_async_email mail.send(msg) File "/home/alexandre/Projetos/microblog/flask/local/lib/python2.7/site-packages/flask_mail.py", line 416, in send message.send(connection) File "/home/alexandre/Projetos/microblog/flask/local/lib/python2.7/site-packages/flask_mail.py", line 351, in send connection.send(self) File "/home/alexandre/Projetos/microblog/flask/local/lib/python2.7/site-packages/flask_mail.py", line 170, in send email_dispatched.send(message, app=current_app._get_current_object()) File "/home/alexandre/Projetos/microblog/flask/local/lib/python2.7/site-packages/werkzeug/local.py", line 297, in _get_current_object return self.__local() File "/home/alexandre/Projetos/microblog/flask/local/lib/python2.7/site-packages/flask/globals.py", line 34, in _find_app raise RuntimeError('working outside of application context') RuntimeError: working outside of application context

  • #31 Miguel Grinberg said 2013-12-31T02:19:50Z

    @Ale: what version of Flask-Mail are you using? I believe if you use 0.7.6, which is the version that I recommended in Part I of this tutorial, you should be fine. Later versions of Flask-Mail require the application context to be active when calling the send() function, so to make it work you will need to do something like this instead:

    @async def send_async_email(app, msg): with app.app_context(): mail.send(msg)

  • #32 Alex said 2013-12-31T21:07:04Z

    @Miguel

    I also received the runtime error that Olga and Zach described. The trivial fix is to wrap mail.send(msg) in send_async_email in the app context:

    def send_async_email(msg): with app.app_context(): mail.send(msg)

  • #33 Dan said 2014-01-01T22:15:58Z

    What did you change to fix the flask-mail error ? Thanks!

  • #34 Daniel said 2014-01-01T22:20:37Z

    ah now I see -- 'with app.app_context: mail.send(...)' -- Thanks!

  • #35 Bogdan said 2014-02-25T17:19:15Z

    Hi Miguel, This tutorial is really amazing. It's the best out there! I am really interested in the multiprocessing variant where you spawn some processes and then keep them alive so they can be "fed" by throwing tasks in the pool from time to time. Can you provide an example or point me to a resource as good as the one you have here where I could learn how to do it?

    Thanks a lot again!

  • #36 Miguel Grinberg said 2014-02-25T17:22:08Z

    @Bogdan: one way is to use the Pool from the multiprocessing package to start a few worker processes. Another very good option (and easier to implement) is to use Celery, which is a package specifically designed for this task.

  • #37 newhouse said 2014-03-07T12:54:51Z

    Hi Miguel,

    First off, I am very, very, very grateful for the time you've put into this (and your other) tutorials. Very helpful and much appreciated.

    For my recent attempt at following the tutorial, I've gone ahead and used the latest versions of just about everything except for SQLAlchemy. So, Flask-Mail is version 0.9.0.

    When running my async send_async_email thread I also got the "working outside of application context" error/warning. I dug around and it seems like you have to import app and use a "with app.app_context():" clause around the "mail.send(msg)". Here's the relevant code in emails.py:

    from app import app, mail

    @async def send_async_email(msg): with app.app_context(): mail.send(msg)

  • #38 Miguel Grinberg said 2014-03-07T15:15:14Z

    @newhouse: Yes, in fact I have implemented an almost identical solution in my book:

    def send_async_email(app, msg): with app.app_context(): mail.send(msg)

  • #39 Alex Leonhardt said 2014-05-27T08:40:35Z

    I think I finally understand decorators! yay! :)

  • #40 Robin said 2014-06-22T17:21:13Z

    @Miguel, I'm still getting the same error that Olga is getting even with your updated code.

    Here's the full traceback:

    reply: '250 2.0.0 OK 1403457521 nx12sm79358076pab.6 - gsmtp\r\n' reply: retcode (250); Msg: 2.0.0 OK 1403457521 nx12sm79358076pab.6 - gsmtp data: (250, '2.0.0 OK 1403457521 nx12sm79358076pab.6 - gsmtp') send: 'quit\r\n' reply: '221 2.0.0 closing connection nx12sm79358076pab.6 - gsmtp\r\n' reply: retcode (221); Msg: 2.0.0 closing connection nx12sm79358076pab.6 - gsmtpException in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 810, in bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 763, in run self.target(*self.args, **self.kwargs) File "/home/robin/github/ieee/calpoly/emails.py", line 13, in send_message mail.send(msg) File "/home/robin/github/ieee/venv/lib/python2.7/site-packages/flask_mail.py", line 416, in send message.send(connection) File "/home/robin/github/ieee/venv/lib/python2.7/site-packages/flask_mail.py", line 351, in send connection.send(self) File "/home/robin/github/ieee/venv/lib/python2.7/site-packages/flask_mail.py", line 170, in send email_dispatched.send(message, app=current_app._get_current_object()) File "/home/robin/github/ieee/venv/lib/python2.7/site-packages/werkzeug/local.py", line 297, in _get_current_object return self.__local() File "/home/robin/github/ieee/venv/lib/python2.7/site-packages/flask/globals.py", line 34, in _find_app raise RuntimeError('working outside of application context') RuntimeError: working outside of application context

  • #41 Miguel Grinberg said 2014-06-23T07:52:29Z

    @Robin: with current versions of Flask-Mail you have to put the send() call inside a "with app.app_context()" block.

  • #42 Jack Maney said 2014-07-15T07:49:04Z

    First of all, thank you for the tutorial. It's an extremely useful resource for learning Flask.

    I got the same "working outside of application context" error that Olga got above. I found a workaround by changing send_async_email to the following:

    @async def send_async_email(msg): with app.app_context(): mail.send(msg)

  • #43 Mihai said 2014-07-25T12:17:54Z

    Hi Miguel,

    what is the role of 'sender' in Message(.., sender=sender,....)? It has no effect, it is ignored. I see the mail to be coming from whatever I have set in my settings file: MAIL_USERNAME = 'your-gmail-username' Thanks.

  • #44 Miguel Grinberg said 2014-07-25T16:16:14Z

    @Mihai: I believe gmail overrides the sender. A less intrusive email server would show what you pass in the "From" header.

  • #45 Howard Edson said 2014-08-01T01:29:38Z

    Miguel - can't thank you enough for this incredible tutorial. In this chapter I was getting the same error message about the application context. In case it is of use to future readers, I wanted to share that I was able to fix it by making two slight changes to emails.py -

    ... from app import app, mail ... @async def send_async_email(msg): with app.app_context(): mail.send(msg) ...

  • #46 B said 2014-09-01T02:53:07Z

    Getting the same error now...

    Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in bootstrap_inner self.run() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run self.target(*self.args, **self.kwargs) File "server4.py", line 445, in send_async_email mail.send(msg) File "/Users/benabelman/Dropbox/python/notify/flasksite/venv/lib/python2.7/site-packages/flask_mail.py", line 416, in send message.send(connection) File "/Users/benabelman/Dropbox/python/notify/flasksite/venv/lib/python2.7/site-packages/flask_mail.py", line 351, in send connection.send(self) File "/Users/benabelman/Dropbox/python/notify/flasksite/venv/lib/python2.7/site-packages/flask_mail.py", line 170, in send email_dispatched.send(message, app=current_app._get_current_object()) File "/Users/benabelman/Dropbox/python/notify/flasksite/venv/lib/python2.7/site-packages/werkzeug/local.py", line 297, in _get_current_object return self.__local() File "/Users/benabelman/Dropbox/python/notify/flasksite/venv/lib/python2.7/site-packages/flask/globals.py", line 34, in _find_app raise RuntimeError('working outside of application context') RuntimeError: working outside of application context

    Again - the email does go through! Any fix for the error?

    Your guide is fantastic by the way!

  • #47 Miguel Grinberg said 2014-09-01T03:05:40Z

    @B: you are using a newer version of Flask-Email than the one I used back when I wrote this article. The current versions required an application context to be set, as shown in comment #45 above.

  • #48 Mostafa Moradian said 2014-10-09T08:39:32Z

    Thanks for threading decorator. It helped me a lot!

  • #49 Andrew said 2014-11-13T05:20:07Z

    My solution to the request context problem was to pass the context, not the app.

    Change the async function:

    def send_async_email(context, msg): with context:

    and if you're using the first example, the line:

    thr = Thread(target=send_async_email, args=app.app_context(), msg])

    or, if you're using the decorator:

    send_async_emailapp.app_context(), msg)

  • #50 Dmitry Belyakov said 2014-11-14T10:03:00Z

    Anyone having outside of context error with latest flask versions: please do not pass current_app proxy directly. Use current_app._get_current_object() instead.

Leave a Comment