2021-05-17T15:01:41Z

Dynamically Update Your Flask Web Pages Using Turbo-Flask

How can you update a part of a web page dynamically, without the user having to hit the refresh button on the browser?

This is a question that is frequently asked on the Internet. If you have some knowledge of JavaScript this is relatively easy to do, more so if you use a front end web framework such as React or Vue. But what if you have a standard web application written in Flask and Jinja templates?

In this article I'm going to introduce you to my Turbo-Flask extension, which will allow your Flask application to easily push asynchronous page updates to the browser, without having to write any JavaScript code. In the screenshot below, note how the CPU load numbers update without the user doing anything.

Turbo-Flask Demo Application

What is Turbo-Flask?

Turbo-Flask is a Flask extension that integrates the turbo.js JavaScript library with your Flask application.

This library bundles a number of different features, all with the goal of making server generated web pages behave more like single-page applications, but without requiring the application to write any front end code in JavaScript. Here is a list of the four modules included in turbo.js:

  • Turbo Drive: to accelerate page navigation by transparently updating pages in the background, without letting the browser ever do a full page reload.
  • Turbo Frames: to only update predefined parts of the page when the user clicks on a link or submits a form.
  • Turbo Streams: to let the server-side application update parts of the page by submitting HTML fragments to the client.
  • Turbo Native: to wrap your application as a native iOS or Android app. This section of turbo.js is out of scope for this article and I have very little experience with it, so I will not discuss it at all.

If you are interested in the Turbo Drive, and Frames features, I suggest you review the turbo.js documentation links above, as these do not require a Flask integration and can be used directly in your Jinja templates.

In this article I want to concentrate on the Turbo Streams feature, which is, in my opinion, the most interesting of the set.

An Example Application

This is going to be a hands-on tutorial, so let's create a short Flask application to which we can later add Turbo-Flask. If you are too lazy to type or copy/paste the code examples below, you can find the complete code for this article in the Turbo-Flask repository on GitHub.

To begin, create a directory in which you will work on this application. You may also want to create a virtual environment and activate it. Once you are ready, install Flask and Turbo-Flask:

(venv) $ pip install flask turbo-flask

Note: The Turbo-Flask package requires Flask version 2.

The Flask application

The code for the base application is shown below. Copy it to a file named app.py:

import random
import re
import sys
from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/page2')
def page2():
    return render_template('page2.html')

@app.context_processor
def inject_load():
    if sys.platform.startswith('linux'): 
        with open('/proc/loadavg', 'rt') as f:
            load = f.read().split()[0:3]
    else:
        load = [int(random.random() * 100) / 100 for _ in range(3)]
    return {'load1': load[0], 'load5': load[1], 'load15': load[2]}

The application defines two routes on the / and /page2 URLs. Both pages are implemented with Jinja templates named index.html and page2.html.

So far this does not show anything out of the ordinary. The last part of the application is something you may not be familiar with. The inject_load() function is decorated with the @app.context_processor decorator from Flask, which allows the application to add symbols to the Jinja scope. The dictionary that this function returns contains three keys, load1, load5 and load15, which Flask will make available to all Jinja templates rendered with the render_template() function.

The three load numbers are a standard metric that can be obtained in Unix based systems. These numbers give you the CPU load of the system in the last 1, 5 and 15 minutes. The method that I use to obtain these numbers is specific to Linux servers:

        with open('/proc/loadavg', 'rt') as f:
            load = f.read().split()[0:3]

To ensure that you can use this example application under other operating systems, the function provides an alternative implementation that just generates three fake load numbers:

        load = [int(random.random() * 100) / 100 for _ in range(3)]

The Base Template

Since the application will have two pages with largely the same structure, it is a good idea to extract the base layout into a base template that both pages can inherit from. Here is the base.html template, which you will need to store in a templates sub-directory:

<!doctype html>
<html>
  <head>
    <title>Turbo-Flask Streams Demo</title>
    <style>
      .load {
        float: right;
        width: 300px;
        border: 1px solid black;
        margin-left: 10px;
        margin-bottom: 10px;
        padding: 10px;
      }
      .load th, .load td {
        padding: 6px;
        text-align: center;
      }
    </style>
  </head>
  <body>
    {% include "loadavg.html" %}
    {% block content %}{% endblock %}
  </body>
</html>

The base template defines a few styles that create the design of the box with the load average metrics as you see in the screenshot at the start of the article. In particular, note how the float: right CSS attribute will make this part of the page appear on the top right corner of the page, wrapped with content.

The body of the template includes the loadavg.html template, which implements this box, and then defines a content Jinja block that the derived pages can use to provide their page bodies.

The Load Averages Template

The template referenced by the base template implementes the box that shows the three load average values as a table. Save the following page in the templates directory with the name loadavg.html:

<div id="load" class="load">
  <table>
    <tr><th></th><th>1 min</th><th>5 min</th><th>15 min</th></tr>
    <tr><th>Load averages:</th><td>{{ load1 }}</td><td>{{ load5 }}</td><td>{{ load15 }}</td></tr>
  </table>
</div>

Here notice the load1, load5 and load15 variables, which were the ones injected into the Jinja context by the application.

The Page Templates

To complete the base version of the application we need to create the two templates that are rendered by the index() and page2() routes. Since the content of the pages does not really matter much, I created a placeholder page with some dummy text in them, plus a link to the other page. Below you can see the index.html and page2.html templates, which you should copy to the templates sub-directory:

{% extends "base.html" %}

{% block content %}
<h1>Main Page</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<p>You can visit <a href="/page2">Page 2</a>.</p>
{% endblock %}
{% extends "base.html" %}

{% block content %}
<h1>Page 2</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<p>Go back to the <a href="/">home page</a>.</p>
{% endblock %}

Both templates inherit from the base.html template created earlier, which means that they are going to have the box with the CPU metrics automatically added in the top-right corner of the page.

Running the Application

The application is now complete. You can start it with the flask run command:

(venv) $ flask run

Open a web browser and type http://localhost:5000 in the address bar. You should now have a two-page application that shows (static) CPU metrics in the top-right corner of the page.

Turbo-Flask Static Demo

Configuring Turbo-Flask

The goal is to have the CPU load values update every five seconds without the user having to refresh the page. The first step in achieving this goal is to add the Turbo-Flask extension to the application.

Turbo-Flask is initialized like most other Flask extensions. In the imports section of app.py, import the Turbo class:

from turbo_flask import Turbo

Right after the Flask application instance is initialized, create an instance of this class and initialized it by passing the Flask application instance:

turbo = Turbo(app)

In the base.html template, add {{ turbo() }} inside the <head> element:

<!doctype html>
<html>
  <head>
    ...
    {{ turbo() }}
  </head>
</html>

The call to the turbo() function in the base template initializes the turbo.js library. While Turbo Drive and Turbo Frames are not the focus of this article, you should know that this is all that is needed from the Python side to enable these modules in your application.

Adding a Background Updater Thread

And now we reach the most interesting part of this tutorial. We want the application to push updates to the CPU load numbers every 5 seconds, so we will need a background thread that can issue these updates to the connected clients.

A convenient way to start this background thread is to do it as a before_first_request handler, as this ensures that the thread will be up and running as soon as the first client connects. The changes to start a background thread on first request are shown below. Add these changes to app.py:

import threading

# ...

@app.before_first_request
def before_first_request():
    threading.Thread(target=update_load).start()

The update_load() function is where the page updates are going to be generated. The function is going to have a loop, and in each loop iteration an update will be pushed to all connected clients. Below you can see the implementation of this function, which you should also add to app.py:

import time

# ...

def update_load():
    with app.app_context():
        while True:
            time.sleep(5)
            turbo.push(turbo.replace(render_template('loadavg.html'), 'load'))

This function is going to call the render_template() function from Flask to generate the updated CPU metrics fragment, so it needs to have an application context. The loop includes a 5 second sleep that sets the frequency of the updates.

An update is sent to all clients with the turbo.push() method. This method has only one required argument, the stream to push to clients. A stream is composed of one or more page update operations. There are 5 supported operations that can be used with Turbo Streams:

  • Append, or turbo.append(content, target): add content at the end of target
  • Prepend, or turbo.prepend(content, target): add content at the start of target
  • Replace, or turbo.replace(content, target): replace target with content
  • Update, or turbo.update(content, target): replace the contents of target with content
  • Remove, or turbo.remove(target): remove target

When you want to push a single operation, you can pass the return value of the corresponding operation directly to turbo.push(), as in the example above. If you would like to send several updates in different parts of the page, you can also pass a list of several operations, and turbo.js will apply all the updates for you.

In this application we want to replace the old version of the CPU metrics with an updated version, so the most appropriate operation is turbo.replace(). The append and prepend options are very useful when the page update needs to add information, without removing any of the existing contents. The update operation is similar to replace, but only the contents of the target element are replaced, leaving the element itself alone. Finally, the remove operation is useful to delete a part of the page.

In all these operations the content argument is the HTML fragment that needs to be updated in the page, given as a string. When working with Flask this is easy to generate, you just need to have a template that renders only this section of the page. In this application we already have the CPU metrics in a separate template because we needed to include it in two different pages, so all we need to do to generate the HTML fragment is to render this template directly in a render_template() call.

The target argument accepted for these operations is the id of the target element in the page that receives the update. If you review the loadavg.html template you will notice that the top-level <div> in this template has id="load".

After you make the updates to the application, restart the Flask server and refresh the page on your browser. You should now see the CPU load numbers update at 5 second intervals. It was easy, right?

The turbo.push() method has an optional argument called to, that can be used to specify which client(s) should receive the update. in this application this argument was not included because the update needs to go to all connected clients. If you are interested in sending updates to a subset of the connected clients the to argument, along with the @turbo.user_id decorator will come in handy.

Great, But How Does This Work?

You may be curious to know how does the turbo.push() method do its magic.

One of the things that the {{ turbo() }} call added in the base template does is open a WebSocket connection with the server. This connection is transparent to the application, as the Turbo-Flask extension manages it for you.

A WebSocket connection is bi-directional, so the server can send data to the client without the client having to ask for it. The turbo.push() method takes advantage of this WebSocket connection to submit your page updates. The turbo.js library is listening on this connection and whenever data is received from the server it executes the operations included in the stream.

Deployment Considerations

Turbo-Flask tries to make the set up and use of the WebSocket endpoint completely transparent to your application. At least during development, you should not need to worry about this, as the extension takes care of integrating WebSocket support with the Flask development web server.

If you intend to deploy a Turbo-Flask application to production you will need to decide on a deployment strategy that is compatible with WebSocket. There are 5 different production-ready configurations that are supported:

  • Gunicorn with the threaded worker
  • Gunicorn with the eventlet worker
  • Gunicorn with the gevent worker
  • The eventlet WSGI web server with monkey patching
  • The gevent WSGI web server with monkey patching

To learn more about these options, consult the deployment documentation for the Flask-Sock package, used by Turbo-Flask to implement the WebSocket route.

Conclusion

I hope this tutorial gave you a good overview of Turbo-Flask and turbo.js, and how to use them to trigger dynamic updates to your Flask web pages.

As mentioned a few times throughout the article, turbo.js is composed of a collection of utilities that speed up navigation, form submissions and updates to web pages rendered in the server, so I encourage you to read the turbo.js documentation to learn about how to take advantage of the remaining parts of this library.

55 comments

  • #26 Jean-Francois Theoret said 2021-09-28T14:19:11Z

    One more question (sorry...). I've integrated the injection code in a blueprint (@blueprint.app_context_processor), and it works fine with the thread running in the app.

    However, when I am in another blueprint and I do a POST (press on a button and call render_template to another html) I get an error in turbo.js:

    Error: Form responses must redirect to another location turbo.ks:17 at D.requestSucceededWithResponse (turbo.js:3) at k.receive (turbo.js:3) at k.perform (turbo.js:3)

    If I remove {{ turbo() }} from my base.html they start working again.

    Any idea why?

  • #27 Miguel Grinberg said 2021-09-28T19:39:21Z

    @Jean-Francois: returning HTML after a form POST is not a good practice. Normally a form POST needs to be responded with a redirect. Turbo enforces that, but there is a way to make it work, if you must do it. Just return a 422 status code with your HTML instead of 200.

  • #28 William Baltus said 2021-10-15T16:05:38Z

    Thanks for posting this! I am using this to show sensor data onto a flask webpage. It works great! However when I continued developing the webpage I ran into an error. Perhaps you could help me?

    I have sensor data being published every 10 seconds onto the webpage. However when I press buttons that render the index but have a different route too quickly, I encounter an error (500 Internal Server Error-- The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application).

  • #29 Miguel Grinberg said 2021-10-15T22:49:14Z

    @William: your server log should have a stack trace with more information about the error. Without that information it is not possible to know what the problem is.

  • #30 Benny said 2021-10-20T10:27:45Z

    Hey @Miguel, Big Chief, everything works great until i try putting it behind an NGINX reverse proxy for gunicorn... any ideas?

  • #31 Miguel Grinberg said 2021-10-20T18:42:37Z

    @Benny: did you configure your nginx to proxy WebSocket connections? Turbo-Flask uses WebSocket to push updates to clients.

  • #32 Nick said 2021-11-03T01:17:25Z

    Hey Miguel,

    When it comes to the client sending data back to the server, is it best to send them as a post request, or should it be through a websocket?

    Does a websocket consistently stay open when using Turbo-Streams, or only when events are sent from the server?

    Thanks for the help!

    Nick

  • #33 Miguel Grinberg said 2021-11-03T09:16:44Z

    @Nick: the WebSocket connection is used by turbo, it is not available for your client to send custom data. At least I think it isn't. You should submit data using your own solution, either HTTP POST requests, or your own WebSocket connection.

  • #34 Jose Laurentino III said 2021-11-18T12:02:05Z

    Wow! After 48+ hours searching for a solution, I landed at this page. It has all to do on how you search. The keyword was "dynamically".

    Thank you very much for spending the time for such a great job. The explanations were divine. It all works just like intended and exactly the solution I was looking for. Coming from PHP as a novice in Python has been a very nice very steep ladder. Works like this one are a gigantic ice-breakers.

    Once again, thank you!

  • #35 David Klein said 2021-12-30T14:18:43Z

    Thank you Miguel. This is a great tutorial. I was able to use this as a guide for an rpi kiosk project i've created and the content is dynamically updated when using the flask server, however, using gunicorn and nginx, the content does not update dynamically. I've read your guide here: https://github.com/miguelgrinberg/flask-sock/blob/main/docs/web_servers.rst and tried but threads and gevent, but the page remains without update. Current work is here: https://github.com/soinkleined/busstop/tree/develop. Was curious if you could suggest what I may be doing wrong.

  • #36 Miguel Grinberg said 2021-12-30T15:14:16Z

    @David: Does the problem exist only when you add nginx, or also when you connect directly to gunicorn? If the problem only starts when you add nginx, then your nginx configuration is likely inappropriate for WebSocket. See https://www.nginx.com/blog/websocket-nginx/ for documentation on how to set up a WebSocket route. The default WebSocket route for Turbo-Flask is /turbo-stream.

  • #37 Aldon Galido said 2022-01-23T14:57:56Z

    Hi Sir! Thank you for this post. I am working on some network visualization, and am getting input from the user on which network they want to check. Within the same template, I am also displaying the statistics (form on the left side, stats on the right side).

    I am trying to do my data requests for visualization in the "update_load" function. I execute my turbo.push commands there because the number of values that I would need to update in real-time vary per network, so instead of using the "inject_load" function to push values to variables, I am explicitly replacing the html in the template with newly generated html strings based on the network.

    Is there a way to pass a variable (in my case, network number) into the "with app.app_context()" block of code from a form?

  • #38 Miguel Grinberg said 2022-01-24T11:46:14Z

    @Aldon: with the design shown in this application, all users receive the same updates, because there is only one background thread that pushes updates. In your solution, I would imagine each user needs their own push thread, because each user makes different selections. Once you have a thread per user, instead of passing the number selection to the thread you can instead stop the old thread and start a new one whenever the number changes, and then you can pass this number as an argument into the thread. Unfortunately what you need is much more complex than the example I show here.

  • #39 Ariane L said 2022-02-07T16:46:46Z

    Hey, first of all thank you for your work, this is exactly what I was looking for. My problem is that I'm working in an offline environment, so downloading turbo.js is failing. I tried to download the turbo.js file and incorporating it via {{turbo(url=url_for('static', filename='turbo/turbo.js') )}} which seems to work at first but then I get the error message "Turbo.connectStreamSource is not a function". Do I need to implement more files or in a other way?

  • #40 Miguel Grinberg said 2022-02-07T18:52:09Z

    @Ariane: any chance the turbo.js file that you are using is incomplete?

  • #41 Ariane L said 2022-02-08T08:50:55Z

    Maybe... I couldn't find the file on the hotwired/turbo github so I downloaded the turbo.js from hotwired/turbo-rails/app/assets. Maybe that's the problem but I couldn't find it anywhere else.

  • #42 Miguel Grinberg said 2022-02-08T14:16:38Z

    @Ariane: The file at https://github.com/hotwired/turbo-rails/blob/main/app/assets/javascripts/turbo.js does have the connectStreamSource symbol in it. Is this the file you are using? Does your application work when you switch to the CDN version of turbo.js?

  • #43 Ariane L said 2022-02-08T16:40:49Z

    It seems like I got it to work. Googleing the problem brought me to this page https://unpkg.com/browse/@hotwired/turbo@7.0.1/dist/ and with turbo.es2017-esm.js it is working somehow. Thanks for helping anyway!

  • #44 kolin said 2022-02-09T01:43:59Z

    Hi Miguel,

    First of all thanks for all of your help with this and many other articles and youtube content.

    Just wondering after seeing so much different ways to do a content refresh on i.e. the index page, what is the best way to go for: server sending information? WebSockets, Turbo, Only AJAX (requests via ajax), SSE?

    My case: fetching data via api , via .py on server startup - this data gets rendered via loop (jinja) to index. All works, but how to handle the refresh of this data? Run the .py in while true, async, not async, SSE, via websockets? I have no bi-directional traffic, so websockets seem overhead. Although SSE can't handle much requests before hardware is :'(. doing it all over in AJAX seems broken..

  • #45 Miguel Grinberg said 2022-02-09T11:00:00Z

    @kolin: There are only the ways for the client and the server to communicate in a web application: HTTP and WebSocket. SSE, polling and long polling are all implementing something on top of HTTP and Ajax. Turbo, Socket.IO and similar libraries use WebSocket. All these methods have pros and cons, if one of them was clearly better than the rest, then there wouldn't be as many still in use. But as I said, under the hood it can only be HTTP or WebSocket, so I would start by choosing between these two and then try one or more of the options under the chosen category.

  • #46 Camobyte said 2022-05-21T14:58:09Z

    i've checked your example in github with todos, now everything makes so much sense. Man this is awesome! Thank you for spreading the word.

  • #47 Joe said 2022-06-01T09:30:40Z

    Hi Miguel, thank you for this. I deployed my app in Heroku using Turbo-Flask. However, I notice that after 30 seconds, the dynamic update stops. Is there a workaround for this? Thank you!

  • #48 Miguel Grinberg said 2022-06-01T18:45:08Z

    @Joe: The Heroku documentation covers this: https://devcenter.heroku.com/articles/websockets#timeouts. You'll need to push something, even a dummy update at less than 30 second intervals to keep the line active.

  • #49 Joe said 2022-06-02T10:03:51Z

    Thank you for the info, Miguel. I ran into another problem when pushing updates. It appears that Heroku keeps crashing when I use Turbo-Flask. Submitting forms takes more than 30 seconds when I have Turbo-Flask in my code which causes it to crash. Is there a way for only one user (ex. admin) to push the updates while other users don't so that it might prevent crashes? I think with @app.before_first_request, there are multiple update pushes happening, one for each user. Thank you!

  • #50 Miguel Grinberg said 2022-06-02T13:54:33Z

    @Joe: I don't understand what you are asking. Shouldn't you investigate why those form posts take so long? That is definitely not normal.

Leave a Comment