The Flask Mega-Tutorial, Part XV: Ajax

This is the fifteenth article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

This will be the last of the I18n and L10n themed articles, as we complete our effort to make our microblog application accessible and friendly to non-English speakers.

In this article we will take a departure from the "safe zone" of server-side development we've come to love and will work on a feature that has equally important server and client-side components. Have you seen the "Translate" links that some sites show next to user generated content? These are links that trigger a real time automated translation of content that is not in the user's native language. The translated content is typically inserted below the original version. Google shows it for search results in foreign languages. Facebook shows it for posts. Today we are adding that very same feature to microblog!

Server-side vs. Client-side

In the traditional server-side model that we've followed so far we have a client (the user's web browser) making requests to our server. A request can simply ask for a web page, like when you click the "Your Profile" link, or it can make us perform an action, like when the user edits his or her profile information and clicks the Submit button. In both types of requests the server completes the request by sending a new web page to the client, either directly or by issuing a redirect. The client then replaces the current page with the new one. This cycle repeats for as long as the user stays on our web site. We call this model server-side because the server does all the work while the client just displays the web pages as it receives them.

In the client-side model we have a web browser that again, issues a request to the server. The server responds with a web page like in server-side, but not all the page data is HTML, there is also code, typically written in Javascript. Once the client receives the page it will display it and then execute the code that came with it. From then on you have an active client that can do work on its own without reaching out to the server. In a strict client-side application the entire application is downloaded to the client with the initial page request, and then the application runs on the client without ever refreshing the page, only contacting the server to retrieve or store data. This type of applications are called Single Page Applications or SPAs.

Most applications are a hybrid between the two models and combine techniques of both worlds. Our microblog application is clearly a server-side application, but today we will be adding a little bit of client-side action to the mix. To do real time translations of user posts the client browser will send requests to the server, but the server will respond with translated texts without causing a page refresh. The client will then insert the translations into the current page dynamically. This technique is known as Ajax, which is short for Asynchronous Javascript and XML (even though these days XML is often replaced with JSON).

Translating user generated content

We now have pretty good support for foreign languages thanks to Flask-Babel. Assuming we can find translators willing to help us, we could publish our application in as many languages as we want.

But there is one element missing. We made the application very friendly and inviting to people of all places, so now we are going to get content in a variety of languages, so our users are likely to come across blog posts that are written in languages they don't understand. Wouldn't it be nice if we could offer an automated translation service? The quality of automated translations isn't great, but in most cases it is good enough to get an idea of what someone said in a language we don't speak, so all our users (even ourselves!) can benefit from such a feature.

This is an ideal feature to implement as an Ajax service. Consider that our index page could be showing several posts, some of which might be in several different foreign languages. If we implemented the translation using traditional server-side techniques a request for a translation would cause the original page to get replaced with a new page showing just the translation for the selected post. After reading the translation the user would have to hit the back button to go back to the post listing. The fact is that requesting a translation isn't a big enough action to require a full page update, this feature works much better if the translated text is dynamically inserted below the original text while leaving the rest of the page untouched. So today we are implementing our first Ajax service!

Implementing live automated translations requires a few steps. First we need to identify the source language for the text we want to translate. Once we know the language we also know if a translation is needed for a given user, because we also know what language the user has chosen. When a translation is offered and the user wishes to see it we invoke the Ajax translation service, which will run on our server. In the final step the client-side javascript code will dynamically insert the translated text into the page.

Identifying the language of a post

Our first problem is identifying what language a post was written in. This isn't an exact science, it will not always be possible to detect a language, so we will consider it a "best effort" kind of thing. We are going to use the guess-language Python module for this. Unfortunately if you are using Python 3 the package guess-language will have issues, but a fork exists that can be used in its place:

flask/bin/pip uninstall guess-language
flask/bin/pip install https://bitbucket.org/spirit/guess_language/downloads/guess_language-spirit-0.5a4.tar.bz2

With this module we will scan the text of each blog post to try to guess its language. Since we don't want to scan the same posts over and over again we will do this once for each post, at the time the post is submitted by the user. We will then store the detected language along with the post in our database.

So let's begin by adding a language field to our Posts table:

class Post(db.Model):
    __searchable__ = ['body']

    id = db.Column(db.Integer, primary_key=True)
    body = db.Column(db.String(140))
    timestamp = db.Column(db.DateTime)
    user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
    language = db.Column(db.String(5))

Each time we modify the database we also need to do a migration:

$ ./db_migrate.py
New migration saved as microblog/db_repository/versions/005_migration.py
Current database version: 5

Now we have a place to store the language of each post, so let's detect the language of each added blog post:

from guess_language import guessLanguage

@app.route('/', methods=['GET', 'POST'])
@app.route('/index', methods=['GET', 'POST'])
@app.route('/index/<int:page>', methods=['GET', 'POST'])
@login_required
def index(page=1):
    form = PostForm()
    if form.validate_on_submit():
        language = guessLanguage(form.post.data)
        if language == 'UNKNOWN' or len(language) > 5:
            language = ''
        post = Post(body=form.post.data, 
                    timestamp=datetime.utcnow(), 
                    author=g.user, 
                    language=language)
        db.session.add(post)
        db.session.commit()
        flash(gettext('Your post is now live!'))
        return redirect(url_for('index'))
    posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False)
    return render_template('index.html',
                           title='Home',
                           form=form,
                           posts=posts)

If the language guessing doesn't work or returns an unexpectedly long result we play it safe and save an empty string to the database, a convention that we will use to signal that a post has an unknown language.

Displaying the "Translate" link

The next step is to show a "Translate" link next to any posts that are not in the language used by the user (file app/templates/posts.html):

{% if post.language != None and post.language != '' and post.language != g.locale %}
<div><a href="#">{{ _('Translate') }}</a></div>
{% endif %}

We are doing this in the post.html sub-template, so any page that displays blog posts will get this functionality. The Translate link will only be visible if we could detect the language of the blog post and the language we detected is different than the language selected by the localeselector function from Flask-Babel, which based on the work we did in the previous article we can now access at g.locale.

This link required us to add a new text, the word "Translate" which needs to be included in our translation files, so we entered it with the translation wrapper to make it visible to Flask-Babel. To get it translated we have to refresh our language catalog (tr_update.py), translate in poedit and recompile (tr_compile.py), as seen in the I18n article.

We don't really know what we will be doing to trigger a translation yet, so for now the link does nothing.

Translation services

Before we can proceed with our application we need to find a translation service that we can use.

There are many translation services available, but unfortunately most are either paid services or free with big limitations.

The two major translation services are Google Translate and Microsoft Translator. Both are paid services, but the Microsoft offering has an entry level option for low volume of translations that is free. Google offered a free translation service in the past but that isn't available anymore. That makes our choice of translation service very easy.

Using the Microsoft Translator service

To use the Microsoft Translator there are a couple of requirements that need to be met:

  • The application developer needs to register with the Microsoft Translator app at the Azure Marketplace. Here the level of service can be selected (the free option is at the bottom).
  • Then the developer needs to register an application. The registered application will get Client ID and Client Secret codes to be used as part of the requests sent to this service.

Once the registration part is complete the process to request a translation is as follows:

  • Get an access token, passing in the client id and secret.
  • Call the desired translation method, either Ajax, HTTP or SOAP, providing the access token and the text to translate.

This really sounds more complicated than it really is, so without going into details here is a function that does all the dirty work and translates a text to another language (file app/translate.py):

try:
    import httplib  # Python 2
except ImportError:
    import http.client as httplib  # Python 3
try:
    from urllib import urlencode  # Python 2
except ImportError:
    from urllib.parse import urlencode  # Python 3
import json
from flask.ext.babel import gettext
from config import MS_TRANSLATOR_CLIENT_ID, MS_TRANSLATOR_CLIENT_SECRET

def microsoft_translate(text, sourceLang, destLang):
    if MS_TRANSLATOR_CLIENT_ID == "" or MS_TRANSLATOR_CLIENT_SECRET == "":
        return gettext('Error: translation service not configured.')
    try:
        # get access token
        params = urlencode({
            'client_id': MS_TRANSLATOR_CLIENT_ID,
            'client_secret': MS_TRANSLATOR_CLIENT_SECRET,
            'scope': 'http://api.microsofttranslator.com', 
            'grant_type': 'client_credentials'})
        conn = httplib.HTTPSConnection("datamarket.accesscontrol.windows.net")
        conn.request("POST", "/v2/OAuth2-13", params)
        response = json.loads (conn.getresponse().read())
        token = response[u'access_token']

        # translate
        conn = httplib.HTTPConnection('api.microsofttranslator.com')
        params = {'appId': 'Bearer ' + token,
                  'from': sourceLang,
                  'to': destLang,
                  'text': text.encode("utf-8")}
        conn.request("GET", '/V2/Ajax.svc/Translate?' + urlencode(params))
        response = json.loads("{\"response\":" + conn.getresponse().read().decode('utf-8') + "}")
        return response["response"]
    except:
        return gettext('Error: Unexpected error.')

This function imports two new items from our configuration file, the id and secret codes assigned by Microsoft (file config.py):

# microsoft translation service
MS_TRANSLATOR_CLIENT_ID = '' # enter your MS translator app id here
MS_TRANSLATOR_CLIENT_SECRET = '' # enter your MS translator app secret here

To use the service you will need to register yourself and the application to receive the codes that go in the above configuration variables. Even if you just intend to test this application you can register with the service for free.

We have added more texts, this time a few error messages. These need translations, so we have to re-run tr_update.py, poedit and tr_compile.py to update our translation files.

Let's translate some text!

So how do we use the translator service? It's actually very simple. Here is an example:

$ flask/bin/python
Python 2.6.8 (unknown, Jun  9 2012, 11:30:32)
>>> from app import translate
>>> translate.microsoft_translate('Hi, how are you today?', 'en', 'es')
u'¿Hola, cómo estás hoy?'

Ajax in the server

Now we can translate texts between languages, so we are ready to integrate this functionality into our application.

When the user clicks the Translate link in a post there will be an Ajax call issued to our server. We'll see how this call is made in a bit, for now let's concentrate on implementing the server side of the Ajax call.

An Ajax service in the server is like a regular view function with the only difference that instead of returning an HTML page or a redirect it returns data, typically formatted as either XML or JSON. Since JSON is much more friendly to Javascript we will go with that format (file app/views.py):

from flask import jsonify
from translate import microsoft_translate

@app.route('/translate', methods=['POST'])
@login_required
def translate():
    return jsonify({ 
        'text': microsoft_translate(
            request.form['text'], 
            request.form['sourceLang'], 
            request.form['destLang']) })

There is not much new here. This route handles a POST request that should come with the text to translate and the source and destination language codes. Since this is a POST request we access these items as if this was data entered in an HTML form, using the request.form dictionary. We call one of our translation functions with this data and once we get the translated text we just convert it to JSON using Flask's jsonify function. The data that the client will see as a response to this request will have this format:

{ "text": "<translated text goes here>" }

Ajax in the client

Now we need to call the Ajax view function from the web browser, so we need to go back to the post.html sub-template to complete the work we started earlier.

We start by wrapping the post text in a span element with a unique id, so that we can later find it in the DOM and replace it with the translated text (file app/templates/post.html):

<p><strong><span id="post{{ post.id }}">{{ post.body }}</span></strong></p>

Note how we construct the unique id using the post's id number. If a given post has an id = 3 then the id of the <span> element that wraps it will be post3.

We are also going to wrap the Translate link in a span with a unique id, in this case so that we can hide it once the translation is shown:

<div><span id="translation{{post.id}}"><a href="#">{{ _('Translate') }}</a></span></div>

Continuing with the example above, the translation link would get the id translation3.

And to make it a nice and user friendly feature we are also going to throw in a spinning wheel animation that starts hidden and will only appears while the translation service is running on the server, also with a unique id:

<img id="loading{{post.id}}" style="display: none" src="/static/img/loading.gif">

So now we have an element called post<id> that contains the text to translate, an element called translation<id> that contains the Translate link but will later be replaced with the translated text and an image with id loading<id> that will be shown while the translation service is working. The <id> suffix is what makes these elements unique, we can have many posts in a page and each will get its own set of three elements.

We now need to trigger the Ajax call when the Translate link is clicked. Instead of triggering the call directly from the link we'll create a Javascript function that does all the work, since we have a few things to do there and we don't want to duplicate code in every post. Let's start by adding a call to this function when the translate link is clicked:

<a href="javascript:translate('{{ post.language }}', '{{ g.locale }}', '#post{{ post.id }}', '#translation{{ post.id }}', '#loading{{ post.id }}');">{{ _('Translate') }}</a>

The template variables obscure this code a bit, but really the function that we are calling is simple. Assuming a post with id = 23 that is written in Spanish and is viewed by a user that uses English then this function will be called as follows:

translate('es', 'en', '#post23', '#translation23', '#loading23')

So this function will receive the source and destination languages and the three DOM elements associated with the post to translate.

We can't write this function in the post.html sub-template because its contents are repeated for each post. We'll implement the function in our base template, so that there is a single copy of it that is accessible in all pages (file app/templates/base.html):

<script>
function translate(sourceLang, destLang, sourceId, destId, loadingId) {
    $(destId).hide();
    $(loadingId).show();
    $.post('/translate', {
        text: $(sourceId).text(),
        sourceLang: sourceLang,
        destLang: destLang
    }).done(function(translated) {
        $(destId).text(translated['text'])
        $(loadingId).hide();
        $(destId).show();
    }).fail(function() {
        $(destId).text("{{ _('Error: Could not contact server.') }}");
        $(loadingId).hide();
        $(destId).show();
    });
}
</script>

We rely on jQuery for this functionality. Recall that we included jQuery back when we styled the application using Twitter's Bootstrap framework.

We start by hiding the element that contains the translate link and showing the spinning wheel image.

Then we use jQuery's $.post() function to send the Ajax request to our server. The $.post function issues a POST request that to the server will be identical to a request issued by the browser when a form is submitted. The difference is that in the client this request happens in the background without causing a page reload. When a response is received from the server the function given as an argument to the done() call will execute and will have a chance to insert the data received into the page. This function receives the response as an argument, so we just overwrite the translate link in the DOM with the translated text, then hide the spinning wheel and finally display the translated text for the user to see. Neat!

If there is any error that prevents the client from getting a response from the server then the function inside the fail() call will be invoked. In this case we just show a generic error message, which also needs to be translated into all languages and thus requires a refresh of our translation database.

Unit testing

Remember our unit test framework? It is always a good idea to evaluate if it makes sense to write tests every time we add a new feature. Invoking the text translation service could be something we could inadvertently break while we work on our code, or it could break due to updates in the service. Let's write a quick test that makes sure we can contact the service and obtain translations:

from app.translate import microsoft_translate

class TestCase(unittest.TestCase):
    #...
    def test_translation(self):
        assert microsoft_translate(u'English', 'en', 'es') == u'Inglés'
        assert microsoft_translate(u'Español', 'es', 'en') == u'Spanish'

We do not have a client-side testing framework at this time, so we will not test the end-to-end Ajax procedure.

Running our testing framework should give us no errors:

$ ./tests.py
.....
----------------------------------------------------------------------
Ran 5 tests in 5.932s

OK

But if you run the test framework without having done the proper configuration for the Microsoft Translator you would get:

$ ./tests.py
....F
======================================================================
FAIL: test_translation (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./tests.py", line 120, in test_translation
    assert microsoft_translate(u'English', 'en', 'es') == u'Inglés'
AssertionError

----------------------------------------------------------------------
Ran 5 tests in 3.877s

FAILED (failures=1)

Final words

And with this we conclude this article. I hope it was as fun to read for you as it was for me to write!

I have recently been alerted of some issues with the database when Flask-WhooshAlchemy is used for the full text search. In the next chapter I will use this problem as an excuse to go over some of my debugging techniques for Flask applications. Stay tuned for episode XVI!

The code for the updated application is available on github, here. For those that do not use github below is a friendly download link:

Download microblog-0.15.zip.

I look forward to see you in the next article!

Miguel

27 comments

  • #1 Dogukan Tufekci said :

    You are simply an amazing person! Thanks for this new article!

  • #2 crazygit said :

    Thanks for this series articles, learned so much.

  • #3 Jimmy Juno said :

    An amazing set of tutorials -- and they just keep coming! Thank you Miguel for your hard work. I've been shopping for a framework for an upcoming project and these non-trivial and clearly written examples have made my choice easy.

  • #4 gason said :

    amazing,thank you very much.

  • #5 A Thankful Reader said :

    This is an amazing tutorial! Thanks for putting in the time and effort to do this.

  • #6 James said :

    Nice step by step tutorial. At this part I've got some problems with the unit tests. It seems its getting the post of the app.db instead of the test.db. I tried printing the post as the first statement of an unit test and I've got the posts of the website. Any ideas?

  • #7 Hari said :

    Thank you so much for your time !! learned so much..

  • #8 dowlf said :

    I am getting the similar behavior as James above in the unit tests, it is getting posts from the app.db instead of test.db. I am pretty stumped by it. I'll post again if I figure something out.

  • #9 Miguel Grinberg said :

    @dowlf: I believe I see what the problem is. The place I chose to configure the testing database (in the setUp() method of the test case) is too late, by that time the application and the database have already been instantiated. I believe this worked when I wrote it, but maybe with newer releases of Flask or other dependencies this does not work. I haven't done this the right way anyway, the configuration should be set correctly from the start. I'll correct the problem and update the article.

  • #10 bs4h said :

    I don't think putting the translate service in your test suite is a good idea. Running the tests shouldn't have any side-effects on the outside world (like fetching URLs or writing non-temporary files to disk), this makes test results less predictable and possibly, non-repeatable. Also, you might simply exceed some API quota ;) Instead, similarly to how an in-memory, temporary DB is set up for SQLAlchemy, you should write a simple "dummy" service exposing an identical API, and provide a simple abstraction layer that will call the "dummy" or the "real" service based on the configuration in effect. As long as your app calls the real service only through the abstraction, the abstraction layer and the dummy service will catch most of the errors in your code during testing. An extra benefit is that your tests will run in 20ms instead of 6 seconds! For big projects short times matter a lot. If the test suite can run in under a few seconds, I just leave a terminal open with "watch ./run_tests" and see a runtime error the moment I save the file.

  • #11 Miguel Grinberg said :

    @bs4h: Hmm. I agree with you. As a general principle you should try to have the unit test suite completely independent of the external world. This functionality has the particularity that it has no logic at all, all that the application does is contact the service using the service provided API endpoints. If I build a mock server that duplicates the API as it is known today then the whole feature is in the test, there is really nothing in the application that can fail. I believe the idea I had back when I wrote this was to ensure that the APIs from the service remain valid more than unit testing my code. This test is in a different class of test, at the very least I should have mark the test as 'skipped' if the service cannot be contacted, for example due to a missing internet connection.

  • #12 Cory Gough said :

    @Miguel: After running this command I see that my special characters are not printing correctly. Do you know what is causing this? translate.microsoft_translate('Hi, how are you today?', 'en', 'es') u'Hola, \xbfc\xf3mo est\xe1s hoy?'

  • #13 Miguel Grinberg said :

    @Cory: You are seeing the foreign characters shown in hexadecimal notation. The opening question mark ¿ is unicode character 0xBF, shown as \xbf by Python. The \xf3 character is ó, and the \xe1 is á. If you render this string to a web page you should see the correct string, which is "Hola, ¿cómo estás hoy?".

  • #14 Julio Guzman said :

    Hi Miguel, Thanks again for all your help. At this part, I found one issue, when I click translate in my app, I receive the following error: ArgumentOutOfRangeException: 'to' must be a valid language Parameter name: to : ID=3811.V2_Json.Translate.4050BE60 I'm trying to figure out what is going on, but it seems that I need your help again!. Best

  • #15 Julio Guzman said :

    Dear Miguel, I found the previous mentioned error using safari. Using Chrome the translator works perfect! Any explanation? Thanks

  • #16 Miguel Grinberg said :

    @Julio: This is the bing translator complaining. You may want to print the request data that you are sending to them, make sure the target language is correct. Maybe Safari has some difference in how it sends the ajax request to the server, compare the request you get from the two browsers, that should give us some more clues.

  • #17 hugleecool said :

    Firstly,Thanks for your great gracious generous work! Then, when I learn this article I got a strange question. Fortunately I've solved this problem. Considering that other people who are careless like me maybe face to the same problem, I decide to write down this comment. Sorry for my bad English~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the section "Using the Microsoft Translator service", on the "appId" key, the value of it is "Bearer " + token.There is a SPACE after Bearer!! If you ignore this, the try's last sentence, response will be printed as "{u'response': u"ArgumentException: The incoming token does not have 'nameIdentifier' : ID=3010.V2_Json.Translate.A997242"}"

  • #18 Alex said :

    Hi!Miguel Thx for your tutorial.I wanna use ajax to upload multiple img in a post, i wanna to add a column(postimg) to Post,which store several images path,and type is str.But I don't know to how to retrieve this data and render to template,How can I deal with it ?

  • #19 Arun said :

    Hi Miguel, Thanks for the brilliant tutorial. I am having a problem with installing guess-language. I can use pip command to install it and it doesn't give any error, but i cannot use it in any of the python programs. It is as if pip install does not install it at all and i get an import error. Any ideas on why this might be? Thanks in advance, Arun.

  • #20 Miguel Grinberg said :

    @Arun: seems to work fine for me. >>> from guess_language import guessLanguage >>> guessLanguage('this is a text in English') 'en'

  • #21 Tim said :

    In adding the unit test, you need to add # -*- coding: utf8 -*- after the #!flask/bin/python line

  • #22 Montreal said :

    This tutorial is as exiting as the space itself! But there is a question I have. How to make new post visible for user without refreshing the page?

  • #23 Miguel Grinberg said :

    @Montreal: the techniques explained in this article can be used to update a page without reloading. Have you tried this code?

  • #24 Lahan said :

    Hi again! I found that the urllib is not adapted to python 3. To make this work I did this: import urllib.parse and then replaced 'urllib.urlencode' with 'urllib.parse.urlencode'. Thank you for this excelent tutorial.

  • #25 Lahan said :

    Still about 'translate.py': I had to replace 'httplib' with 'http.client', because httplib nas been replace with http.client in Python 3. :)

Leave a Comment

Note: all comments are screened before they are published. Thank you for your patience!