The Flask Mega-Tutorial, Part X: Full Text Search

This is the tenth article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Recap

In the previous article in the series we've enhanced our database queries so that we can get results on pages.

Today, we are going to continue working on our database, but in a different area. All applications that store content must provide a search capability.

For many types of web sites it is possible to just let Google, Bing, etc. index all the content and provide the search results. This works well for sites that have mostly static pages, like a forum. In our little microblog application the basic unit of content is just a short user post, not a whole page. The type of search results that we want are dynamic. For example, if we search for the word "dog" we want to see blog posts from any users that include that word. It is obvious that until someone searches for that word there is no page that the big search engines could have indexed with these results, so clearly we have no choice other than rolling our own search.

Introduction to full text search engines

Unfortunately support for full text search in relational databases is not well standardized. Each database implements full text search in its own way, and SQLAlchemy at this time does not have a full text search abstration.

We are currently using SQLite for our database, so we could just create a full text index using the facilities provided by SQLite, bypassing SQLAlchemy. But that isn't a good idea, because if one day we decide to switch to another database we would need to rewrite our full text search capability for another database.

So instead, we are going to let our database deal with the regular data, and we are going to create a specialized database that will be dedicated to text searches.

There are a few open source full text search engines. The only one that to my knowledge has a Flask extension is Whoosh, an engine also written in Python. The advantage of using a pure Python engine is that it will install and run anywhere a Python interpreter is available. The disadvantage is that search performance will not be up to par with other engines that are written in C or C++. In my opinion the ideal solution would be to have a Flask extension that can connect to several engines and abstract us from dealing with a particular one in the same way Flask-SQLAlchemy gives us the freedom to use several database engines, but nothing of that kind seems to be available for full text searching at this time. Django developers do have a very nice extension that supports several full text search engines called django-haystack. Maybe one day someone will create a similar extension for Flask.

But for now, we'll implement our text searching with Whoosh. The extension that we are going to use is Flask-WhooshAlchemy, which integrates a Whoosh database with Flask-SQLAlchemy models.

Python 3 Compatibility

Unfortunately, we have a problem with Python 3 and these packages. The Flask-WhooshAlchemy extension was never made compatible with Python 3. I have forked this extension and made a few changes to make it work, so if you are on Python 3 you will need to uninstall the official version and install my fork:

$ flask/bin/pip uninstall flask-whooshalchemy
$ flask/bin/pip install git+git://github.com/miguelgrinberg/flask-whooshalchemy.git

Sadly this isn't the only problem. Whoosh also has issues with Python 3, it seems. In my testing I have encontered this bug, and to my knowledge there isn't a solution available, which means that at this time the full text search capability does not work well on Python 3. I will update this section once the issues are resolved.

Configuration

Configuration for Flask-WhooshAlchemy is pretty simple. We just need to tell the extension what is the name of the full text search database (file config.py):

WHOOSH_BASE = os.path.join(basedir, 'search.db')

Model changes

Since Flask-WhooshAlchemy integrates with Flask-SQLAlchemy, we indicate what data is to be indexed for searching in the proper model class (file app/models.py):

from app import app

import sys
if sys.version_info >= (3, 0):
    enable_search = False
else:
    enable_search = True
    import flask.ext.whooshalchemy as whooshalchemy

class Post(db.Model):
    __searchable__ = ['body']

    id = db.Column(db.Integer, primary_key=True)
    body = db.Column(db.String(140))
    timestamp = db.Column(db.DateTime)
    user_id = db.Column(db.Integer, db.ForeignKey('user.id'))

    def __repr__(self):
        return '<Post %r>' % (self.body)

if enable_search:
    whooshalchemy.whoosh_index(app, Post)

The model has a new __searchable__ field, which is an array with all the database fields that will be in the searchable index. In our case we only want to index the body field of our posts.

We also have to initialize the full text index for this model by calling the whoosh_index function. Note that since we know that the search capability currently does not work on Python 3 we have to skip its initialization. Once the problems in Whoosh are fixed the logic around enable_search can be removed.

Since this isn't a change that affects the format of our relational database we do not need to record a new migration.

Unfortunately any posts that were in the database before the full text engine was added will not be indexed. To make sure the database and the full text engine are synchronized we are going to delete all posts from the database and start over. First we start the Python interpreter. For Windows users:

flask\Scripts\python

And for everyone else:

flask/bin/python

Then in the Python prompt we delete all the posts:

>>> from app.models import Post
>>> from app import db
>>> for post in Post.query.all():
...    db.session.delete(post)
>>> db.session.commit()

Searching

And now we are ready to start searching. First let's add a few new posts to the database. We have two options to do this. We can just start the application and enter posts via the web browser, as regular users would do, or we can also do it in the Python prompt.

From the Python prompt we can do it as follows:

>>> from app.models import User, Post
>>> from app import db
>>> import datetime
>>> u = User.query.get(1)
>>> p = Post(body='my first post', timestamp=datetime.datetime.utcnow(), author=u)
>>> db.session.add(p)
>>> p = Post(body='my second post', timestamp=datetime.datetime.utcnow(), author=u)
>>> db.session.add(p)
>>> p = Post(body='my third and last post', timestamp=datetime.datetime.utcnow(), author=u)
>>> db.session.add(p)
>>> db.session.commit()

The Flask-WhooshAlchemy extension is nice, because it hooks up into Flask-SQLAlchemy commits automatically. We do not need to maintain the full text index, it is all done for us transparently.

Now that we have a few posts in our full text index we can issue searches:

>>> Post.query.whoosh_search('post').all()
[<Post u'my second post'>, <Post u'my first post'>, <Post u'my third and last post'>]
>>> Post.query.whoosh_search('second').all()
[<Post u'my second post'>]
>>> Post.query.whoosh_search('second OR last').all()
[<Post u'my second post'>, <Post u'my third and last post'>]

As you can see in the examples above, the queries do not need to be limited to single words. In fact, Whoosh supports a pretty powerful search query language.

Integrating full text searches into the application

To make the searching capability available to our application's users we have to add just a few small changes.

Configuration

As far as configuration, we'll just indicate how many search results should be returned as a maximum (file config.py):

MAX_SEARCH_RESULTS = 50

Search form

We are going to add a search form to the navigation bar at the top of the page. Putting the search box at the top is nice, because then the search will be accessible from all pages.

First we add a search form class (file app/forms.py):

class SearchForm(Form):
    search = StringField('search', validators=[DataRequired()])

Then we need to create a search form object and make it available to all templates, since we will be putting the search form in the navigation bar that is common to all pages. The easiest way to achieve this is to create the form in the before_request handler, and then stick it in Flask's global g (file app/views.py):

from forms import SearchForm

@app.before_request
def before_request():
    g.user = current_user
    if g.user.is_authenticated():
        g.user.last_seen = datetime.utcnow()
        db.session.add(g.user)
        db.session.commit()
        g.search_form = SearchForm()

Then we add the form to our template (file app/templates/base.html):

<div>Microblog:
    <a href="{{ url_for('index') }}">Home</a>
    {% if g.user.is_authenticated() %}
    | <a href="{{ url_for('user', nickname=g.user.nickname) }}">Your Profile</a>
    | <form style="display: inline;" action="{{ url_for('search') }}" method="post" name="search">{{ g.search_form.hidden_tag() }}{{ g.search_form.search(size=20) }}<input type="submit" value="Search"></form>
    | <a href="{{ url_for('logout') }}">Logout</a>
    {% endif %}
</div>

Note that we only display the form when we have a logged in user. Likewise, the before_request handler will only create a form when a user is logged in, since our application does not show any content to guests that are not authenticated.

Search view function

The action field of our form was set above to send all search requests the the search view function. This is where we will be issuing our full text queries (file app/views.py):

@app.route('/search', methods=['POST'])
@login_required
def search():
    if not g.search_form.validate_on_submit():
        return redirect(url_for('index'))
    return redirect(url_for('search_results', query=g.search_form.search.data))

This function doesn't really do much, it just collects the search query from the form and then redirects to another page passing this query as an argument. The reason the search work isn't done directly here is that if a user then hits the refresh button the browser will put up a warning indicating that form data will be resubmitted. This is avoided when the response to a POST request is a redirect, because after the redirect the browser's refresh button will reload the redirected page.

Search results page

Once a query string has been received the form POST handler sends it via page redirection to the search_results handler (file app/views.py):

from config import MAX_SEARCH_RESULTS

@app.route('/search_results/<query>')
@login_required
def search_results(query):
    results = Post.query.whoosh_search(query, MAX_SEARCH_RESULTS).all()
    return render_template('search_results.html',
                           query=query,
                           results=results)

The search results view function sends the query into Whoosh, passing a maximum number of search results, since we don't want to be presenting a potentially large number of hits, we are happy showing just the first fifty.

The final piece is the search results template (file app/templates/search_results.html):

<!-- extend base layout -->
{% extends "base.html" %}

{% block content %}
  <h1>Search results for "{{ query }}":</h1>
  {% for post in results %}
      {% include 'post.html' %}
  {% endfor %}
{% endblock %}

And here, once again, we can reuse our post.html sub-template, so we don't need to worry about rendering avatars or other formatting elements, since all of that is done in a generic way in the sub-template.

Final words

We now have completed yet another important, though often overlooked piece that any decent web application must have.

The source code for the updated microblog application is available below:

Download microblog-0.10.zip.

As always, the above download does not include a database or a flask virtual environment. See previous articles in the series to learn how to create these.

I hope you enjoyed this tutorial. If you have any questions feel free to write in the comments below. Thank you for reading, and I will be seeing you again in the next installment!

Miguel

58 comments

  • #1 neoragex said :

    Thank you for the excellent tutorials. Keep up the good work Miguel. And also obligatory here (only one chance to do it)... Furst!!!!!!!!!11 :)

  • #2 Vadim said :

    In model Post misprint: 'return '<Post %r>' % (self.text)' Must be: 'return '<Post %r>' % (self.body)'

  • #3 Miguel Grinberg said :

    @Vadim: you are correct, thanks for pointing it out. I have corrected the code above.

  • #4 George Mabley said :

    I feel like a pest asking all of these questions, but I can't seem to find any information about this error. Previously I added a feature to delete posts with a view that finds the post through its id, then removes it with db.session.delete(). However, after adding search, calling that view function gives the error: InvalidRequestError: Object '<Post at 0x3fe04a8>' is already attached to session '1' (this is '5'). Have you ever experienced a problem similar to this? Is there a way to bring the object to the current session? Thanks for helping so much!

  • #5 Miguel Grinberg said :

    George, are you using the development server like I am or do you have a different web server? All the references I see for this error are for multithreaded servers, the error occurs when the same object is attached to two sessions at the same time. Note that due to the way the application is at this stage you are likely to encounter race conditions with a multithreaded server, there is no proper locking to prevent two threads from modifying the same record and colliding.

  • #6 George Mabley said :

    I am using basically the same setup that you are, just with a few added views and model tweaks. I was able to delete before adding search, and can also comment out the __searchable__ and whoosh_index lines to get my delete function to work. Should I just read up on ways to lock the sessions, or is it unrealistic with the way the site is set up currently?

  • #7 Miguel Grinberg said :

    @George: can you show me the code that you have written, in particular the delete post view function and the changes to the Post model? You paste the code on pastebin or similar and share the URL here. Thanks!

  • #8 George Mabley said :

    Sorry for the delayed response. Here are the two parts of the code. http://pastebin.com/0TDx8Gz8

  • #9 George Mabley said :

    While adjusting for clarity on pastebin, I forgot to change the delete(u) to delete(post), but that is not the problem with the code. My bad!

  • #10 Miguel Grinberg said :

    @George: I get the error too. I haven't figured out exactly where the problem is, but after some debugging I noticed there is more than one session object active. As a workaround you can add 'db.session.close_all()` before the db.session.delete() call and that removes those extra sessions. I'll continue debugging to try to figure out where those extra sessions come from.

  • #11 Miguel Grinberg said :

    There is a bug in Flask-WhooshAlchemy that is causing this. I have forked the project and fixed it at https://github.com/miguelgrinberg/Flask-WhooshAlchemy. Hopefully the author will apply this fix or something similar soon. In the meantime you can download my fixed version and install it manually over the original. I will explain the bug and how I found it in an upcoming debugging article, not the following one but probably the one after that.

  • #12 Richard Austin said :

    Thank you for this great set of tutorials. I tried flask a while back, but got stuck trying to implement various features, including search. So this was very helpful! One small note though: This page is missing a couple of import statements (SearchForm & MAX_SEARCH_RESULTS). Nothing serious!

  • #13 Miguel Grinberg said :

    @Richard: thanks, I have added the missing imports.

  • #14 James said :

    Since this version the unit test 'test_follow_posts' isn't working anymore. I've downloaded microblog-0.10.zip. and ensured the test is included in tests.py. Any ideas?

  • #15 James said :

    If I comment out the following lines the test is working again: # __searchable__ = ['body'] .... #whooshalchemy.whoosh_index(app, Post)

  • #16 James said :

    You can ignore my previous comments. After the flask-whooshalchemy fix mentioned in Part XVI the unit test is working.

  • #17 Guillermo Nuñez said :

    This is exactly what I was looking for a couple of my projects. But on one of them, the site has already been running for almost a year. Is it possible to make it work with existing data?

  • #18 Miguel Grinberg said :

    @Guillermo: there are utilities that can look at a database and generate compatible SQLAlchemy models. Search for SQLSoup and sqlautocode, for example.

  • #19 Gurom said :

    Dear Miguel Grinberg! One more time :) Many thanks for your greate job! I have a problem after lesson "Part X: Full Text Search ". Please look at the following error: ________________________ File "......./app/views.py", line 44, in index Display the sourcecode for this frameOpen an interactive python shell in this frameposts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False) AttributeError: '_QueryProxy' object has no attribute 'paginate' ________________________ This mistake happens after your code (microblog-0.10.zip.) too. Do you have any idea were is problem?

  • #20 Miguel Grinberg said :

    @Gurom: install my fork of the Flask-WhooshAlchemy project, the official version has a bug that causes that error.

  • #21 Joshua Grigonis said :

    Downloading/unpacking Flask-WhooshAlchemy Could not find a version that satisfies the requirement Flask-WhooshAlchemy (from versions: 0.3a.macosx-10.6-x86_64, 0 .1a, 0.2a, 0.4a, 0.51a, 0.53a, 0.54a, 0.55a, 0.5a, v0.52a) Cleaning up... No distributions matching the version for Flask-WhooshAlchemy Storing complete log in C:\Users\joshua.grigonis\pip\pip.log

  • #22 Joshua Grigonis said :

    I was able to install WhooshAlchemy from git.

  • #23 Farshid P. said :

    How do you install from git ? I downloaded Miguel's version, and clearly see the difference in the code he has made. Do I just move the flask_whooshalchemy.py to the site-packages and replace ? Or do I move the the whole folder and run the setup.py ?

  • #24 Farshid P. said :

    Figured out how to 'install' the edited Whoosh fork from Miguel's github ( https://github.com/miguelgrinberg/Flask-WhooshAlchemy )that fixes the _QueryProxy bug. Just copy and past the Flask_WhooshAlchemy.py in the site-packages folder under flask/lib directory and run it (python Flask_WhooshAlchemy.py). Shazam fixed the bug.

  • #25 Cory Gough said :

    The lessons are great! But for some reason I am having trouble with pagination. It actually didn't give me this error until I made the text search changes, but now I am getting the below exception. I even download your code base and it is giving me the same issue. Do you have any idea why? As alway thank you for your help! File "/home/cory/Documents/microblog-0.10/app/views.py", line 43, in index posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False) AttributeError: '_QueryProxy' object has no attribute 'paginate'

Leave a Comment

Note: all comments are screened before they are published. Thank you for your patience!