2018-03-20T18:20:43Z

The Flask Mega-Tutorial Part XVI: Full-Text Search

This is the sixteenth installment of the Flask Mega-Tutorial series, in which I'm going to add a full-text search capability to Microblog.

For your reference, below is a list of the articles in this series.

Note 1: If you are looking for the legacy version of this tutorial, it's here.

Note 2: If you would like to support my work on this blog, or just don't have patience to wait for weekly articles, I am offering the complete version of this tutorial packaged as an ebook or a set of videos. For more information, visit courses.miguelgrinberg.com.

The goal of this chapter is to implement a search feature for Microblog, so that users can find interesting posts using natural language. For many types of web sites, it is possible to just let Google, Bing, etc. index all the content and provide search results through their search APIs. This works well for sites that have mostly static pages, like a forum. But in my application the basic unit of content is a user post, which is a small portion of the entire web page. The type of search results that I want are for these individual blog posts and not entire pages. For example, if I search for the word "dog" I want to see blog posts from any users that include that word. Obviously a page that shows all blog posts that have the word "dog" (or any other possible search term) does not really exist as a page that the big search engines can find and index, so clearly I have no choice other than to roll my own search feature.

The GitHub links for this chapter are: Browse, Zip, Diff.

Introduction to Full-Text Search Engines

Support for full-text search is not standardized like relational databases are. There are several open-source full-text engines: Elasticsearch, Apache Solr, Whoosh, Xapian, Sphinx, etc. As if this isn't enough choice, there are several databases that also provide searching capabilities that are comparable to dedicated search engines like the ones I enumerated above. SQLite, MySQL and PostgreSQL all offer some support for searching text, and NoSQL databases such as MongoDB and CouchDB do too.

If you are wondering which of these can work within a Flask application, the answer is all of them! That is one of the strengths of Flask, it does its job while not being opinionated. So what's the best choice?

From the list of dedicated search engines, Elasticsearch is one that stands out to me as being fairly popular, in part due to its popularity as the "E" in the ELK stack for indexing logs, along with Logstash and Kibana. Using the searching capabilities of one of the relational databases could also be a good choice, but given the fact that SQLAlchemy does not support this functionality, I would have to handle the searching with raw SQL statements, or else find a package that provides high-level access to text searches while being able to coexist with SQLAlchemy.

Based on the above analysis, I'm going to use Elasticsearch, but I'm going to implement all the text indexing and searching functions in a way that is very easy to switch to another engine. That will allow you to replace my implementation with an alternative one based on a different engine just by rewriting a few functions in a single module.

Installing Elasticsearch

There are several ways to install Elasticsearch, including one-click installers, zip file with the binaries that you need to install yourself, and even a Docker image. The documentation has an Installation page with detailed information on all these options. If you are using Linux, you will likely have a package available for your distribution. If you are using a Mac and have Homebrew installed, then you can simply run brew install elasticsearch.

Once you install Elasticsearch on your computer, you can verify that it is running by typing http://localhost:9200 in your browser's address bar, which should return some basic information about the service in JSON format.

Since I will be managing Elasticsearch from Python, I will also be using the Python client library:

(venv) $ pip install elasticsearch

You may also want to update your requirements.txt file:

(venv) $ pip freeze > requirements.txt

Elasticsearch Tutorial

I'm going to start by showing you the basics of working with Elasticsearch from a Python shell. This will help you familiarize with this service, so that you can understand the implementation that I will discuss later.

To create a connection to Elasticsearch, create an instance of class Elasticsearch, passing a connection URL as an argument:

>>> from elasticsearch import Elasticsearch
>>> es = Elasticsearch('http://localhost:9200')

Data in Elasticsearch is written to indexes. Unlike a relational database, the data is just a JSON object. The following example writes an object with a field called text to an index called my_index:

>>> es.index(index='my_index', id=1, body={'text': 'this is a test'})

For each document stored, Elasticsearch takes a unique id and the JSON object with the data.

Let's store a second document on this index:

>>> es.index(index='my_index', id=2, body={'text': 'a second test'})

And now that there are two documents in this index, I can issue a free-form search. In this example, I'm going to search for this test:

>>> es.search(index='my_index', body={'query': {'match': {'text': 'this test'}}})

The response from the es.search() call is a Python dictionary with the search results:

{
    'took': 1,
    'timed_out': False,
    '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
    'hits': {
        'total': 2, 
        'max_score': 0.5753642, 
        'hits': [
            {
                '_index': 'my_index',
                '_type': 'my_index',
                '_id': '1',
                '_score': 0.5753642,
                '_source': {'text': 'this is a test'}
            },
            {
                '_index': 'my_index',
                '_type': 'my_index',
                '_id': '2',
                '_score': 0.25316024,
                '_source': {'text': 'a second test'}
            }
        ]
    }
}

Here you can see that the search returned the two documents, each with an assigned score. The document with the highest score contains the two words I searched for, and the other document contains only one. You can see that even the best result does not have a great score, because the words do not exactly match the text.

Now this is the result if I search for the word second:

>>> es.search(index='my_index', body={'query': {'match': {'text': 'second'}}})
{
    'took': 1,
    'timed_out': False,
    '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
    'hits': {
        'total': 1,
        'max_score': 0.25316024,
        'hits': [
            {
                '_index': 'my_index',
                '_type': 'my_index',
                '_id': '2',
                '_score': 0.25316024,
                '_source': {'text': 'a second test'}
            }
        ]
    }
}

I still get a fairly low score because my search does not match the text in this document, but since only one of the two documents contains the word "second", the other document does not show up at all.

The Elasticsearch query object has more options, all well documented, and includes options such as pagination and sorting, just like relational databases.

Feel free to add more entries to this index and try different searches. When you are done experimenting, you can delete the index with the following command:

>>> es.indices.delete('my_index')

Elasticsearch Configuration

Integrating Elasticsearch into the application is a great example of the power of Flask. This is a service and Python package that does not have anything to do with Flask, yet, I'm going to get a pretty good level of integration, starting from the configuration, which I'm going to write in the app.config dictionary from Flask:

config.py: Elasticsearch configuration.

class Config(object):
    # ...
    ELASTICSEARCH_URL = os.environ.get('ELASTICSEARCH_URL')

Like with many other configuration entries, the connection URL for Elasticsearch is going to be sourced from an environment variable. If the variable is not defined, I'm going to let the setting be set to None, and I'll use that as a signal to disable Elasticsearch. This is mainly for convenience, so that you are not forced to always have the Elasticsearch service up and running when you work on the application, and in particular when you run unit tests. So to make sure the service is used, I need to define the ELASTICSEARCH_URL environment variable, either directly in the terminal or by adding it to the .env file as follows:

ELASTICSEARCH_URL=http://localhost:9200

Elasticsearch presents the challenge that it isn't wrapped by a Flask extension. I cannot create the Elasticsearch instance in the global scope like I did in the examples above because to initialize it I need access to app.config, which only becomes available after the create_app() function is invoked. So I decided to add a elasticsearch attribute to the app instance in the application factory function:

app/__init__.py: Elasticsearch instance.

# ...
from elasticsearch import Elasticsearch

# ...

def create_app(config_class=Config):
    app = Flask(__name__)
    app.config.from_object(config_class)

    # ...
    app.elasticsearch = Elasticsearch([app.config['ELASTICSEARCH_URL']]) \
        if app.config['ELASTICSEARCH_URL'] else None

    # ...

Adding a new attribute to the app instance may seem a little strange, but Python objects are not strict in their structure, new attributes can be added to them at any time. An alternative that you may also consider is to create a subclass of Flask (maybe call it Microblog), with the elasticsearch attribute defined in its __init__() function.

Note how I use a conditional expression to make the Elasticsearch instance None when a URL for the Elasticsearch service wasn't defined in the environment.

A Full-Text Search Abstraction

As I said in the chapter's introduction, I want to make it easy to switch from Elasticsearch to other search engines, and I also don't want to code this feature specifically for searching blog posts, I prefer to design a solution that in the future I can easily extend to other models if I need to. For all these reasons, I decided to create an abstraction for the search functionality. The idea is to design the feature in generic terms, so I will not be assuming that the Post model is the only one that needs to be indexed, and I will also not be assuming that Elasticsearch is the index engine of choice. But if I can't make any assumptions about anything, how can I get this work done?

The first thing that I need, is to somehow find a generic way to indicate which model and which field or fields in it are to be indexed. I'm going to say that any model that needs indexing needs to define a __searchable__ class attribute that lists the fields that need to be included in the index. For the Post model, these are the changes:

app/models.py: Add a \_\_searchable\_\_ attribute to the Post model.

class Post(db.Model):
    __searchable__ = ['body']
    # ...

So here I'm saying that this model needs to have its body field indexed. But just to make sure this is perfectly clear, this __searchable__ attribute that I added is just a variable, it does not have any behavior associated with it. It will just help me write my indexing functions in a generic way.

I'm going to write all the code that interacts with the Elasticsearch index in a app/search.py module. The idea is to keep all the Elasticsearch code in this module. The rest of the application will use the functions in this new module to access the index and will not have direct access to Elasticsearch. This is important, because if one day I decided I don't like Elasticsearch anymore and want to switch to a different engine, all I need to do is rewrite the functions in this module, and the application will continue to work as before.

For this application, I decided that I need three supporting functions related to text indexing: I need to add entries to a full-text index, I need to remove entries from the index (assuming one day I will support deleting blog posts), and I need to execute a search query. Here is the app/search.py module that implements these three functions for Elasticsearch, using the functionality I showed you above from the Python console:

app/search.py: Search functions.

from flask import current_app

def add_to_index(index, model):
    if not current_app.elasticsearch:
        return
    payload = {}
    for field in model.__searchable__:
        payload[field] = getattr(model, field)
    current_app.elasticsearch.index(index=index, id=model.id, body=payload)

def remove_from_index(index, model):
    if not current_app.elasticsearch:
        return
    current_app.elasticsearch.delete(index=index, id=model.id)

def query_index(index, query, page, per_page):
    if not current_app.elasticsearch:
        return [], 0
    search = current_app.elasticsearch.search(
        index=index,
        body={'query': {'multi_match': {'query': query, 'fields': ['*']}},
              'from': (page - 1) * per_page, 'size': per_page})
    ids = [int(hit['_id']) for hit in search['hits']['hits']]
    return ids, search['hits']['total']['value']

These functions all start by checking if app.elasticsearch is None, and in that case return without doing anything. This is so that when the Elasticsearch server isn't configured, the application continues to run without the search capability and without giving any errors. This is just as a matter of convenience during development or when running unit tests.

The functions accept the index name as an argument. In all the calls I'm passing down to Elasticsearch, I'm using this name as the index name and also as the document type, as I did in the Python console examples.

The functions that add and remove entries from the index take the SQLAlchemy model as a second argument. The add_to_index() function uses the __searchable__ class variable I added to the model to build the document that is inserted into the index. If you recall, Elasticsearch documents also needed a unique identifier. For that I'm using the id field of the SQLAlchemy model, which is also conveniently unique. Using the same id value for SQLAlchemy and Elasticsearch is very useful when running the searches, as it allows me to link entries in the two databases. Something I did not mention above is that if you attempt to add an entry with an existing id, then Elasticsearch replaces the old entry with the new one, so add_to_index() can be used for new objects as well as for modified ones.

I did not show you the es.delete() function that I'm using in remove_from_index() before. This function deletes the document stored under the given id. Here is a good example of the convenience of using the same id to link entries in both databases.

The query_index() function takes the index name and a text to search for, along with pagination controls, so that search results can be paginated like Flask-SQLAlchemy results are. You have already seen an example usage of the es.search() function from the Python console. The call I'm issuing here is fairly similar, but instead of using a match query type, I decided to use multi_match, which can search across multiple fields. By passing a field name of *, I'm telling Elasticsearch to look in all the fields, so basically I'm searching the entire index. This is useful to make this function generic, since different models can have different field names in the index.

The body argument to es.search() includes pagination arguments in addition to the query itself. The from and size arguments control what subset of the entire result set needs to be returned. Elasticsearch does not provide a nice Pagination object like the one from Flask-SQLAlchemy, so I have to do the pagination math to calculate the from value.

The return statement in the query_index() function is somewhat complex. It returns two values: the first is a list of id elements for the search results, and the second is the total number of results. Both are obtained from the Python dictionary returned by the es.search() function. If you are not familiar with the expression that I'm using to obtain the list of IDs, this is called a list comprehension, and is a fantastic feature of the Python language that allows you to transform lists from one format to another. In this case I'm using the list comprehension to extract the id values from the much larger list of results provided by Elasticsearch.

Is this too confusing? Maybe a demonstration of these functions from the Python console can help you understand them a bit more. In the following session, I manually add all the posts from the database to the Elasticsearch index. In my test database, I had a few posts that had the numbers "one", "two", "three", "four" and "five" in them, so I used that as a search query. You may need to adapt your query to match the contents of your database:

>>> from app.search import add_to_index, remove_from_index, query_index
>>> for post in Post.query.all():
...     add_to_index('posts', post)
>>> query_index('posts', 'one two three four five', 1, 100)
([15, 13, 12, 4, 11, 8, 14], 7)
>>> query_index('posts', 'one two three four five', 1, 3)
([15, 13, 12], 7)
>>> query_index('posts', 'one two three four five', 2, 3)
([4, 11, 8], 7)
>>> query_index('posts', 'one two three four five', 3, 3)
([14], 7)

The query that I issued returned seven results. When I asked for page 1 with 100 items per page I get all seven, but then the next three examples shows how I can paginate the results in a way that is very similar to what I did for Flask-SQLAlchemy, with the exception that the results come as a list of IDs instead of SQLAlchemy objects.

If you want to keep things clean, delete the posts index after you are doing experimenting with it:

>>> app.elasticsearch.indices.delete('posts')

Integrating Searches with SQLAlchemy

The solution that I showed you in the previous section is decent, but it still has a couple of problems. The most obvious problem is that results come as a list of numeric IDs. This is highly inconvenient, I need SQLAlchemy models so that I can pass them down to templates for rendering, and I need a way to replace that list of numbers with the corresponding models from the database. The second problem is that this solution requires the application to explicitly issue indexing calls as posts are added or removed, which is not terrible, but less than ideal, since a bug that causes a missed indexing call when making a change on the SQLAlchemy side is not going to be easily detected, the two databases will get out of sync more and more each time the bug occurs and you will probably not notice for a while. A better solution would be for these calls to be triggered automatically as changes are made on the SQLAlchemy database.

The problem of replacing the IDs with objects can be addressed by creating a SQLAlchemy query that reads those objects from the database. This sounds easy in practice, but doing it efficiently with a single query is actually a bit tricky to implement.

For the problem of triggering the indexing changes automatically, I decided to drive updates to the Elasticsearch index from SQLAlchemy events. SQLAlchemy provides a large list of events that applications can be notified about. For example, each time a session is committed, I can have a function in the application invoked by SQLAlchemy, and in that function I can apply the same updates that were made on the SQLAlchemy session to the Elasticsearch index.

To implement the solutions to these two problems I'm going to write a mixin class. Remember mixin classes? In Chapter 5, I added the UserMixin class from Flask-Login to the User model, to give it some features that were required by Flask-Login. For the search support I'm going to define my own SearchableMixin class, that when attached to a model, will give it the ability to automatically manage an associated full-text index. The mixin class will act as a "glue" layer between the SQLAlchemy and Elasticsearch worlds, providing solutions to the two problems I stated above.

Let me show you the implementation, then I'll go over some interesting details. Note that this makes use of several advanced techniques, so you will need to study this code carefully to fully understand it.

app/models.py: SearchableMixin class.

from app.search import add_to_index, remove_from_index, query_index

class SearchableMixin(object):
    @classmethod
    def search(cls, expression, page, per_page):
        ids, total = query_index(cls.__tablename__, expression, page, per_page)
        if total == 0:
            return cls.query.filter_by(id=0), 0
        when = []
        for i in range(len(ids)):
            when.append((ids[i], i))
        return cls.query.filter(cls.id.in_(ids)).order_by(
            db.case(when, value=cls.id)), total

    @classmethod
    def before_commit(cls, session):
        session._changes = {
            'add': list(session.new),
            'update': list(session.dirty),
            'delete': list(session.deleted)
        }

    @classmethod
    def after_commit(cls, session):
        for obj in session._changes['add']:
            if isinstance(obj, SearchableMixin):
                add_to_index(obj.__tablename__, obj)
        for obj in session._changes['update']:
            if isinstance(obj, SearchableMixin):
                add_to_index(obj.__tablename__, obj)
        for obj in session._changes['delete']:
            if isinstance(obj, SearchableMixin):
                remove_from_index(obj.__tablename__, obj)
        session._changes = None

    @classmethod
    def reindex(cls):
        for obj in cls.query:
            add_to_index(cls.__tablename__, obj)

db.event.listen(db.session, 'before_commit', SearchableMixin.before_commit)
db.event.listen(db.session, 'after_commit', SearchableMixin.after_commit)

There are four functions in this mixin class, all class methods. Just as a refresher, a class method is a special method that is associated with the class and not a particular instance. Note how I renamed the self argument used in regular instance methods to cls, to make it clear that this method receives a class and not an instance as its first argument. Once attached to the Post model for example, the search() method above would be invoked as Post.search(), without having to have an actual instance of class Post.

The search() class method wraps the query_index() function from app/search.py to replace the list of object IDs with actual objects. You can see that the first thing this function does is call query_index(), passing cls.__tablename__ as the index name. This is going to be a convention, all indexes will be named with the name Flask-SQLAlchemy assigned to the relational table. The function returns the list of result IDs, and the total number of results. The SQLAlchemy query that retrieves the list of objects by their IDs is based on a CASE statement from the SQL language, which needs to be used to ensure that the results from the database come in the same order as the IDs are given. This is important because the Elasticsearch query returns results sorted from more to less relevant. If you want to learn more about the way this query works, you can consult the accepted answer to this StackOverflow question. The search() function returns the query that replaces the list of IDs, and also passes through the total number of search results as a second return value.

The before_commit() and after_commit() methods are going to respond to two events from SQLAlchemy, which are triggered before and after a commit takes place respectively. The before handler is useful because the session hasn't been committed yet, so I can look at it and figure out what objects are going to be added, modified and deleted, available as session.new, session.dirty and session.deleted respectively. These objects are not going to be available anymore after the session is committed, so I need to save them before the commit takes place. I'm using a session._changes dictionary to write these objects in a place that is going to survive the session commit, because as soon as the session is committed I will be using them to update the Elasticsearch index.

When the after_commit() handler is invoked, the session has been successfully committed, so this is the proper time to make changes on the Elasticsearch side. The session object has the _changes variable that I added in before_commit(), so now I can iterate over the added, modified and deleted objects, and make the corresponding calls to the indexing functions in app/search.py for the objects that have the SearchableMixin class.

The reindex() class method is a simple helper method that you can use to refresh an index with all the data from the relational side. You saw me do something similar from the Python shell session above to do an initial load of all the posts into a test index. With this method in place, I can issue Post.reindex() to add all the posts in the database to the search index.

After the class definition I made two calls to SQLALchemy's function db.event.listen(). Note that these calls are not inside the class, but after it. The purpose of these two statements is to set up the event handlers that will make SQLAlchemy call the before_commit() and after_commit() methods before and after each commit respectively.

To incorporate the SearchableMixin class into the Post model I have to add it as a subclass, and I also need to hook up the before and after commit events:

app/models.py: Adding the SearchableMixin class to the Post model.

class Post(SearchableMixin, db.Model):
    # ...

Now the Post model is automatically maintaining a full-text search index for posts. I can use the reindex() method to initialize the index from all the posts currently in the database:

>>> Post.reindex()

And I can search posts working with SQLAlchemy models by running Post.search(). In the following example, I ask for the first page of five elements for my query:

>>> query, total = Post.search('one two three four five', 1, 5)
>>> total
7
>>> query.all()
[<Post five>, <Post two>, <Post one>, <Post one more>, <Post one>]

Search Form

This was very intense. The work that I've done above to keep things generic touches on several advanced topics, so it may take you time to fully understand it. But now I have a complete system to work with natural language searches for blog posts. What I need to do now is integrate all this functionality with the application.

A fairly standard approach for web-based searches is to have the search term as a q argument in the query string of the URL. For example, if you wanted to search for Python on Google, and you want to save a couple of seconds, you can just type the following URL in your browser's address bar to go directly to the results:

https://www.google.com/search?q=python

Allowing searches to be completely encapsulated in a URL is nice, because these can be shared with other people, who just by clicking on the link have access to the search results.

This introduces a change in the way I showed you to handle web forms in the past. I have used POST requests to submit form data for all the forms the application has so far, but to implement searches as above, the form submission will have to go as a GET request, which is the request method that is used when you type a URL in your browser or click a link. Another interesting difference is that the search form is going to be in the navigation bar, so it needs to be present in all pages of the application.

Here is the search form class, with just the q text field:

app/main/forms.py: Search form.

from flask import request

class SearchForm(FlaskForm):
    q = StringField(_l('Search'), validators=[DataRequired()])

    def __init__(self, *args, **kwargs):
        if 'formdata' not in kwargs:
            kwargs['formdata'] = request.args
        if 'csrf_enabled' not in kwargs:
            kwargs['csrf_enabled'] = False
        super(SearchForm, self).__init__(*args, **kwargs)

The q field does not require any explanation, as it is similar to other text fields I've used in the past. For this form, I decided not to have a submit button. For a form that has a text field, the browser will submit the form when you press Enter with the focus on the field, so a button is not needed. I have also added a __init__ constructor function, which provides values for the formdata and csrf_enabled arguments if they are not provided by the caller. The formdata argument determines from where Flask-WTF gets form submissions. The default is to use request.form, which is where Flask puts form values that are submitted via POST request. Forms that are submitted via GET request have the field values in the query string, so I need to point Flask-WTF at request.args, which is where Flask writes the query string arguments. And as you remember, forms have CSRF protection added by default, with the inclusion of a CSRF token that is added to the form via the form.hidden_tag() construct in templates. For clickable search links to work, CSRF needs to be disabled, so I'm setting csrf_enabled to False so that Flask-WTF knows that it needs to bypass CSRF validation for this form.

Since I'm going to need to have this form visible in all pages, I need to create an instance of the SearchForm class regardless of the page the user is viewing. The only requirement is that the user is logged in, because for anonymous users I am not currently showing any content. Instead of creating a form object in every route, and then passing the form to all the templates, I'm going to show you a very useful trick that eliminates duplication of code when you need to implement a feature across the entire application. I already used a before_request handler before, back in Chapter 6, to record the time of last visit for each user. What I'm going to do is create my search form in that same function, but with a twist:

app/main/routes.py: Instantiate the search form in the before\_request handler.

from flask import g
from app.main.forms import SearchForm

@bp.before_app_request
def before_request():
    if current_user.is_authenticated:
        current_user.last_seen = datetime.utcnow()
        db.session.commit()
        g.search_form = SearchForm()
    g.locale = str(get_locale())

Here I create an instance of the search form class when I have an authenticated user. But of course, I need this form object to persist until it can be rendered at the end of the request, so I need to store it somewhere. That somewhere is going to be the g container, provided by Flask. This g variable provided by Flask is a place where the application can store data that needs to persist through the life of a request. Here I'm storing the form in g.search_form, so then when the before request handler ends and Flask invokes the view function that handles the requested URL, the g object is going to be the same, and will still have the form attached to it. It's important to note that this g variable is specific to each request and each client, so even if your web server is handling multiple requests at a time for different clients, you can still rely on g to work as private storage for each request, independently of what goes on in other requests that are handled concurrently.

The next step is to render the form to the page. I said above that I wanted this form in all pages, so what makes more sense is to render it as part of the navigation bar. This is, in fact, simple, because templates can also see the data stored in the g variable, so I don't need to worry about adding the form as an explicit template argument in all the render_template() calls in the application. Here is how I can render the form in the base template:

app/templates/base.html: Render the search form in the navigation bar.

            ...
            <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
                <ul class="nav navbar-nav">
                    ... home and explore links ...
                </ul>
                {% if g.search_form %}
                <form class="navbar-form navbar-left" method="get"
                        action="{{ url_for('main.search') }}">
                    <div class="form-group">
                        {{ g.search_form.q(size=20, class='form-control',
                            placeholder=g.search_form.q.label.text) }}
                    </div>
                </form>
                {% endif %}
                ...

The form is rendered only if g.search_form is defined. This check is necessary because some pages, such as error pages, may not have it defined. This form is slightly different than the ones I did previously. I'm setting its method attribute to get, because I want the form data to be submitted on the query string with a GET request. Also, the other forms I created had the action attribute empty, because they were submitted to the same page that rendered the form. This form is special because it appears in all pages, so I need to tell it explicitly where it needs to be submitted, which is a new route that is specifically dedicated to handling searches.

Search View Function

The last bit of functionality to complete the search feature is the view function that receives the search form submission. This view function is going to be attached to the /search route, so that you can send a search request with a http://localhost:5000/search?q=search-words, just like Google.

app/main/routes.py: Search view function.

@bp.route('/search')
@login_required
def search():
    if not g.search_form.validate():
        return redirect(url_for('main.explore'))
    page = request.args.get('page', 1, type=int)
    posts, total = Post.search(g.search_form.q.data, page,
                               current_app.config['POSTS_PER_PAGE'])
    next_url = url_for('main.search', q=g.search_form.q.data, page=page + 1) \
        if total > page * current_app.config['POSTS_PER_PAGE'] else None
    prev_url = url_for('main.search', q=g.search_form.q.data, page=page - 1) \
        if page > 1 else None
    return render_template('search.html', title=_('Search'), posts=posts,
                           next_url=next_url, prev_url=prev_url)

You have seen that in the other forms I used the form.validate_on_submit() method to check if the form submission was valid. Unfortunately that method only works for forms submitted via POST request, so for this form I need to use form.validate() which just validates field values, without checking how the data was submitted. If the validation fails, it is because the user submitted an empty search form, so in that case I just redirect to the explore page, which shows all blog posts.

The Post.search() method from my SearchableMixin class is used to obtain the list of search results. The pagination is handled in a very similar way to that of the index and explore pages, but generating the next and previous links is a little bit trickier without the help of the Pagination object from Flask-SQLAlchemy. This is where the total number of results passed as a second return value from Post.search() is useful.

Once the page of search results and pagination links are calculated, all that is left is to render a template with all this data. I could have figured out a way to reuse the index.html template to display search results, but given that there are a few differences I decided to create a dedicated search.html template that is dedicated to show search results, taking advantage of the _post.html sub-template to render the search results:

app/templates/search.html: Search results template.

{% extends "base.html" %}

{% block app_content %}
    <h1>{{ _('Search Results') }}</h1>
    {% for post in posts %}
        {% include '_post.html' %}
    {% endfor %}
    <nav aria-label="...">
        <ul class="pager">
            <li class="previous{% if not prev_url %} disabled{% endif %}">
                <a href="{{ prev_url or '#' }}">
                    <span aria-hidden="true">&larr;</span>
                    {{ _('Previous results') }}
                </a>
            </li>
            <li class="next{% if not next_url %} disabled{% endif %}">
                <a href="{{ next_url or '#' }}">
                    {{ _('Next results') }}
                    <span aria-hidden="true">&rarr;</span>
                </a>
            </li>
        </ul>
    </nav>
{% endblock %}

If the rendering logic for the previous and next links gets a bit confusing it might help to review the Bootstrap documentation for the pagination component.

Search Results

What do you think? This was an intense chapter, where I presented some fairly advanced techniques. Some of the concepts in this chapter may take some time to sink in. The most important take away from this chapter is that if you want to use a different search engine than Elasticsearch, all you need to do is re-implement the three functions in app/search.py. The other important benefit of going through this effort is that in the future, if I need to add search support for a different database model, I can simply do so by adding the SearchableMixin class to it, the __searchable__ attribute with the list of fields to index and the SQLAlchemy event handler connections. I think it was well worth the effort, because from now on, it is going to be easy to deal with full-text indexes.

334 comments

  • #101 Miguel Grinberg said 2018-11-20T10:05:54Z

    @Krzysztof: I do not know how any of these new Flask-Whooshalchemy clones work, all I know is that the original was flawed. The error that you get appears to be because you are using "current_app" in a place where there is an app context. If you replace that with the actual application instance I think you will bypass the error.

  • #102 Philip Nwaobasi said 2018-11-27T00:07:36Z

    Hello Miguel/Everyone. Can anyone help with this if they have previously encountered it? My understanding of this line:

    app.elasticsearch = Elasticsearch([app.config['ELASTICSEARCH_URL']]) \ if app.config['ELASTICSEARCH_URL'] else None

    Was that if the ELASTICSEARCH_URL was not provided in the app.config file, it will stop the Elasticsearch functionality. However, when i comment out the URL, i get the traceback below:

    Traceback (most recent call last): File "c:\users\workstation\appdata\local\programs\python\python36\lib\site-packages\flask_compat.py", line 35, in reraise raise value File "D:\Projects\truthspearmint\spearmint.py", line 2, in app = create_app() File "D:\Projects\truthspearmint\app__init__.py", line 35, in create_app if app.config['ELASTICSEARCH_URL'] else None KeyError: 'ELASTICSEARCH_URL'

    Which seems to be suggesting that the URL must be provided. I am having trouble with the Elasticsearch functionality, as i constantly get this error:

    elasticsearch.exceptions.AuthorizationException: TransportError(403, 'cluster_block_exception', 'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];')

    Nothing i have tried so far has remedied this, so i wanted to switch off Elasticsearch for the time being, so i can continuing working on my app.

    Any help will be sincerely appreciated. Thank you

  • #103 Miguel Grinberg said 2018-11-27T10:17:51Z

    @Philip: the configuration variable needs to always exist. What you can omit is the environment variable from where its value comes from. If the env variable is missing the configuration variable is going to be set to None, and that is what disables the search code.

  • #104 bonny said 2018-11-29T20:55:41Z

    Hey Mr Miguel, thank you for the tutorial it's I really love it. I would like you to help me explain what could be the cause of this error

    RequestError: TransportError(400, u'search_phase_execution_exception', u'For input string: "kampala"')

    Thank you

  • #105 Miguel Grinberg said 2018-11-29T22:30:07Z

    @bonny: you need to check the Elasticsearch logs to see if there are any errors or clues. You can also try recreating your database by deleting the index, and then using the reindex() method shown in this article to repopulate the index again.

  • #106 Krzysztof said 2018-12-02T23:17:35Z

    Hello Miguel,

    I managed to bypass an error and a .toc file has been created on a startup. However, having applied all these changes in the Models, flask_whooshalchemy is supposed to hook up into flask_sqlalchemy, e.g. to modify its classes. Without it, there is no possibility of searching via Model.query.seach(). The necessary changes are stored in the QueryProxy class in the init file for flask_whooshalchemy: https://github.com/blakev/Flask-WhooshAlchemy3/blob/master/flask_whooshalchemy/init.py Would you be able to suggest a way to implement these changes? I might be wrong, but I assumed that by simply importing a module such as flask_whooshalchemy we prompt the init to be run, thus importing all the elements contained within. Please let me know if these assumptions were incorrect.

  • #107 Miguel Grinberg said 2018-12-03T09:33:44Z

    @Krzysztof: I really don't know. The flask-whooshalchemy extension is old, it may not even work with recent versions of Flask. I suggest you use my new solution, which you can implement using whoosh instead of elasticsearch, without relying on flask-whooshalchemy.

  • #108 xccx said 2018-12-16T04:45:08Z

    Hi Miguel, thanks for this tutorial. I want to ask a question: The g.search_form.validate() is always false. But if I delete the two sentences: if not g.search_form.validate(): return redirect(url_for('main.explore'))

    I can get the right results. So, is the validate() is still available?

    Thank you very much!

  • #109 Miguel Grinberg said 2018-12-16T12:28:16Z

    @xccx: the validate() method for the search form should return False only when the search field is empty. You may want to check how you defined and rendered your form versus my version, you may have differences there. In particular, this form is different than others in that it disables CSRF token validation, the reason is discussed in the article. Maybe you missed that part?

  • #110 Rohit Dalal said 2018-12-17T07:17:23Z

    Hello Sir,

    I have two queries -

    1) class SearchableMixin(object): @classmethod def search(cls, expression, page, per_page): ids, total = query_index(cls.tablename, expression, page, per_page) if total == 0: return cls.query.filter_by(id=0), 0

    if total == 0 why not just return [], 0 is there any specific need to make one extra db query searching id = 0 which will always be empty

    2) When searching some random text which is not there in posts getting following error - elasticsearch.exceptions.RequestError elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'For input string: "adawdwadawwd"')

    Please tell a way to resolve this issue.

    Thank you for this awesome tutorials.

  • #111 Miguel Grinberg said 2018-12-17T09:26:17Z

    @Rohit: 1) the return value must be a database query, not a list of results. 2) what version of elasticsearch are you using? I do not see those errors here.

  • #112 Dima said 2018-12-17T18:24:50Z

    Miguel, this is a great tutorial. I heavily use it on my work. One question - could you please explain how to update nginx config if I want to use elasticsearch? So far I have the following, but it does work. I have error (111: Connection refused) while connecting to upstream ``` upstream elasticsearch { server 127.0.0.1:9200; keepalive 64; }

    server { listen 8080; location /search { proxy_pass http://elasticsearch; proxy_http_version 1.1; proxy_set_header Connection "Keep-Alive"; proxy_set_header Proxy-Connection "Keep-Alive"; } }

  • #113 Miguel Grinberg said 2018-12-17T23:30:53Z

    @Dima: Elasticsearch is an internal service, it does not need to be exported to the outside world by nginx. The correct thing to do is to not include elasticsearch in nginx, as I do it in the deployment chapter a bit later.

  • #114 David A said 2019-01-04T20:46:28Z

    Hi Miguel, I am a big fan of this tutorial series. I am working through it with windows 10 and I have come across this issue in this chapter: Traceback (most recent call last): File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 2309, in call return self.wsgi_app(environ, start_response) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 2295, in wsgi_app response = self.handle_exception(e) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 1741, in handle_exception reraise(exc_type, exc_value, tb) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask_compat.py", line 35, in reraise raise value File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 2292, in wsgi_app response = self.full_dispatch_request() File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request rv = self.handle_user_exception(e) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 1718, in handle_user_exception reraise(exc_type, exc_value, tb) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask_compat.py", line 35, in reraise raise value File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request rv = self.dispatch_request() File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask\app.py", line 1799, in dispatch_request return self.view_functionsrule.endpoint File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\flask_login\utils.py", line 261, in decorated_view return func(*args, kwargs) File "C:\Users\flyff\Desktop\Flask Megatutorial\microblog\app\main\routes.py", line 144, in search current_app.config['POSTS_PER_PAGE']) File "C:\Users\flyff\Desktop\Flask Megatutorial\microblog\app\models.py", line 15, in search ids, total = query_index(cls.__tablename__, expression, page, per_page) File "C:\Users\flyff\Desktop\Flask Megatutorial\microblog\app\search.py", line 26, in query_index 'from': (page - 1) * per_page, 'size': per_page}) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\elasticsearch\client\utils.py", line 76, in _wrapped return func(*args, params=params, kwargs) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\elasticsearch\client__init__.py", line 660, in search doc_type, '_search'), params=params, body=body) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\elasticsearch\transport.py", line 318, in perform_request status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\elasticsearch\connection\http_urllib3.py", line 186, in perform_request self._raise_error(response.status, raw_data) File "c:\users\flyff\desktop\flask megatutorial\microblog\venv\lib\site-packages\elasticsearch\connection\base.py", line 125, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index')

    Another possibly related issue is after using query, total = Post.search('one two three four five', 1, 5) I get: NameError: name 'Post' is not defined

  • #115 Miguel Grinberg said 2019-01-04T23:24:10Z

    David: were you able to run the elasticsearch exercises from the article in the console? The error indicates the elasticsearch index does not exist, so it seems you did not add anything to the index.

  • #116 Goga said 2019-01-05T00:49:13Z

    Hi. Miguel. I am having the following problem. if i try to launch flask shell in cmd, I get RuntimeError: Working outside of application context. if i start python shell and do : from app.search import add_to_index, remove_from_index, query_index and then Post.query.all() I get name 'Post' is not defined. then i do : from app import create_app and then app = create_app() I again get Working outside of application context . and the same result with

    from app import db, create_app db.create_all(app=create_app())

    So what confuses me is how am I supposed to push app context if when I am trying to do that, I am already required to have app context.

  • #117 Miguel Grinberg said 2019-01-05T14:50:29Z

    @Goga: are you following the tutorial chapters in order? Sounds like you might be skipping things. Go back to Chapter 4 and make sure you can complete your database and shell set up.

  • #118 David A said 2019-01-25T19:35:42Z

    Hi Miguel, I am running into an issue when trying to search. I get the following error: elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index'). Does this error mean I need to delete and recreate the indexes? If so could you guide me. Also note I am on Windows. Would this be related to another issue I have in which Post is not defined when trying to execute this section of the tutorial: >>> from app.search import add_to_index, remove_from_index, query_index

    for post in Post.query.all(): ... add_to_index('posts', post)

  • #119 Miguel Grinberg said 2019-01-25T23:45:26Z

    @David: The "Post is not defined" could be that you haven't imported the Post model. Once you fix that you will be able to generate the elasticsearch index, and once you have an index created you will not get the 'index not found" error anymore.

  • #120 Flaviu said 2019-02-26T09:46:35Z

    Hi Miguel,

    Many thanks for your tutorial! I think in your tutorial or your book (I use both of them) you split the databases and the config between test, dev and prod environment. How would you recomment doing the same with elastic search environment?

    I was thinking using prefixes like test-, dev-, and prod- in front of the index names. Would that be a good idea? I couldn't think of something else.

    Regards, Flaviu

  • #121 Miguel Grinberg said 2019-02-26T19:07:21Z

    @Flaviu: you can use different elasticsearch databases if that's possible. In general you do not want to use the same database for multiple purposes. But if you must use a single database, then a prefix on all the index names is a decent solution.

  • #122 Roger said 2019-02-27T09:23:14Z

    @Miguel, thanks for this tutorial, I'd like to set ik-analyzer as my ElasticSearch analyzer instead of default Standard analyzer. It works directly on HTTP request: curl -H 'Content-Type: application/json' -XGET 'localhost:9200/_analyze?pretty' -d '{"analyzer":"ik_max_word","text":"Some Keywords"}' then I try to combine this analyzer "ik_max_word" into Mega-Tutorial project, I could not find the effective config, can you give some hints about it? thank you!

  • #123 Miguel Grinberg said 2019-02-27T10:48:51Z

    @Roger: I have never done this myself, but looking at the docs, it appears all you need to do is create a mapping from the fields in your index to this analyzer plugin. Since my usage in this chapter is fairly simple, I decided not to explicitly create an index. But in your case, you will need to create the index and configure it with the proper mapping. The Python client has an index creation function: https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create.

  • #124 Samuel said 2019-04-10T13:09:25Z

    Hi, Miguel ! Thanks for this whole body of information ! I have learned alot. I have one issue and maybe you can throw some light. My SQLAlchemy models have many-to-many relationships(ex. multiple Plants can have multiple elements). I have tried to add the relationship attribute for the elements to the searchable variable, but I got the error SerializationError, because they are coming from a Select Multiple list and every element was passed as an individual object. I have tried to convert to JSON, but that doesn't work, so I deleted those relationships attr from the searchable, but in the console ofc they don't get indexed so I cannot search for a plant which has an element as Nitrogen. How can this be solved ? I was thinking to modify something in the SearchableMixin probably? Thank you !

  • #125 Miguel Grinberg said 2019-04-10T17:15:25Z

    @Samuel: the elasticsearch index is used to search text, not to search relationships. Are you sure what you need can't be done strictly within SQLAlchemy and your relational database?

Leave a Comment