2017-06-12T15:46:32Z

Migrating from Flask-Script to the New Flask CLI

Flask CLI

In release 0.11, Flask introduced new command-line functionality based on Click, which includes the flask command. Before then, Flask did not provide any support for building command-line interfaces (CLIs), but Flask-Script provided similar functionality as a third party extension.

It's been more than a year since the Flask CLI has been released, and I still see a lot of projects out there based on Flask-Script. My guess is that there aren't really any important reasons that motivate people to migrate, since Flask-Script worked well, or at least well enough. But the reality is that Flask-Script hasn't had an official release since 2014 and appears to be unmaintained. In this article I want to show you how I migrated the Flasky application from my Flask book from Flask-Script to Click (one of the changes that are coming in the second edition of the book!), so that you can learn what the differences are, and decide if it is time to migrate your applications.

First, A Confession

You may think that with this article I want to convince you that the Flask CLI is the greatest thing since sliced bread, but if you think that, you are actually wrong. Back in 2014, when Armin Ronacher introduced the Click integration in Flask, I was (and still am) opposed to the idea of adding yet another dependency to the Flask core project, and suggested that Click-based CLIs be introduced as an extension that can become a fair competitor to Flask-Script. This is all water under the bridge now, but you can see an exchange I had with Armin on this topic here: https://github.com/smurfix/flask-script/issues/97.

So really, I believe forcing a CLI based on Click was a mistake, and against the "pick the best tool for each task" spirit Flask is known for. Unfortunately, making the Click-based CLIs part of the Flask core crushed any steam the Flask-Script maintainers had left, and at least from looking at their GitHub repository, my impression is that the project is now dying a slow death.

What's Wrong With Flask-Script?

Flask-Script has, in my opinion, one important flaw. But interestingly enough, it is a problem that is directly caused by an old design issue in the Flask reloader, also present if you start your application with app.run(), without using Flask-Script.

If you start your application in debug mode, there will be two Flask processes running. The first process is watching the source files for changes, and the second is the actual Flask server. When any source files change, the watcher process kills the server process, and then starts another one, which will now use the updated source files.

The problem occurs when you inadvertently introduce a syntax error in one of your source files. The watcher process will not know any better, so it will kill the older server, and launch a new one. But the new server is not going to start, since Python will raise an error while parsing the modified file. When the server process exits in error, the watcher process gives up and exits too, forcing you to restart the whole thing once you fix the error in your code.

This works much better when you start the reloader with the new flask run command. The application is not imported until the first request is sent, and at that point, if an error occurs while importing the source files, it is handled exactly as if it was an error that occurred during runtime. This means that if you have the web-based debugger enabled, you will get the error reported there. The new watcher process is also smarter, and does not exit if the server process dies. Instead, it will continue to monitor source files, and try to start the server again after more changes are made.

While this was an annoyance, I got used to it and was always prepared to restart the application when it exited due to an error on my part.

The New Flask CLI

Is the Flask CLI better than Flask-Script? Let's start with a review of the default commands that all Flask applications get and see how they fare.

When you install Flask 0.11 or newer, you get a flask command installed on your virtual environment:

(venv) $ flask
Usage: flask [OPTIONS] COMMAND [ARGS]...

  This shell command acts as general utility script for Flask applications.

  It loads the application configured (through the FLASK_APP environment
  variable) and then provides commands either provided by the application or
  Flask itself.

  The most useful commands are the "run" and "shell" command.

  Example usage:

    $ export FLASK_APP=hello.py
    $ export FLASK_DEBUG=1
    $ flask run

Options:
  --version  Show the flask version
  --help     Show this message and exit.

Commands:
  run    Runs a development server.
  shell  Runs a shell in the app context.

When using Flask-Script, you had to create a driver script, typically called manage.py. Looking at the above, you can see that ./manage.py runserver maps to flask run, and ./manage.py shell maps to flask shell. Not much of a difference, right?

There is actually one gotcha. The manage.py and flask commands differ in the way they discover the Flask application instance. For Flask-Script, the application is provided to the Manager class as an argument, either directly or in the form of an application factory function. The new Flask CLI however, expects the application instance to be provided in the FLASK_APP environment variable, which you typically set to the filename of the module that defines it. Flask will look for an app or application object in that module and use that as the application.

Unfortunately, there is no direct support for application factory functions in the new Flask CLI. The approach that you need to follow to use a factory function is to define a module that calls the factory function to create the app object, and then reference that module in FLASK_APP. This is similar in concept to the wsgi.py module in Django applications.

If you wanted to support the new Flask CLI for Flasky, you could write a flasky.py file as follows:

import os
from app import create_app

app = create_app(os.getenv('FLASK_CONFIG') or 'default')

Note that I copied these lines from manage.py, which we are not going to use once we are fully migrated to the new CLI.

The "flask run" Command

After adding the flasky.py module, you can start the Flask development server with the following command:

$ export FLASK_APP=flasky.py
$ flask run

If you are using Microsoft Windows, the FLASK_APP environment variable is set in a slightly different way:

> set FLASK_APP=flasky.py
> flask run

The flask run command has options to enable or disable the reloader and the debugger, and also to set the IP address and port where the server will be listening for client requests. The options are not identical to those from Flask-Script, but they are close enough. You can use flask run --help to see all the available options.

As I'm sure you are aware, most Flask applications have this at the bottom of the main script:

if __name__ == '__main__':
    app.run()

This is what actually starts the server when you don't use any CLI support. If you leave this in your script and start using the new CLI it is not going to cause any problem, but it isn't really needed, so you can go ahead and remove it.

The "flask shell" Command

The shell command is basically the same, but there is a small difference in how you define additional symbols that you want auto-imported into the shell context. This is a feature that can save you a lot of time when working on an application. Normally you add your model classes, database instance and other objects you are likely to interact with in a testing or debugging session in the shell.

For Flask-Script, the Flasky application had the following shell context definition:

def make_shell_context():
    return dict(app=app, db=db, User=User, Follow=Follow, Role=Role,
                Permission=Permission, Post=Post, Comment=Comment)
manager.add_command("shell", Shell(make_context=make_shell_context))

As you see above, the make_shell_context() function is referenced in the add_command call that defines the shell option. Flask-Script calls this function right before starting a shell session, and includes all the symbols returned in the dictionary in it.

The Flask CLI offers the same functionality, but uses a decorator to identify the function that provides shell context items (there can be multiple functions, actually). To have the equivalent functionality to the above, I've expanded the flasky.py module:

import os
from app import create_app, db
from app.models import User, Follow, Role, Permission, Post, Comment

app = create_app(os.getenv('FLASK_CONFIG') or 'default')

@app.shell_context_processor
def make_shell_context():
    return dict(app=app, db=db, User=User, Follow=Follow, Role=Role,
                Permission=Permission, Post=Post, Comment=Comment)

So as you see, the function is identical, but needs the @app.shell_context_processor decorator so that Flask knows about it.

Commands From Flask Extensions

A big part of the success of Flask-Script was that it allowed other Flask extensions to add their own commands. I have actually taken advantage of this functionality in my Flask-Migrate extension.

So how do you migrate those commands to the Click CLI? Unfortunately you depend on the extension author to do the migration for you. If you use any extensions that work with Flask-Script that haven't been updated to also work with the Flask CLI, you are pretty much out of luck, you will need to continue using Flask-Script, at least when you interact with the extensions in question. If you use Flask-Migrate, you do not have to worry, as I have updated it to support both the Click and Flask-Script CLIs. Just make sure you are using a recent version.

Specifically for Flask-Migrate, ./manage.py db ... directly translates to flask db .... In the Flask-Script version, the Flask-Migrate extension was initialized in manage.py. For the Flask CLI version, we can move that to the new flasky.py module, which I'm sure you realized by now that it is becoming a replacement of Flask-Script's manage.py, with the only difference that it is not a script you execute directly. Here is that module with the Flask-Migrate integration added:

import os
from app import create_app, db
from app.models import User, Follow, Role, Permission, Post, Comment
from flask_migrate import Migrate

app = create_app(os.getenv('FLASK_CONFIG') or 'default')
migrate = Migrate(app, db)

@app.shell_context_processor
def make_shell_context():
    return dict(app=app, db=db, User=User, Follow=Follow, Role=Role,
                Permission=Permission, Post=Post, Comment=Comment)

As an example, you can upgrade the Flasky database to the latest revision with the following command, but remember that you need to have FLASK_APP set for it to work:

$ flask db upgrade

Application Commands

Another nice feature in Flask-Script was that it allowed you to write your custom tasks as functions, which then were automatically converted to commands in the CLI just by adding a decorator. In Flasky, I've used this feature to add a few commands. For example, here is how I implemented the ./manage.py test command for Flask-Script:

@manager.command
def test(coverage=False):
  """Run the unit tests."""
  # ...

The @manager.command decorator was all that was needed to expose the function as a test command, and it even detected the coverage argument and added it as a --coverage option.

Decorator-based commands are actually Click's bread and butter, so this is something that can be migrated without a problem. The equivalent function for the Flask CLI can go in flasky.py, and would be as follows:

import click

# ...

@app.cli.command()
@click.option('--coverage/--no-coverage', default=False, help='Enable code coverage')
def test(coverage):
    """Run the unit tests."""
    # ...

The @app.cli.command() decorator provides an interface to Click. As in Flask-Script, commands are executed with an application context installed, but with the Flask CLI the app context can be disabled if you don't need it. The actual code of the function does not need to change, but note how the --coverage option needs to be given explicitly using the @click.option decorator, unlike the Flask-Script case, in which the option is automatically derived from the function argument list.

The same approach can be taken with the other two custom functions that I have in my Flasky application. The complete implementation of the flasky.py module is below:

import os
from app import create_app, db
from app.models import User, Follow, Role, Permission, Post, Comment
from flask_migrate import Migrate
import click

app = create_app(os.getenv('FLASK_CONFIG') or 'default')
migrate = Migrate(app, db)

@app.shell_context_processor
def make_shell_context():
    return dict(app=app, db=db, User=User, Follow=Follow, Role=Role,
                Permission=Permission, Post=Post, Comment=Comment)

@app.cli.command()
@click.option('--coverage/--no-coverage', default=False, help='aaa')
def test(coverage=False):
    "Test coverage"
    # ...

@app.cli.command()
@click.option('--length', default=25, help='Profile stack length')
@click.option('--profile-dir', default=None, help='Profile directory')
def profile(length, profile_dir):
    """Start the application under the code profiler."""
    # ...

@app.cli.command()
def deploy():
    """Run deployment tasks."""
    # ...

Conclusion

In this article I showed you the functionality offered by the new Flask CLI that will be useful if you want to migrate your Flask-Script based applications. There are a couple more things that you can do that I haven't included here, however. The documentation describes how you can create your own driver scripts, similar to the manage.py of Flask-Script, in case you don't want to use the flask command (this is surprisingly harder than it sounds!). It also tells you how to register commands through "entry points" defined in your project's setup.py file, if you use one for your project. Be sure to check the CLI documenation to learn about those if you are interested.

So what do you think? Will you be migrating your Flask-Script applications soon? Let me know below in the comments!

41 comments

  • #26 Anastasios Selalmazidis said 2018-03-06T23:38:50Z

    Hi Miguel,

    nice article on how to migrate from Flask-Script to Flask-Cli. I wanted to mention that you can combine the manager.py file with the new Cli module like they do at realpython, https://github.com/realpython/flask-skeleton/blob/master/manage.py

    This way you don't need to set an environment variable and you can keep your manager.py file.

    What do you think ?

  • #27 Miguel Grinberg said 2018-03-07T03:30:41Z

    @Anastasios: That is an interesting choice to structure your application, they are making the CLI be the main object, which then creates the application through the factory function. I'll have to think about that, not sure I like it at first sight, because maybe you don't care about a custom CLI, and this is forcing you to create one.

  • #28 Wesam said 2018-07-30T17:00:30Z

    Hi Miguel, Thanks for the great explanations.. I am new to flask and I have a (possibly very naive) question.

    I used to run my app with: if name == 'main': app.run()

    this worked also with flask-script.

    But when I try using Flask CLI, I get that the import paths are a bit different. Assuming all my project sits in folder 'TopLevel'. Then with Flask CLI I need to add 'TopLevel.' before all absolute imports.

    Am I doing something wrong here? Is there a way I can avoid adding this to all my imports?

    Thanks in advance, Wesam

  • #29 Miguel Grinberg said 2018-07-30T21:06:02Z

    @Wesam: I don't understand what you mean. The imports should be the same regardless of how you start your application. The top level directory should not be part of the import path, because imports start from the current directory.

  • #30 Flurin Conradin said 2018-09-25T07:23:04Z

    Hello Miguel

    Thanks for this post.

    When I run "flask run", I get the error: A valid Flask application was not obtained from "package_name.wsgi:app"

    Would you happen to know why?

    I set FLASK_APP to "package_name.wsgi:app".

    My application runs as expected when I start it using a gunicorn server. Happy to provide details.

  • #31 Miguel Grinberg said 2018-09-25T18:19:59Z

    @Flurin: impossible for me to know without seeing the app. Flask can sometimes be a bit tricky, try using just the module name, like FLASK_APP=package_name.wsgi.py.

  • #32 ogah said 2019-03-06T16:35:53Z

    I was trying to implement role model into the microblog tutorial. Everything works well when I initiate the roles function from the flask shell command. ie>>> Role.insert_roles(). But when I add this code import click from app import create_app,db,cli from app.models import User, Post,Permission, Role,Comment app=create_app() cli.register (app) @app.shell_context_processor def make_shell_context(): return {'db':db,'User':User,'Post':Post,'Role':Role,'Permission':Permission,'Comment':Comment}

    @app.cIi.command() def deploy(): Role.insert_roles()

    to the microblog.py so that the role will be initiated when I run $flask run. The role table return no result. Pls guide me on this.

  • #33 Miguel Grinberg said 2019-03-07T10:01:54Z

    @ogah: are you expecting that the command will run when you issue "flask run"? That's not how custom commands work, you created a command called "deploy", so you run it with "flask deploy".

  • #34 Yang said 2019-03-31T06:47:57Z

    Hello, I had a syntax error when changing the sqlite database to a mysql database but I couldn't find out why.

    SQLALCHEMY_DATABASE_URI = mysql+pymysql://root:123456@localhost/Blog SyntaxError: invalid syntax

  • #35 Miguel Grinberg said 2019-03-31T09:33:02Z

    @Yang: you need to enclose your URL with quotes, since it is a string.

  • #36 Nithin said 2019-07-30T14:11:47Z

    @Miguel. Is it possible to use "flask run" to launch a dev server that is capable of supporting websockets?

    Like in the example https://github.com/heroku-python/flask-sockets (by Kenneth Reitz), uses the pywsgi from gevent.

  • #37 Miguel Grinberg said 2019-07-31T09:30:40Z

    @Nithin: the "flask run" command launches the development web server from Flask, which does not support WebSocket. It would be possible to add WebSocket support to this web server I guess, but nobody has done it to my knowledge.

  • #38 Charles said 2020-01-09T16:42:13Z

    Hi, Miguel.

    I want to reindex every three days using elasticsearch. Could you give me any suggestions how to do that? I deploy my flask application in Ubuntu. Do I need to run reindex() in Flask-Script? Or is there any methods to reindex automatically every three days?

    Thanks.

  • #39 Miguel Grinberg said 2020-01-09T23:21:23Z

    @Charles: what is the purpose of reindexing every 3 days? You'd normally reindex when the normal database and the elasticsearch database are out of sync. But in any case, if you are set on doing this, the easiest way is to add a cron job on your host that run the reindex command.

  • #40 Charles said 2020-01-10T04:22:26Z

    @Miguel. I get new data using some spiders every 3 days(almost 30000 columns) and want to use elasticsearch to index the new latest data which allows users to search them. I put the old data and new data in the same table, because I wlant to show some line charts of historical data. Do you have other ideas to achieve that? Or do I need to use elasticsearch or not?

  • #41 Miguel Grinberg said 2020-01-10T08:06:21Z

    @Charles: a reindex will generate the ES index from scratch, covering both the old and the new data. If you have a mechanism to update the ES index every time a new item is added to the SQLAlchemy database (such as the one I present in my tutorial) then your index will be updated, there is no need to use the reindex function.

    Now whether you need ES or not, that's impossible for me to answer. It depends on what kind of search functionality you want to offer. There are literally dozens of options for database searching, ES is just one of them.

Leave a Comment