2013-04-15T06:14:46Z

The Flask Mega-Tutorial, Part XVII: Deployment on Linux (even on the Raspberry Pi!)

(Great news! There is a new version of this tutorial!)

This is the seventeenth article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Today we are reaching a huge milestone in the life of our microblog application. We are going public with it, as we will be looking at different ways in which we can deploy it and make it accessible to our users.

In this article I will explore the traditional hosting options, and will be looking at hands-on deployment on a Linux box and on the widely popular Raspberry Pi minicomputer. In the next article we will be looking at deployment on cloud services.

To begin we will be looking at the more traditional of all deployment options. We will install the application behind a web server on a dedicated host that we will manage.

But where can we find a host? These days there are many cheap server offers out there. The cheapest options are for a VPS (virtual private server), which is virtual machine that looks like a standalone and dedicated server to you, but is really sharing the physical hardware with a few others like it. You can check on lowendbox.com for deals if you want to get one to experiment with (note that I'm not affiliated with them, I do not endorse them or make money if you click the link above).

Another easy way to obtain a host is to install a virtual machine on your regular PC. If you are a Windows or Mac user and would like to experience deploying the application to a Linux server without spending a dime, then this is the way to go. You can install VirtualBox on your system, download an ISO image for the Linux distribution of your choice and then create a VM on which you install this ISO image.

Let's talk a bit about operating system choices. I think there are four decent choices for hosting a web server:

  • Linux
  • BSD
  • OS X
  • Windows

From a technical point of view you could setup a Python based web server on any of the above, so how do we choose?

In my opinion the choice isn't too hard to make. For me it is important that we are on a platform that is open source, because the community at large does a much better job at keeping a platform safe and secure than what a single company can do on a closed source product. We also want to be on a platform that has a large server installation base, because that gives us more chances of finding solutions to our problems. Based on the above criteria, I consider Windows and OS X inferior to Linux and BSD as servers.

So now we are down to two options. The choice of Linux vs. BSD is an easy one for me: It does not matter. Both are excellent options for hosting a web server. Depending on the hosting provider you may have Linux or BSD options to choose from, so I just simply use what's available.

There are many Linux and BSD distributions out there, so how to pick one? This is largely a matter of preference, but if we are going to try to use what the majority use we can look at the distributions according to some of their major characteristics, and then pick a popular distribution in the group of choice:

  • RPM based (RedHat, CentOS, Fedora)
  • Debian based (Debian, Ubuntu, Mint)
  • BSD based (FreeBSD, NetBSD, OpenBSD)
  • others

I'm sure my simplistic grouping of distros will offend some. I'm also sure many will have the need to mention that Mac OS X is BSD based and as such should be in my list. Keep in mind that this classification is just my own, I don't expect everyone will agree with me.

As an exercise in preparing this article I tested a few Linux distros (I'm using a VPS that does not offer BSD choices). I tried Fedora, Ubuntu, Debian and CentOS. None of them was straightforward, unfortunately, but out of all of these, the CentOS install was the one with less complications, so that's the one I'm going to describe here.

If you have never done this you may find the setup task to be extremely laborious, and you would be correct if you think that. The thing is, once the server is setup it takes very little effort to keep it running.

Ready? Here we go!

Hosting on CentOS 6

I'm going to assume that we have a clean install of CentOS 6 on a dedicated server or VPS. A VM in your PC also works, but go ahead and do a standard install before starting.

Client side setup

We will be managing this server remotely from our own PC, so we need a tool that we can use to login into this system and run commands. If your PC runs Linux or OS X then you already have OpenSSH installed. If you are on Windows then there are two very decent ways to get SSH functionality:

  • Cygwin (select the OpenSSH package in the installer)
  • Putty

Cygwin is much more than SSH, it provides a Linux like environment inside your Windows box. Putty, on the other side, is just an SSH tool. Since I want to cover the most ground, the instructions below will be for Linux, OS X and Cygwin users. If you use Putty on Windows, then there is a little bit of translation required.

Let's begin by logging in to our shiny new CentOS 6 box. Open a command line prompt on your PC (a Cygwin prompt if you are on Windows) and run the following:

$ ssh root@<your-server>

This is saying that you want to login to the system as user root, the admin of the system. You should replace <your-server> with the IP address or hostname of your server. Note the $ is the command line prompt, you do not type that.

You will be asked to provide the password for user root. If this is a dedicated server or VPS that you purchased, you must have been asked about this password during the setup process. If this is your own VM then you chose a root password during installation.

Installing software packages

Now that we are logged in, we need to install all the software that we will be using to deploy our server, which obviously include Python and a web server. We will also switch from sqlite to a more robust MySQL database.

The command to install software on CentOS is called yum. We can install all the packages we need using a single command:

$ yum install python python-devel python-virtualenv httpd httpd-devel mysql-server mysql-devel git gcc sudo

Some of these could be already installed, but it doesn't hurt to be safe and request all the packages we need, yum will only install what's missing.

The packages we just installed are:

  • python and python-devel: the Python interpreter and its development package
  • virtualenv: the Python script that creates virtual environments
  • httpd and httpd-devel: The Apache web server and its development package
  • mysql-server and mysql-devel: The MySQL database server and its development package
  • git: source code version control system (we will use it to download and update the application)
  • gcc: the C/C++ compiler (needed to compile Python extensions)
  • sudo: a tool that helps users run commands as other users.

Creating a user account to host the application

We will now setup a user account on which we will host our application. In case the reasons aren't clear, the root account has the maximum privileges, so we could easily delete or destroy important parts of the system by mistake. If we work inside a restricted account then all the important parts of the system will be out of reach so we are protected against mistakes. A second, maybe more important reason to not use the root account is that if a hacker gets hold of this account he will own our server. We want to hide the root account as much as possible, I like to disable root account logins completely on production servers.

We will call this new account apps, assuming we could host several applications inside it. Since the applications will be run by the web server account, we will make the group of our apps account the group of the web server account (which was created for us when Apache was installed), so that the web server has permission to read and write on these files. If you are not familiar with Unix user permissions, then an introduction is available over at this Wikipedia page.

The command to create the apps account on CentOS is:

$ adduser -g apache apps

The -g option makes the primary group of this account the apache group, which on CentOS is the group of the apache user account. As part of the account setup you will have to enter a password for this account.

The adduser command will create a home directory for the new account, at /home/apps. Unfortunately, when we go check the permissions of this directory we find it to only be accessible to the owner:

$ ls /home -l
total 7
drwx------ 2 apps  apache 4096 Apr  7 11:46 apps

We decided we want the apache group users to be able to freely work on the files on this account, so we have to give the group enough permissions:

$ chmod 775 /home/apps

Consult the documentation for the chmod tool for detailed information. The 775 permisions will allow the owner and the group full access, and will also give others permission to read and execute, but not write to this directory.

Password-less logins

The next step is to be able to login to this account from our PC without using a password. The SSH tool supports another type of authentication, called public key. If you are on a non-Windows OS it is very likely that you already have keys installed on your system.

Now open a terminal window on your own PC (Cygwin's bash shell for Windows users). To check if you have keys installed run the following command:

$ ls ~/.ssh
id_rsa  id_rsa.pub

If the directory listing shows files named id_rsa and id_rsa.pub like above then you are good to go. If you don't have these files then you have to run the following command:

$ keygen -t rsa

This application will prompt you to enter a few things, for which I recommend you accept the defaults by pressing Enter on all the prompts. If you know what you are doing and want to do otherwise, you are certainly welcome to.

After this command runs you should have the two files listed above. The file id_rsa.pub is your public key, the data in this file can be shared with others without creating a security risk. The id_rsa file is your private key, so you should never share the contents of this file with anyone.

We will now add our public key to our server, so that it knows who we are. Still on our local PC we start by printing the public key file:

$ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwaK4JVuY6PZAr8HocCIOszrLIzzCjO0Xlt9zkFNKvVpP1B92u3JvwwiagqR+k0kHih2SmYnycmXjAcE60tvu+sIDA/7tEJZh4kO4nUYM5PJ17E+qTqUleBXQM74eITydq/USkOqc5p++qUUgA60gUUuNum3igbZiNi71zK4m8g/IDywWYk+5vzNt2i7Sm8NEuauy/xWgnWhCBXZ/tXfkgWgC/4HzpmsfO+nniNh8VgTZp8Q+y+4psSE+p14qUg7KdDbf0Wo/D35wDkMvto96bIT8RF0np9dTkFj8TgNW8inP+6MC+4vCd8F/NpESCVt8hRlBVERMF8Xv4f/0+7WT miguel@miguelspc

Now we need to copy this data to the clipboard and then switch to the CentOS system, where we will issue these commands:

$ mkdir /home/apps/.ssh
$ echo <paste-your-key-here> > /home/apps/.ssh/authorized_keys

For my server and public key the following commands are necessary:

$ mkdir /home/apps/.ssh
$ echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwaK4JVuY6PZAr8HocCIOszrLIzzCjO0Xlt9zkFNKvVpP1B92u3JvwwiagqR+k0kHih2SmYnycmXjAcE60tvu+sIDA/7tEJZh4kO4nUYM5PJ17E+qTqUleBXQM74eITydq/USkOqc5p++qUUgA60gUUuNum3igbZiNi71zK4m8g/IDywWYk+5vzNt2i7Sm8NEuauy/xWgnWhCBXZ/tXfkgWgC/4HzpmsfO+nniNh8VgTZp8Q+y+4psSE+p14qUg7KdDbf0Wo/D35wDkMvto96bIT8RF0np9dTkFj8TgNW8inP+6MC+4vCd8F/NpESCVt8hRlBVERMF8Xv4f/0+7WT miguel@miguelspc > /home/apps/.ssh/authorized_keys

These commands write your public key to the file authorized_keys in the server. This makes your public key known to OpenSSH on the server. The remaining task is to secure the .ssh directory and the authorized_keys file inside it:

$ chown -R apps:apache /home/apps/.ssh
$ chmod 700 /home/apps/.ssh
$ chmod 600 /home/apps/.ssh/authorized_keys

These commands change the ownership of these files to the apps account and then make the directory and file only accessible by the new owner.

The password-less login should now be working. We will now logout of the root account of our server and then login again, but this time we will login as the apps user:

$ ssh apps@<your-server>

If everything goes well you should not need to enter a password to obtain access.

Installing the application

We will now use git to download and install microblog into the server. If you are not familiar with git I recommend that you read git for beginners.

The application must be available in a git server that can be reached from our web server. I will be using my own github hosted application. You are welcome to use it as well, or if you prefer you can also clone it and make one that is yours.

To install microblog in our server we just issue a git clone command:

$ git clone https://github.com/miguelgrinberg/microblog.git
$ cd microblog

The latest version of the application has a few changes from how we left it in the previous article.

First, there are a few new files at the top level, runp-sqlite.fcgi, runp-mysql.fcgi, killpython and requirements.txt. The *.fcgi scripts are starter scripts that are invoked by servers that use the FastCGI protocol. The killpython script will help us restart the application after we upgrade it. The requirements.txt file lists all the dependencies that we need to install to run our application. We will look at all these later.

Another interesting change is in our config.py file. Up to now we initialized our database as follows:

SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db')

Now we have this instead:

if os.environ.get('DATABASE_URL') is None:
    SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db')
else:
    SQLALCHEMY_DATABASE_URI = os.environ['DATABASE_URL']

This is a simple change that enables us to override the database the application will use by setting an environment variable. You will see how we take advantage of this change in the next section.

Setting up MySQL

The sqlite database that we've been using all this time is great for simple applications, but when we get into a full blown web server that can serve multiple requests at a time it is a better idea to use a more robust database. So we will setup a MySQL database for microblog.

We have installed MySQL already, so all that is left is to create a database and a user that has permissions on it. To manage our database server we use the mysql tool:

$ mysql -u root -p
Enter password: (enter the mysql root password, or empty if one is not defined)
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.1.67 Source distribution

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

And now that we are in the mysql prompt we can go ahead and create a database and a user, both named apps:

mysql> create database apps character set utf8 collate utf8_bin;
mysql> create user 'apps'@'localhost' identified by 'apps';
mysql> grant all privileges on apps.* to 'apps'@'localhost';
mysql> flush privileges;
mysql> quit;

Note that you will need to pick a password for the apps MySQL user in the identified by section. For simplicity I have picked apps as password, but on a real installation a hard to guess password should be used. Do not confuse the database apps user with the system's apps user.

Initializing the application

Now that we have a database ready we can initialize microblog.

We start by creating and populating the Python virtualenv:

$ virtualenv flask
$ flask/bin/pip install -r requirements.txt
$ flask/bin/pip install mysql-python

Note that in addition to installing all the dependencies listed in the requirements.txt file we add the MySQL support, which is required by SQLAlchemy when connecting to that database server.

Then we create the database:

$ DATABASE_URL=mysql://apps:apps@localhost/apps ./db_create.py

Note how we set an environment variable to point to our new MySQL database instead of the sqlite default.

Next we compile all the messages in our translation database:

$ ./tr_compile.py

And to finalize, we enable group write access to the two folders that need to be written to by the web server:

$ chmod -R g+w search.db tmp

The search.db folder is used by our Whoosh full text search database. The tmp directory should be left open so that new files can be created there when needed.

Setting up Apache

The last thing to do is to configure our Apache web server.

We wil be using the mod_fcgid module to handle the FastCGI communication with our application. Many Linux distributions offer a package for mod_fcgid, but unfortunately CentOS is not one of them, so we will build this module from its source code.

Here is the list of commands that build and install the current release of mod_fcgid:

$ wget http://mirror.metrocast.net/apache/httpd/mod_fcgid/mod_fcgid-2.3.9.tar.gz
$ tar xvzf mod_fcgid-2.3.9.tar.gz
$ cd mod_fcgid-2.3.9
$ APXS=/usr/sbin/apxs ./configure.apxs
$ su
(enter root password)
$ make install

You can consult the mod_fcgid documentation if you want details on the above commands.

With this module installed we can now configure our server. For this we need to edit the Apache configuration file. Linux distributions do not agree on a standard location and name for this file, so you will need to figure out where this file is. On CentOS, the config file is located at /etc/httpd/conf/httpd.conf.

The changes that we need to make are simply to define our server. At the bottom of the configuration file we add the following (from the root account, since this file is not writable to regular users):

FcgidIPCDir /tmp
AddHandler fcgid-script .fcgi
<VirtualHost *:80>
    DocumentRoot /home/apps/microblog/app/static
    Alias /static /home/apps/microblog/app/static
    ScriptAlias / /home/apps/microblog/runp-mysql.fcgi/
</VirtualHost>

The FcgidIPCDir sets the directory where the file sockets will be created. I've found that on CentOS this was going by default to a directory where the apache user did not have write permissions, so I'm putting all these files on /tmp instead.

Next we use AddHandler to tell apache that any files that have a .fcgi extension should be treated as FastCGI files and routed through the mod_fcgid module that we just installed. Remember the new runp-mysql.fcgi file that we have in our root folder? This file uses the flipflop Python module as an adapter so that our application can speak the FastCGI protocol. Let's have a quick look at this file:

#!flask/bin/python
import os
os.environ['DATABASE_URL'] = 'mysql://apps:apps@localhost/apps'

from flipflop import WSGIServer
from app import app

if __name__ == '__main__':
    WSGIServer(app).run()

Apache will execute this script to start our application. Note how we insert the MySQL database name in the environment, so that Apache can see it.

The <VirtualHost> section defines the host that will run our web server. The *:80 denomination indicates that any requests that arrive at the server for any hostnames on port 80 will be handled by this virtual server. The configuration can have multiple <VirtualHost> sections, each defining a different server. To differentiate the servers you can use different IP addresses, domain or sub-domain names or ports.

The definition of our virtual host is pretty simple. The DocumentRoot statement tells Apache where to look for static files. Any requests for files will be served our of this folder, so for example, when browsers ask for a /favicon.ico file to show the little icon next to the URL Apache will try to find it here. Unfortunately all the static files that we use in our application have a /static prefix, so to prevent Apache from looking for another static folder we use the Alias statement, which just says that anything that starts with /static should go directly to our folder. Finally, the ScriptAlias statement tells Apache that when a request that starts with / arrives (basically all requests that are not for static files) then use the script indicated as second argument to handle it, and that is our .fcgi script.

The Apache config file supports many more options that I will not mention here. I recommend that you review the Apache documentation to decide what options make sense for your server.

To activate the changes we made we need to restart the apache server, again from the root account:

$ service httpd restart

And now from the browser on your PC you should be able to access microblog at the address http://<your-server>.

Installing application updates

The last remaining thing we are going to look at for this server is how to roll out an update of the application.

Let's assume we have deployed the application and it has been running happily for a while on our server. Now it's time to push an update, which could fix some bugs we found, or just add features.

We will use git and our own tools for this. If we login to the web server using the apps account we can upgrade to the latest release from the git repository by running:

$ cd microblog
$ git pull
$ DATABASE_URL=mysql://apps:apps@localhost/apps ./db_upgrade.py
$ ./tr_compile.py

The git pull operation downloads any new or updated files from our git server. We then upgrade our database and recompile our translation files, in case any of those need updating. That's it!

Once we have the updates in place we need to tell apache to restart the FastCGI processes, and this gets a bit tricky, because we are logged in as user apps, an unprivileged account, while the FastCGI processes are owned by the apache user.

For this we are going to use sudo. The sudo tool allows users to selectively run applications as other users. We don't really want to give our apps account a lot of power, we will limit it to just be able to send kill signals to the FastCGI processes started by the apache user.

The killpython script that we have in the microblog directory does this work:

killall /home/apps/microblog/flask/bin/python

The problem is, if we run this under the apps account we won't have permission to kill the processes owned by the apache user. To enable apps to kill these processes we have to make a small change in file /etc/sudoers:

apps    ALL=(apache) NOPASSWD:/home/apps/microblog/killpython
Defaults: apps    !requiretty

Note you will need to make this change as user root, since /etc/sudoers is a system file.

The first cryptic command gives the apps user permission to run the killpython command as user apache without having to provide a password. The second line allows sudo commands for user apps to be issued from a script in addition to doing it from an interactive console. You can read the man page for sudoers for detailed information on the syntax of sudo configuration file.

Now when logged in as apps we can kill the python jobs as follows:

$ sudo -u apache ./killpython

If the FastCGI processes are killed Apache will restart them the next time a request comes for microblog.

To make upgrading our server a bit more simple we can write a client side script that upgrades and restarts all together. The script would run the following:

ssh apps@<your-server> "cd microblog;git pull;sudo -u apache ./killpython;DATABASE_URL=mysql://apps:apps@localhost/apps ./db_upgrade.py;./tr_compile.py"

If you store the above line in a script then you can roll out an application upgrade with a single command!

What's left to do

I will not discuss a few more mundane operations that are recommended to make a server that is open to the hostile Internet world more secure. Things such as:

  • Not allowing root remote logins
  • Disabling any services that are not used, like FTP, CIFS, etc.
  • Setting up a firewall
  • Keep the server up to date on security fixes.
  • Etc.

I will leave these as an exercise to the interested reader, since they are largely unrelated to this tutorial.

Hosting on the Raspberry Pi

The Raspberry Pi is a revolutionary little Linux computer that costs $35. It has very low power consumption so it is the perfect device to host a home based web application that can be online 24/7 without tying up a full blown computer.

There are several Linux distributions that run on the Raspberry Pi. We'll set up our application on Raspbian, the official distribution.

As a side benefit, note that Raspbian is a derivative of Debian, so the instructions below will apply with little or no modification for Debian/Ubuntu based servers.

We will now follow the same steps we followed for CentOS to get the RPi server setup.

Client side setup

A Raspberry Pi is a pretty decent computer to work in. You can connect an HDMI monitor, keyboard and mouse and work directly on it to configure our web server. Also like we did for CentOS or actually any other Linux distribution you can have the Raspberry Pi just connected to the network and then access it remotely from a PC over ssh. See the client side setup section above for CentOS for instructions on how to setup a client PC with ssh.

Note that I will not cover how to install and setup the Raspberry Pi here, there are plenty of articles elsewhere for that. I will assume the Raspberry Pi is already running and connected to the network.

Installing software packages

Since the Pi is a limited power machine we will not do a full blown Apache/MySQL installation like we did for CentOS. Instead, we will take a lightweight approach. For web server we will use Lighttpd (pronounced "Lighty"), a small web server with a very good and efficient FastCGI implementation. For our database we will stay with sqlite.

With CentOS we used yum to install packages. In Debian derivatives the package manager is called apt-get:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install python python-dev python-virtualenv lighttpd git

The update line downloads an updated package list to your Pi. The upgrade line upgrades all installed packages to their latest versions. It is recommended that you issue these two commands regularly, to keep your Raspberry Pi's software up to date. Finally the install statement installs the packages that we need for our task.

Creating a user account to host the application

We said that we will assume our Raspberry Pi server will be in a trusted network. Since security is not a major concern we will not bother creating a dedicated account and instead will install the app directly on the default pi account.

If your Raspberry Pi will be connected to the internet then you should follow the approach we used for CentOS and make a different account that has no privileges, as this ensure that if an intruder manages to access the account the damage that he or she can cause will be limited.

Password-less logins

Having a public key authentication procedure is also less important if we assume the application will run on a small intranet, but should you need such functionality the procedure is identical to what we did for the CentOS system.

Installing the application

The application installs very easily, again using git:

$ git clone git://github.com/miguelgrinberg/microblog.git
$ cd microblog

Initializing the application

Since we will be using a sqlite database in this host we can go ahead and initialize microblog as we have done in past articles for our development server:

$ virtualenv flask
$ flask/bin/pip install -r requirements.txt
% flask/bin/pip install mysql-python
$ ./db_create.py
$ ./tr_compile.py

Note that the setup.py script will try to install the MySQL support and that will fail, but this is all right since we will not use it.

The files were all checked out under the pi user ownership. But as we have seen for CentOS, the web server on the Pi will run under a different account. Since security isn't a concern for this server, we will take a much simpler approach:

$ chmod -R 777 *

The above chmod statement makes all the files writable by all users. Again, this would be a really bad thing to do on an open server, but for a server where you have full control it saves you from the trouble of dealing with user permissions.

I want to make it clear that the Raspberry Pi is perfectly capable of supporting a configuration based on user groups and user permissions similar to what we created above for the CentOS box. We are choosing this simplified setup because we will be using this server in a controlled environment.

Setting up Lighttpd

Lighttpd comes with native support for FastCGI, so we do not need to worry about installing a separate module for it.

All we need to do is append the definition of our website to the configuration file, which is located at /etc/lighttpd/lighttpd.conf:

fastcgi.server = ("/microblog" =>
    ((
        "socket" => "/tmp/microblog-fcgi.sock",
        "bin-path" => "/home/pi/microblog/runp-sqlite.fcgi",
        "check-local" => "disable",
        "max-procs" => 1
    ))
)

alias.url = (
    "/microblog/static/" => "/home/pi/microblog/app/static/",
)

And with this our application should be online at http://aaa.bbb.ccc.ddd/microblog, with aaa.bbb.ccc.ddd being the IP address of your Raspberry Pi.

But there are a few things to note in the above configuration, and also a couple of things that didn't initially work and required some changes in the application side.

The fastcgi.server statement is the one that defines the behavior of our FastCGI server, which we are exposing under the /microblog URL. The reason we are not exposing the application at the root URL is simple, we may want to host more than one application, so putting all the URLs under a /microblog root effectively works as a namespace.

Inside the FastCGI definition we provide a socket name in the /tmp directory and the path to our runp-sqlite.fcgi file, which Lighttpd will execute to start up FastCGI processes. The check-local option tells Lighttpd to send requests to the FastCGI server even if the request path matches a file on disk. The max-procs limits the number of FastCGI processes to 1, which for a small server is enough and avoids potential problems with multiple concurrent writers to the sqlite database.

The alias.url section provides the mapping that enables Lighttpd to server our static files. The alias maps any requests for /microblog/static/... to the correct location on disk where we store our static files.

The runp-sqlite.fcgi script is pretty much the same as the one we used for CentOS, but without overriding the database setting:

from flipflop import WSGIServer
from app import app

if __name__ == '__main__':
    WSGIServer(app).run()

Some necessary fixes

Testing this setup reveals a couple of bugs in our application.

First, our javascript and css static files are not being served. The problem becomes evident after inspecting the HTML source of the login page. The base.html template references these files via hardcoded paths that begin with /static, but we said are putting this application inside a /microblog prefix. For example, see how we had the link to the CSS file:

<link href="/static/css/bootstrap.min.css" rel="stylesheet" media="screen">

Here we really want /microblog/static/bootstrap.min.css, but of course we can't add /microblog to the base.html file because that will break the development server that runs on our PC. The solution is to let Flask generate these URLs using our old friend url_for:

<link href="{{ url_for('.static', filename='css/bootstrap.min.css') }}" rel="stylesheet" media="screen">

After updating all the js and css files as above the site serves the login page with all the auxiliary static files.

But trying to login reveals another problem, Right after logging in we get a 500 error code. Luckily for us we have logging, so a quick look at /home/pi/microblog/tmp/microblog.log shows the following error:

ProgrammingError: (ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id -1260325776 and this is thread id -1243548560

What is this? We are not running multiple threads ourselves but obviously there are multiple threads in our application. The only difference between our development server and this one is that we are now using flipflop for the FastCGI server. Looking at the code in this module we can easily find that by default flipflop runs a multithreaded web server.

A possible solution would be to use the single-threaded FastCGI server from flipflop but that could affect the performance if the server will have multiple concurrent users. Another, more interesting way to handle this problem is to enable threading in sqlite (which appears to be fully supported). Multithreading can be enabled in SQLAlchemy by setting the check_same_thread option to False in our config.py configuration file:

SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db') + '?check_same_thread=False'

With this change we are allowing multiple threads to make changes to the database, the sqlite library takes care of synchronizing multiple accesses.

And with these small changes we have a fully enabled web application running on our Raspberry Pi!

Installing application updates

To deploy application updates we can just login to the Raspberry Pi and issue the commands that update the source files, database and translation files:

$ cd microblog
$ git pull
$ ./db_upgrade.py
$ ./tr_compile.py

If you setup password-less login to your Pi then of course you can write a script wrapper that does all this in a single command.

Final words

The updated application is available, as always, on my github page. Alternatively you can download it as a zip file below:

Download microblog 0.17.

In the next article we will be looking at deployment on cloud services as an alternative to the traditional options we explored today.

I will see you then. Thank you!

Miguel

101 comments

  • #76 Miguel Grinberg said 2015-07-16T06:26:39Z

    @Wasch: well, the reply looks like a redirect, it's not an error. The content of the page is not going to be right because the WSGI environment is missing, but the application appears to run. So really this doesn't say much, just that your fcgi file is able to execute, so the problem must be elsewhere.

  • #77 Wasch said 2015-07-16T08:09:46Z

    Thanks @Miguel, I tried setting up your microblog and got a 302 FOUND error message with a working redirect to the login page, but the page would not serve when i pointed my browser to 192.168.XXX.XXX/microblog. Static pages were also a no go. I don't understand how the WSGI environment is missing (am running this on the raspberry pi), i'll check through online resources, but any pointers you may have would be welcome.

    PS: I also had to switch to flup from flipflop.

  • #78 Miguel Grinberg said 2015-07-17T05:34:27Z

    @Wasch: static pages do not go through the Flask application, you need to configure them in the web server. Regarding the redirect I'm not sure I understand. You can connect to the main page, but then the redirect to the login page fails? What is the login URL you are being sent? Check if it is wrong that may give you some clues.

  • #79 Konrad said 2015-09-24T11:59:18Z

    First of all thank you very much for this tutorial. I had a problem with "Traceback (most recent call last): File "./db_create.py", line 2, in from migrate.versioning import api ImportError: No module named migrate.versioning" The solution was to clone repo from branch version 0.17

    because master is somewhat different - I had no time to investigate which change caused it but insted of cloning with commend $ git clone git://github.com/miguelgrinberg/microblog.git I cloned branch with: git clone -b version-0.17 https://github.com/miguelgrinberg/microblog.git and it helped

  • #80 Neil said 2015-12-04T21:41:55Z

    Just wanted to thank you again for the excellent tutorial. The site is now up and running on an AWS EC2 instance. I will be evangelizing this tutorial to anyone interested in learning Flask or backend development in general!

  • #81 Phill said 2016-01-02T21:19:24Z

    Liking your blog! I had trouble getting Lighttpd up and running on an RPi. The first problem was this error:

    Duplicate config variable in conditional 0 global: alias.url

    I fixed this by changing the line: alias.url = ( "/microblog/static/" => "/home/pi/microblog/app/static/", )

    to: alias.url += ( "/microblog/static/" => "/home/pi/microblog/app/static/", )

    That allowed me to start the server.

    Then I kept getting 404's and had to: sudo lighttpd-enable-mod fastcgi sudo service lighttpd reload

  • #82 Christer said 2016-01-18T00:36:23Z

    Thanks for the tutorial on Flask. I am rather new on Flask and try by step and step learn Flask through examples and reading. I have installed Flask on Raspberry and it works great with static template. On my Rasperry pi I have a python application running that collecting temperature. I want to get information from temperature python application to show on my Flask webserver. I can not figur out how I can retrieve information from on other running python application into Flask evironment. The temperature is constantly changing and that must reflekt on the website, not once but multiple value on temperature change on website. Would glad if you could give me some direction how that could be solved.

  • #83 Miguel Grinberg said 2016-01-27T15:37:37Z

    @Christer: You have a few options. One is to have the temperature service offer an API endpoint that returns the current temp. The Flask app can send a request to the other service to get the current value and relay it to the client. Another option is to save the temperature routinely to a file, which the Flask app can read as needed.

  • #84 Mike Zhao said 2016-02-22T02:28:07Z

    Hi Miguel,

    I followed your tutorial and now am trying to deploy my app to Heroku. However, I'm getting a DatabaseAlreadyControlled Error during the init process from the file db_create.py. I've tried deleting my database multiple times but it still doesn't seem to work. Can you help me? I've included the github link if you would like to see my code: https://github.com/zzhao36/chara2.

    2016-02-22T02:23:17.582534+00:00 app[init.1]: Traceback (most recent call last): 2016-02-22T02:23:17.582549+00:00 app[init.1]: File "db_create.py", line 12, in 2016-02-22T02:23:17.582554+00:00 app[init.1]: api.version_control(SQLALCHEMY_DATABASE_URI, SQLALCHEMY_MIGRATE_REPO) 2016-02-22T02:23:17.582555+00:00 app[init.1]: File "", line 2, in version_control 2016-02-22T02:23:17.582643+00:00 app[init.1]: File "/app/.heroku/python/lib/python2.7/site-packages/migrate/versioning/util/init.py", line 160, in with_engine 2016-02-22T02:23:17.582690+00:00 app[init.1]: return f(a, *kw) 2016-02-22T02:23:17.582705+00:00 app[init.1]: File "/app/.heroku/python/lib/python2.7/site-packages/migrate/versioning/api.py", line 250, in version_control 2016-02-22T02:23:17.582764+00:00 app[init.1]: ControlledSchema.create(engine, repository, version) 2016-02-22T02:23:17.582768+00:00 app[init.1]: File "/app/.heroku/python/lib/python2.7/site-packages/migrate/versioning/schema.py", line 141, in create 2016-02-22T02:23:17.582819+00:00 app[init.1]: table = cls._create_table_version(engine, repository, version) 2016-02-22T02:23:17.582822+00:00 app[init.1]: File "/app/.heroku/python/lib/python2.7/site-packages/migrate/versioning/schema.py", line 189, in _create_table_version 2016-02-22T02:23:17.582876+00:00 app[init.1]: raise exceptions.DatabaseAlreadyControlledError 2016-02-22T02:23:17.582891+00:00 app[init.1]: migrate.exceptions.DatabaseAlreadyControlledError

  • #85 ruslan said 2016-06-27T22:39:31Z

    Hello Miguel! Thank you for detail and helpful manual!

    I follow your manual from part 1 and finally installed the application in Debian 8 + lighttpd + sqlite. Debian instance works well under MS Hyper-V (2008 R2).

    I found one small item in code that move FastCGI to "Error 500".

    actual from version 0.7 till version 0.17

    File app/"init.py >> line 54 file_handler = RotatingFileHandler('tmp/microblog.log', 'a',

    Application can't find directory 'tmp/...' because of missed "/". I changed it to '/tmp/...' and updated code works w/o error: file_handler = RotatingFileHandler('/tmp/microblog.log', 'a',

    If this notice is matter, please update the manual in next reduction. I put the same comments to part 7 when code was initially presented.

    regards, ruslan

  • #86 jeh0753 said 2016-06-29T22:59:17Z

    Miguel, I'm having an issue here related to password-less login. I'm running this on CentOS 7, and found that if I give 775 permissions to the apps folder, I am denied permission to enter the apps account. When I adjusted the /etc/.ssh/sshd-config file to turn strictmode off, I was able to get in anyway, but from my research I found that so long as StrictMode is enabled (which it should be), the maximum permissions for the /home/apps/ folder needs to be a maximum of 755. Is there a work-around for this?

  • #87 Miguel Grinberg said 2016-07-14T18:18:35Z

    @rusian: the tmp directory is local to the project, it is not the global /tmp. But if you want logs to go to the system wide /tmp your change is fine.

  • #88 Miguel Grinberg said 2016-07-14T18:44:01Z

    @jeh0753: not sure I understand why 775 vs 755 is important. This is a security feature of SSH. If you want to relax security on your account you can, but that means you have to tell SSH to not check your account (i.e. disable StrictMode).

  • #89 Jake Hawkesworth said 2016-07-20T00:19:55Z

    Miguel, thanks for this great, challenging tutorial.

    For some reason, tmp/microblog.log is not being created by the RotatingFileHandler, and the db_create.py step is throwing me a "no such file" error. When I comment the RotatingFileHandler out, I immediately get a new error, sqlalchemy.exc.OperationalError: (OperationalError) 1045, which I haven't been able to figure out. I'm running Python 2.7 and Centos 7. Does this sound like a package version issue?

  • #90 Jake said 2016-07-21T01:39:22Z

    @Miguel - Thanks so much for your reply. I was able to get past the 775 755 issue; it turned out that I had incorrect settings on the home folder for apps. This was a great link that helped me through it, if others run into issues where they can't get ssh to work without StrictMode turned off: http://callumpy.co.uk/blog/centos-public-key-authentication-setup-guide-and-tips/

    I sent a followup question yesterday; I've been stuck on the db_compile.py step for a few days; can't seem to crack it!

  • #91 Jake said 2016-07-21T02:17:45Z

    @Miguel - update on my problem. @rusian had the idea of switching the / in tmp, and that solved my issue with the log file. I'm still running into error 1045 with db_create.py: sqlalchemy.exc.OperationalError: (OperationalError) (1045, "Access denied for user 'apps'@'localhost' (using password: YES)") The issue traces back to: File "./microblog/db_create.py", line 7, in db.create_all()

    This is weird, because I've been able to successfully log in to mysql's apps account without a password. Any ideas?

  • #92 Jake said 2016-07-23T05:11:15Z

    @Miguel - solved my problem. The code above for db_create assumes the password to be 'apps'. I had no password at all, so once I got rid of that part of the line of code, the problem went away!

  • #93 Soham said 2016-07-23T16:57:08Z

    Hi Miguel, many thanks for this amazing tutorial. I encountered a problem while setting up the server: while executing the command 'make' to compile mod_fcgid, i get this error: Makefile:29: /rules.mk: No such file or directory make: *** No rule to make target `/rules.mk'. Stop. Would be glad if you could help me. Thanks

  • #94 Miguel Grinberg said 2016-07-28T05:38:11Z

    @Soham: not familiar with the problem you are having. Maybe a good idea to ask on stack overflow.

  • #95 Souvik Ray said 2017-04-01T10:09:55Z

    I don't understand one thing...I have linux installed and coded the app in linux only.Then why do I need a VPS?Can't I directly host my site on Heroku?

  • #96 Miguel Grinberg said 2017-04-02T05:38:25Z

    @Souvik: Yes. Heroku is one option, a VPS is another. There are actually even more options for deploying your app. You just need to pick the one you like the best.

  • #97 Sun Qingyao said 2017-08-16T03:01:10Z

    Hello, I'm following your great tutorial, and it's really educative. However, I got an error with flipflop: “OSError: [Errno 88] Socket operation on non-socket”. Is it an issue of Apache, CentOS, or flipflop itself?

    Please see this Stack Overflow question for a MCVE:

    https://stackoverflow.com/q/45703959/5399734

  • #98 Miguel Grinberg said 2017-08-16T06:21:30Z

    @Sun: You can't run the fastcgi script from the terminal. This script is supposed to be executed by Apache. Typically you have it configured in a ScriptAlias directive in your Apache config file.

  • #99 Victor Johnson said 2017-08-30T08:31:51Z

    Can i deploy this microblog in a shared hosting which has a cpanel?? if not why? can you explain me why?

  • #100 Miguel Grinberg said 2017-08-30T22:25:22Z

    @Victor: If you can run Python scripts on your hosting account then I don't see why you would not be able to run this application. Unfortunately shared hosting is not a platform that I will be writing about, I instead prefer to focus on more modern cloud hosting options that are more flexible.

Leave a Comment