2013-04-15T06:14:46Z

The Flask Mega-Tutorial, Part XVII: Deployment on Linux (even on the Raspberry Pi!)

This is the seventeenth article in the series in which I document my experience writing web applications in Python using the Flask microframework.

The goal of the tutorial series is to develop a decently featured microblogging application that demonstrating total lack of originality I have decided to call microblog.

NOTE: This article was revised in September 2014 to be in sync with current versions of Python and Flask.

Here is an index of all the articles in the series that have been published to date:

Today we are reaching a huge milestone in the life of our microblog application. We are going public with it, as we will be looking at different ways in which we can deploy it and make it accessible to our users.

In this article I will explore the traditional hosting options, and will be looking at hands-on deployment on a Linux box and on the widely popular Raspberry Pi minicomputer. In the next article we will be looking at deployment on cloud services.

To begin we will be looking at the more traditional of all deployment options. We will install the application behind a web server on a dedicated host that we will manage.

But where can we find a host? These days there are many cheap server offers out there. The cheapest options are for a VPS (virtual private server), which is virtual machine that looks like a standalone and dedicated server to you, but is really sharing the physical hardware with a few others like it. You can check on lowendbox.com for deals if you want to get one to experiment with (note that I'm not affiliated with them, I do not endorse them or make money if you click the link above).

Another easy way to obtain a host is to install a virtual machine on your regular PC. If you are a Windows or Mac user and would like to experience deploying the application to a Linux server without spending a dime, then this is the way to go. You can install VirtualBox on your system, download an ISO image for the Linux distribution of your choice and then create a VM on which you install this ISO image.

Let's talk a bit about operating system choices. I think there are four decent choices for hosting a web server:

  • Linux
  • BSD
  • OS X
  • Windows

From a technical point of view you could setup a Python based web server on any of the above, so how do we choose?

In my opinion the choice isn't too hard to make. For me it is important that we are on a platform that is open source, because the community at large does a much better job at keeping a platform safe and secure than what a single company can do on a closed source product. We also want to be on a platform that has a large server installation base, because that gives us more chances of finding solutions to our problems. Based on the above criteria, I consider Windows and OS X inferior to Linux and BSD as servers.

So now we are down to two options. The choice of Linux vs. BSD is an easy one for me: It does not matter. Both are excellent options for hosting a web server. Depending on the hosting provider you may have Linux or BSD options to choose from, so I just simply use what's available.

There are many Linux and BSD distributions out there, so how to pick one? This is largely a matter of preference, but if we are going to try to use what the majority use we can look at the distributions according to some of their major characteristics, and then pick a popular distribution in the group of choice:

  • RPM based (RedHat, CentOS, Fedora)
  • Debian based (Debian, Ubuntu, Mint)
  • BSD based (FreeBSD, NetBSD, OpenBSD)
  • others

I'm sure my simplistic grouping of distros will offend some. I'm also sure many will have the need to mention that Mac OS X is BSD based and as such should be in my list. Keep in mind that this classification is just my own, I don't expect everyone will agree with me.

As an exercise in preparing this article I tested a few Linux distros (I'm using a VPS that does not offer BSD choices). I tried Fedora, Ubuntu, Debian and CentOS. None of them was straightforward, unfortunately, but out of all of these, the CentOS install was the one with less complications, so that's the one I'm going to describe here.

If you have never done this you may find the setup task to be extremely laborious, and you would be correct if you think that. The thing is, once the server is setup it takes very little effort to keep it running.

Ready? Here we go!

Hosting on CentOS 6

I'm going to assume that we have a clean install of CentOS 6 on a dedicated server or VPS. A VM in your PC also works, but go ahead and do a standard install before starting.

Client side setup

We will be managing this server remotely from our own PC, so we need a tool that we can use to login into this system and run commands. If your PC runs Linux or OS X then you already have OpenSSH installed. If you are on Windows then there are two very decent ways to get SSH functionality:

  • Cygwin (select the OpenSSH package in the installer)
  • Putty

Cygwin is much more than SSH, it provides a Linux like environment inside your Windows box. Putty, on the other side, is just an SSH tool. Since I want to cover the most ground, the instructions below will be for Linux, OS X and Cygwin users. If you use Putty on Windows, then there is a little bit of translation required.

Let's begin by logging in to our shiny new CentOS 6 box. Open a command line prompt on your PC (a Cygwin prompt if you are on Windows) and run the following:

$ ssh root@<your-server>

This is saying that you want to login to the system as user root, the admin of the system. You should replace <your-server> with the IP address or hostname of your server. Note the $ is the command line prompt, you do not type that.

You will be asked to provide the password for user root. If this is a dedicated server or VPS that you purchased, you must have been asked about this password during the setup process. If this is your own VM then you chose a root password during installation.

Installing software packages

Now that we are logged in, we need to install all the software that we will be using to deploy our server, which obviously include Python and a web server. We will also switch from sqlite to a more robust MySQL database.

The command to install software on CentOS is called yum. We can install all the packages we need using a single command:

$ yum install python python-devel python-virtualenv httpd httpd-devel mysql-server mysql-devel git gcc sudo

Some of these could be already installed, but it doesn't hurt to be safe and request all the packages we need, yum will only install what's missing.

The packages we just installed are:

  • python and python-devel: the Python interpreter and its development package
  • virtualenv: the Python script that creates virtual environments
  • httpd and httpd-devel: The Apache web server and its development package
  • mysql-server and mysql-devel: The MySQL database server and its development package
  • git: source code version control system (we will use it to download and update the application)
  • gcc: the C/C++ compiler (needed to compile Python extensions)
  • sudo: a tool that helps users run commands as other users.

Creating a user account to host the application

We will now setup a user account on which we will host our application. In case the reasons aren't clear, the root account has the maximum privileges, so we could easily delete or destroy important parts of the system by mistake. If we work inside a restricted account then all the important parts of the system will be out of reach so we are protected against mistakes. A second, maybe more important reason to not use the root account is that if a hacker gets hold of this account he will own our server. We want to hide the root account as much as possible, I like to disable root account logins completely on production servers.

We will call this new account apps, assuming we could host several applications inside it. Since the applications will be run by the web server account, we will make the group of our apps account the group of the web server account (which was created for us when Apache was installed), so that the web server has permission to read and write on these files. If you are not familiar with Unix user permissions, then an introduction is available over at this Wikipedia page.

The command to create the apps account on CentOS is:

$ adduser -g apache apps

The -g option makes the primary group of this account the apache group, which on CentOS is the group of the apache user account. As part of the account setup you will have to enter a password for this account.

The adduser command will create a home directory for the new account, at /home/apps. Unfortunately, when we go check the permissions of this directory we find it to only be accessible to the owner:

$ ls /home -l
total 7
drwx------ 2 apps  apache 4096 Apr  7 11:46 apps

We decided we want the apache group users to be able to freely work on the files on this account, so we have to give the group enough permissions:

$ chmod 775 /home/apps

Consult the documentation for the chmod tool for detailed information. The 775 permisions will allow the owner and the group full access, and will also give others permission to read and execute, but not write to this directory.

Password-less logins

The next step is to be able to login to this account from our PC without using a password. The SSH tool supports another type of authentication, called public key. If you are on a non-Windows OS it is very likely that you already have keys installed on your system.

Now open a terminal window on your own PC (Cygwin's bash shell for Windows users). To check if you have keys installed run the following command:

$ ls ~/.ssh
id_rsa  id_rsa.pub

If the directory listing shows files named id_rsa and id_rsa.pub like above then you are good to go. If you don't have these files then you have to run the following command:

$ keygen -t rsa

This application will prompt you to enter a few things, for which I recommend you accept the defaults by pressing Enter on all the prompts. If you know what you are doing and want to do otherwise, you are certainly welcome to.

After this command runs you should have the two files listed above. The file id_rsa.pub is your public key, the data in this file can be shared with others without creating a security risk. The id_rsa file is your private key, so you should never share the contents of this file with anyone.

We will now add our public key to our server, so that it knows who we are. Still on our local PC we start by printing the public key file:

$ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwaK4JVuY6PZAr8HocCIOszrLIzzCjO0Xlt9zkFNKvVpP1B92u3JvwwiagqR+k0kHih2SmYnycmXjAcE60tvu+sIDA/7tEJZh4kO4nUYM5PJ17E+qTqUleBXQM74eITydq/USkOqc5p++qUUgA60gUUuNum3igbZiNi71zK4m8g/IDywWYk+5vzNt2i7Sm8NEuauy/xWgnWhCBXZ/tXfkgWgC/4HzpmsfO+nniNh8VgTZp8Q+y+4psSE+p14qUg7KdDbf0Wo/D35wDkMvto96bIT8RF0np9dTkFj8TgNW8inP+6MC+4vCd8F/NpESCVt8hRlBVERMF8Xv4f/0+7WT miguel@miguelspc

Now we need to copy this data to the clipboard and then switch to the CentOS system, where we will issue these commands:

$ mkdir /home/apps/.ssh
$ echo <paste-your-key-here> > /home/apps/.ssh/authorized_keys

For my server and public key the following commands are necessary:

$ mkdir /home/apps/.ssh
$ echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwaK4JVuY6PZAr8HocCIOszrLIzzCjO0Xlt9zkFNKvVpP1B92u3JvwwiagqR+k0kHih2SmYnycmXjAcE60tvu+sIDA/7tEJZh4kO4nUYM5PJ17E+qTqUleBXQM74eITydq/USkOqc5p++qUUgA60gUUuNum3igbZiNi71zK4m8g/IDywWYk+5vzNt2i7Sm8NEuauy/xWgnWhCBXZ/tXfkgWgC/4HzpmsfO+nniNh8VgTZp8Q+y+4psSE+p14qUg7KdDbf0Wo/D35wDkMvto96bIT8RF0np9dTkFj8TgNW8inP+6MC+4vCd8F/NpESCVt8hRlBVERMF8Xv4f/0+7WT miguel@miguelspc > /home/apps/.ssh/authorized_keys

These commands write your public key to the file authorized_keys in the server. This makes your public key known to OpenSSH on the server. The remaining task is to secure the .ssh directory and the authorized_keys file inside it:

$ chown -R apps:apache /home/apps/.ssh
$ chmod 700 /home/apps/.ssh
$ chmod 600 /home/apps/.ssh/authorized_keys

These commands change the ownership of these files to the apps account and then make the directory and file only accessible by the new owner.

The password-less login should now be working. We will now logout of the root account of our server and then login again, but this time we will login as the apps user:

$ ssh apps@<your-server>

If everything goes well you should not need to enter a password to obtain access.

Installing the application

We will now use git to download and install microblog into the server. If you are not familiar with git I recommend that you read git for beginners.

The application must be available in a git server that can be reached from our web server. I will be using my own github hosted application. You are welcome to use it as well, or if you prefer you can also clone it and make one that is yours.

To install microblog in our server we just issue a git clone command:

$ git clone https://github.com/miguelgrinberg/microblog.git
$ cd microblog

The latest version of the application has a few changes from how we left it in the previous article.

First, there are a few new files at the top level, runp-sqlite.fcgi, runp-mysql.fcgi, killpython and requirements.txt. The *.fcgi scripts are starter scripts that are invoked by servers that use the FastCGI protocol. The killpython script will help us restart the application after we upgrade it. The requirements.txt file lists all the dependencies that we need to install to run our application. We will look at all these later.

Another interesting change is in our config.py file. Up to now we initialized our database as follows:

SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db')

Now we have this instead:

if os.environ.get('DATABASE_URL') is None:
    SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db')
else:
    SQLALCHEMY_DATABASE_URI = os.environ['DATABASE_URL']

This is a simple change that enables us to override the database the application will use by setting an environment variable. You will see how we take advantage of this change in the next section.

Setting up MySQL

The sqlite database that we've been using all this time is great for simple applications, but when we get into a full blown web server that can serve multiple requests at a time it is a better idea to use a more robust database. So we will setup a MySQL database for microblog.

We have installed MySQL already, so all that is left is to create a database and a user that has permissions on it. To manage our database server we use the mysql tool:

$ mysql -u root -p
Enter password: (enter the mysql root password, or empty if one is not defined)
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.1.67 Source distribution

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

And now that we are in the mysql prompt we can go ahead and create a database and a user, both named apps:

mysql> create database apps character set utf8 collate utf8_bin;
mysql> create user 'apps'@'localhost' identified by 'apps';
mysql> grant all privileges on apps.* to 'apps'@'localhost';
mysql> flush privileges;
mysql> quit;

Note that you will need to pick a password for the apps MySQL user in the identified by section. For simplicity I have picked apps as password, but on a real installation a hard to guess password should be used. Do not confuse the database apps user with the system's apps user.

Initializing the application

Now that we have a database ready we can initialize microblog.

We start by creating and populating the Python virtualenv:

$ virtualenv flask
$ flask/bin/pip install -r requirements.txt
$ flask/bin/pip install mysql-python

Note that in addition to installing all the dependencies listed in the requirements.txt file we add the MySQL support, which is required by SQLAlchemy when connecting to that database server.

Then we create the database:

$ DATABASE_URL=mysql://apps:apps@localhost/apps ./db_create.py

Note how we set an environment variable to point to our new MySQL database instead of the sqlite default.

Next we compile all the messages in our translation database:

$ ./tr_compile.py

And to finalize, we enable group write access to the two folders that need to be written to by the web server:

$ chmod -R g+w search.db tmp

The search.db folder is used by our Whoosh full text search database. The tmp directory should be left open so that new files can be created there when needed.

Setting up Apache

The last thing to do is to configure our Apache web server.

We wil be using the mod_fcigd module to handle the FastCGI communication with our application. Many Linux distributions offer a package for mod_fcgid, but unfortunately CentOS is not one of them, so we will build this module from its source code.

Here is the list of commands that build and install the current release of mod_fcgid:

$ wget http://mirror.metrocast.net/apache/httpd/mod_fcgid/mod_fcgid-2.3.9.tar.gz
$ tar xvzf mod_fcgid-2.3.9.tar.gz
$ cd mod_fcgid-2.3.9
$ APXS=/usr/sbin/apxs ./configure.apxs
$ su
(enter root password)
$ make install

You can consult the mod_fcgid documentation if you want details on the above commands.

With this module installed we can now configure our server. For this we need to edit the Apache configuration file. Linux distributions do not agree on a standard location and name for this file, so you will need to figure out where this file is. On CentOS, the config file is located at /etc/httpd/conf/httpd.conf.

The changes that we need to make are simply to define our server. At the bottom of the configuration file we add the following (from the root account, since this file is not writable to regular users):

FcgidIPCDir /tmp
AddHandler fcgid-script .fcgi
<VirtualHost *:80>
    DocumentRoot /home/apps/microblog/app/static
    Alias /static /home/apps/microblog/app/static
    ScriptAlias / /home/apps/microblog/runp-mysql.fcgi/
</VirtualHost>

The FcgidIPCDir sets the directory where the file sockets will be created. I've found that on CentOS this was going by default to a directory where the apache user did not have write permissions, so I'm putting all these files on /tmp instead.

Next we use AddHandler to tell apache that any files that have a .fcgi extension should be treated as FastCGI files and routed through the mod_fcgid module that we just installed. Remember the new runp-mysql.fcgi file that we have in our root folder? This file uses the flipflop Python module as an adapter so that our application can speak the FastCGI protocol. Let's have a quick look at this file:

#!flask/bin/python
import os
os.environ['DATABASE_URL'] = 'mysql://apps:apps@localhost/apps'

from flipflop import WSGIServer
from app import app

if __name__ == '__main__':
    WSGIServer(app).run()

Apache will execute this script to start our application. Note how we insert the MySQL database name in the environment, so that Apache can see it.

The <VirtualHost> section defines the host that will run our web server. The *:80 denomination indicates that any requests that arrive at the server for any hostnames on port 80 will be handled by this virtual server. The configuration can have multiple <VirtualHost> sections, each defining a different server. To differentiate the servers you can use different IP addresses, domain or sub-domain names or ports.

The definition of our virtual host is pretty simple. The DocumentRoot statement tells Apache where to look for static files. Any requests for files will be served our of this folder, so for example, when browsers ask for a /favicon.ico file to show the little icon next to the URL Apache will try to find it here. Unfortunately all the static files that we use in our application have a /static prefix, so to prevent Apache from looking for another static folder we use the Alias statement, which just says that anything that starts with /static should go directly to our folder. Finally, the ScriptAlias statement tells Apache that when a request that starts with / arrives (basically all requests that are not for static files) then use the script indicated as second argument to handle it, and that is our .fcgi script.

The Apache config file supports many more options that I will not mention here. I recommend that you review the Apache documentation to decide what options make sense for your server.

To activate the changes we made we need to restart the apache server, again from the root account:

$ service httpd restart

And now from the browser on your PC you should be able to access microblog at the address http://<your-server>.

Installing application updates

The last remaining thing we are going to look at for this server is how to roll out an update of the application.

Let's assume we have deployed the application and it has been running happily for a while on our server. Now it's time to push an update, which could fix some bugs we found, or just add features.

We will use git and our own tools for this. If we login to the web server using the apps account we can upgrade to the latest release from the git repository by running:

$ cd microblog
$ git pull
$ DATABASE_URL=mysql://apps:apps@localhost/apps ./db_upgrade.py
$ ./tr_compile.py

The git pull operation downloads any new or updated files from our git server. We then upgrade our database and recompile our translation files, in case any of those need updating. That's it!

Once we have the updates in place we need to tell apache to restart the FastCGI processes, and this gets a bit tricky, because we are logged in as user apps, an unprivileged account, while the FastCGI processes are owned by the apache user.

For this we are going to use sudo. The sudo tool allows users to selectively run applications as other users. We don't really want to give our apps account a lot of power, we will limit it to just be able to send kill signals to the FastCGI processes started by the apache user.

The killpython script that we have in the microblog directory does this work:

killall /home/apps/microblog/flask/bin/python

The problem is, if we run this under the apps account we won't have permission to kill the processes owned by the apache user. To enable apps to kill these processes we have to make a small change in file /etc/sudoers:

apps    ALL=(apache) NOPASSWD:/home/apps/microblog/killpython
Defaults: apps    !requiretty

Note you will need to make this change as user root, since /etc/sudoers is a system file.

The first cryptic command gives the apps user permission to run the killpython command as user apache without having to provide a password. The second line allows sudo commands for user apps to be issued from a script in addition to doing it from an interactive console. You can read the man page for sudoers for detailed information on the syntax of sudo configuration file.

Now when logged in as apps we can kill the python jobs as follows:

$ sudo -u apache ./killpython

If the FastCGI processes are killed Apache will restart them the next time a request comes for microblog.

To make upgrading our server a bit more simple we can write a client side script that upgrades and restarts all together. The script would run the following:

ssh apps@<your-server> "cd microblog;git pull;sudo -u apache ./killpython;DATABASE_URL=mysql://apps:apps@localhost/apps ./db_upgrade.py;./tr_compile.py"

If you store the above line in a script then you can roll out an application upgrade with a single command!

What's left to do

I will not discuss a few more mundane operations that are recommended to make a server that is open to the hostile Internet world more secure. Things such as:

  • Not allowing root remote logins
  • Disabling any services that are not used, like FTP, CIFS, etc.
  • Setting up a firewall
  • Keep the server up to date on security fixes.
  • Etc.

I will leave these as an exercise to the interested reader, since they are largely unrelated to this tutorial.

Hosting on the Raspberry Pi

The Raspberry Pi is a revolutionary little Linux computer that costs $35. It has very low power consumption so it is the perfect device to host a home based web application that can be online 24/7 without tying up a full blown computer.

There are several Linux distributions that run on the Raspberry Pi. We'll set up our application on Raspbian, the official distribution.

As a side benefit, note that Raspbian is a derivative of Debian, so the instructions below will apply with little or no modification for Debian/Ubuntu based servers.

We will now follow the same steps we followed for CentOS to get the RPi server setup.

Client side setup

A Raspberry Pi is a pretty decent computer to work in. You can connect an HDMI monitor, keyboard and mouse and work directly on it to configure our web server. Also like we did for CentOS or actually any other Linux distribution you can have the Raspberry Pi just connected to the network and then access it remotely from a PC over ssh. See the client side setup section above for CentOS for instructions on how to setup a client PC with ssh.

Note that I will not cover how to install and setup the Raspberry Pi here, there are plenty of articles elsewhere for that. I will assume the Raspberry Pi is already running and connected to the network.

Installing software packages

Since the Pi is a limited power machine we will not do a full blown Apache/MySQL installation like we did for CentOS. Instead, we will take a lightweight approach. For web server we will use Lighttpd (pronounced "Lighty"), a small web server with a very good and efficient FastCGI implementation. For our database we will stay with sqlite.

With CentOS we used yum to install packages. In Debian derivatives the package manager is called apt-get:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install python python-dev python-virtualenv lighttpd git

The update line downloads an updated package list to your Pi. The upgrade line upgrades all installed packages to their latest versions. It is recommended that you issue these two commands regularly, to keep your Raspberry Pi's software up to date. Finally the install statement installs the packages that we need for our task.

Creating a user account to host the application

We said that we will assume our Raspberry Pi server will be in a trusted network. Since security is not a major concern we will not bother creating a dedicated account and instead will install the app directly on the default pi account.

If your Raspberry Pi will be connected to the internet then you should follow the approach we used for CentOS and make a different account that has no privileges, as this ensure that if an intruder manages to access the account the damage that he or she can cause will be limited.

Password-less logins

Having a public key authentication procedure is also less important if we assume the application will run on a small intranet, but should you need such functionality the procedure is identical to what we did for the CentOS system.

Installing the application

The application installs very easily, again using git:

$ git clone git://github.com/miguelgrinberg/microblog.git
$ cd microblog

Initializing the application

Since we will be using a sqlite database in this host we can go ahead and initialize microblog as we have done in past articles for our development server:

$ virtualenv flask
$ flask/bin/pip install -r requirements.txt
% flask/bin/pip install mysql-python
$ ./db_create.py
$ ./tr_compile.py

Note that the setup.py script will try to install the MySQL support and that will fail, but this is all right since we will not use it.

The files were all checked out under the pi user ownership. But as we have seen for CentOS, the web server on the Pi will run under a different account. Since security isn't a concern for this server, we will take a much simpler approach:

$ chmod -R 777 *

The above chmod statement makes all the files writable by all users. Again, this would be a really bad thing to do on an open server, but for a server where you have full control it saves you from the trouble of dealing with user permissions.

I want to make it clear that the Raspberry Pi is perfectly capable of supporting a configuration based on user groups and user permissions similar to what we created above for the CentOS box. We are choosing this simplified setup because we will be using this server in a controlled environment.

Setting up Lighttpd

Lighttpd comes with native support for FastCGI, so we do not need to worry about installing a separate module for it.

All we need to do is append the definition of our website to the configuration file, which is located at /etc/lighttpd/lighttpd.conf:

fastcgi.server = ("/microblog" =>
    ((
        "socket" => "/tmp/microblog-fcgi.sock",
        "bin-path" => "/home/pi/microblog/runp-sqlite.fcgi",
        "check-local" => "disable",
        "max-procs" => 1
    ))
)

alias.url = (
    "/microblog/static/" => "/home/pi/microblog/app/static/",
)

And with this our application should be online at http://aaa.bbb.ccc.ddd/microblog, with aaa.bbb.ccc.ddd being the IP address of your Raspberry Pi.

But there are a few things to note in the above configuration, and also a couple of things that didn't initially work and required some changes in the application side.

The fastcgi.server statement is the one that defines the behavior of our FastCGI server, which we are exposing under the /microblog URL. The reason we are not exposing the application at the root URL is simple, we may want to host more than one application, so putting all the URLs under a /microblog root effectively works as a namespace.

Inside the FastCGI definition we provide a socket name in the /tmp directory and the path to our runp-sqlite.fcgi file, which Lighttpd will execute to start up FastCGI processes. The check-local option tells Lighttpd to send requests to the FastCGI server even if the request path matches a file on disk. The max-procs limits the number of FastCGI processes to 1, which for a small server is enough and avoids potential problems with multiple concurrent writers to the sqlite database.

The alias.url section provides the mapping that enables Lighttpd to server our static files. The alias maps any requests for /microblog/static/... to the correct location on disk where we store our static files.

The runp-sqlite.fcgi script is pretty much the same as the one we used for CentOS, but without overriding the database setting:

from flipflop import WSGIServer
from app import app

if __name__ == '__main__':
    WSGIServer(app).run()

Some necessary fixes

Testing this setup reveals a couple of bugs in our application.

First, our javascript and css static files are not being served. The problem becomes evident after inspecting the HTML source of the login page. The base.html template references these files via hardcoded paths that begin with /static, but we said are putting this application inside a /microblog prefix. For example, see how we had the link to the CSS file:

<link href="/static/css/bootstrap.min.css" rel="stylesheet" media="screen">

Here we really want /microblog/static/bootstrap.min.css, but of course we can't add /microblog to the base.html file because that will break the development server that runs on our PC. The solution is to let Flask generate these URLs using our old friend url_for:

<link href="{{ url_for('.static', filename='css/bootstrap.min.css') }}" rel="stylesheet" media="screen">

After updating all the js and css files as above the site serves the login page with all the auxiliary static files.

But trying to login reveals another problem, Right after logging in we get a 500 error code. Luckily for us we have logging, so a quick look at /home/pi/microblog/tmp/microblog.log shows the following error:

ProgrammingError: (ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id -1260325776 and this is thread id -1243548560

What is this? We are not running multiple threads ourselves but obviously there are multiple threads in our application. The only difference between our development server and this one is that we are now using flipflop for the FastCGI server. Looking at the code in this module we can easily find that by default flipflop runs a multithreaded web server.

A possible solution would be to use the single-threaded FastCGI server from flipflop but that could affect the performance if the server will have multiple concurrent users. Another, more interesting way to handle this problem is to enable threading in sqlite (which appears to be fully supported). Multithreading can be enabled in SQLAlchemy by setting the check_same_thread option to False in our config.py configuration file:

SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db') + '?check_same_thread=False'

With this change we are allowing multiple threads to make changes to the database, the sqlite library takes care of synchronizing multiple accesses.

And with these small changes we have a fully enabled web application running on our Raspberry Pi!

Installing application updates

To deploy application updates we can just login to the Raspberry Pi and issue the commands that update the source files, database and translation files:

$ cd microblog
$ git pull
$ ./db_upgrade.py
$ ./tr_compile.py

If you setup password-less login to your Pi then of course you can write a script wrapper that does all this in a single command.

Final words

The updated application is available, as always, on my github page. Alternatively you can download it as a zip file below:

Download microblog 0.17.

In the next article we will be looking at deployment on cloud services as an alternative to the traditional options we explored today.

I will see you then. Thank you!

Miguel

96 comments

  • #1 Laurent Rivard said 2013-06-20T22:14:26Z

    When I try to connect to the mySQL tool. I haven't set any password, but I get the error <code> ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) </code> Any ideas? Thanks a lot for the tutorial, I really enjoyed it!

  • #2 Miguel Grinberg said 2013-06-21T01:36:58Z

    @Laurent: chances are the MySQL service isn't running, or it is running but its socket file is incorrectly set. If you google the error message you will find lots of ideas on how to debug this.

  • #3 Theladdie said 2013-06-24T10:55:59Z

    Can you help me. I have tried the setup for the raspberry pi and I ammended the lighttpd.conf and restarted the server but I am not able to get the page... Just 404 not found. I got the microblog from git and ran setup, db_create and tr_compile But now I am lost

  • #4 Miguel Grinberg said 2013-06-25T03:52:38Z

    @Theladdie: did you check the lighttpd logs for errors? I can't really tell you what's wrong, either the configuration does not point to the place where microblog is installed, or maybe you are using the wrong URLs, I would try to get lighttpd to server regular files first to make sure it is working.

  • #5 Marc Vanelverdinghe said 2013-07-24T18:14:49Z

    Thank you for this tutorial whiwh is invaluable as for me Flask is totally new. I have a project where I want to use the Raspberry PI's Piface board from the internet. when including "import piface.pfio " I get an error that this module is not found. How do I add this module to the virtual environment? When starting the standard Python version importing the piface module is not a problem. Any ideas? Thanks!

  • #6 Miguel Grinberg said 2013-07-25T05:41:19Z

    @Marc: you have two options. You can install piface in your virtual environment along with Flask, or you can create the virtual environment passing --system-site-packages as argument to virtualenv. This option makes the packages installed globally also visible to the virtual environment.

  • #7 Thai Nguyen said 2013-08-08T05:34:17Z

    really great mega guide Miguel and i am glad that i found your blog. i am currently at part 5 of your guide. do you have by any chance an example of where you connect to a MS SQL-Server 2008 using slqalchemy?

  • #8 Miguel Grinberg said 2013-08-08T05:38:40Z

    @Thai: with SQLAlchemy it does not really matter which database you use, all implement the same model setup and operations.

  • #9 Thai Nguyen said 2013-08-09T02:18:21Z

    you are absolutely right with regards to the db conn. setup. it was more straight forward then anticipated. again thank you for writing up this very helpful guide!

  • #10 Rob Juffermans said 2013-10-09T13:22:30Z

    Hi Miguel, I think you should update setup.py with: subprocess.call([os.path.join('flask', bin, 'pip'), 'install', 'flask-wtf<0.9.0']) Otherwise you could get: ImportError: No module named flask.ext.wtf.TextField

  • #11 ryan said 2013-10-31T20:33:19Z

    Miguel, thanks for the great tutorial. I was able to follow you all the way through except to the point of setting up the httpd in centos. No matter how hard i tried, i cant get the service running. it always shows the service as dead. Any ideas. i enabled most causes like SElenux, firewall, and what ever help i could get from other forums. thanks and best regards

  • #12 ryan said 2013-10-31T21:35:27Z

    hi Miguel want to add one information for you. I can get the httpd run with out any problem until i configure mod_fcigd hope this helps you regards

  • #13 ryan said 2013-11-01T11:50:57Z

    hi Miguel, continuing my update. after much reading i was able to make my httpd service run with fcgi. But then i got hit with this problem. [Fri Nov 01 07:32:41 2013] [error] [client 127.0.0.1] script not found or unable to stat: /home/apps/microblog/runp-mysql.fcgilogin any thoughts thanks

  • #14 ryan said 2013-11-01T12:24:44Z

    hi Miguel, my update, with a smile. I was able to finally deploy successfully on centos thank you

  • #15 jeromef said 2013-11-21T17:30:36Z

    Thanx a lot Rob ! But i actually have a problem, i only get the header (title "microblog" and "home" link).... then a "File not found" In the code it seems that the {% block content %}{% endblock %} is where the code stop... any suggestion ? i forget to mentionned how impressed im by this tuto... and wait for the book !

  • #16 Miguel Grinberg said 2013-11-21T17:47:57Z

    @jeromef: you can download my version of the code and compare against yours to find the problem.

  • #17 jeromef said 2013-11-22T11:21:20Z

    Moreover for a good deployment (enabling fast cgi) on raspberry i had to add this line in the /etc/lighttpd/lighttpd.conf : include "/etc/lighttpd/conf-available/10-fastcgi.conf" but always only the header and no content block (File Not Found)... any help ?

  • #18 rm said 2013-12-23T14:45:00Z

    Hi Miguel, thank you for this. Was wondering how you implement roles into the system. Specifically, some best practices for it, perhaps even a way to structure applications for it. I'm guessing the templates could have a series of <if ROLE> then <>, but there must be a better way to do this? Thanks.

  • #19 Miguel Grinberg said 2013-12-23T17:44:48Z

    @rm: I did not go into a lot of detail on that on this tutorial, but I actually have an entire chapter of the coming book where I show a possible implementation of user roles. How you embed this into the application depends. Sometimes you have to use conditionals in templates, so that you can use the same template for, say, admins and non-admins. Other times you can use a decorator that only allows the specified roles into the view function. For roles that apply to an entire blueprint you can add a before_request function that checks the role. Hope this helps!

  • #20 Cory Gough said 2014-01-21T01:30:15Z

    @Miguel: I'm trying to deploy on my Pi, but when I run runp-sqlite.fcgi I get the following message: WSGIServer: missing FastCGI param REQUEST_METHOD required by WSGI! WSGIServer: missing FastCGI param SERVER_NAME required by WSGI! WSGIServer: missing FastCGI param SERVER_PORT required by WSGI! WSGIServer: missing FastCGI param SERVER_PROTOCOL required by WSGI! .... and then a lot of stack traces. Do you know what might be causing this? Thank you for your continuing help. I'm looking forward to getting this up and running!

  • #21 Cory Gough said 2014-01-30T02:47:33Z

    @Miguel: When I created my db I ran: DATABASE_URL=mysql://apps:apps@localhost/apps ./db_create.py I think I was supposed to do: DATABASE_URL=mysql://apps:<my_password>@localhost/apps Now when I run the application it tries to store the first user, but says this in the log: ProgrammingError: (ProgrammingError) (1146, "Table 'apps.user' doesn't exist"). Do you know what is causing this?

  • #22 Miguel Grinberg said 2014-01-30T05:36:36Z

    @Cory: did the db_create.py command finish without errors? It appears it did not create the tables. Note that using "apps" as a password was for example purposes, you are correct in that you need to replace it with a more secure password.

  • #23 Cory Gough said 2014-01-31T01:44:54Z

    @Miguel: db_create.py ran without errors. When I setup mysql I used my own password. But when I ran db_create.py I set the DATABASE_URL to the apps password in your example. I didn't realize I needed to change it until later (not sure if that is the issue). Is there a way to recreate the db or something else that can be done? When I rerun db_create.py is says: migrate.exceptions.DatabaseAlreadyControlledError

  • #24 Miguel Grinberg said 2014-01-31T01:54:19Z

    @Cory: Delete the database and recreate it using your favorite mysql admin tool. It seems it is currently in a bad state where the migration information is there but the tables are not.

  • #25 Cory Gough said 2014-02-01T19:54:24Z

    @Miguel: Turns out this was a simple one! When setting the DATABASE_URL I needed to export it so that the db_create.py script would see it. Otherwise, it didn't see it and was just creating a sqlite db. The good news is I just made my first post on my VPS! Thank you for helping me through your tutorial and taking the time to make it! I look foward to your book and continuing to expand on my own microblog! PS(if it helps anyone) I also made the following changes from the github code to get it working for me: Add "import os" to runp-mysql.fcgi flask/bin/pip install MySQL-python; Because it was not in the setup script. flask/bin/pip uninstall Flask flask/bin/pip install Flask==0.9 Reference https://github.com/miguelgrinberg/microblog/issues/4

Leave a Comment