Sunday, August 25, 2013

How to Setup a Tornado Server, with Web Sockets Support, on Production, Behind NGINX

Ubuntu 13.04, Digital Ocean

I am not an expert on servers setup, so if you see anything I could have done better please leave a comment and I'll try it out on my server and improve this tutorial. So far this setup works very nice for me. Ok, let's get this started.

Step 1 - Create User

First things first, let's ssh into the server and because this is the first time, i will do it as a root user:

$ ssh root@<your server ip>

Say yes to continue connecting, enter your password and we are in.

On Digital Ocean, the password that you will receive is a default and therefore, first thing, we need to change that:

$ passwd

Next thing, i will create a user for myself:

$ adduser dev

Define new password for the user and you can leave everything else blank or fill it in, as you wish.
Next step is to give my new user root privileges:

$ visudo

Find the section called 'User privilege specification', it will look something like this:

# User privilege specification

under it add your user, granting all root privileges:


Enter Cntr x to exit and save the file.

Step 2 - Configure SSH

Although this step is optional, it is very much advisable to do this for security reasons. 
Caution! Misplacing following information can bar your entrance from your own server!
That said, we are all grown people and if you can't keep your login creds secure and in mind(or securely backed up somewhere), then this is no business for you.

Open the configuration file:

$ nano /etc/ssh/sshd_config

Find the following sections and change the information as follows:

Port 5050
Protocol 2
PermitRootLogin no

1. Port -  default is 22. You can change this to anything between 1025 and 65536. Whatever you choose, remember it (!) you will use it to login in the future. 
2.  PermitRootLogin - change this form 'yes' to 'no'. This will disallow root login. From now on, you will be able to login with your user only. I also found out, that on Digital Ocean, you will still be able to login with root user through a terminal on

Next, add following lines at the bottom of the config file:

UseDNS no
AllowUsers dev

AllowUsers will allow only this users to login to the server. Don't forget to substitute 'dev' for whatever you called your user.

Finally, reload the ssh to make the changes take effect:

$ reload ssh

To test this, open new terminal and try to login with your user this time:

$ ssh -p 5050 dev@<your server ip>

If something went wrong and your root access terminal was closed and you can't login with your user, you can go back to and login with root using their terminal and go over the sshd_config file, fixing whatever might have gone wrong.

Step 3 - Setup SSH Login to Server

Creating a new user for yourself, and a new password, both for root and your new user, and preventing root login access, and changing the port, and defining allowed users to login to the server, is all very well and good...BUT! There is always a but ;) Your password can still be very well cracked by brute force attack. Here come into picture SSH keys that are nearly impossible to guess by brute force alone.

Make sure that you have a .ssh/ (chmod 700) folder under /home/dev/ (a.k.a ~ ) on your server and chmod 600 for .ssh/* (for all files inside). And now let's make our server even more secure, and easier to access from your computer, by creating a pair of ssh keys.

On your computer

$ ssh-keygen -t rsa
$ Enter file in which to save the key (/demo/.ssh/id_rsa): ~/.ssh/tornado_server_rsa
$ Enter passphrase (empty for no passphrase): 

I usually leave this empty, so hit enter.
Now let's add this key to the list of known identities:

$ ssh-add ~/.ssh/tornado_server_rsa

Verify that it went ok:

$ ssh-add -l

Should show the new key.
At this point you should have a pair of public/private keys for your server. What is left is to pass the public key to the server. 2 ways to do this.

$ ssh-copy-id dev@<server ip>


$ cat ~/.ssh/ | ssh -p 2012 dev@<server ip> ''cat >> ~/.ssh/authorized_keys''

No matter what you chose, in both cases you should see some communication going on, asking you if you are sure you want to continue connecting and such. The usual ssh stuff.

After this step you should be able to login to your server using ssh keys without the prompt for password. A much more secure and easier way.

Step 4 - Installing Necessary Packages

$ sudo apt-get install python-setuptools libcurl4-gnutls-dev libexpat1-dev gettext libz-dev libssl-dev build-essential git-core git nginx keychain

$ sudo easy_install pip
$ sudo pip install tornado
$ sudo pip install supervisor

Setting up keychain (optional)

This step is good if you keep the code under your /home folder.
What keychain does for us is help us manage our ssh keys. It drives both ssh-agent and ssh-add, and can maintain a single ssh-agent process across mutliple login sessions. It means that you will have to enter your passphrase only once each time your machine is booted.

Add following to your .bashrc:

### The --clear option make sure Intruder cannot use your existing SSH-Agents keys
### i.e. Only allow cron jobs to use password less login
/usr/bin/keychain --clear $HOME/.ssh/*_rsa
source $HOME/.keychain/$HOSTNAME-sh

If you didn't create .bash_aliases file yet, now is the time to do so:

$ touch ~/.bash_aliases
$ nano ~/.bash_aliases

Add following lines to the file:

alias ssh='eval $(/usr/bin/keychain --eval --agents ssh -Q --quiet ~/.ssh/*_rsa) && ssh'

This checks to see if keychain is runnig and if not starts it, adding all your rsa keys under .ssh folder.

alias git='eval $(/usr/bin/keychain --eval --agents ssh -Q --quiet ~/.ssh/*_rsa) && git'

Why aliases?
Keychain is tacking itself to every command you might need to use keys. In our case ssh and git. First time you run those command, you will enter the passwords and for afterwards keychain will do this for you.

Step 5 - Get the Code

Pulling the code from Github is much easier done by ssh. So, let's create a pair of keys to use with git. 

$ sudo ssh-keygen -t rsa

Enter a passphrase if you want, otherwise hit enter.
I usually save the keys in named files:

$ Enter file in which to save the key (/demo/.ssh/id_rsa): ~/.ssh/github_rsa

Why sudo ssh-keygen? Because i'm going to save the code under /srv/ folder which is accessible for write only to root users, therefore usual 'git clone' will not work. You can create an ssh key without sudo, but then it won't work with 'sudo git clone' that you will have to use to pull the code.

Now let's create a folder that will hold the code:

$ sudo mkdir /srv/www
$ sudo mkdir /srv/www/myapp

Go to the folder that will hold the code, in my case that is myapp/ :

$ cd /srv/www/myapp
$ sudo ssh-agent /bin/bash
root@server$ git clone git@....

Cntr+d to exit.

Step 6 - Setup NGINX

Create a user for nginx:

$ sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx
$ sudo nano /etc/nginx/nginx.conf

Configure the file according to this gist.

The server configuration, under http {}, you can alternatively add to 
/etc/nginx/sites-available/myapp (you will have to create it first), then run:

$ sudo ln -s /etc/nginx/sites-available/myapp /etc/nginx/sites-enabled/myapp

In both cases, delete default server block:

$ sudo rm -r /etc/ngingx/sites-enabled/default

Step 7 - Configure Supervisor

Supervisor will help you keep your tornado server running.

$ sudo mkdir /etc/supervisor/conf.d/tornado.conf
$ sudo nano /etc/supervisor/conf.d/tornado.conf

Inside this file add following lines:

command = python /srv/www/myapp/ --port=8001
stderr_logfile = /var/log/supervisor/tornado-stderr.log
stdout_logfile = /var/log/supervisor/tornado-stdout.log 
autostart = true
autorestart = true

command = python /srv/www/myapp/ --port=8002
stderr_logfile = /var/log/supervisor/tornado-stderr.log
stdout_logfile = /var/log/supervisor/tornado-stdout.log 
autostart = true
autorestart = true

command = python /srv/www/myapp/ --port=8003
stderr_logfile = /var/log/supervisor/tornado-stderr.log
stdout_logfile = /var/log/supervisor/tornado-stdout.log 
autostart = true
autorestart = true

command = python /srv/www/myapp/ --port=8004
stderr_logfile = /var/log/supervisor/tornado-stderr.log
stdout_logfile = /var/log/supervisor/tornado-stdout.log 
autostart = true
autorestart = true

Cntr+x, save and exit the file.
Make sure that all log file directories exist, if need be create them.

Step 8 - Start NGINX and Supervisor

$ /etc/init.d/nginx start
$ sudo supervisord

To validate that your app is up and running, access your domain. If you only see nginx default template, then something is wrong in your nginx configuration and nginx is not forwarding the requests to your app. In that case, check nginx error log and tornado-stderr.log.

Monday, August 12, 2013

Django Class Based Views and Inline Form Sets

Django 1.5.1, Python 2.7.5

Ok, so you moved on to working with class based views, got a usual form going on, and everything went smooth, but now you want to use more than one form on the same page and, even better, one of the forms is actually a form set. Cool! Here is a real example of how I made it work on one of the projects I was working on.

For the sake of the example, a few words of what I have and what I want to achieve.
I want to let my users manage theirs sponsors. Simple in itself and a basic need. My users want to be able to define a sponsor and it's sponsorship dates. From this I have 2 models: Sponsor and Sponsorship, connected with one to many field, aka ForeignKey.
Let's say we defined the models, and take  a look at :

from django.forms import ModelForm
from django.forms.models import inlineformset_factory

from models import Sponsor, Sponsorship

class SponsorForm(ModelForm):

    class Meta:
        model = Sponsor


class SponsorshipForm(ModelForm):
     class Meta:
        model = Sponsorship

SponsorShipsFormSet = inlineformset_factory(Sponsor, Sponsorship,
                                            form=SponsorshipForm, extra=2)

Notice that I defined the formset in and not inside the view.

Ok, so I have the models, I have the forms, now I need a view to work it all out.

class CreateSponsor(SponsorMixin, CreateView):
    form_class = SponsorForm
    template_name = 'sponsor_form.html'

    def get_context_data(self, **kwargs):
        data = super(CreateSponsor, self).get_context_data(**kwargs)
        if self.request.POST:
            data['sponsorships'] = SponsorShipsFormSet(self.request.POST)
            data['sponsorships'] = SponsorShipsFormSet()
        return data

    def form_valid(self, form):
        context = self.get_context_data()
        sponsorships = context['sponsorships']
        with transaction.commit_on_success():
            form.instance.created_by = self.request.user
            form.instance.updated_by = self.request.user
            self.object =

        if sponsorships.is_valid():
           sponsorships.instance = self.object

        return super(CreateSponsor, self).form_valid(form)

    def get_success_url(self):
        return reverse('sponsors')

Let's go over what is going on here. My main form is SponsorForm and the view will take care of that pretty much by itself. Notice get_context_data(), on get it will create an unbound SponsorShipsFormSet and on post instantiate it with the data in self.request.POST. But if you remember to instantiate inline formset you also need, on get and on post, to define an instance, which is the main model instance. You don't see it here because it's a create new instance view, there is no instance yet to which i want to bind the formset.
Next is form_valid method. I get the context of the view, and extract from it the formset, which now holds the data the user entered. I make sure that i first save the main form - SponsorForm, that will create an instance i need for the inline formset. Notice that i call is_valid() method on sponsorships, that's because the class doesn't take care of it for me (this is SponsorForm class, for it, it happens automatically). All that is left to do is to bind the formset to this new instance and we are done ;)

Here is a gist for better readability.