OpenVZ how-to: Your own free VPS infrastructure in 10 minutes

As I wrote in part 1 of this series of posts, creating your own dedicated Linux “cloud” for free on a root server is easier than you might think. It for sure has been easier for me than I thought before actually doing it. In the first part of this post I’ll explain my choices, but if you want you can just skip to the how-to part.

My first step was to choose a virtualization technology. There are many free open source solutions that allow you to create and manage a virtualization infrastructure on a Linux host; these solutions can broadly be divided into two different approaches:

  • OS Level (aka kernel level) virtualization
  • Full virtualization

To simplify things, we can say that with full virtualization technologies you can have virtual machines that run any operating system that would work on the virtualized hardware, while with kernel level virtualization you can only have containers (which aren’t full virtual machines) that are compatible with your kernel.

In my case I was only interested in Linux virtual servers on a Linux host, so both approaches would have worked. On a performance and resource usage level, kernel virtualization imposes less overhead, but doesn’t guarantee the same level of isolation between the virtual servers. On my own private cloud isolation isn’t so much a concern, so I decided that kernel level virtualization would be my first choice, if it would result easy enough to set up and manage.

OpenVZ is the most used solution for free kernel level virtualization on Linux – as a matter of fact, most low cost VPS solutions offered on the internet are based on it. So I did my research about how to install and manage it. As usual for free open source technologies, there are a plethora of solutions; in the end I chose two possible options as the most promising:

  • Using a full stack solution, either Proxmox or OpenNode
  • Installing OpenvVZ and a web control panel myself

I would have liked to use one of the full stack solutions, both because of the ease of installation they promise, and because with those it would have been easier to also use full virtualization, if needed, at a later time. But both Proxmox and OpenNode offer an ISO installer as their main option, and I couldn’t use that on my hetzner EQ root server; moreover, even if I could, I then would have had to configure the network myself. This is why I decided to install OpenVZ myself on one of the pre-configured Linux OS images that were available for my dedicated server.

The Linux version I know best is Ubuntu Server, as this is what I used on all my own virtual servers until now. The second one is CentOS, which is the default at my day job – but there I don’t do system administration. So I first tried to install OpenVZ on Ubuntu server, and then on CentOS – installing a new OS image in my dedicated server only takes a few minutes, so making different tries was very easy.

From my search on the internet and my own attempts I found out that installing OpenVZ on either of those two OSs isn’t as easy as it could be – or, at least, as easy as I wished it to be, as I found some problems in the repository versions of OpenVZ (these of course could have been fixed at the time of reading this article). From what I read, Debian is the most supported Linux flavor for OpenVZ, so that’s what I tried next.

The recipe

Installing, configuring and using OpenVZ on Debian was really very easy. The following are the detailed steps I used. After writing a draft of this post I tried them and it took less than 10 minutes from start to finish.

  1. Start with a clean, minimal Debian 6.0 installation – I used the pre-built image I found for my hetzner EQ server
  2. Login as root
  3. Install OpenVZ with its management tools and its special kernel, 64-bit version (this is good for both AMD and Intel 64 bit CPUs):
    apt-get install linux-image-openvz-amd64 vzctl vzquota vzdump
  4. Create a symlink to easily find the OpenVZ files:
    ln -s /var/lib/vz /vz
  5. Edit sysctl config
    nano /etc/sysctl.conf

    to make it like this:

    ### Hetzner Online AG installimage
    # sysctl config
    net.ipv4.ip_forward=1
    net.ipv4.conf.all.rp_filter=1
    net.ipv4.icmp_echo_ignore_broadcasts=1 

    net.ipv4.conf.default.forwarding=1
    net.ipv4.conf.default.proxy_arp = 0
    kernel.sysrq = 1
    net.ipv4.conf.default.send_redirects = 1
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.eth0.proxy_arp=1

  6. Use the new options:
    sysctl -p
  7. Restart the server:
    reboot
  8. Download a pre configured container image for my VPS – I was interested in ubuntu server:
    cd /vz/template/cache
    wget http://download.openvz.org/template/precreated/ubuntu-10.04-x86_64.tar.gz

    You can find more images here: http://wiki.openvz.org/Download/template/precreated

At this point you are ready to use your VPS. But I wanted to make management easier, so I decided to immediately install a web panel to control OpenVZ. My choice was OpenVZ web panel, which you can install with these commands:

cd /root
wget -O – http://ovz-web-panel.googlecode.com/svn/installer/ai.sh | sh

There you go! Your web panel is already started and you can find it at port 3000 of your server.

To create your first VPS at this point you only need an IP from your provider and use the web panel:

  1. Open http://<yourip>:3000/ in your browser
  2. Login using admin/admin – don’t forget to change the default password!
  3. Click on “Physical Servers->localhost”
  4. Click “Create virtual server”
  5. Input a server ID – use a number over 100, for example 101 – IDs from 1 to 100 are reserved and you’ll have problems if you use one of them
  6. Choose your OS template – you’ll find only ubuntu 10 if you followed my instructions
  7. Choose your server template or leave the default
  8. Type the IP you got from your provider (I’m talking about an IPv4 here, but on a next post I’ll write about using IPv6)
  9. Choose a host name, like “myhostname”
  10. Click on “Additional settings” and choose how much RAM, disk and CPU you want for the new machine
  11. Choose a password and click “create”

That’s it! Your VPS is ready to use. Just connect to it via ssh, and you can use it as you wish, just the same as if you bought it from any VPS provider. You can also clone your new server with minimal effort from the web panel: You just need to select it, click Tools->Clone, type the new IP for the clone and go on. If you don’t have one more public IP from your provider, and you don’t need it, you can use a private one like 10.10.10.2

If you’re going to do anything serious with your virtual cloud of VPSs, though, there are at least two more things you’ll want to set up: Backups and a firewall.

A backup strategy is really important here, because you’re working on your own physical server now, and if it fails and any data is lost you’re on your own. Moreover, if you are using a low cost dedicated server like mine, you should always keep in mind that they aren’t based on highly redundant, fault tolerant server hardware. There is still software RAID 1 for the disks, but that’s about it for redundancy.

The good news is that backing up a whole VPS is very easy with OpenVZ. There are three different ways to do it:

  • Simple dump – long downtime: The VPS will be stopped while the backup is going on
  • Suspend mode – very short downtime: The system will suspend the VPS and resume it a few seconds later
  • LVM2 snapshot – no downtime: A snapshot is created without suspending the VPS at all

The third method, of course, is the best one. But it requires you to configure your file system in a special way, and for most situations the few seconds of downtime of the suspend mode are acceptable – they are for me. So I went this way.

To backup a running VPS (container) from the command line (or a batch script) with the second method you just type:

vzdump –suspend NNN

where NNN is the ID of the server you want to backup. This will create a file named vzdump-openvz-NNN-YYY_MM_DD-hh_mm_ss.tar in the /vz/dump folder.

By restoring this backup (with vzrestore) you get back your full VPS in the state it was when backed up. You just need to move the backup file to a secure location (preferably after compressing it – with bzip2 you could have a 75% reduction of the file size); In my case I got a 100 GB FTP backup space from hetzner where I move my backups.

To protect your server/s you should also set up a firewall, at least for the host: By gaining control of it, an hacker could gain control to all your VPSs. This isn’t OpenVZ specific though, so I’ll not cover it in this post.

Conclusion

Setting up my own private “virtual cloud” has been much easier than I thought – and much cheaper. The most time consuming part was finding the information on various wikis, blog posts and other sources. With this post I hope to give something back to the community by packaging the updated information in an hopefully easy to use format, even if I’m aware that all this will become outdated at the speed of the internet.

Cheers

My own dedicated “cloud” with 24 GB of RAM for 89 Euro (127 USD)/Month – part 1

As I already explained in my last post, hashtagify’s servers need to keep most of the data they handle in memory: This is the only way to answer an AJAX request about an hashtag so quickly that the user perceives the interaction as instantaneous and engaging. 

The downside to this, of course, is that I need servers with a lot of memory, and those usually don’t come very cheap. To keep the costs down I had to find a cheap “cloud” Virtual Private Server provider that would allow me to add RAM without also having to pay for more CPU, disk and bandwidth that I didn’t need: With my usual otherwise excellent VPS provider, linode.com, the amount of memory I needed would have been just too expensive for my budget.

After some search I found what I was looking for in glesys.com. For the first month or so this solution worked very well, as I could increase the RAM as needed without having to restart my server, and I was even able to double it at one point for just one hour to make a very memory intensive calculation, paying for just that hour of use. But hashtagify is becoming a very memory hungry application, especially now that I’ve also started collecting users and URLs data, and I wanted to try an even cheaper solution.

During my search for a cloud service I had stumbled upon a couple of tempting dedicated unmanaged server offers with boatloads of RAM and a comparatively cheap price; namely, they were from server4you.comgiga-international.com and hetzner.de – curiously, they’re all from German-speaking countries, but who cares. At first I went with glesys anyway because I didn’t need all that memory right away, and because I knew VPSs better.

But now, having an actual need for more memory, I went back to my bookmarks and studied those offers in depth. I found out that hetzner has got very good reviews and, even if I don’t like setup fees, I decided to try their EQ8 offer: For 89 euro/month, and 150 euro setup, you get a quad-core I7 CPU, 2x 1500 GB disks, “unlimited” bandwidth – that is, unlimited if you stay under 5 TB – and, most importantly for me,  a whopping 24 GB of RAM.

With those resources I could host all my different websites needing a VPS, both the Ruby on Rails ones like mortgagecalculator3.com and hashtagify.me which uses node.js, and still have a lot to spare for future needs. I could see only two drawbacks:

  • The need to manage the virtualization infrastructure myself
  • The risk connected with relying on a physical server that, of course, could break at any time.

After a quick search about open source virtualization technologies, I came to the conclusion that the first problem was easy enough to solve. There were many solutions to try that promise an easy setup; I was sure that at least one of them would have worked out.

The second one is inherently more serious. Real cloud services, like google app engine, are based on the idea that your software runs on unidentified, fully managed redundant hardware that you don’t have to worry about at all. A dedicated server is exactly the opposite. But most “cloud” offers you find on the internet are nothing more than VPSs with more flexible management services, prices and possibly an API (hence the quotation marks I use when referring to this flavor of “cloud”).

With these “cloud” offers you still get the full management of the physical host (server) by the provider, and in case of server failure they should guarantee you a smooth transition to a new server,  but what you get is still just a flexible VPS. The fail-over to a different host is something that wouldn’t be too difficult to organize yourself with a sensible backup policy, or, even better, a clusterized and redundant DB (if your data, as it very often is, resides in a DB that offers that kind of possibility).

Redundancy based on commodity hardware is exactly, from what we know, how Google manages its own cloud services. And the cheap dedicated server offers I found are based on commodity hardware, not on highly redundant, fault tolerant “server” hardware – I think that’s the main reason for their cheapness, considering that the network availability and service, at least for what I could read online and see myself for now, is very good at least for hetzner.

So, if it was possible to easily migrate VPSs from a broken server to a new one with a minimum downtime, or, even better, to have synchronized multiple VPSs on different physical servers, you would get the perfect solution for bootstrapped startups (like hashtagify) or anybody who needs some VPSs with a lot of resources for a very cheap price: This software redundancy would make up for the relatively low MTBF of the commodity hardware.

As it turned out, even with free and open source technologies migrating a VPS from one physical server to a new one is actually very easy. And, at least if you use Redis (like hashtagify.me does), it is also very very easy to have many synchronized DB instances both for scalability (if you need it) and hot failover.

So in the end I ordered my 24 GB dedicated server last Saturday, and on Monday I had my login data. I expected to use a fair amount of time to learn how to set everything up and move hastagify to my own “cloud”, but after only maybe 6/7 hours stolen here and there from my vacation everything was up end working fine. There is still enough work I’ll have to do to also migrate my Ruby on Rails applications to the new server, but being already able to clone a new instance of the hashtagify server with just a few clicks and for free is a very satisfying, albeit entirely geeky, sensation!

The way I did this is what I’ll write in part 2 of this already too long article. I guess that, when done in the right sequence, all the steps necessary to set up a virtual infrastructure like hashtagify’s would only take little more than half an hour, starting from scratch and using only free open source software. It’s not at all difficult, but as usual having to wade among all the different possibilities and a dozen of half-outdated how-tos is a pain, so I hope to relieve it from other people – at least until my own post will become outdated.

Cheers from Russia.

Disclosure: I asked hetzner if they have a referral program, but they don’t. If they had, I would have applied to it prior to publishing this post ;)

Redis or: How I learned to stop worrying and love the unstable version

Numbers on Twitter are huge. Even with just a 1% sampling of tweets, even working only on tweets with an hashtag, you reach big numbers very fast.

If you want to to offer a highly interactive and very responsive experience you’ll need to keep most of those numbers in memory, and well indexed. So you’re going to need a lot of RAM: I was aware of that when I decided to use Redis as my main DB, and I purposefully chose a parsimonious data structure and a “cloud” – whatever that means – solution that would allow me to increase the amout of RAM when needed without breaking my shoestring budget.

What I didn’t know was that, with just over one GB and a half of data, as measured by Redis itself, I was going to need double that amount of memory not to have my app crash (which happened twice yesterday). Multiply that amount for two servers for a minimum of redundancy, and you’ll need 6 GB of precious RAM for just 1.3 million tags worth of data…

In the past I had found that, after a restart, Redis would use much less memory than before. So, after the obvious quick fix to get back the service up and running – increasing the amount of RAM – I tried to do just that, using the load balancer to keep the service up while restarting the two istances. But the gains from a restart were very short lived, and I didn’t want to pay for that amount of memory and to have to worry that even that wouldn’t be enough during my upcoming 2 weeks trip to Russia.

Checking the info command output after the restarts, I noticed that what was increasing very fast after the new start was the mem_fragmentation_ratio, which started around 1.3 but quickly reached and surpassed 2. After a quick search, it turned out that there is a known memory fragmentation problem with Redis on Linux, caused by malloc.

I had already noticed the disquieting field in the past, but hadn’t realized just how bad the situation was. A ratio over two to one looked really too much to me, especially when a quick search on the internet revealed that a ratio of 1.4 was already considered abnormal.

Delving deeper into the issue, I found out that the fragmentation problem is especially bad with sorted lists, and hashtagify uses a lot of sorted lists: As a matter of fact, of the 4 million keys used right now, more than 90% are sorted lists!

Different versions of Redis were made to find a solution to this issue, but I didn’t find a “definitive” post/article/howto about which is the best one. On the other hand, I noticed that version 2.4, which is not final yet, is incorporating one of those solutions, the use of jemalloc, so I tried it on one of my servers.

Things immediately improved; it’s early to be sure, but for now after many hours the fragmentation ratio is only 1.02. Halving my memory requirements with just a simple version upgrade is not bad at all! After all, I should be able to leave for Russia without the fear of being bankrupt, or with a frozen server, at my return :)

I just wished it was easier to learn about this before, but this is the nature of open source software. Now I just hope that this post will help at least someone else not making my mistake of using the “official” 2.2 release.

Cheers.

UPDATE 2011-06-25

After 6 days, 2 hours and 4 minutes of uptime, during which 33,245,499 commands where executed, the fragmentation is still 1.02. I can now confirm that jemalloc rocks :)

Greetings from Russia