jejt / jmons

Engineering Tommorrow

Category Archives: Cloud

Continual Integration and Continual Deployment and the Cloud


One of the big buzz phrases at the moment seems to be Continual Integration Development. If you’re developing and wanting to deploy ‘as the features are ready’, and you have a cloud you have two main options, which both have pros and con’s but:

New Image per Milestone

Most cloud systems work by you taking an ‘image’ of a pre-setup machine, then booting new instances of these images. Each time you get to a milestone, you setup a new image, and then setup your auto scaling system to launch these instances rather then the old one, but you have to shut down all your old images and bring them up as new ones.

Pro’s: The machines come up in the new state quickly.
Con’s: For each deployment, you have to do quite a bit more work making the new image. Each deployment requires shutting down all the old images and bringing up new replacements.

SCM Pull on Boot

Make one image, and give it access to your SCM (i.e. git / svn etc). Build in a boot process that brings up the service but also fetches the most recent copy of the ‘live’ branch.

Pro’s: You save a lot of time in deployments – deployments are triggered by people committing to the live branch, rather then by system administrators performing the deployments. Because they are running SCM, updating all the currently running images is as simple as just running the fetch procedure again.
Con’s: You do need to maintain two branches: a live and a dev branch, and merge (some SCM’s might not like this). Also, your SCM hosting has to be able to cope with when you get loads (i.e. when new computers get added). Your machines come up a little slower as they have to do the fetch before they are usable.

I opted for the second route: we use Git, so we can clone quickly to the right branch. We’ve also added in git hooks that make sure any setup procedures (copying the right settings file in) are done when the computer comes up. Combining this with a fabric script to update all the currently running boxes is a dream.

What Cloud Computing System to Use?


So you’re sitting at work, and you have to build a new system, and for once you don’t have any previous code or language forcing you to write one way or another, and you know this is going to get big – maybe not Twitter or Google big, but certainly big enough to give you a good old headache. The big question becomes what technology to use. Firstly, I apologise that this is early 2011, so if you’re reading this in two or three years (or even six months) then the technology will all change again – I’m not planning on updating this particular post as the stuff changes, but I might make new ones.

So the first decision to make is what Cloud Computing system you’re going to use – are you doing lots of queries, or just a few queries and lots of processing. I’m presuming the first one, but the latter one is quite interesting – it deals with universities and researchers trying to run on massive data sets and producing reports.

Your main contenders are:

  • Some collection of *nux or Windows servers
  • Proprietary cloud compute services

The first category might mean more work for you and your sysadmins – it really does point towards the requirements of a sysadmin but gives you a lot more flexibility in your choice of languages and systems, where as the latter might mean that you are able to do with out those, and also (depending on the service) have access to a lot of tools and power without having to use any other third party services.

The main propietary services at the moment seem to be:

  • Google’s Apps
  • Microsoft Azure

Now – both of these platforms are quite seductive, they have a lot of benefits – mainly that you don’t have to be a sysadmin to deploy and maintain the system, that you can access quite complicated things such as shared and persistent storage, caching and database pooling without having to actually spend days reading various manuals for everything.

The downside though? You are locked to one provider and their billing methods – Apps has a very strange billing mechanism to do with the number of users (Which if you’re producing something for a lot of users, that might be very expensive), and because you’re locked into that service provider, there isn’t another provider who you can goto for alternative pricing, and because of this, I think a lot of smaller businesses make a commercial decision to go with the more traditional style hosting.

Traditional clusters (such as provided by Amazon and Rackspace) seem to provide a collection of tools along side. Content distribution networks for static content such as images and javascipt, as well as tools for monitoring and automatically scaling the systems. The advantages of the traditional route is that its easy to run up a local copy at your location and develop away, which means when you are looking at doing architectural changes, these are much easier to stage to live.

In my opinion, it seems that commercial reasons for going down cloud hosting of systems which are ‘traditional linux/windows’ boxes have massive advantages, but do require more systems-administration work.