One of the big buzz phrases at the moment seems to be Continual Integration Development. If you’re developing and wanting to deploy ‘as the features are ready’, and you have a cloud you have two main options, which both have pros and con’s but:
New Image per Milestone
Most cloud systems work by you taking an ‘image’ of a pre-setup machine, then booting new instances of these images. Each time you get to a milestone, you setup a new image, and then setup your auto scaling system to launch these instances rather then the old one, but you have to shut down all your old images and bring them up as new ones.
Pro’s: The machines come up in the new state quickly.
Con’s: For each deployment, you have to do quite a bit more work making the new image. Each deployment requires shutting down all the old images and bringing up new replacements.
SCM Pull on Boot
Make one image, and give it access to your SCM (i.e. git / svn etc). Build in a boot process that brings up the service but also fetches the most recent copy of the ‘live’ branch.
Pro’s: You save a lot of time in deployments – deployments are triggered by people committing to the live branch, rather then by system administrators performing the deployments. Because they are running SCM, updating all the currently running images is as simple as just running the fetch procedure again.
Con’s: You do need to maintain two branches: a live and a dev branch, and merge (some SCM’s might not like this). Also, your SCM hosting has to be able to cope with when you get loads (i.e. when new computers get added). Your machines come up a little slower as they have to do the fetch before they are usable.
I opted for the second route: we use Git, so we can clone quickly to the right branch. We’ve also added in git hooks that make sure any setup procedures (copying the right settings file in) are done when the computer comes up. Combining this with a fabric script to update all the currently running boxes is a dream.
So you’re sitting at work, and you have to build a new system, and for once you don’t have any previous code or language forcing you to write one way or another, and you know this is going to get big – maybe not Twitter or Google big, but certainly big enough to give you a good old headache. The big question becomes what technology to use. Firstly, I apologise that this is early 2011, so if you’re reading this in two or three years (or even six months) then the technology will all change again – I’m not planning on updating this particular post as the stuff changes, but I might make new ones.
So the first decision to make is what Cloud Computing system you’re going to use – are you doing lots of queries, or just a few queries and lots of processing. I’m presuming the first one, but the latter one is quite interesting – it deals with universities and researchers trying to run on massive data sets and producing reports.
Your main contenders are:
- Some collection of *nux or Windows servers
- Proprietary cloud compute services
The first category might mean more work for you and your sysadmins – it really does point towards the requirements of a sysadmin but gives you a lot more flexibility in your choice of languages and systems, where as the latter might mean that you are able to do with out those, and also (depending on the service) have access to a lot of tools and power without having to use any other third party services.
The main propietary services at the moment seem to be:
- Google’s Apps
- Microsoft Azure
Now – both of these platforms are quite seductive, they have a lot of benefits – mainly that you don’t have to be a sysadmin to deploy and maintain the system, that you can access quite complicated things such as shared and persistent storage, caching and database pooling without having to actually spend days reading various manuals for everything.
The downside though? You are locked to one provider and their billing methods – Apps has a very strange billing mechanism to do with the number of users (Which if you’re producing something for a lot of users, that might be very expensive), and because you’re locked into that service provider, there isn’t another provider who you can goto for alternative pricing, and because of this, I think a lot of smaller businesses make a commercial decision to go with the more traditional style hosting.
Traditional clusters (such as provided by Amazon and Rackspace) seem to provide a collection of tools along side. Content distribution networks for static content such as images and javascipt, as well as tools for monitoring and automatically scaling the systems. The advantages of the traditional route is that its easy to run up a local copy at your location and develop away, which means when you are looking at doing architectural changes, these are much easier to stage to live.
In my opinion, it seems that commercial reasons for going down cloud hosting of systems which are ‘traditional linux/windows’ boxes have massive advantages, but do require more systems-administration work.
So recently a friend at work (the esteemed DJ Hedflux) had a problem with his speakers – he has a pair of powered speakers, and he outputs sound down a normal stereo cable into the speakers. Being a professional DJ, nothing in his system is particually cheap (he’s not spending £100 on cables, but he’s not buying £1.50 speaker modules from the market) so we can quickly rule out shoddy work and bad connections inside of the devices.
Anyways, he worked out that it was a combination of his computer and his touch lamp causing it – doing things on his computer that required lots of processing, and having the dimmer lamp “dimmed” caused the noise to appear and dissapear.
This is very common, and also, very easy to fix.
How to fix:
The solution is very simple, and very cheap. You’ll need a ferrite ring, which you wind the audio cable in just a couple of times, like in the above picture.
Maplins sell these for a couple of pounds, usually find them in the Radio section:
Image stolen from http://www.gbrcaa.org/ntoa/Filters,%20Chokes%20and%20OIs.htm on which site you can also find more information about other kinds of chokes.
In Hedflux’s case, its possible that his computer internals arn’t all grounded to the case correctly, or the case to the power supply, so its generating more electrical noise then ideal. The dimmer lamp however is a common source of Electromagnetic Interferance (EMI).
Because he’s using a set of powered speakers, the loudspeaker cable is acting as an antenna, and the amplification circuit is having a side effect of acting as a little radio, which is picking up the EMI from the computer and the lamp, and turning it into the annoying hissing noise.
The ferrite bands that he added to the audio cable basically change the frequencies of that cable, effectivly filtering the annoying frequencies out.
Why don’t these cables come with the bands installed? Well, the point is that it dosn’t STOP the frequencies, it shifts the resonance frequency of the cable. The cable and speakers will still produce noise if the EMI comes in on a different frequency, so it was just bad luck that the EMI in his room was the same frequency as the cable / speaker setup.
So I’ve started on a tinymush server, and its quite interesting scripting objects together, so here is my first object (which is actually version two of it) because the first version was a little cumbersome. Please note this is ready for copy and pasting into the mush (I’ve escaped the ; with backslashes)
@desc Board=An oak framed chalk board ready to be written on. It contains the words of wisdom from the teachers, or doodles from the students.[ifelse([hasattr(me,text)],It currently reads: [eval(me,text)],It is currently blank)]. Feel free to "write on board z" or "erase board"
@set BOARD = COMMANDS
&C_WRITE Board=$write on board *:&text me=%0\;pose has just been written on
&C_ERASE Board=$erase board:&text me\;pose causes clouds of dust to rise as it is erased
It has two commands “write on board x” and “erase board” which reset an attribute &text. I suspect I should change the pose lines to emit lines. But apart from that, I think its pretty cool.
Ok, so theres this massive need I’ve found for good distributed bug tracking which is actually platform independent. And this brings me around to my gripe of the week. Platform Independence.
Firstly, I classify myself as an indie developer – that means that I use open source and free software tools in my tool chains, and I use developer / free to use sdk’s to develop applications.
So I’m not able to afford any of the larger programs and tools which are used – things like the Visual Studio or the Team system. Things like Bug Management and Tracking, things like support and ticketing. To some extent things like release managment.
And why would I want to? There are plenty of open source, free, and community based tools out there which do of these things. The problem is however is that they often don’t work across all platforms. They often have requirements or don’t “fit together properly”.
Last night I found a quote by Doug Vargas (I dont know who the guys is or I would have linked to it):
“It’s easy to cry ‘bug’ when the truth is that you’ve got a complex system and sometimes it takes a while to get all the components to co-exist peacefully”
So with this in mind, this is where things begin to get a bit more tricky. The first problem is that I don’t work on one project – I don’t have a single project which is “my baby” I dont have a single passion in life which takes my complete and utter concentration, instead I have a few – simple single use tools which I reuse in other projects. I have a collection of libraries and tasks and I treat each one as a new project and I keep them seperate.
So when it comes to using some of these tools – they are really only for one project. For instance, trac – awesome huh? But I dont want to have to configure one for every application that I write. And this is where some of the harder hitting application come in, but there really isn’t much for the small indie people.
Then we find that half these tools only work in linux. Or windows. Or they work in Cygwin and they require some bizaare libraries. Its very easy to write a tool which does what *you* want and then not worry about everyone else – in fact this is probably the only way that the tool gets written.
So I’m starting work on a platform-independant distributed bug tracking app (this is because I want to track bugs inside of the projects) and I’ve decided to write it in C# which means it will be dependant upon Mono or .Net being installed on the target platform.
I’ve worked out that I need a scriptable level inbetween my application and the SCM – because I want to make it independant of the SCM I am at the moment tying it into Git, however I can see how I need to make it indepenant by having a script wrapper around the SCM (which might be Git or might be SVN or something like that) which would provide the answers to a collection of questions that I want to put to it.
Its getting things to coexist peacfully which I think is the big Programming problem.
I’m having a real problem understanding the idea of an SDK. The entire concept is becoming more and more spleurgh (if you don’t mind my making up words). The concept makes sense when tit ties in with a particular IDE. And here’s the problem – why should anything tie into an IDE? The advantage of using something like Ant is that it allows you to build both inside an IDE and outside.
I built a “build bot” a while ago which took its own parameters and called csc.exe from the command line to help do building – because my knowledge was small, it seems that I had missed the point of msbuild. This takes the same project files that Visual Studio produces to do the same builds.
The only problem I have now is that I don’t know if there is a mono alternative to msbuild. If there is, that would be awesome, becuase I can define the build just as I would a java ant build, and know that it would work no matter what kind of system you where going to build for.
So is a SDK actually more then libraries? When it comes to mobile development, they seem to be libraries, maybe some tools which integrate into a particular IDE (for example Eclipse) and maybe also an emulator or test tools of some kind. Perhaps also, with an SDK is just a marketing thing – that you expect with an SDK a higher level of support.
Why am I writing this? Well, I’m currently struggling with the Windows SDK for .Net 3.5 – I’m having massive problems with it and its compatibility with the Visual Studio C# Express. It would be nicer if instead of complex SDK installers (and from my experience, it is the installer which is the problem) they had Library versions or just “unzippable” folders to get the tools out.
Anyways. Windows SDK and Visual Studio do not like to be installed in that order – Visual Studio first, THEN the Windows SDK (or at least, if you’re using Express editions because you’re an indie development like me)
Uhh.. ok. The biggest problem I have with EAGLE Cad (and I only have the freeware version here – my job lets me use design tools at work, but most of the stuff I talk about on here is actually for home, and I’m trying to finally get some of it online as part of my “open source” roots.) is…
the lack of parts. Too often I come across a custom part that I want to incorperate in my circuit. When I draw my circuits by hand, thats not a problem, but to get these things online, I need to digitise them, and scan’s of my lab book just really arn’t suitable.
This time, I found the need for a particular Triode – the 6S7L – now, before people start asking me questions about Triodes and Valves in general, I don’t know, this is the first time I’ve ever used them and I’m getting quite excited over the prospect. But, Eagle Cad just dosn’t contain the parts for some of these devices, so more and more I’ve been building my own versions.
Because EagleCad is designed to go from a schematic straight to a board layout, I’ve found that I’ve been using the schematics as close as possible, but I really have been just throwing pins at the package file, meaning that if anyone where to convert my schamtics to boards then they will certainly a) be wrong and b) could be quite ammusing.
is the current working diagram. Yes I stole it. The book in question actually, “22 Radio and Receiver Projects for the Evil Genius” was a Christmas present, and I’m getting quite annoyed with the number of typing mistakes I’m finding in it. I wouldn’t say its the best book – you have to be a genius (or be good at pretending to be one) to understand some of the bits in it, but forging onwards … high voltage valve radio soon.
It seems that some readers have problems when you are using cards with both mifare and ISO14443 cards (such as the Nokia 6131 or 6212). Devices have to … Guess which interface to use and from our experance different devices they have differing prioritys.
The omnikey 5321 we use here actually requires a registry mid to get the driver to change it’s order – not something you can do on the fly but also not as bad as some others which we have not yet found solutions to.
So a few years back I was doing a fair bit of sys-admining, and a fair bit of programming. I would say probably 50-50 in my time. Since then I actually went down the route of doing a lot more programming, and as little sys-admming as possible. Im a firm beliver that programmers should be forced to do *Some* systems administration so that they understand the concepts and ideas behind security and the other issues that Sys-admins have to deal with, but I also believe that programmers shouldn’t be the sysadmins of the boxe’s they deploy to.
Of course in a small organisatio, you don’t get those kinds of options – its very hard to justify the expense of a specialist sys-admin where every month is another extension on the mortgage of the CEO (or whoever your startup financial backer is).
Anyways, my problem is that on my personal server, I’m not actually very comfortable in claiming to be the sys-admin of it anymore. I havn’t got a clue whats going on with it any more. Four years of solid programming with only minimal systems-administration in the mix. So this is why, when trying to get some of my new code (in Python – you hear that – I’ve broken my “all languages beginning with P are evil” habit) to start on the server, I broke *everything*.
Thank you, goodnight.
So you’ve all seen the video of me making the Prototype 1.2 (back in this post) but now we’re actually gearing up to make a real productised version of it. What this means is turning a prototype system into a single board working version. This presents me with a couple of challenges – mainly over licensing and IP.
I will be the first to recognise that what I’ve done in terms of hardware is hardly rocket science – I’ve taken a module from here, and a module from there, and wired the two together. My touch of genius has come in the form of the code sitting on the microprocessor (module 2) which is driving and controlling the first module.
Module 1 – the really tricky bit, actually isn’t a problem – we’ve paid for a license to use this circuit, and we pay a small royalty for each product we produce with it on.
Module 2 – the microprocessor board, is proving a little more difficult to source – my current build is based upon Arduino, that is I have developed a shield containing Module 1 on it. I was hoping that the license for Arduino would enable me to adapt the circuit design and build the entire board as one. However, Module 2 is released under Creative Commons Attribute Share-Alike License, which basically means if I use their schematics, I have to license the entire board as that.
And looking for other, more friendly “Open Source” schematics, is prooving very tricky to find, so I resorted to a different tactic – just designing the entire microprossor module myself. Actually, it wasn’t that hard – if your design (from the docs of the microprocessor you are building for) then there is only very small descions you can take. It turns out (after comparing my design against the Arduino, and about three other Atmel based prototyping microcontroller boards), that there is only very very slight differences in terms of schematic – and this will also be reflected in layout – following standard design layout “logic” will lead you to the same conclusion.
mmmm, final version sounds so near.