Making our life better with Hudson (?)

It's been about 10 months since I joined HubSpot.  One of the first things I started doing in my first few months here was to make our Hudson environment happier.  Get it running  in some decent shape, get the unit test building, and get it emailing in cases of real failure.  We made some significant progress on it.  We now have over 70 Java projects in our Hudson environment.  We even have our production python code unit testing happily in Hudson!  We also have an extremely annoying flashing light in our dev area when a build has failed test.  At some point in the future, I want flashing lights from poor code coverage/code quality also, but thats a bit down the road.

Unfortunately, our Hudson setup is a bit sad in some areas.  We are using it as a means to run unit test on checkin, but we aren't doing any jar deploys.  Without deploying, scaling our hudson environment horizontally has been a problem.  We scaled it vertically - a bigger server and more executors, but now there are random issues with fingerprint files going bad.

The biggest headache has been our massive dependency chain.  Project Z depends on Project Y which depends on Project X.  Hudson does wonderful things to help with this, especially with maven projects (which all our java projects are). When code is checked in, project X builds, and then project Y builds, followed by Project Z.  Now we can see when Patrick deletes code in project X that he broke Project Z.  It's a disaster the other way around.  Project X and Y are both checked in at the same time, and Hudson decides to build them both.  Well, Project Y needed somethign new from project X, so project Y fails and our lights start flashing.  The other annoying one is if Project Z is building, and Project X decides to start building.  Project X builds faster than Project Z ,and does its 'mvn install' while project Z is running.  A bunch of tests now fail in project Z.

I'm sure a lot of our problems can be solved by taking the effort to make sure we can deploy the jars and not mess up local environments or test environments.  With jar's deployed, the 'mvn install' won't overwrite a jar file in use.  It will also let us scale horizontally, which has it's whole set of bonuses!

For now, Hudson is a mixed victory for us.  We gain a great deal by the test running, but we do so with a decent amount of headaches.  Hopefully we can fix these and have more happy fun times with Hudson in the future!

For more information about Hudson, check here.  It's free, [mostly] wonderful, and there is a plugin for almost everything!  In fact, I'll make a post about all the wonderful Hudson plugins we use at some point in the future.