If you work in software development, you’ll be used to having source control store your code in a central location. A continuous integration program will pick up new code changes, compile your application, run your tests and produce a new version of your application. This encourages developers to keep the build ‘green’, to test their code locally before putting it out to the wider organisation (or world).

What happens when the build fails? Ther are many reasons for failure, from accidental to reckless. Many teams have adopted the practice of applying a penalty, of some form, to the developer who breaks a build. A quick check on this (archived) Stack Overflow question shows that this practice is pretty common. I’ll quote some answers:

  • “We have a bee outfit… You have to wear it until the end of the day.”
  • “Take No Prisoners. I maintain a nerf arsenal. Revert the check-in and aim for the face.”
  • “USB foam dart gun. It is the geek way!”
  • “A giant cardboard cutout of Jar-Jar Binks”
  • “We sometimes use fingerpointing”
  • “We have a rubber chicken that hangs outside the cubicle of the person who most recently broke the build”.
  • “WALL OF SHAME”
  • “We have just started throwing shoes at the offender”
  • “Wear a dress for a day”

You may call this a laugh, office banter whilst getting a serious message across. I call it ritual humiliation, because that’s what it is. Verbal humiliation, social humiliation. In some cases, what could be reasonably classed as assault. That’s not what you expect from professional, inclusive environments. I do not stand for it. You should not either. Based on the responses above, I’m glad the question was deleted from StackOverflow.

Builds break. When it happens, all you need to do is make sure the developer who broke it fixes it right away. Help them out if it’s something non-trivial. Make it a learning experience - perhaps this wasn’t some reckless check-in, but an accidental oversight. They could have forgotten to save their .csproj file when adding a new code file, so everything works locally but fails on the build server. If it was reckless and this is a frequent occurrence from that person, have a private conversation with them, but don’t humiliate them. We should demand better than that. Your CI server is another line of defence and sometimes it’s going to flag build failures for you.

We can do better. Here are a couple of suggestions that I have, purely from my experience.

Gated Check-ins

When you push your code to the server, it doesn’t actually ‘accept’ the change unless a build passes. Should a build fail, you’ll get a report and the change is rejected. Gated check-ins will ensure that you don’t have a broken build and/or revert commits scattered throughout your code history. Furthermore, you can make this part of your code review approval process! Gerrit, an open source code review tool, will handle this process for you.

Feature Branching

A developer may want to push code to the server, to back it up, or allow someone else to work on it. That may be experimental, in progress code. We don’t want a build failure for that, nor do we want that code on master. The solution is to create a feature branch and commit your work-in-progress changes to that branch. It can be re-synced with the master branch periodically to avoid diverging too much from other changes. When it’s ready, it can be merged back into the master branch. You may still want a CI build to be generated for every commit to that branch, just so your team-mates have a snapshot of the branch status. But, it doesn’t matter if that build fails! All that matters is that the build passes when it comes to re-integrate with master. The Atlassian toolset, Stash for code hosting and reviews, plus Bamboo for builds, works very nicely here. Just like Github, you can create ‘pull requests’ to re-integrate a feature branch back into master. Bamboo can be configured to build feature branches. When it comes to accepting and merging the pull request, the two tools integrate, so if the ‘tip’ of the feature branch doesn’t have a working build against it, then you can’t merge.