I really don't like doing a ton of choosing. I don't know a ton of developers that do. I think it's because when you choose you expose yourself to the risk of being wrong. I don't think I know any developers that like being wrong.
A few weeks ago, I was going to have to make a choice. We weren't sure if we wanted to implement our new social media interactions with links on the right column or buttons at the top of app. Both are good options, so we should set up an A/B test, right? We'll get some data and we'll have an answer. Product owners and other MBA types absolutely love this kind of stuff. They can make a graph and put it on a power point. I like it because I don't have to choose, I know which choice works. The world is warm and fuzzy.
Except...
There's always an "except"; except at HubSpot our A/B testing wasn't very effective or glamourous. Generally, tests were implemented with modular arithmetic based on an arbitrary parameter like an ID and when we really took a hard look at the results, we realized that grabbing a number from a hat would have produced roughly the same results.
That was until one of our resident smart people, Dan Milstein (@danmil), stepped up and created an awesome testing solution. Now, implementing a test is as simple as adding one line to your code. Dan's solution is able to insure a very close even distribution of the availible options, is extremely fast and allows you to set up reporting. Nirvana reached.
After about 5 weeks our testing showed that the results were very close but the buttons won out. Preliminary poking with some of our heavier users has shown that they prefer the buttons. A very big win without any fretting or unnecessary meetings. Isn't working at HubSpot great? Yes, we're hiring.