Code Screen Problems Don't Have to be Unspeakably Terrible

Many software developers are familiar with a “fizzbuzz”-type problem in coding interviews. The idea is to ask a fairly trivial coding question of candidates to make sure they are competent coders. This can be done on-site, or in a phone interview format. Although well-intentioned, these kinds of questions are annoying and insulting for qualified candidates, can give false negatives as a result of trivial errors (especially if asked on a whiteboard) and are tedious for interviewers to ask.

In recent years, many companies have forgone these dull onsite interviews in favor of automated code screening tools like CoderFights or HackerRank. These provide an online IDE in which candidates can solve programming questions in a browser. We used one of these for years - here’s the problem description we used:

A palindrome is a word that reads the same backward as forward. For instance "civic" or "deed" are palindromes while "toyota" is not.

Your task is to write a function that rearranges letters in a string to become a palindrome, or "-1" if it's not possible to rearrange to be a palindrome.

For example: "cecarar" can be rearranged to obtain "racecar", which is a palindrome. "aab" can be rearranged to obtain "aba", which is a palindrome. Note that palindromes don't need to be real words from the dictionary. "abc" cannot be rearranged to become a palindrome. Therefore the correct output is the string "-1".

You will receive an array of strings to rearrange. You should return an array of strings converted to palindromes, or "-1" if a palindrome cannot be made.

That’s a reasonable technical problem to solve, but it has one very troublesome flaw: it looks absolutely nothing like the work we actually do day-to-day at HubSpot. We were finding that many candidates could pass a test like this which assesses basic coding competence but still were underwater when it came to the basics of building web apps.

We’ve tried hard to make our interview process more closely match the environment of doing real work in other ways, like cutting any contrived whiteboard coding in favor of in-person coding questions exclusively on computers (ideally on the candidate’s actual laptop with their IDE and documentation of choice).

So we wondered... could we do that for a pre-screen coding test too? What if we could create a test that could automate some of the assesment, but was in the realistic environment of writing code in your language of choice on your own machine rather than the rigid web-based IDE?

Turns out, yes. Well, we think so at least.

As of mid-last year, the coding project we use isn’t through a test provider, but instead an API and associated documentation. We email a start link to a candidate with a link to an overview page. Once they land on the page and click through to start, we provide the documentation on how to use a source API, and provide a spec for how to transform the data from that API and send it back to another API.

We start a timer when they click start, and when they send in the right response (or run out of time), we send an automated email with a link to upload code.

Here are the rules we provide:

This gives us a “semi-automatic” assessment - we know if candidates are able to complete the problem in time, but lets us dive into the code as well. Best of all, it lets us give candidates a problem which looks a lot like a problem we actually deal with all the time: take data from some API, perform some processing, and send it to another API. It’s way more realistic, less anxiety inducing, and ultimately far more reflective of candidates’ ability to work at HubSpot than assessing identification of palindromes.

We send out anonymous post-interview surveys to candidates after they interview, and here’s some real, uncensored feedback we’ve gotten:

"The written technical test was, by far, the best technical test I've ever done; it was clear, concise, challenging but also reassuring given you know that your solution works at the end."

"I really liked the challenge! I believe it really shows both the knowledge of data structures & algorithms, and web development skills of the candidate. While looking for a summer internship, I've completed a lot of challenges and code tests but this was by far the best."

We’re happy to be part of the trend of moving software engineering interviews away from riddles and CS puzzles and more toward real, practical work. If that kind of a process sounds cool to you, we’d love to get to know you better and show you firsthand - we’re hiring software engineers to work on a bunch of problems that have nothing to do with finding palindromes.