What We've Learned About Interviewing Engineers

At HubSpot, we have grown from one to 75 developers in a few short years. This is what we've learned.

Why Interview

There is no one brilliant solution to the interviewing problem.

Deciding if someone should spend 40+ hours a week with you for the next several years is a very hard question to answer based on a couple hours of contact. The one tried-and-true method we have is the interview.

Interviewing is based more on tradition and "doing what the guy who interviewed you did," than hard science. With rare exception, it's impossible to tell if you are hiring the best candidates, or missing some of the best. Each company has their own formula, and even the worst systems (looking at you 90s era Microsoft) are allowed to persist for decades.

There is no one brilliant solution to the interviewing problem. Evaluating the developers you have hired is an unsolved problem, much less trying to evaluate everyone you don't hire. It's best to avoid the "clever" solutions and stick with a simple formula.

The Interview Loop

An interview is an opportunity for the candidate to expose to you their abilities.

You only have an hour, the amount of information gained per minute is worth trying to optimize. Try keeping this model in your head during the interview:

• Think "What more do I need to know to make a decision?"
• Ask a question
• Listen to the answer
• Repeat until you have no big unknowns about their ability or fit

An interview is an opportunity for the candidate to expose to you their abilities. You have to get off of topics quickly when you have hit their limit, to give them a chance to demonstrate all of the skills they do have.

Your Purpose

One of the more terrifying moments of a young developer's career is the day they are conducting their first interview, and their mind blanks looking for a question to ask. Nothing makes a company look like more of a joke than an interviewer failing at their own interview. Let's start by defining our purpose:

• Decide if the candidate should work at your company.
• Make the candidate want to work at your company.

You should try to make every question and statement work to accomplish one of those goals.

Deciding

Before the Interview

The test saves us a week of developer time for each candidate we end up hiring.

Each stage of the interview process costs proportionally more in developer time. Most interview processes follow a funnel, where each step winnows out as many candidates for as little investment of time and money as is possible.

So lets create a semi-realistic example of a hiring funnel:

• Review Resume / Web Presence - 100% of applicants - 15 minutes
• Phone Screen - 50% of applicants - 1 hour
• In Person Interviews - 10% of applicants - 6 hours
• Offer - 2.5% of applicants
• Acceptance - 1% of applicants

With these ratios about 135 developer hours, three weeks of full time work, goes into finding a single employee. At HubSpot we have started to use a programming test web app which asks candidates to spend a few hours solving problems and writing code. The test saves us a week of developer time (using the figurative example above) for each candidate we end up hiring.

Anything you can do to make the early steps filter out more unsuitable candidates will help your efficiency. Getting candidates to write code in your phone screen (particularly with a tool that lets you execute it live like CoderPad) is going to help filter out those people who won't pass an in person interview.

If the candidate is not actively looking for a new job, you may have to play it slow. Ask them to dinner, to come in just to chat, or meet up with them at an event.

Parts of an Interview

The basic components of an interview:

• Small talk to make the candidate comfortable
• Technical questions where they write code
• Behavioral questions where they talk about past projects
• Answer any questions and sell the job

Small Talk

Not every technical candidate is going to be Stephen Fry.

An interview is uncomfortable. How the candidate feels in the interview though will determine what they think about your organization. They aren't going to want to work with people who made them feel nervous and uncomfortable.

Keep it personal, and talk about what you do specifically, rather than talking in the abstract. To make it not feel like an inquisition, to share as much about yourself as you ask about them. Don't read the conversation like a script, talk like you would over beers, leaving room for the candidate to enter the conversation as you go.

Frankly, not every technical candidate is going to be Stephen Fry. Some people, even more so technical people, are naturally awkward around people they don't know well. If they aren't comfortable answering questions in the abstract ("what do you like to do for fun"), ask concrete questions ("have you ever been sailing"). The more factual the question, the easier it is to answer for someone who is nervous.

Of course, if you truly can't have a conversation with the canidate, he or she may not be a great culture fit for your organization.

Sample Script

Intro

"Hi, my name is Zack Bloom, I'm the frontend tech lead on the contacts team. Contacts is one of the six major "apps" that form HubSpot. Do you know much about HubSpot?"

What We Do

"HubSpot is like Google Apps for marketing and sales. Each developer has ownership over some parts of the product or infrastructure. For example, I'm responsible for the frontend of the contacts database and all the tools people use to find and understand the contacts they're marketing and selling to. It, like most of our apps, is built in CoffeeScript as Backbone app serving data from the Java APIs that our backend developers build."

Small Talk Prompts

"I tried Kite Surfing for the first time this weekend, it was really cool. Have you ever tried any sports on the water?"

"I'm really into Starcraft 2 right now, are you playing any video games right now?"

"Is this your first time in Boston? I remember when I first moved I couldn't navigate two blocks without getting lost. Have you gotten lost yet?"

Technical Questions

The purpose of the interview is to get an accurate opinion of the candidate at whatever skill level they have, not demonstrate your superiority.

Virtually any technical interview should include some amount of code writing. You can't become good at calculus unless you know algebra front to back, similarly, we expect that if you can't answer basic questions very quickly, you are very likely not a stellar programmer. This is not a perfect test, some organizations have had a lot of success with take-home projects or using short-term contract work to evaluate candidates before making a final decision.

Our attitude is that we are hiring engineers who will be solving problems, not scientists who will be theorizing about them, so we avoid overly theoretical or 'word puzzle' type questions. The goal should always be to evaluate the candidate on topics which are as close to what will be required of them as is possible.

Your feedback during the questions is important as well. If the candidate sees you shaking your head, getting exasperated or getting up and writing a different solution on the board yourself, they are going to be discouraged at best, hate you at worst. The purpose of the interview is to get an accurate opinion of the candidate at whatever skill level they have, not demonstrate your superiority. If you feel like your time was wasted, that's something to take up with the person who did the phone screen or reviewed the resume, not with the candidate who's just trying to do his or her best.

Question Examples

Knowledge Based

A developer is not proficient unless they have grasped some basic concepts in his or her language of choice. The difficulty of these questions is set based on how strong the candidate claims to be, but every candidate with some amount of experience should be very skilled in at least one language.

"Do you know what the GIL is in Python? In what way does it limit what Python can do?"

"Can you explain how scoping works in JavaScript? How does this work?"

"What's the difference between a div and a span?"

Coding

These questions are essentially to winnow out canidates who can't write code. They also provide great launching points to enter into more in depth discussions. Here's an example of how that might go (in an abbreviated way):

"How would you find the first duplicate string in a list"

"I'll sort it, and then scan through."

"Great, what would the efficiency of that be?"

"Hmmm... O(n log n) for the sort + O(n) for the scan"

"Can you do it faster?"

"I could keep track of all the strings I'd seen, maybe using a hashtable?"

"Great, can you write that up in whatever language you're comfortable with?"

Writes the code, with some back and forth to fix issues

"What's the efficiency of that?"

"O(n), but it really depends the hash function and what happens with collisions"

"What if you had 100 TB of data and a hundred servers?"

A great candidate might start talking about Bloom filters, what map/reduce steps might work, or what the minimum amount of data you would need to send over the wire might be.

Experience Questions

You can't trust a resume.

People become great engineers by engineering lots of things. These questions center around real or hypothetical "big projects" in the candidates past or invented by the interviewer. They help us to confirm what real experience the candidate has (you can't trust a resume), and allow the candidate to demonstrate how they approach problems.

"Do you know what reddit is? How would you go about building it? How would you support merging a bunch of subreddits together to form a user's homepage? How would pagination work?"

"How would you build YouTube? How would the video be ingested? What are the biggest scaling issues? Would you treat videos that get millions of views differently than ones which only get a few?"

"Looking at your resume, I see you built a missile defense system to protect us from alien invasion. What parts were you responsible for? How did the system work? So aliens are real?"

Selling

The six hours the candidate spends with the interviewers is likely the only real contact they will ever have with the company before making a decision.

It doesn't matter if your interviewing process is perfect, if no one accepts your offers. In too many companies the interviewers are seen as just evaluators, with no responsibility to educate the candidate about the company. The six hours the candidate spends with the interviewers is likely the only real contact they will ever have with the company before making a decision.

Many engineers see selling as a duplicitous process dominated by men with bad mustaches and worse morals. But, assuming that your company is actually a place worth working at, selling is just education. It's teaching the candidate about what it's like to work with you, how it's better than their other options and why they should take the job. If you can't make a convincing argument to that effect, you shouldn't be interviewing (and you might want to look into getting a different job yourself).

Selling Guidelines

• Be somebody someone would want to work with
• Educate the candidate on why working at your company is better than working elsewhere

Being Nice

If someone had told you that you were an idiot back then, do you think you would be working with them today?

When you are doing an interview, you are the companies representative. If they hate you, they will hate the company. You don't necessarily represent all of the company, so it's alright to tune your energy level and behavior to match that of the candidate within reason, but the most important thing is to be honest.

It's not worth lying. The one thing worse than not hiring a candidate is having to fire someone, or having them quit. They will cost you more on the way out than they did coming in.

That being said, it is your place to paint the best, but accurate, picture of your company that you can. Be excited about the vision of both the organization and the dev team, and try to get the candidate excited about that vision. Share the great things you've done, and plan to do. If you haven't done much yet, talk about what great things you want to do, or would do if you had the resources.

It should go without saying, but treat the candidate with respect. Be punctual and respectful. Don't check your email during the interview. You may do a dozen interviews a week, but for them it represents a pivotal moment in their life. Remember when you were interviewing for your first job; how nervous you were.

Give positive, but truthful, feedback, even if they aren't stellar yet. An example might be:

"I really appreciate how focused you were during the interview.I know how motivated you are, and we all have been at your experience level at one point, but we just don't have the mentorship resources right now to take you on right now.Keep working on projects and developing your skills, and we'd love to talk to you again in six months."

We were all at the point in our career when we thought global variables were a fine idea and the hardest problem we would ever have is implementing quick sort. If someone had told you that you were an idiot back then, do you think you would be working with them today? Sometimes you can take a candidate with less experience, it they're smart enough and you have the resources to work with them, sometimes you can't, but either way, be honest so they can move on to the next phase of their career.

Closing

You need to listen before you can talk.

Selling comes down to convincing them that the job you're offering is better than other jobs they may have on the table. The variables you can play with are the enjoyment and challenge of the work, the quality of the coworkers, the prestige of the company and the salary/benefits. The points you choose to extenuate depend on the audience and what they tell you. If they tell you they use subversion, talk about how you use git. If they don't get much freedom in their job, talk about how much autonomy your position offers. You need to listen before you can talk.

Before you can convince them of anything though, they need to trust you. They need to believe that what you're telling them is the truth. The best way to do this is to actually have a genuine relationship with the person. If you are active in the community and put in the effort to maintain connections with fellow devs you meet, you will have a network which allows you to potentially hire them, and other people they may know. If you don't have the luxury of a long term relationship, build a rapport as quickly as you can, speak plainly and don't sell too hard, just talk about the company as you would with a friend.

RFC

Please share your interviewing tips in the comments below, on Hacker News, or tweet us @HubSpotDev. You should follow us as well.