Launching HBase on ARM
At HubSpot, we recently did the work to launch our largest data store on Amazon Web Services’ new Graviton servers. It took time to get that data ...
GitHub is an amazing company and an amazing startup story. We're not only customers of the product, we're huge fans.
Recently, we had Zach Holman (@holman), an engineer at GitHub come and give a HubTalk at HubSpot. HubTalks are regularly scheduled events where we have internal or external speakers give a talk on a variety of topics at the HubSpot headquarters in Cambridge, MA. (We have a brand-spankin' new meeting space that's perfect for these kinds of talks).
The title of Zach's talk was "Product us The Byproduct" -- but he chatted about a lot of different topics.
A video of the talk is included below, along with a full transcript (for those of you like me, that like to read/scan instead of watch).
Here are some of my favorite bits from the talk.
Insights from Zach Holman of GitHub
1. I think you should worry more about building the damn thing and worry less about what you’re actually building. A product should be a byproduct of the people, process, and technology of your company.
2. There are thousands of developers who are extremely smart and talented, all of whom I would never, ever hire here
3. One thing we try to figure out at Github is to try and hire people bothered my suck.
4. “Any time you interview a potential hire, you need to ask yourself not only if they’re talented or collaborative but also if they’re capable of literally running the company, because they will be.” ~Valve Employee Handbook
5. Cox and Blake in 91 concluded that groups exposed to minority views were more creative than the more homogeneous, majority groups.
6. The overall median proportion of female executives in successful companies is 7.1%, compared to 3.1% at unsuccessful companies. ~VentureSource
7. “increases in masculine wording were sufficient to decrease women’s job appeal ratings and their anticipated belongingness in specific occupations.”
8. I want to get to know my co-workers families because their kids are almost always more interesting than their parents.
9. once you've launched everything, it’s time to get the reaction of people. Which is problematic for one reason, people on the internet are dicks.
10. Any sort of small feature that we launch, we’ll blog about it. Because it’s good, you get people talking about it. And it’s the whole long tail effect where you fix some small bug that’s been eating away at some people and most people don’t care but they’ll be some people that say “yes! I’ve been looking for this for two years.”
11. Nothing great was ever not shipped.
Full Transcript
Dharmesh: So Zach’s - the reason he’s here. I traded emails and he’s awesome by association, someone I respect who is awesome said, “Zach’s awesome. He’s going to be in Boston, you should do something,” I’m like, “sure”. So with that let me introduce Zach.
Applause
Zach: That’s probably my favorite introduction of all time, that’s hilarious. I need a microphone, hold on. Cool. Everybody hear me?
All right, the title of my talk is - “Sure”, no ha ha. The title of my talk is - “The Product is the Byproduct” and by that, ya know, I think you can create really great product but it’s better to promote a great product. Wait, I should stop talking like that. I hate when people get up on stage and start talking about a product like it’s some amazing, angelic thing.
Here’s what I actually mean, I think you should worry more about building the damn thing and worry less about what you’re actually building. By that I also mean a product should be a byproduct of the people, process, and technology of your company. So focus less on the actual thing and more on the stuff surrounding what you’re building. I really think if you foster a really good environment, you’re more likely to create a good product. If you look around at really good companies, people with a really good culture behind them, more often than not they tend to be successful. There’s a lot of companies with horrible culture that still are successful but that is America.
So I’m Zach Holman, this is a picture from a charity dodgeball tournament in the Fall. This is great because you put pictures of yourself in small shorts and since it’s for charity everyone thinks it’s awesome. Dodgeball is amazing, I was just happy about that. I do work for Github, I am a Developer there and I talk a lot about how we work, about how open source works, that’s kind of tangential. All of this talk sort of stemmed from looking around about how Github sort of does stuff, and we’ll just start from there.
Team. Team is a critical part of your company. Everyone gets on stage and says people are important, so I’m going to do that as well, and say that people are important. People are extremely important. (points to presentation) This is my team, this is my team circa a year ago, we probably added them on 70 or 80, we’re at 153 total right now. And this team is really important to me. Kyle Neath, our Director of Design at Github said something in Campfire, just off-handedly which I thought was really interesting. He said there are thousands of developers who are extremely smart and talented, all of whom I would never ever hire here. And I think that’s really indicative of what we try to find at Github. It’s not just about smarts anymore, it’s about how you think about things, how you come to things. How you deal with problems, you know, it’s not necessarily I.Q. points anymore, it’s the whole spectrum of stuff.
One thing we try to figure out at Github is to try and hire people bothered my suck. By that I mean, basically you want fixers, you want somebody who, if they deal with a certain application on a day to day basis, they’re bothered by stuff that sucks, they’re bothered things that isn’t quite designed as well as it could be, isn’t quite as fast as it could be. There’s a lot of developers or designers who will just, you know, if that not necessarily their 100% priority, they’re just not going to fix it because that’s not in their job description. We don’t want those types of people. We want fixers.
We’re big fans of Valve. Valve “leaked” their handbook last year. Fantastic marketing for them, I love that, but we’re really excited about this handbook because we found that Valve was working, ended up working the same way that we have over the years and they’re twice the size of us. So that’s really interesting to see them, sort of confirm the way that we’re working. And in their handbook they said this, “Any time you interview a potential hire, you need to ask yourself not only if they’re talented or collaborative but also if they’re capable of literally running the company, because they will be.”
I think more often than not people don’t recognize that fact. Even like the low level developer on your team, they end up making of ton of small decisions that impact you down the line. And people think, oh, it’s not that critical but they tend to be, like these very small decisions, like oh, should this code be this way. And if feel like you can trust everyone to say, ok everyone else is dead and now you have to run the company, that changes the amount reliability you feel in letting people make those small choices.
So I hire broad people, just like that in general, rather than just hiring the smart and the best and brightest, it’s just kind of silly. I like really broad, interesting people. Diverse people is also a side of that. This room is better than most, a lot of white dudes my age in here, which is basically all I ever talk to which is really sad. But diversity really matters, and there’s some really interesting studies surrounding this. Cox and Blake in 91, they concluded that groups exposed to minority views were more creative than the more homogeneous, majority groups. And I think that makes sense anecdotally too. If somebody comes from different backgrounds or different values than you, and you recruit them on your team, you’re going to end up with a more interesting solution that will tackle more peoples view points.
For more up to date stuff, VentureSource looked at a whole bunch of different startups and looked at the success of those startups over time. And they found that the overall median proportion of female executives in successful companies is 7.1%, compared to 3.1% at unsuccessful companies. All of this sort of stems from this idea that this really impacts your bottom line. There’s just some people in life that you just have to go back to money, and say yes, this will make a better product, this will make...more diversity in your company will make you more profitable. I think that’s definitely really true.
Building a company that’s friendly to everyone is not as valued as highly as it should be in our industry. There’s a lot of different ways you can sort of deal that, building this company that you want to work at.
One is Tone, just how do you talk about yourselves to other people. There’s another study that was really fascinating, they just talk about wording. They said, “increases in masculine wording were sufficient to decrease women’s job appeal ratings and their anticipated belongingness in specific occupations.” Bunch of academia words. Basically what I mean is stuff like this, like “Badass” “Killer” “Rockstar” “Dominant Force in the Industry”. You see those all the time and you compare those jobs posting that will talk about “Happiness in the workplace, honest, or working stuff, building stuff together. I would much rather work there, and that’s the critical point stuff because it works for men too. I would much rather work at some place talking about happiness than talking about badass beasting on code or something like that.
It’s, it’s, it comes down to this for me, it’s like if you’re building a company and it appeals to more people, then go in that direction. So basically just consider your approach, also really hate brogrammers, I just want to underline that, I hate all that.
Cool, Team is important, Culture is also really important. All of this is just no! (referring to a slide) Crushing code and all that stuff, it’s just not a culture you really want to have. Good culture attracts really good people. And I believe that a billion percent, that doesn’t work with math but I believe it a billion percent over. Turns out like once you have really good people, they like mate and have kids and offspring and stuff like that. And it’s weird how industry in particular forgets that point, families are critical, you know. So being family friendly is really important for us at Github. We have, we usually get together twice a year to have these big company wide summits. And we try to make a really big point to have these sort of family friendly outings because I want to get to know my co-workers families because their kids are almost always more interesting then their parents.
So consider that, how do you build your actual events in a way to make your company more interesting for everyone. There’s a lot of ways that Github does it ourselves, this idea of flexibility. One is just location, we’re about two thirds remote employees at Github, which I think is interesting, and of those we maybe a third that live in San Francisco and of those, maybe half end up coming to the office every single day. It’s just a really weird but its really flexible way of work. And I think that really helps us in attracting a really family friendly-centric focus.
Hours are really important for us, we let people work any sort of hours because, man have you ever had... if you’re not a morning person and you have this guy say you gotta come in at 9 in the morning for this stand-up meeting, and you’re just like, no. Like, I’m going to be asleep in my head until 2pm and just dragging me in is not going to help. So we have very flexible hour perspective of work at Github. We let people work when they want to work. And finally just a sane workload, this whole idea of doing 140 hour weeks, especially like the Gaming industry where they thinks that’s a healthy way to work, that sucks. We much prefer the idea that if you work reasonable hours that you’re going to be much more creative, you’re going to be much more...you’ll produce really good work if you work not-crazy hours.
So basically you limit yourself with all of these things, you limit yourself by saying you have to be in an office, you limit yourself by saying you have to have set work hours, or work a billion hours a week. One of my favorite stories that relates to all of this is the notion of OS X on Intel, and all of this happened because a self-demoted engineer wanted his son Max to be able to live closer to Max’s grandparents.
The story is, this engineer’s like, he’s living out in Cupertino and he’s like, man I really want to go to Boston or something like that because my family is near there. My kid wants to be with his grandparents and all that. And he talks to his boss, he’s like, you know I kinda want to move out there but is there a project I can work on, I kind of want to see if I can get OS X on an Intel platform and see what goes on, and the boss is like yeah, just go away just work on that. And they forget about him for like 16-18 months because you know big companies and stuff like that, then finally somebody is like, oh man we gotta figure out what this guy’s been doing out there on the East Coast the whole time. So they bring him in and he runs OS X on his Intel Windows box type of deal. And his boss is like, alright alright, then he goes and gets his boss, and he gets his boss, and he gets his boss and then there was like, Steve Jobs is on the next flight to Tokyo or something to talk to... to figure out ways to deal with all this stuff. Because it was awesome and really exciting and it all stemmed from the boss’ ability to let this really smart engineer work on what he wanted to, work on something that he found was interesting and work form a place that he wouldn’t of otherwise been able to work at. This all stemmed because they were very flexible.
I don’t know the guy personally but, I feel like if his manager would have said no you can’t move, he would have been like, alright bye. And then we wouldn’t have had OS X on Intel and maybe we wouldn’t have had the iPhone and then obviously we wouldn’t have Android without the iPhone (laughter). It weirdly all stemmed from one specific person. So that stuffs really important.
I want to talk about a little bit about desire and building desire. Marketing is not a bad word, and I feel like that’s probably nothing crazy to say here in this room. But a lot of other places, like Developers like to say, “ugh the Marketing people, they’re all just idiots”. But it’s not a bad word because I think instead what what we hate is the disingenuous. This idea that someone’s trying to sell you something that they just don’t want themselves.
So this guy goes up to you and he’s like, oh I really want you to buy my distributed social cat network. And you’re like wait, hold on let just ask you a couple of questions first. Like one, do you even have a cat? And you just know he’s going to be like, no well you should do it anyway. And you’re like alright. Don’t we want to centralize cats? Because their hard enough when they start darting around all over the place, you want to centralize cats not distribute it. Would you even use this yourself? And I think that’s the most critical question, that’s why we don’t inherently like Marketing is you just feel like people are trying to shovel this crap on you that they don’t actually want. So being genuine about this is the most fascinating part about all of this.
A really good example recently, I gave this talk a month or so ago and I gave a shout-out to OnePassword and there was a OnePassword dude in the audience and he was just like, “yeah this is amazing!” and he wasn’t even listening to what I was saying, but it was good. The story with OnePassword is funny because I use OnePassword for all passwords and it’s awesome and I love the app. They sent me an email like last Summer or something, check out a new point version. And you know, I skimmed the features and I didn’t care. I like the app but it wasn’t something that I was really fascinated with but I’m reading through and it’s like the founder or the CEO or something and he’s talking about these really incremental improvements that I don’t care about, and he’s talking about them with such enthusiasm. He’s like, we got the whole team together, we’re working on this, we’re so excited about what we got coming. And even though I didn’t really care about what he was trying to market to me at that point, I came away really impressed with how excited he was about the product. And I went from somebody who really liked the product into someone who really sort of admired and thought that they genuine, what they were trying to sell to me. And I thought that was really just important, because you see that much anymore, strangely enough.
Being vocal is not bad. I think it’s interesting to have people vocally talk about what they’re working on. Blogging small improvements is really interesting, I think Github does that a lot more than anyone else. Any sort of small feature that we launch, we’ll blog about it. Because it’s good, you get people talking about it. And it’s the whole long tail effect where you fix some small bug that’s been eating away at some people and most people don’t care but they’ll be some people that say “yes! I’ve been looking for this for two years.” Then they share to all their friends and some of those people are really excited about it, you don’t get that unless you blog about these small incremental changes. All to often a lot of companies just say, alright we’re only going to blog our 1.0 launch or 2.0 launch, we’re only going to do press for these special events. You don’t have to be that way.
Leveraging regular press is a really powerful thing in general. This is one of the most interesting things I’ve seen in Tech. Steve Jobs went on the stage in January and gave this Keynote where he introduced the iPhone and then on June 29th they launched it, and people tend to dis-remember that they go back and say, he went on stage did an awesome Keynote and people started buying it a whole bunch. The truth is a lot more interesting, there was a lot of stuff that Apple did in between those, February 26th they released the first TV ad which didn’t even show the iPhone, it was just a bunch people picking up a phone saying hello. June 3rd they had TV ads, June 6th they had another ad, WWDC was on June 11th where they announced the iPhone SDK. Then they talked about and improved battery that they were putting into it a week later. The same day they announced a new glass screen for the iPhone. June 20th they announced the YouTube app was going to be on it. Then there was another TV ad. There was a guided tour they announced, then they published all of the press reviews and all of the old school journalistic endeavors. Then on June 26th, AT&T announced the plans for the iPhone.
When you look at all of this together, it becomes much more clear that this is a very concerted effort. Especially in the last month leading up to the iPhone. The Apple PR was every single day, week whatever, was announcing something new which got the headlines almost every single day. I think that’s fascinating, all these things, to their credit are interesting things at least. So people are talking about it, it’s not disingenuous.
So talking about stuff regularly I think is important and amazing.
Ok, once you’ve launched everything, that’s cool and then it’s time to get the reaction of people. Which is problematic for one reason, people on the internet are dicks. Man they’re just really assholes. I sometimes frequent Hacker News, and I see this all the time where someone, “I just spent six months on this, it’s an open source project, it’s awesome, it’s totally free. You’re going to love it.” But then someone comes in and they’re like, “Fuck Youuuuuuuuu”. It’s the saddest thing of our industry. It’s probably this one guy, had like one table in html or something, and somebody dug into source and was like, “aww now I hate you’re whole product for this, like I hate you”. It’s terrible, that’s Hacker News for you. But’s it’s our whole industry.
So the question I feel, is how much should you care about this? How much should you worry about the reaction that you’ll get from people when you push stuff out? And the answer is obviously, zero. But that’s really hard, it’s really hard to sort of ignore all this feedback from people, and ignore people saying, “fuck you that was a dumb idea”. So I’ve sort of started thinking about this a lot and I sort of developed an informal thing called Care Theory, basically saying, should I care, theory. How do you respond to feedback? I think there’s a bunch of different ways to look at this. Mostly, I think there are three specific groups that you should worry about when somebody is reacting to something that you’ve built.
One is fans, one is skeptics, and the other are haters. And these percentages are really relative. So it depends on the product launch, sometimes you can have a lot more skeptics and haters. Sometimes you can have a whole bunch of fans. Sometimes you’re going to launch something and it’s all just haters. But it’s important to recognize who these people are. Fans I think are the users, people that are using your stuff, and that’s all great. Skeptics are people that could be users of products, and haters are not users, typically. Should you worry about what they’re saying? Fans I think, yes. Skeptics I think, maybe you should worry about it. Haters I think, maybe you should worry about it.
Skeptics you gotta be worried about sometimes because they always say, if. So I mean they’re like, “I’ll use it IF you add cats to your product”. Why would I put cats in here, but alright I’ll add cats. Then they’ll say, alright, I’ll use it if you add cats and ajax to it. Then you’re like, ok I’ll just add ajax. Then they’re like, now I’ll only use it if we use websockets, too. And just like, this is silly. So be wary of “if”.
Github experienced this a lot earlier on before I joined three years ago. Because when they were just getting started, Github got all this stuff from really larger established open source projects. Where they’d say well, you look good but you’re not doing this and we’d use Github “if” you added this. And to their credit the founders were like alright that’s interesting. And then they just didn’t usually add it unless it seemed like a really good idea. And then people ended up going anyway. We found this to happen a lot, in a lot of different things we would work on. People think that they desperately need something because that’s how they’ve always done it and they end up coming around. So just be wary about “if”. Be wary about (when) somebody says, we will definitely use you only if you add “X”.
I don’t even know what that slide is talking about. (laughter) I want to talk about haters, that’s what I really want to talk about. Because haters are also really interesting. I think haters are also broken up into three separate groups. One is just unreachable people, this is totally fine. People might hate on your stuff because they aren’t ever going to use your product. You know a ‘social distributed cat network’, maybe you don’t have a cat. You’re going to be unreachable and that’s fine, they’re just never going to use you. So don’t worry about them too much. There’s another group that may come around when you succeed, there’s always this whole, you know, the iPod left space (in-audible) lame. But they end up buying one anyway, four years later. And there is finally, assholes who are just assholes on the internet. This is the group you should worry about (high-lighting the may come around group). These are the people that they may come around once you hit market penetration, once you’ve demonstrated that your product is actually worth while, so worry them. Don’t worry as much about the assholes and the unreachable people because it’s just not worth your time.
Not all feedback is really equal, and that’s important. The thing kills me the most, is seeing somebody that is doing awesome, creates a really great product and they’re just really worried about thousands of people yelling over Twitter for something you said, or something like that. Try not to worry to much about reaction, and when you do, worry about who is specifically saying that. Who is, identify what audience they’re from because not all feedback equal.
And remember that “Nothing great was ever not shipped”. It’s important to realize that everything goes through these steps, every single thing that has ever been shipped by anybody has gone through haters, and people who love it, and people who are uncertain about it. So overall, I mean, building products is really hard. I think we all sort of inherently know that, it’s a difficult problem but I think you can really build it indirectly, I think you can build really great products just by building a really good environment. And that involves a whole bunch of different stuff. It involves people, culture, building desire, managing reaction. You can’t just say, I’m going to have a great culture then just never market your product. People need to see your product. You have to be able to realize when to throw down into battle, just don’t attack the haters all the time because you think you have to do that. Recognize where the hate is coming from and then just work on building a really good product that way. So thanks, that’s what I’ve got. (applause)
I ended kind of early, so if you folks have questions and stuff, I’d be happy to talk forever
Question 1: so you’ve been about three years now, in those three years Github has raised like a ton of capital, talk about how the culture has changed as a result of growth and having access to cash, or are you folks buying small planets, what’s going on?
Zach: What was the last part?
Question 1: Are you buying small planets?
Zach: Not publicly, so the money thing is really fascinating I think, as an employee who just digs Github and what the founders have done over the years. We went 4 years or so without taking any financing at all because we’ve been profitable pretty much since day one. Which really changes your perspective on stuff. So we had 4 years of doing whatever we wanted and not having to answer anybody because we were profitable and we didn’t have to answer to anybody. When we took money, that changed the game because we could take money on our own terms. We could just say, no, and then go off and make money. That’s different because you end up being able to take really good terms and still take that money but still be able to retain a lot of culture, like I don’t feel beholden to anyone. I can go on stage and talk shit about Andreessen Horowitz because, you know, whatever we have good terms and we still own the company and stuff like that. So I don’t feel like it’s a culture thing at all for us, I think we’ve done a really good job at both identifying our own culture, which I think is the... a big first step to figure out what you actually value, and then it’s...it’s, I think a lot of the part of it is just hiring the right people too. If we had something where like somebody said alright, now we have to change everything and go back this really weird way of working, we wouldn’t be able to do that because everyone would quit because people are passionate about how we work. So that’s been really nice for us because we’ve just not had to deal with culture change at all.
Question 2: So, if I’m correct you 153 people now and you’ve hired about half of them in the last year.
Zach: Yeah, sounds about right.
Question 2: Two thirds are working from home, so how do you teach that culture to 50 (workers) in the office?
Zach: It’s, it’s... so I get that question fairly frequently and a lot harder to go from a company that everybody works in the office, to going really distributed, for us it was really nice because we spent two years without an office. So, and we were very small at that point. When we got an office, we were about 9 people so from that point we had very ingrained culture of letting people work from home, meaning we have a huge emphasis on chat, a huge emphasis working via email rather than...that’s why we don’t have any meetings and stuff at the office because you know nobody is in the office. So for us it’s been a matter of just building a culture and going from that point. I think it’s a lot harder to say, we have a...everyone’s in the office today, how do we start being more remote? But I think it’s just identifying what we do that requires being physically in the office and then just start removing that. If it’s a lot of meetings, start asking, can we do this on an email thread, can we do this over chat rather than having to be in the office. And then just, piece by piece going from there I guess.
Question 3: What’s the demographics like at Github?
Zach: Demographics how?
Question 3: (Like the diversity pool and like ages and stuff)?
Zach: I have no idea on the age anymore, that’s gotten a lot more diverse faster than other statistics I think. I know we’ve doubled female engineering staff in the last month, which has been good but it still a lot lower. (crowd mumbles inaudibly) What is it like 6 or something like that? Our problem is...the problem I have with a lot of companies that say their diverse in terms of gender is that they tend to stick their females in specific roles. So like the stereotypical is secretarial and stuff. Which I’m saying is bad by any stretch of the imagination, I’m saying that if you look at the development teams, they’re like 100% male. And they’re still saying, we’re very diverse and stuff. For us it’s a little bit trickier because we don’t have managers, we don’t really have all these other areas. So we’re such a... I don’t know, 120 technical workers or something like that of the 150. So it’s a different pool to sort of hire from. So it’s something that we still suck at and we’re trying to improve dramatically, it just takes a long time.
Question 4: So you’re obviously very passionate about the culture of Github, is everyone there as passionate about it as you?
Zach: I believe so, there’s a... yeah people get angry about stuff, which I think is really good. This idea that people tend to really hate process in general, and every now and then there will be somebody suggesting something that maybe will add some sort of process for good reasons. And people will jump on that right away and say, you know do we actually need this? And I think Github, more than anyone else, is able reflect upon that better, like, I have to do this every day, can we just automate this away? And we’re really good at doing that sort of stuff. We’re good at hiring people that worry about that stuff, too I think.
Question 5: What are some examples of processes’ you actually (run)?
Zach: So, the (pool request?) in general, the ability to merge code into the primary branch is kind of central to everything. The process there very generally is, you open the (pool request?), you get +1’s or thumb’s up from people that are responsible for that code and then you merge it. And the...kinda the good process beyond that is we force responsibility on you to A.) merge it and B.) make sure the tests are written for that and you also support that feature as well. I think we have a good emphasis in our product, in our processes’ in general about responsibility. Putting the responsibility on you to be responsible for your code, rather than saying alright, I push it to Key Way, Key Way is going to make sure their systems and I don’t have to worry about it anymore. So that’s one side of things, we also do...we have an internal team app which is sort of like a Twitter / Way to collect ideas for what it takes stuff long term. Again, not really much of a process, just kind of post when ever you have either done something or posted an idea for a longer term change, or something like that.
Question 6: You said you like Valve, the thing that my mind can’t handle is, that everyone works on what ever they want and they just move the (desk?) to the next group and now they’re working on that team and they get up to the next floor any time they want to, are you folks like that?
Zach: Yes. It works fairly well, we tend to say it’s the land of best argument I guess, and one benefit of the way that it works is that things tend to go down in pairs, a natural back-end, front-end pair. You want someone to do the design, some one do the back-end of a specific feature. Most features will include a dozen people but it’s usually two people driving it. And to get to that point we usually have a back-end guy saying, I really want this feature, some one work on it with me please. And you have to do a really good argument of saying, check this out, this is going to be really great and for a designer to come work with you on it is at least saying the designer thinks it’s a good idea. So from there it’s sort of a good way of saying, well if no one wants to work on this with me, it’s probably not a good idea. As an aside, I think we do a really good job at allowing people to not work on stuff too. Which I think is almost as important as letting people work on what they want to work on. Specifically I’m thinking about, there’s been a lot of side projects at Github that have started out really dumb and small, and ended up being really critical to what we do. One of which is Hubot, another one is Boxen, another one is all of our deployment and testing infrastructure. Those all stem from somebody saying, I don’t really want to work on my day to day stuff because it’s tiring for me right now or I just want to work on this new technology stack or whatever. And letting people do that sort of thing, saying work on what you want to work on, ends up building some really interesting stuff, you know I don’t want to work at a company without a Hubot anymore because it’s that mission critical to how we work. So I think that’s been really good for us.
Question 7: Can you talk about your deployment infrastructure?
Zach: Yeah so, it’s kind of tied to testing a lot. Where if you push a commit, like the very basic example, I push a commit right now to getup/Github, our main dot com repository, if I push that to the primary branch the tests will run automatically and as soon as they’re run they’ll all push...they’ll deploy the code. Usually everything is done on a branch, and once the branch is ready to go all the tests are green and merged in to deployed again. I mean, it depends on what specifically you want to talk about I guess. All of our tests are Jenkins related infrastructure. Deployment is handled from this (ruby app), it’s called Heaven which is just Capistrano wrapper, it’s just an API that Hubot can talk to. And then that knows...you say like Hubot deploy get up to production. Hubot talks to this Heaven API, Heaven knows all of the boxes that we deploy to, all of the hundred boxes or whatever, and then it actually runs all of the commands. So basically we have all these automated steps tied onto one simple command that everyone can deploy from. Does that help anything, I don’t know. It’s complicated.
Question 8: I guess I’m kind of curious what happens when things sort of go off the rails organizationally speaking, like you have all this independence and flexibility and a very flat sort of management, if any management at all. What happens when you get a pair of people who are off on some totally hair-brained scheme, is that cool? Do they get six months?
Zach: Umm, it depends, sometimes hair-brained schemes are amazing. But there are times when you want some level of control I guess. One of the interesting things we’ve realized as we’ve grown bigger and we sort...we realized we sort of modeled ourselves after open-source. We’re sort of an open-source company, and by that I mean, we have a...there are natural maintainers of an open-source project and they are sort of the spiritual leaders / in charge of the bare metal, bare bones, like yes I want your code in my project. And in that case those are the people who I would feel like should be in charge of saying, you’ve gone too long down the rabbit hole on this one, come up for air and figure out a way to ship this immediately or can you figure out some way to take this where you can get out the door and work on some stuff. Usually there’s some level of friction that appears at some point where either you’re getting upset because you’re six months down and you haven’t actually shipped anything yet, or someone else is getting upset because you haven’t shipped anything yet. I think...we have a really strong culture of like we want people to ship stuff, we want people to keep pushing out new code and I think if it gets down to that...if it gets that far down where nobody’s shipping, you just start having conversations.
Comment from person asking question: Is this all peers looking at their peers?
Zach: Yeah, yup and I think that’s really...it’s worked very well for us. People tend to know how they’re...who’s pulling their own weight or if they’re just...you know the product they’re working on, the feature just doesn’t make any sense. And I think along side that we have a good culture of people saying no. And we try with the team app internally, pushing ideas and stuff out. You want to get people to say no as soon as possible, so people don’t spend six months on something that won’t ultimately ship.
Question 9: Along the lines of that same note, it strikes me that you folks have very focused products and you know who your user is and your customer is, how are you folks, from a product perspective, my guess is that you’re really good at saying no to features and functions that’s just not going to happen. (rest of question inaudible)
Zach: One of them is just hiring people who say no, like literally there...sometimes they’re jerks. In a nice way, but you want people who are very comfortable with that, I guess the other part of that is just building steps along the way to make people...to say no early. Like I was saying earlier. But other than that...I guess really no secret or magic formula, I guess, just say no.
Question 10: After the culture, do you folks have any job titles or has that (evolved?)
Zach: No, we have the mandated, CEO, CTO and stuff like that. We used to have directors but we kind of dropped those titles, like Director of Design and stuff. Because it’s...for us it’s mostly come down to project oriented leadership rolls, again like this idea of an open-source project. The spiritual leaders to a project tends to be the ones with the final say on something. But usually there are very few times where it escalates to the point where we need somebody external of that project to say...just to come in and put their foot down, and say like no do this some other way.
Question 11: Github seems to be synonymous with open-source software, (rest of question inaudible)
Zach: Yes. (laughter from audience) No. (more laughter from audience) We tend not to talk about future stuff to much.
Question 12: What are the kind of challenges faced by how you (inaudible)?
Zach: So one is technical, I came on 3 years ago to work on Github FI which would eventually become Github Enterprise. And dealing with this idea of...back then it was way more extreme, it was me versus ten developers who were coding ridiculous amounts every day and I would have to try and figure out, does this code work in our installable product, how do I have to change it, and it’s just like...the balance between all of that became really difficult but that something we wanted to consider and once we ended up launching Github Enterprises, could we build a model where we can kind of easily ship both products at the same time. Because they are the same product, they are the same code base except, Github Enterprises affected the super-set of features (inaudible) L-Dap, all stuff like that. So a lot of it was figuring out the best ways to deal with that, we ended up doing it...we solved that by doing the same thing we do with dot com, like we have staff features in dot com all over the place. The last time I counted we have like 40 or 50 staff only features that I have that you folks don’t have yet. We used that same sort of idea that, well you know if this is Enterprise, do this other code path, and stuff like that. So that was part of it, the other part of it is non-technical inherently. One is just sales people, that’s always been the concern is that you bring in people that aren’t douches. I don’t know how many sales people are here but, stereotypically, not you folks who ever are the sales people here. But they tend to be a lot more focused on the money, which is their job and stuff, but we still want people to be focused on the product. And I talk about our developers being able to say now, our sales people are really good at saying no, because they get the “If” question all the time, like we’ll add it only if you add ‘X’ and we’ll pay you X-hundreds of thousands of dollars, and they’re just like, no. And it’s weird when big enterprise companies, they’ll usually just be like, my boss just made me ask that, we’re just going to buy it any way. So that’s been good. That’s probably been the biggest fear I’ve had for enterprise, like they’re going to...like just because it’s such a different market it would end up adding a whole bunch of (cruff?) to the app, but because both the sales people and the technical developers on Github Enterprise have been able to say no we’re not going to add that or no we’re going to take that idea and make it more focused, it’s been able to make that side of the code base, not crazy at all. So it’s a bunch of stuff, culture and the tech side of things.
Question 13: Could I ask a follow-up question? So you have two funnels, right? Is one more important or strategic than the other? How do you manage that tension?
Zach: Umm, not really. It’s really a matter of just...because they both bring in money and they both have varying long term ideas of money. We’ve usually try to persuade people to post on dot com just because it’s easier and it’s going to be less headaches and it’s a better product idea for you folks. But I mean, if people still want this idea of hosting on their own, we’ll support that too. So it’s a matter of...umm, I think that was a fear that we were going to have this split in the company and I think almost to my surprise we’ve been able to say it’s just one product, and figuring out ways to make it one product rather than divert. Because diverging is really much harder.
Question 14: You said you folks don’t like to talk about future things, why is that (rest of question inaudible)?
Zach: Umm, it’s a number of things. One of the classic...you folks can probably remind me...it’s the computer in the ’80’s (answer from crowd The Osborne), yes, the Osborne, when they announced the next version and no one bought the current version so they had no money to develop the next version. So it’s very low level stuff like that. It’s also umm, if I say we’re going to add ponies or something at Github, you end up setting expectations, like, I want ponies, I’ve had unicorns forever but now I want ponies. And it sucks for us because we are so...work on what you want...and it’s, it’s...the phrase we usually say is, “it’s not shipped until it’s good”, type of deal. Sometimes the ponies may not arrive until 6 months from now or sometimes a year from now. And even...we’d had stuff that are definitely going to ship next week and it ends up slipping 8 months because that’s just how the cards played out. So we’d rather...instead of setting expectations, just don’t set any expectations and surprise and delight and it ends up being. And it ends up being people are super excited, like the stuff that we’ve had that have been really big ships, like when we did Issues, two or three years ago because the old one was so horrible. It was weird because we we’re using it internally for 6 months and I was just like, this is so awesome but I can’t tell everyone but I really want to. But when we finally shipped it people were like, this is amazing. I’m so happy that you finally fixed this and stuff.
Question 15: How do you determine when something is right? (rest of question inaudible)
Zach: Yeah it’s...so I’ve been asked that question a couple of times now and I still don’t necessarily know how to answer it, because I think it’s kind of weird because we have these two sides of things. One we’re really shipping ‘culture company’, we want to ship stuff, ship stuff, ship stuff, that’s really important. On the other hand we don’t want to ship stuff unless it’s ready to be shipped. So these are kind of odd ends and to be honest I don’t know how it works. I don’t know how we end up shipping stuff at all. But it’s...I think it’s both of those things, and there have been a few people we’ve had at Github were I don’t think they had as much fire under their feet to ship stuff. I think it’s not necessarily a matter their skills or anything like that, it’s just they didn’t operate in such a clear open atmosphere like that. And it becomes a problem, you have to have people ready to...like I want to get this out there as much as possible. I think along side that, something we do pretty well is celebrating ships, and I think that important because once you ship you get to write the blog post yourself, so you get some fame and glory or whatever, it’s a way of saying, yeah I did this. And we try and celebrate, you shipped this thing to Github.com today, and we’ll have a team update and we’ll say like, yeah I and you can post that. And just getting that sort of celebration from the rest of the team is really kind of contagious. So I think that end of things is just like getting people to get really excited about shipping stuff is sort of that side of stuff.
Question 16: Here at Hubspot, we work in teams of 3 people (inaudible) but they own that kind of (API?) or that project, are they responsible for the engineers as well. How do you folks deal with the operations the reliability (inaudible)?
Zach: So we have an Ops Team which...their ultimately in charge of all the Up-Time and Scalability and stuff at Github.com, it’s...like I was saying earlier...we operate in much the same way, where you own that feature, and owning is not just deploying the production. Owning is...you deploy the production and it’s also on you to realize the scale and implications of what you’ve done. So we have a lot of tooling around, data base cores and stuff. That’s a very simple one. If the data base is dying under the load, and it’s probably your fault...you have to fix that basically. There’s a side of that and also just the Ops Team, their good at telling you if...this is crumbling under the load you have to get on this and start working on it and they start working together and dealing with all that.
Question 17: So you talked a lot about feedback and kind of how you have the breakdown of (inaudible), how do you think that correlates to (rest of question inaudible)
Zach: I think the (Bride?) groups are kind of similar...you may have different...it’s weird because Github is like...we’re both blessed to have developers as our audience and cursed, just because we have stuff were, you know, we’ll get dissed for our CSS and say if you fix this, this will fix this browser. And we’re like, that’s awesome. And then we have the same people who are like, yeah I can fix this in an hour, why can’t you just add me to the repo? And we’re like, it’s not that simple. But I think...people still have the same motivations and stuff regardless of the industry they happen to be in. So I think if you can still identify the same type of people, there’s still grumpy people in Sales and Marketing. I think it’s just much more a matter of identifying what their state in the marketplace is. Is their feedback something I need to be worried about or not?
Question 18: (most of question is inaudible) (basically asked if certain products would be for the masses or not)
Zach: That’s an interesting question, I think umm...developers get all the cool stuff, at least a decade ahead of time, like Git is amazing. And if you talk about the high level too, like my mom’s a lawyer, and I talk to her partners and stuff, they’re always like, that sounds amazing, like, I want that. And then you just think like, yeah then you tag this (blob as a tag and here’s the shot-one?), and does not work. People have been trying to come up with a more user friendly way...not even Github in particular, just all version control systems and it’s difficult. I think TimeMachine is a really good example of using really powerful underlying technology and making it more accessible to most people. So I think like, yeah I really think like developers do get the cool stuff early and I think that proves that this is a really good way to work. The real question is how do you make it just digestible by the masses. And that’s going to be really difficult.
Question 19: I really liked it when you said that, something doesn’t ship until it’s good. Do you ever have difficulty on people agreeing what “good” is?
Zach: Yeah! A lot, and it depends like, usually...it really depends on the context, sometimes you paint the shed and you have to step back and say like, wow this doesn’t really matter at all. Sometimes there are cases where people get really into in-depth conversations about stuff, and that alone is usually a good sign that it’s not right. If there’s any discussion on it at all, if it’s really passionate discussion. It’s probably a good sign that something is wrong and at least take a step back and say, is there another way to do this or what do we want...how do we want to take it from here? But it really depends on the situation I think.
Question 20: What’s your favorite staff feature on Github?
Zach: (laughter) Man, if I brought up Github.com right now I’d blow your minds. It’s got (dash) -next at the end of it on the branch name. It’s like all of our features I guess, but it’s awesome. That’s all I can say. (laughing)
Question 21: How deep do your designers need to be in the software? Who’s responsible for the code?
Zach: Yeah we do, I think almost all of our designers...we have a couple of illustrators now who aren’t really code oriented at all, but of our designers...they at least know HTML, CSS, mostly JavaScript too. Which I think...that’s definitely been a conscience decision of who we’ve hired because we want people to be able to...you know you end up building designs differently when you actually know how they’ll end up being created. And it’s....like I was saying earlier, we tend to do 2 person teams at least for dot com for designer and back-end, so at least on that one I’ve teamed up with some designers before who I know can do the (ruby?) to do that. But it’s nice to say, alright, you handle that, you’ll probably do it better and stuff like that. We have a lot of designers who’ve picked up (Ruby?), this is the first time they’ve used (Ruby?) before, and even though they don’t necessarily throw down on (Ruby?) in the app, it’s been super helpful for them to actually understand the background of things. We’ve been trying to figure out better ways to sort of do in-house training of each other just because, even though you won’t ship on it, it’s still interesting to learn and stuff like that.
Question 22: How do projects get open-sourced, like how did Hubot get open-sourced and why, for example is the deploy infrastructure not open-sourced?
Zach: It really needs a winner...umm not a winner, a champion. Because people have asked stuff like that before and it’s just like, open-source is not just a switch, because if it is, we would have open-sourced a ton of stuff in the past. But you have to have open-source documentation, figuring out like what the best way to deal with like peoples weird environments and stuff like that. And then the biggest part is just supporting all the bugs and issues and stuff like that. Which has bitten us in the past after like...year two, somebody becomes less interested in that project and they move on to other things and you have stale open-source project and that makes everyone upset. So for Hubot, that was sort of designed from the beginning because we were, we liked how that changed our company. We were really...we could end up doing...it transformed how we chatted basically. So that perspective was like yeah, we want to have other people working that way as well, same with Boxen. It was like, this changed how we set up computers, let’s open-source this and let’s let everyone set up their computer just as well. But the deployment stuff and the testing stuff...umm well our testing stuff is open-source under (Jenky?), but our deployment stuff is more...that’s really difficult because that’s tied to Github.com stuff, that’s weird, finicky app related stuff that maybe we only run into. And then everybody’s got weird deployment stuff too. So it just really depends on somebody wants to do the work to do all of that stuff. And usually we end up with people that would rather just ship products, rather than dealing with open-source issues that doesn’t directly relate to Github.com and stuff like that so.
Question 23: Alright, related closing question. I’m going to put you on the spot a little bit, talking about...you folks are an open-source powered company, you think a lot about culture, you love the Valve document. Would you consider open-sourcing your culture code and publishing things that you use internally to run the company essentially?
Zach: Yeah, but it’s, it’s...that’s the reason why I give a lot of these talks. Is just because that’s sort of...because a lot what we do is not necessarily a document or something like that or an application or anything like that. So...
Dharmesh: You gotta have something though, let’s say when you have 20 new people join in the next 2 or 3 months, like osmosis is not that useful, right?
Zach: We’re starting to figure out...we’re always trying to figure out the best way to deal with all that stuff is. For a long time we had a really detailed guide for like...this is how this is all run and stuff like that. Then came the update and we had some different opinion, like it changed a whole bunch. So like we never really had a document that we can point to and say this is how we do stuff. It think the credit to Valve is that they figured out how that’s what they wanted their document to look like.
Dharmesh: I have an idea, what if you just documented the first 3 days of a Github employee, like all the things I went through, how they acquire stuff, what kind of questions they have, their email threads, what they looked at...and publish that, that would be cool. Anyway, thanks so much for your time!