The Design Process That Helped Us Ship a New Product in 11 Weeks
Talk to any engineering team about their process and you’ll hear a common theme: speed. There’s an entire culture around shipping fast and deploying ...
Apologies for the title. Here's what I wanted the title to be:
Evolution of a Web Developer: From Newbie to Ninja Not-So-Newbie
because I don't like using the term ninja (or rockstar or guru). But, putting the striketrhough in an article title is ill-advised.
Some of you may have come out of the womb hacking shell scripts to turn off that Internet-enabled baby video surveilliance monitor that your parents used to watch what you're doing on their iPad in the kitchen (visualize eTrade baby hacking away on a terminal app on his Android phone). But, I suspect most of us started more modestly than that and moved up the learning curve (some faster than others). The following is a somewhat fictional, somewhat true recollection.
Favor request: Please do not misguidedly think we have a bunch of newbies on the HubSpot dev team. Quite the opposite. Most of them did come out of the womb hacking shell scripts. At the office, they speak some weird alien language that every now and then has words in it that make me suspect they're talking about software development. But, a lot of the times, I think it's a diabolical plot to make me keep feeling like a rookie.
In any case, on with the show. Note that the time elapsed between steps varies dramatically. I never had the talent for literal cadence (or whatever it is you call it when the time between things is consistent).
Evolution of a Web Developer: From Newbie to Ninja Not-So-Newbie
1. I put the words print “Hello World” into a file called "helloworld.php" and I can make it show up in a browser. Woo hoo!
2. Well, that was kind of lame. But, now I'm actually doing something dynamic and can include the current date and time. “Hello World on 2011-08-10 00:47:02.355000”
3. Learned how to pass parameters via the URL in the query string. It's kind of like the command-line argc, argv thing — but not.
4. Look ma, I can run database queries!
5. Using Javascript to actually run some code on the client-side. alert ('Hello World!”).
6. PHP is ugly. Decide to use CodeIgniter to make it less so. Considered Cake as well, but went with CodeIgniter mainly because I had friends that had used it. Took me a couple of hours to get a handle on things.
7. Realized that my friends will forever make fun of me if I stay on PHP. And, Ruby as a language feels a little strange. Maybe it was designed for hipsters. If I had a dog, it would bark at Ruby code.
8. Python and I experience a "love at first sight" moment (well, the love might have been unrequited at the time, but still). The language is elegant and clean and all the things a language should be.
9. Went through all of “Python The Hard Way” just for fun.
10. The Django tutorial isn't too shabby. Only downside is that it made me want to just keep going with the polling app (I've always wanted to write my own polling software!) I wonder how much influence development tutorials have on actual apps that get developed. That could explain why there are so many custom blogging apps out there.
11. Writing CRUD queries from scratch is for the birds, learned about ORMs and database abstractions in Django.
12. Wait, what, Django's ORM doesn't support composite primary keys? That sucks! I'm just going to write my own database layer.
13. Reconsidering decision to write my own database layer. Maybe in the early days, it's OK to have auto-incrementing primary keys. I can always “fix it later”.
14. Discovered jQuery. Have no idea how humanity even existed before jQuery.
15. Used right-click, “Inspect Element” for the first time. FireBug forever changes my life.
16. Processing POST requests is almost as easy as GET requests. Only cooler and no length limitations.
17. Man, spammers suck. Need to go find a good CAPTCHA library. I think recaptcha is a pretty cool idea.
18. Heard that concatanating strings to form SQL queries is a bad idea. Looking up how to do bind parameters and escaping the strings. I don't care which ORM you use, you're going to someday need to write direct SQL queries.
19. Session management makes stateless applications more statefully.
20. Database calls are slow. Memory access is fast. Memcached FTW!
21. AJAX makes web apps oh so much cooler. (Quietly blessing John Resig again for creating jQuery. Super happy that I didn't talk him out of his crazy idea when I met him years ago).
22. Considered using SQLAlchemy. Seemed powerful, but all I really want is for Django's ORM to not suck, not do something completely different from the active record pattern.
23. Learned how to invoke other websites and applications from my own code. This could get dangerously fun. Or maybe, just dangerous.
24. Ah, the beautiful serendipitous wonderfulness that is Hacker News. Learned about the Python requests library. I will now judge all libraries based on this work of beauty.
25. Learned that everybody else's HTML sucks and is not XHTML compliant and won't parse worth a damn.
26. Learned that trying to process in-the-wild HTML with regex is close to pure sin. Figured out that Beautiful Soup is indeed, quite beautiful. (But not very soupy).
27. Wrote a note to self to learn how CSS is actually meant to be used. For the 5th time.
28 Learn that my HTML sucks, is not XHTML compliant and won't parse woth a damn.
29. Spend half a day looking for libraries and frameworks that make CSS suck less. Decide none of them are “mainstream” enough yet. A little voice inside my head is telling me I'm missing out on something important.
30. Ooh. Came up with a clever way to use Javascript to insert dynamic hidden fields into my forms to stop the spammers without aggravating users with CAPTCHA. Later learn that this was not an original idea and had been done like, a million times before.
31. Read up on cookies. Then non-expiring cookies. Then cross-domain cookie issues. Then statistics on how many users turn off cookies in their browser. Then blog article about how Europe might make cookies illegal. Then just went and ate a chocolate chip cookie. Felt much better.
32. Learn there's actually a way to return things other than HTML from a web application. Not sure why I'd ever want to do that, but you never know.
33. Wait, wait, wait — you mean I can create a dynamic image from my code? <img> tags don't have to point to static JPG/PNG files on my web server? That's awesome! I can build my own meme generator. Woo hoo!
34. Wouldn't have believed it if I hadn't seen it myself, but SASS does make CSS syntacticaly awesome(er).
35. So, I have these objects in a simple two-level hierarchy. And I have a relational database (MySQL) and Django has an Object Relational Mapper (ORM). So, you'd think that the ORM helps you take the objects and map them to database tables in somewhat natural ways. Au contraire mon frere! ({pardon my French). It gets many of the basics wrong.
36. Consider doing A Very Bad Thing. Simply taking data about my "child" objects and munging them into JSON and sticking them into a single database column Avoids some of the issues with how Django handles things, but introduces other problems (like not being able to query on any of those object properties. Plus, I have this feeling that a kitten is going to die somewhere if I do this.
37. Start reading up on MongoDB. Seems simple enough. Get it up and running in a few minutes. Inserting and querying in a few more. Now, I can have my objects stored as documents with the hierarchy loosely represented. And, it's query-able. I don't have anything against SQL (we go way back), but MongoDB feels like a more natural fit than trying to munge my simple objects into a relational database.
38. In my meanderings for ways to use MongoDB in cleaner, simpler ways, come across DictShield. It's a wonderfully useful and simple library. And @j2labs is a super-nice guy who gives near instantaneous support (including answering basic questions that have nothing to do with DictShield)
39. Learned the hard way that MongoDB 32-bit edition has a database size limit of 2GB. Decided to stop being a cheapskate and spin up a 64-bit Amazon EC2 instance. Besides, I'm not playing around any more. The database is growing fast (it involves pulling down small parts of the interwebsociosphere.
40. Lamenting the fact that MongoDB doesn't have full text search (even MySQL had that). Reading up on Elastic Search and just saw that Amazon now has some new hosted search service that looks pretty functional. (Impressed how quickly those Amazon folks put new things out there – who knew that an online bookstore would rule the cloud computing world?)
41. Digging into the Python Natural Language Processing Toolkit. Makes my head hurt. Learning curve is a bit steep (or I'm a bit impatient). All I need to do is some simple Bayesian classification. (Famous last words). Considered the new Google Prediction API, and though it has promise, is almost completely unusable for my simple use case (do some simple tweet classification).
42. Starting to read "Natural Language Processing with Python". Don't grok all of it, but that's OK. I can feel myself getting a wee bit smarter.
Clearly, there's still a lot I have to learn – and I skipped a bunch of steps along the way. So, where are you in the "evolution"? Are you like at step #83 or something? Any other major milestones you've hit along the way that you'd like to share? I'll award the two funniest submissions a $50 Amazon gift certificate, so you can read whatever you need to read to get to the next step in your development.
Please consider voting this up on my favorite news site if you enjoyed it.