HubSpot's approach to front-end development: Part 2

In part 1 about HubSpot’s approach to front-end development, we talked about common issues with sharing front-end code (JS, CSS, etc) across teams and how dependency management can help.

However, dependency management of front-end code is a relatively new thing. There have been a few projects that have started taking that path, but developers haven’t yet taken into account how front-end web code is subtly different from typical desktop or server-side code.

For the past year at HubSpot, all of our web apps have been built using something we call Asset Bender (historically we’ve also called it Static3). And it is a crucial piece of infrastructure that helps us ship better, more consistent web apps even faster.

So what is Asset Bender?

Asset Bender isn’t a single tool. Rather, it is a suite of several tools that work in concert. It includes:

1. A dependency management system for Javascript and CSS
2. A local server, package manager, and precompiler
3. Build tools that optimize and link the project's dependencies
4. A server-side runtime that allows on-the-fly dependency updates

Dependency management by magic URLs 

At its core, Asset Bender performs dependency management by changing existing URLs in your code and inserting version numbers. I like to call these "magic URLs" because this conversion happens automatically, without any intervention on your part.

For example, let’s say that you had some URLs like this in your code:

/* CSS */
background-image: url(/app1/static/img/background.png)

// Javascript
jQuery.get("/app2/static/js/some-code.js")

<!-- HTML -->
<a href="/app3/static/html/page2.html">Page 2</a>

After getting built and deployed by Asset Bender, they might look like this:

/* CSS */
background-image: url(/app1/static-1.7/img/background.png)

// Javascript
jQuery.get("/app2/static-3.42/js/some-code.js")

<!-- HTML -->
<a href="/app3/static-2.13/html/page2.html">Page 2</a>

I’m jumping over a lot of steps here, but bear with me. The basic idea is is that asset URLs in code look like:  /<app-name>/static/<asset-path>. And when deployed they look like:  /<app-name>/static-<resolved-version>/<asset-path>.

Version numbers

In Asset Bender, version numbers have two parts, a major and a minor version number (<major>.<minor>). Like other versioning systems, the major number denotes significant, breaking changes. And it is each application’s responsibility to increment the major version number for “major” changes.

On the other hand, the minor version number is for what it seems, “minor” updates that are backwards-compatible. However, unlike major version numbers, you never need to manually increment minor version numbers. Instead they are automatically incremented every single build. So the version numbers that come out of our Jenkins builds look like this:

1.1, 1.2, 1.3, …, 1.9, 1.10, 1.11, …

Having these automatically increasing minor version numbers can lead to some pretty silly versions (we have ones like 2.5764), but it works well with our “constantly shipping” philosophy. Not only would it be a pain in the butt to make developers have to manually change the minor version number frequently, but it also makes every single build referenceable. That gives us a lot of flexibility when depending on specific versions or rolling back.

Now that you understand the basic building blocks—version numbers, and versioned URLs—let’s start to walk through the various pieces of Asset Bender. It all starts with the local dev server.

The local server

You start the dev server with a simple command, bender run. By default, that starts up a local HTTP server on port 3333 that logs to your terminal. But it does more than just serve files, it will also:

1. Compile SASS, CoffeeScript, Handlebars templates, and others
2. Combine multiple files and/or directories together into a single Javascript or CSS file
3. Automatically adjust your asset URLs to point to the exact version of the dependencies you’ve downloaded

Also, it is useful to note that all of the above are done on-the-fly with each request. That means that:

1. You don’t have to run any other filesystem watcher processes
2. Errors are in-your-face in the browser
3. You never run into the dreaded, "I pressed refresh but my changes didn't show up" issue (which can tend to happen with large, complicated SASS files).

An example project

Let’s say you are working on a local project called app1 that has a directory structure like so:

app1/
  static/
    static_conf.json    -> Configuration (dependencies, etc)

    img/… 
    coffee/
      app.coffee        -> Javascript entry point
      main.coffee       -> Most of the app code
      some-plugin.js    -> A custom jquery plugin

    sass/
      app.sass          -> CSS entry point
      other-styles.css  -> More styles

And here are the contents of app.coffee and app.sass (the main two assets that your app includes):

app.coffee

#= require ./some-plugin.js
#= require_tree /other-app/static/js/components/
#= require ./main.coffee

app.sass

//= require ./other-styles.css
//= require /another-app/static/sass/shared/base.sass

@import "/another-app/static/sass/shared/contants.sass"

#my-awesome-app
  // best styles ev4r! …

If you’ve ever used Rails 3.x before, this will be familiar (Asset Bender is built on top of Sprockets, which powers the Rails Asset Pipeline).  But for those of you that haven’t, those require directives act as a preprocessor to include other files or a directory full of files inline into the current file when served. Also, they can refer directly to SASS or CoffeeScript, compiling the contents before inlining it.

That’s cool, but it is standard with Rails. The thing that Asset Bender brings to the table is referencing files in other projects (those lines with other-app and another-app above).

Depending on your setup, you may or may not have the source for other-app and another-app checked out on your machine. But for now, let’s assume app1 is the only project you’ve checked out.  How will this code get access to those projects? 

Installing dependencies

As things are right now, if you requested http://localhost:3333/app1/static/coffee/app.coffee or http://localhost:3333/app1/static/sass/app.sass, you’d get an error because those dependencies haven’t been downloaded yet. However, you can solve that by running bender update-deps.

That command will look at the configuration of app1 (aka static_conf.json) and download the latest version of each dependency to a local archive folder. So, after running that command in our example, the archive folder might look like this:

~/.bender-archive/
    other-app/
      recommended   -> version “pointer” file
      edge          -> version “pointer” file

      static-2.19/  -> specific version of other-app’s source
        js/…

    another-app/
      recommended   -> version “pointer” file
      edge          -> version “pointer” file

      static-1.6/   -> specific version of another-app’s source
        sass/…

We will talk about those pointer files later. The important thing to understand now is that the archive folder collects specific versions of Asset Bender dependencies. This is roughly similar to how Python modules end up in the site-packages/ folder or JARs installed by Maven end up in ~/.m2/repository/.

Also, the Asset Bender server will automatically serve those dependencies. That means you can visit a URL like http://localhost:3333/other-app/static-2.19/js/components/some-file.js (notice that dependencies need to be referred to by a version number).

Bringing it all together

OK. Now that those dependencies are installed, we can successfully request app.coffee and app.sass from app1. And when you hit http://localhost:3333/app1/static/coffee/app.coffee, Asset Bender will interpret this line:

#= require_tree /other-app/static/js/components/

as:

#= require_tree /other-app/static-2.19/js/components/

Essentially, Asset Bender recognizes when you are referring to a dependency and automatically replaces the appropriate version. And version replacement can happen practically anywhere: in compiled SASS and CoffeeScript code, in require directives and SASS @import statements, and in any plain HTML, JS, or CSS you might have.

All of this simply means that you never need to worry about versions in your code, Asset Bender will handle it for you. And if you ever need to upgrade to newer versions of your dependencies, all you need to do is run bender update-deps again.

Continue on to part 3 (available soon) to learn more about Asset Bender’s internals, how those versions get resolved, and how all of this works on production. But, if you’d like to skip ahead and see a preview of the code, you can check out the Asset Bender repo on Github (but don’t worry, we’ll get to it if you keep on reading).