Here at HubSpot, we ship new code quite often. Development at such a rapid pace only works if you can ensure as little disruption as possible (if any) for your customers by using strategic patterns and techniques. We have a system in place that allows us to toggle features on a per-customer basis and we recently applied it to successfully rewriting an existing, high-trafficked API. We implemented this with our own homegrown gating system, and our nginx loadbalancing tier, but the general pattern could be applied to other architectures, too. Here’s a rundown of how we did this and the steps you can take to address issues with a new system before they impact your overall customer base.
Step 1: Create internal proxy in nginx Load Balancer
Step 2: Check gate early in request filter in new API
Step 3: Reroute all existing API traffic to new API load balancer
Step 4: Validate, Verify, Vanquish (the old API)
As you can see, using an internal proxy at the load balancer can be a powerful technique for rolling out new code in a controlled fashion. An added benefit is that you aren't tying up your new API application server with the I/O of proxying requests to the existing API during the rollout period.