Houston, we have a problem
There are some performance drawbacks to this approach. Because of the update policy of always, each build needs to check whether there's a new version of every snapshot. To do this, Maven needs to fetch the snapshot metadata and its checksum from the remote repository (Nexus, in our case) so that it can compare to the cached data in the local repository. This requires at least two round-trips to Nexus for each snapshot dependency. If an app has hundreds of snapshot dependencies, this latency starts to add up. And for our Dublin engineers, where each of these round-trips is transatlantic, the latency is downright unusable.
We were able to partially mitigate this issue by writing a local proxy that would cache responses for snapshot metadata and return the cached data if a snapshot hadn't changed. Each developer ran this proxy locally and configured Maven to use it instead of hitting Nexus directly. This sort of worked. But it was never perfect, because the cache invalidation wasn’t as reliable as we liked and our HTTP proxy just wasn’t very good. And even with the local proxy, snapshot resolution still added a few seconds of overhead compared to running the build in offline mode. But it was better than nothing, so local development relied on the proxy and our CI environment relied on low latency to the remote repository (which meant running all of our builds in the same region as our Nexus server).
The snapshot accelerator
So we started designing a system that allows us to hook into Maven and skip the metadata fetch for a dependency if we know the snapshot hasn't changed. This is a similar idea to the proxy, except that it allows us to bypass the metadata HTTP requests entirely (which is noticeably faster, possibly because Maven isn't great at parallelizing this sort of work). This design also means we don't have to proxy requests to Nexus, which eliminates some complexity and operational issues. We also redesigned the way snapshot versions are tracked so that the system is reliable enough to use in our CI environment.
We have open-sourced this system as the maven-snapshot-accelerator.
Putting it all together
mvn -B deploy com.hubspot.snapshots:accelerator-maven-plugin:0.3:report
This will invoke the accelerator plugin after the deploy phase in order to report the new snapshot version to the accelerator API.
- -Daether.connector.resumeDownloads=false - To prevent one build from trying to resume the download of another concurrent build and potentially corrupting the local repository
- -Daether.artifactResolver.snapshotNormalization=false - To make Maven use the fully resolved snapshot JARs for everything, which should be immutable and not change out from under us
(As a side note, you can tell from that graph that our engineers work from roughly 10am to 6pm. So next time you're interviewing somewhere, don't ask about work-life balance. Just ask to see their Nexus graphs.)
And here is a chart showing the Nexus request latency during the same time period:
You can see that each spike in traffic had a corresponding spike in latency, and now both graphs are much more stable.
If you have any thoughts, feel free to leave a comment below and don't forget to check out the maven-snapshot-accelerator on GitHub.