At HubSpot, pretty much all of our backend services are written in Java. As part of keeping our Java stack up-to-date, we've been working on the upgrade from Java 8 to Java 11 for a long time. Doing this sort of upgrade smoothly and safely across a codebase as large as ours, while not disrupting the workflow of our hundreds of backend engineers, is no small feat. This post will cover a snag we hit when we thought we were finally done.
We'd been building all of our apps with Java 11 for over a year (but still targeting Java 8 bytecode). We'd been running all of our apps on Java 11 in production for five months. We'd been targeting Java 11 in all of our leaf modules (ie, not libraries) for three weeks. We upgraded ASM, CGLIB, and ByteBuddy everywhere. We fixed all of the dependency issues with jaxb. We forked Spark 2.4 to make it work on Java 11. We thought we had done our due diligence and were ready to target Java 11 everywhere.
So we went ahead and made the change to target Java 11 globally (which we announced far in advance). Services picked up this change throughout the day as they rebuilt and redeployed. A few hours later, a report came in of an issue in our QA environment, with this stacktrace included (class names replaced for brevity):
The InvalidClassException suggests that the error is related to Java serialization. Before this, if you asked me whether HubSpot uses Java serialization to persist data, I would've said "Of course not, it's not 2009 anymore!" But it turns out that a relatively important service was accidentally using Java serialization to store data in HBase, and the class it was serializing did not declare a serialVersionUID. And for some reason, the Java 11 upgrade changed the computed serialVersionUID, meaning that this service could no longer read data out of HBase. Luckily, our automated monitoring and alerting detected the issue in our QA environment, so there was no impact to our production environment. In response, we rolled back the upgrade and started to investigate.
As a sanity check, we first confirmed that the Java 8 and Java 11 versions of the class did in fact have different serialVersionUIDs. We checked this using the serialver tool included with the JDK.
So, targeting Java 11 is changing the serialVersionUID, but why? To be fair, the Javadoc for Serializable makes it clear that serialVersionUID can break across compiler implementations:
"[I]t is strongly recommended that all serializable classes explicitly declare serialVersionUID values, since the default serialVersionUID computation is highly sensitive to class details that may vary depending on compiler implementations, and can thus result in unexpected InvalidClassExceptions during deserialization. Therefore, to guarantee a consistent serialVersionUID value across different Java compiler implementations, a serializable class must declare an explicit serialVersionUID value."
But looking at the specification for computing a serialVersionUID, it wasn't clear why it would be different.
The source code hadn't changed, so the class name, modifiers, interfaces, fields, constructors, and methods should have all been the same. However, we were able to see the problem using javap, the class file disassembler included with the JDK.
At HubSpot, we generate immutable data classes and builders at compile time using a library called Immutables. However, the generated code isn't very pretty to look at, so I crafted a heavily simplified version:
First, we compile this class targeting Java 11 and then run it through javap:
Next, we do the same thing targeting Java 8:
And here we see the problem; when targeting Java 8, an extra package-private constructor is included. And the specification for Java serialization says that all non-private constructors are included in the serialVersionUID calculation. So now we're starting to see what's going on; when targeting Java 11, a package-private constructor is removed, which changes the computed serialVersionUID. But what's the deal with this constructor? Where is it coming from and why doesn't it appear in the Java 11 class file?
The first thing to note is that SomeClass has a builder, which is an inner class. And this inner class calls the private constructor of SomeClass. But the builder is compiled to a separate class file, which won't be allowed to access this private constructor at runtime. In order to support this, the Java compiler (before Java 11) generated a synthetic, package-private constructor. The builder class can then use this package-private constructor, rather than invoking the private constructor directly. In Java 11, however, JEP-181 landed. This JEP enhanced the class file format and JVM to directly support these inner class accesses, which eliminates the need to generate a synthetic constructor. And now we fully understand why our serialVersionUID changed when targeting Java 11.
In the short term, we plan to use our tooling for automated refactoring to find serializable classes that don't declare a serialVersionUID and set an explicit value (equal to the current computed value). Next, we will start failing the build if any serializable classes are detected that don't declare a serialVersionUID. These changes should allow us to finish rolling out Java 11 safely. In the longer term, we plan to find and remove any lingering usages of Java serialization, as there are almost always better options available.