CoreRT and MessagePack - Real World Example

While the benchmarks I recently posted showed a lot of promise for combining MessagePack C# with CoreRT, I wanted to run it with a real world example to confirm that this is worth the effort, so this evening I updated Last 'n Furious, now available on itch.io.

The original version was 97MB, which zipped down to 41MB.  The second iteration - using ReadyToRun, PublishTrimmed and Warp as detailed here - came to 32MB.  The new CoreRT version is 45MB which zips down to just 16MB, so we have halved the size.

Where previously I'd been happy to get in-game in about a second, the CoreRT version now shows the main menu screen in half that time.  It feels snappy like a native executable.  The improvements can be clearly seen side-by-side:

As always, there's more work to be done to integrate MessagePack fully into XAGE's pipeline, but the results already justify the effort involved.

Experimenting with MessagePack

Serialization keeps me up at night.

For many genres, saving the game state may be as simple as recording what the current level is, along with a score.  For others, like RPGs and Adventure Games, things get a lot more complicated when you're having to save the state of every item in the game at any given point.

There are many serialization mechanisms available to choose from.  For XAGE, the needs are:

• Performance - the faster the better.
• File size - too large a payload can typically affect performance, especially when disk IO is a bottleneck.
• Ease of use - from both an engine and game perspective, the user should be free to think about serialization as little as possible.  So no contracts or schemas, ideally no per-field attributes, and built-in version tolerence.
• Platform support - must work on all platform, including those with restrictions on JIT.

Early versions of XAGE used XML for the game state (and still do within the editor tools, as it is version-control friendly) but evolved to use Protobuf-net, a C# library that allows you to perform contractless serialization in Google's binary format.  This improved performance and file size significantly, but had one major drawback - lack of AOT support, required by some platforms like iOS and UWP.

Protobuf-net initially had AOT support by pre-generating a serialisation library, but official support for this was dropped.  The author has been waiting for Roslyn Generators in order to embed the serialization algorithms in at compile time.  Unfortunately this has not yet materialised and remains on the future roadmap.

Other serialization libraries exist with AOT support like Ceras, but without version tolerence, so have not been considered.

One I've had my eye on for a while is MessagePack for C#, which has long promised fast serialization times and low payload sizes with LZ4 compression.  Recently their AOT solution - a seperate executable called mpc.exe to generate the serialization logic as C# code - has become available as an MSBuild task, essentially allowing you to automate this process.

MessagePack's benchmarks always looked promising, but it's always important to test using your own data structure - in my case XAGE's main GameContent class.  For this I used BenchmarkDotNet which has become the industry standard for getting consistent and meaningful .NET benchmarks.  With this I was able to get results comparing the standard XML Serializer, Protobuf-net and various flavours of MessagePack (all using string keys rather than integer keys, for version tolerence):

• LZ4:  Where the payload is compressed using this performant compression library.
• MPC:  Where the serialization logic is generated by mpc.exe at build time, rather than determined and emitted at runtime.
• CoreRT:  Using the CoreRT AOT runtime, instead of .NET Core 3.1 (where use of mpc.exe is required)

However it wasn't all plain sailing as:

1. I couldn't use the BenchmarkDotNet nuget package as the latest SimpleJob overloads weren't in place, allowing you to combine a ColdStart test with the CoreRT runtime.
2. I couldn't use the MessagePack nuget package for the CoreRT tests due to some emit issues that are outstanding.  Compiling from scratch using some conditional symbols (NET_STANDARD_2_0; UNITY_2018_3_OR_NEWER; ENABLE_IL2CPP) resolved this.
3. All my classes had to be decorated with MessagePack attributes, and all public properties not needed with [IgnoreMember].
4. Protobuf-net's surrogate mechanism (for types like XNA's Vector2 and Color classes) is not supported and had to be reworked to be more generic in order to support MessagePack.

Once these issues were resolved, I was able to test with the largest GameContent data I had, and the results were surprising:

Note that these are 'Cold Start' benchmarks - i.e. single iteration tests with no warmup, repeated many times over.

Here the benefit of pre-generating the serialization logic is clear to see (MPC), but the performance of the CoreRT incarnations blew the rest away.  What took XMLSerializer 1247ms and Protobuf 848ms took MessagePack just 36ms from cold.  Adding LZ4 compression to reduce the filesize to 255K only brought this up to 40ms.  I dumped the GameContent as JSON in the benchmark cleanup methods to confirm the data was actually being loaded and saved correctly, as I didn't believe the results at first.

While it would take some time to fully integrate MessagePack into XAGE, it currently appears to be the best option in the absence of protobuf AOT support.  I just hope that .NET 5's AOT strategy works as well as CoreRT.