Big fat update (FNA3D, UWP, new functionality)

A new version of XAGE has been released on itch.io to alpha users (keys are still available to anyone who wants one):

• Automatic UWP project generation (i.e. you can now have your games target Xbox One & Windows Store).
• A new backend renderer: FNA3D.  The version currently used is still limited to OpenGL renderer but new ones - e.g. Directx11 and Vulkan - are on the way, which is an exciting development as it will further help future-proof the platform, as well as bring performance improvements.  To help with this (and free up some disk space on my poor laptop), I've set up some automated builds of the UWP dlls, via Github Actions.
• Recent engine changes around gamepad support, viewports & cameras etc, as well as various bugfixes.

All template games have been updated accordingly.  At some point documentation will need a lot of focus, but in the meantime let me know of anything that needs attention.  I've added a project on github to track all items considered necessary for a public release.

Viewports & Cameras

AGS v3.5 brought some new functionality, including dedicated mechanisms for handling Viewports and Cameras.  Put simply, this allows you to control which parts of the room to display on the screen, and allows you to do things like split-screens and simple zooms, and stuff like this:

Implementing this took longer than I thought, as I had to rework XAGE's rendering flow several times to get it right.  There's a slight overhead as we're now performing two rendering passes - one to project the game onto a room-sized rendertarget, and another to project the cameras onto the viewports on the final rendertarget, which gets displayed on the screen.  Having said that, Last & Furious still only consumes under 5% GPU on my old HD4400, so the overhead is minimal.

Next up is working on a port of Rellax which, among other things, will help me finish up some of the missing GUI controls like Sliders and Listboxes.

Official Console Support

While the port of Last 'n Furious from AGS had been completed last August, it was a good candidate for getting the first XAGE game out Xbox One.  It needed a bit of work, but not much:

• Controller support had to be exposed to the game scripts and incorporated into the menus and car handling.  This turned out to be pretty straightforward as the decision was made to simply expose the GamePadState via the scripting interface, as well as a new OnButtonPress event for convenience.

• Basic Xbox Live functionality (sign in) had to be incorporated in order to pass certification.  This turned out to be slightly fiddly, but from a developer point of view is pretty simple to incorporate into a game.  Under the hood, XAGE has a new basic PlatformService interface that - in theory - can be injected with various common services (e.g. Xbox Live, SteamWorks).  This may be padded out in future to support other common functionality like presence, achievements, leaderboards etc. in a general way, so as not to overly complicate the core game script.

With these changes in place, Last 'n Furious was approved for release on Xbox One (as well as the Windows 10 Store):  https://www.microsoft.com/en-gb/p/last-n-furious/9p3c8dq7qfsh

While the desktop platforms support one-click publish, the UWP equivalent is a bit more involved (Microsoft makes you jump through a few hoops - creating developer accounts, Xbox Live sandboxes etc).  It is all documented on their own site, but I'll work on putting the hooks in place so that XAGE does as much of this for you as it can.  Once this process has been trialled on some of the other ported games, I'll push out an update to XAGE Editor for alpha users.

Space Pool Alpha

Another AGS port has been completed - this time Steve McCrea's Space Pool Alpha, which isn't an adventure game at all but rather a vector-based pool game with spaceships.



The XAGE CoreRT build can be played on itch.io.  The source code is available on github (with thanks to Steve).  An update has been pushed for XAGE alpha users so they can clone/download within XAGE itself, and tinker with it that way. 

Like Last & Furious, this was again a more technical port, where the challenge was getting the DynamicSprite and DrawingSurface performance up to an acceptable level.  The results are similar - lower CPU usage than the AGS version, and higher GPU to reflect the work being offloaded onto the graphics card.

I especially like these sorts of ports - smaller and not narrative driven - as I can iterate quickly and they allow me to tackle some of the more complicated items on the outstanding list. 

CoreRT and MessagePack - Real World Example

While the benchmarks I recently posted showed a lot of promise for combining MessagePack C# with CoreRT, I wanted to run it with a real world example to confirm that this is worth the effort, so this evening I updated Last 'n Furious, now available on itch.io.

The original version was 97MB, which zipped down to 41MB.  The second iteration - using ReadyToRun, PublishTrimmed and Warp as detailed here - came to 32MB.  The new CoreRT version is 45MB which zips down to just 16MB, so we have halved the size.

Where previously I'd been happy to get in-game in about a second, the CoreRT version now shows the main menu screen in half that time.  It feels snappy like a native executable.  The improvements can be clearly seen side-by-side:

As always, there's more work to be done to integrate MessagePack fully into XAGE's pipeline, but the results already justify the effort involved.

Experimenting with MessagePack

Serialization keeps me up at night.

For many genres, saving the game state may be as simple as recording what the current level is, along with a score.  For others, like RPGs and Adventure Games, things get a lot more complicated when you're having to save the state of every item in the game at any given point.

There are many serialization mechanisms available to choose from.  For XAGE, the needs are:

• Performance - the faster the better.
• File size - too large a payload can typically affect performance, especially when disk IO is a bottleneck.
• Ease of use - from both an engine and game perspective, the user should be free to think about serialization as little as possible.  So no contracts or schemas, ideally no per-field attributes, and built-in version tolerence.
• Platform support - must work on all platform, including those with restrictions on JIT.

Early versions of XAGE used XML for the game state (and still do within the editor tools, as it is version-control friendly) but evolved to use Protobuf-net, a C# library that allows you to perform contractless serialization in Google's binary format.  This improved performance and file size significantly, but had one major drawback - lack of AOT support, required by some platforms like iOS and UWP.

Protobuf-net initially had AOT support by pre-generating a serialisation library, but official support for this was dropped.  The author has been waiting for Roslyn Generators in order to embed the serialization algorithms in at compile time.  Unfortunately this has not yet materialised and remains on the future roadmap.

Other serialization libraries exist with AOT support like Ceras, but without version tolerence, so have not been considered.

One I've had my eye on for a while is MessagePack for C#, which has long promised fast serialization times and low payload sizes with LZ4 compression.  Recently their AOT solution - a seperate executable called mpc.exe to generate the serialization logic as C# code - has become available as an MSBuild task, essentially allowing you to automate this process.

MessagePack's benchmarks always looked promising, but it's always important to test using your own data structure - in my case XAGE's main GameContent class.  For this I used BenchmarkDotNet which has become the industry standard for getting consistent and meaningful .NET benchmarks.  With this I was able to get results comparing the standard XML Serializer, Protobuf-net and various flavours of MessagePack (all using string keys rather than integer keys, for version tolerence):

• LZ4:  Where the payload is compressed using this performant compression library.
• MPC:  Where the serialization logic is generated by mpc.exe at build time, rather than determined and emitted at runtime.
• CoreRT:  Using the CoreRT AOT runtime, instead of .NET Core 3.1 (where use of mpc.exe is required)

However it wasn't all plain sailing as:

1. I couldn't use the BenchmarkDotNet nuget package as the latest SimpleJob overloads weren't in place, allowing you to combine a ColdStart test with the CoreRT runtime.
2. I couldn't use the MessagePack nuget package for the CoreRT tests due to some emit issues that are outstanding.  Compiling from scratch using some conditional symbols (NET_STANDARD_2_0; UNITY_2018_3_OR_NEWER; ENABLE_IL2CPP) resolved this.
3. All my classes had to be decorated with MessagePack attributes, and all public properties not needed with [IgnoreMember].
4. Protobuf-net's surrogate mechanism (for types like XNA's Vector2 and Color classes) is not supported and had to be reworked to be more generic in order to support MessagePack.

Once these issues were resolved, I was able to test with the largest GameContent data I had, and the results were surprising:

Note that these are 'Cold Start' benchmarks - i.e. single iteration tests with no warmup, repeated many times over.

Here the benefit of pre-generating the serialization logic is clear to see (MPC), but the performance of the CoreRT incarnations blew the rest away.  What took XMLSerializer 1247ms and Protobuf 848ms took MessagePack just 36ms from cold.  Adding LZ4 compression to reduce the filesize to 255K only brought this up to 40ms.  I dumped the GameContent as JSON in the benchmark cleanup methods to confirm the data was actually being loaded and saved correctly, as I didn't believe the results at first.

While it would take some time to fully integrate MessagePack into XAGE, it currently appears to be the best option in the absence of protobuf AOT support.  I just hope that .NET 5's AOT strategy works as well as CoreRT.

Putting lipstick on Sisyphus

Development has been somewhat chaotic lately.  Instead of focusing on one or two particular areas, I've been hammering away at my TODO list - all the little bugs and enhancements and quality-of-life improvements needed to get XAGE in a shippable state.  I'm regularly pushing out new builds of the tools to itch.io to get into good habits as part of an established release process.

Having said that, a number of recent changes have a common theme, which is either to reduce the CPU usage (particularly startup time) or reduce the disk/memory requirements.  Often these optimisations work against each other, so it has taken some experimentation to determine which combinations achieve the best balance.

• The custom BitArray ("BooleanMatrix", for fast alpha detection on textures) was changed to use a RoaringBitset implementation.  This massively reduces the amount of storage space needed, both file and memory, at a slight performance cost at runtime.

• The pipeline was changed so that the PNG spritesheets are published as LZ4 compressed 32-bit bitmaps.  While testing with Cart Life I'd noticed some frame drops while loading in a 4K spritesheet mid-game.  The LZ4 compression has an extremely fast decompression rate, and having the raw bitmap data removes the need to decode from the original PNG in-engine.  This loads in textures much more quickly at a cost of moderately increased file sizes.

• Profiling indicated that FAudio, the audio engine used by FNA, was slow to initialise.  By warming this up on program entry on a throwaway thread, we're able to shave a second or so from the initial startup time.

• The GameSettings file is now stored as protobuf file, removing a minor XML deserialization performance overhead (a few 100ms).  

• Using the 'PublishTrimmed' option on the game project has the linker remove unused code, which reduces the final package size.

• Using the 'PublishReadyToRun' option - a sort of mini AOT solution - reduces the start up time, at the cost of increasing the final package size.

• Warp was used to produce the final package (as the 'PublishSingleFile' option is not mature enough yet).  Packaging it up in this way increases the very first start up time but reduces the final package size and makes it easier to distribute and run.

I've released an updated version of Last 'n Furious with the above improvements.  The result is a single executable that is smaller (32MB) than the previous version (40MB zipped).  After the initial run, the game starts up in about 1 second, which is about as good as it gets until we are AOT-compiling the final executable in .NET 5.

CPU usage is down slightly, hovering around 2-3% CPU mid-race on my old laptop compared to 4-5% on the previous version.  Memory usage is also down to about 180MB compared to 220MB before.

Mac and Linux builds of the game also exist, though are not yet available due to an outstanding texture issue that only seems to affect the Mac platform - this should be resolved once I get my hands on some actual Apple hardware, like one of the ten richest kings in Europe.

Getting there.