Nodecraft always has been built by gamers for gamers. Almost a decade ago, we were disappointed in the state of almost every hosting provider, forcing people to purchase multiple servers for each of the games they play. So, we introduced an innovative feature to help this known as Save and Swap, allowing players to manage their servers in the same way they actually play games. No extra game server subscription is required, no matter how many times you hop between Minecraft and Palworld, or anything else you want to play.
Now, it’s time to upgrade again.
Scaling Backups
In previous dev updates, we've walked you through some of the technology we built, including our mass-migration in 2019 from AWS S3 to Backblaze B2 (the amount of data we store today is more than an order of magnitude higher), but today, the core process for how a customer’s game server instance gets archived and stored looks like this:
Throughout this process, the Bare Metal Node and Orchestration Layer communicate regularly for instructions and progress, but the two slowest parts of this process are:
Creating Backup Archives
Uploading Backup Archives
Create Game Server Archives
When we first launched this feature, Windows 7 was still the dominant operating system for gamers. We opted to use the very stable and standard compression format, .zip. It’s something almost everyone is familiar with, and most operating systems are able to open it without any third party software.
It was always important to us that the archive format we use was easy to use for users. We always make the archives we create of customer game servers available for download to them, so if a customer wanted to download their server files and run them locally, or even move to another provider entirely, we didn’t want to create any kind of artificial lock-in.
The zip format has never been the most efficient though, so recently with the added support for newer archive formats natively in Windows 11 with the 23H2 update, we felt it was time for us to move to a newer archive format and began testing.
Archive Format Testing
The Nodecraft dev team spent extensive time testing various archive formats and compression against a bunch of real-world use-cases. We found for testing that an ARK: Survival Evolved server with a handful of mods, was a great representation of a more “extreme” use-case for our customers, with its mixture of large/small file-sizes, some heavily compressible, others not so much, and general total filesize. And then a smaller Minecraft server with a well-developed world and a handful of mods/plugins represented the more "average" use-case, specifically a Pixelmon server in this case.
You can see some of the results of our testing below, including timing for our previous default, zip. We did a lot more extensive testing with different CPUs, thread usage, parallelisation, and compression rates, but with average workloads and usage across our servers, you can see some of the most important results below averaged over a few runs:
ARK Server Example
| Size | Compression Time | Decompression Time | |
|---|---|---|---|
| Original | 18GB | n/a | n/a |
| Zip | 8.3GB | 14 minutes, 50 seconds | 5 minutes, 11 seconds |
| Gzip (level 5) | 8.3GB | 9 minutes, 38 seconds | 6 minutes, 5 seconds |
| Gzip (level 4) | 8.4GB | 7 minutes, 52 seconds | 3 minutes, 18 seconds |
| XZ (level 2) | 6.5GB | 46 minutes, 37 seconds | 7 minutes, 52 seconds |
| XZ (level 0) | 6.8GB | 24 minutes, 11 seconds | 8 minutes, 27 seconds |
| Zstandard (level 6) | 7.6GB | 5 minutes, 27 seconds | 5 minutes, 28 seconds |
| Zstandard (level 4) | 7.8GB | 2 minutes, 34 seconds | 4 minutes, 44 seconds |
| Zstandard (level 2) | 8.2GB | 1 minute, 16 seconds | 5 minutes, 9 seconds |
Minecraft Server Example
| Size | Compression Time | Decompression Time | |
|---|---|---|---|
| Original | 1.2GB | n/a | n/a |
| Zip | 878MB | 0 minutes, 35 seconds | 0 minutes, 8 seconds |
| Gzip (level 5) | 878MB | 0 minutes, 40 seconds | 0 minutes, 8 seconds |
| Gzip (level 4) | 878MB | 0 minutes, 35 seconds | 0 minutes, 6 seconds |
| XZ (level 2) | 861MB | 5 minutes, 31 seconds | 0 minutes, 41 seconds |
| XZ (level 0) | 871MB | 3 minutes, 13 seconds | 0 minutes, 42 seconds |
| Zstandard (level 6) | 862MB | 0 minutes, 24 seconds | 0 minutes, 1 second |
| Zstandard (level 4) | 866MB | 0 minutes, 4 seconds | 0 minutes, 1 second |
| Zstandard (level 2) | 871MB | 0 minutes, 2 seconds | 0 minutes, 1 second |
Archive Format Choice
We found that tar + zstd level 4 provided an incredible uplift in compression time, as well as a nice reduction in archive size. During our testing on multiple occasions, the archive finished so quickly that we all looked around the room and expected an error to have happened, but no, it was just that fast.
With that, .tzst was the obvious winner, and our choice moving forward. With native Windows 11 support, and easy macOS support via Keka, we weren’t sacrificing any user experience, and just getting faster and smaller backups. A quick user guide later for how to work with tzst Zstandard compressed archives, and we were off to the races. Zstandard opens up a lot of opportunities for us in the future too, such as using custom dictionaries for specific games or use-cases, reducing file sizes even further.
Game Server: Upload Step
Uploading the backup to offsite remote storage with B2 was the other slow step in this process, and although we haven’t made any massive changes here, the smaller archive filesizes help a lot! We’re also always working to improve network connectivity throughout our server fleet.
Some of these efforts can be seen in the following graph, showing average backup time before and after these changes. You can observe a significant decrease in the time it takes for an upload now with the reduced filesizes.
Game Server: Deploy Step
This isn’t a step we discussed before, but when deploying a game server at Nodecraft, you’re effectively downloading and extracting a previously created backup. If that backup's archive is smaller, then extracting it is quicker (both of which are true with our new archive format). Deploying your servers to play again at Nodecraft is faster now!
In practice we measured a similar ~20% decrease in time spent deploying, meaning users can Save and Swap game servers even faster, and get back to doing the most important thing: playing games.
Improvements
To summarise, a list of improvements Nodecraft customers can now expect:
Instance backups are up to ~20% smaller, meaning your backup storage can stretch much further
Archiving game servers will happen more quickly, up to ~30% quicker
Extracting archives will happen more quickly
Deploying game servers will happen more quickly
A whole bunch of speed improvements, optimisations, and larger effective backup space, at no additional cost!
What's next?
These updates have been live for quite some time, meaning whenever you backup, archive, or even deploy an instance with Nodecraft now, the process will be measurably faster.
The team at Nodecraft continues to lead the industry innovations for multiplayer game server hosting in dedicated servers and Player-Created Game Servers. The innovation to reduce game server archive and deploy time was actually a prerequisite to some exciting new product changes coming in early 2025. If you're a server owner or someone who wants to create game servers for you and your friends, now has never been a more exciting time to create your own server.