Alright, now for the hardware portion of this talk!
I hope you are prepared for some serious behind the scenes facts and info.
OFFICIAL SERVER HARDWARE:
The majority of our official servers are running
- Intel Xeon E3-1270 V6
- 3.80GHz Quad Core.
- 64 GB of memory
There is some slight variation, but this is the standard we use. But keep these numbers in mind
QUICK OVERVIEW OF WHAT AFFECTS A GAME SERVER’S PERFORMANCE:
- CPU usage
- memory usage
- bandwidth/internet connection usage.
Bandwidth/internet connection:
If the servers are trying to receive or send 1 TB of data per second, when the connection only allows 1 MB, it’s obvious that things would go wrong.
Or if there’s a point between the server and client where packets are dropped, it would force the server or client to have to re-send data. Which would make the gaming experience bad.
But we are pretty safe here, we haven’t had issues with this at all.
Memory usage:
For memory usage we just have to make sure that the servers have enough memory to not need to swap out to disk. And with 64GB it just depends on how many servers are running on the machine. See conclusion.
CPU usage:
On this front it’s all about how many instructions can be done in the shortest amount of time. And there are two ways of doing it:
- Clockrate of the core. (Newer generations of CPUs can also do more instructions per cycle, so that also has an effect.)
- Multiple cores
CPU clock rate is not increasing as fast as it used to, and you can read up on your own as to why that is. But the fastest clock rates you generally get nowadays is around 4.00 GHz.
So the new tactic has been to increase the number of cores a CPU. That’s why we have CPUs like the AMD Threadripper with 16 cores(hyperthreaded).
But multiple cores only helps with work that can be multithreaded. (Multi-threaded means that parts of the program can run on different cores in parallel)
HOW THIS AFFECTS CONAN EXILES AND OTHER UE4 GAMES:
Unreal Engine 4 does utilize some multithreading, but it runs the vast majority of its gameplay logic on a single thread called the GameThread.
This means that it has to run on a single core. And it is the costliest thread to run for the game, which affects the player experience the most.
Since the engine is designed around this principle of having all of the gameplay run in the GameThread, it means that having many cores doesn’t really improve the server performance.
Since there is some actions (like loading of assets) that are multithreaded, it makes sense to have some cores, but having server hardware with 32 cores wouldn’t help.
Small note on topic of GameThread cpu utilization:
If anyone has ever inspected the CPU usage of a Conan Exiles server that is not running at 30 SFPS, and thought it odd that none of the cores are maxed out at 100%:
It’s because the GameThread is only ever executed by one core at the time, but it can swap which core is executing it quite frequently. So across a Quad Core machine, the GameThread will be executed at 100% of the time.
But in turns by the 4 cores, so you are unlikely to see any of them report 100% usage. This does not mean that the GameThread could be running any faster. If it were to be locked to a single core, you’d be seeing 100% utilization.
CONCLUSION:
What does this mean for our selection of server hardware?
It means we need to go for CPUs with high single-thread performance scores, that are compatible with the infrastructure provided by GPortal, and within reasonable cost.
Which is why we are going for the Intel Xeon E3-1270 V6, which has a high clockrate of 3.80 GHz, and only 4 cores. Since multi-cores do not help our performance issues.
How many servers do we run per machine?
Three. Because that means it can run 3 GameThreads at 100% utilization, and still have one core to deal with the actual multithreaded parts of the engine that these 3 servers are using.
This way they do not impede each other. The moment we were to run a 4th server, we would see them start to slow down the other servers running on the machine.
And since the machines have 64 GB of ram, it means we have 64/3 = ~21.3 GB per server. So we don’t end up with any swapping of memory to disk. So no performance loss because of memory either.
We’ve basically gone for the best hardware we could get that fits what Conan Exiles needs. It is not possible to get server hardware that is five times more powerful.
**We’re continuously looking into improving performance, and moving things out of the GameThread and making them multithreaded is a good way of doing so. **
But it’s not done with a snap of your fingers, and a lot of things are just not feasible to move out of it in UE4.