That was my point, with thralls not beeing somewhat both reliable and useful during game, the most looked for are basically worstations named thralls which don’t need feeding or cares.Perhaps por pvp they make a difference, but in the end there is 1on1 more kind of thing.
All right, lets get a bit technical shall we?
Conan Exiles has a very big world with everything running on one server.
The world has 6000+ NPCs roaming the world, not counting player thralls and pets. Actually, probably even more by now.
The simple fact is that running NPC behavior on all of these NPCs all the time on such a big world is not something current hardware can handle with UE4.
And it only gets worse when you add in players and their dynamic content, such as buildings, placeables, thralls and pets.
We have however done a lot of trickery to counter this. And the performance we have today is leaps improved from the early days. We are also constantly working on improving this and have more optimization coming with one of the next patches.
The “static world” that you are describing is part of a system we call the AI LOD (Level of Detail) system. Which turns down the amount of CPU resources an actor (Placeable, building or NPC) receives based on their proximity to a player.
Outside of a certain range they go into a suspended mode where they are not given any CPU cycles, and basically go into a sleeping mode.
We do this because there is, in most cases, no point in having the actor run at full speed without a player nearby. And we have “catch up” methods in place where they are needed to make sure they account for the suspended time.
Example: A crafting station.
The crafting station needs to tick and update the progress of whatever is being crafted at real time when a player is standing nearby, so that you can see the progress bar move if you are inspecting it, or see the fire in your furnace turn off the moment the crafting is done.
If a player can not even see the crafting station, it can do this much slower or even not at all. When a player returns to the area and wakes it up, it needs to catch up on X amount of minutes of work. And when you inspect it, you will see the same result as if you had been standing there for those X minutes. (The reality is slightly different, we run a few ticks of crafting here and there even if the player is not around).
PLEASE BE CAREFUL WITH THIS AND TRY AT YOUR OWN RISK:
If an admin wants to have some “fun” and see what happens when nothing is allowed to be suspended, and all logic will run at full speed and all NPCs will start roaming around, you can attempt this command: ForceAILOD 0.
Could be a fun experiment for you to see what our optimizations have actually contributed on this front. You can turn it back on with ForceAILOD -1. It’s very possible that you will not be able to do so, so be prepared to restart your server. This shouldn’t influence anything on the server per se.
tldr; NPCs remaining fully awake when players aren’t nearby is not worth no one being able to play.
However as mentioned, there are still more performance improvements to be made and coming your way 
12 Likes
The AI itself is completly fine, if you play SP it works flawless IMo, but a good AI does not help if the perfomance o the servers is so poor that they do not move at all most of the time, server performance really is the issue, and as stated befor, coding optimization won´t do the job. Get rid of like 80% of the servers noone plays on anyways currently and buy servers which are five times as powerful instead, trust me those numbers are not overkill.
I think I love you, Tascha!
5 Likes
We all do 
She’s our community mom.
7 Likes
Hi Tasha, ty again for your reply. Srry for being negative again, nothing personal towards you.
You are confirming in your message that the ‘Official’ servers are not able to run the game. You also confirm that your devs have been programming around the “static world”(catch up mechanism), in fact wasting resources and time and introducing a buggy mechanism that is prone to fail when the server is under stress. It was introduced as a performance fix, it could be not further away from the truth.
The only thing I see are excuses, again: stress testing, technical choices and running your game on your official servers that are up to the task is Funcom’s responsibility. Too complex for your internal resources? Get a UE4 consultant. Too many bugs to fix for your internal resources? Get some freelancers, heck pay some modders who have already fixed some of your issues themselves. Turning off a feature is not a performance fix. Suggestion: rename the servers from ‘Official’ to ‘FREE cheap server’ with a message that not all features are enabled and that it cannot cope with 40 players and the many structures on them.
2 Likes
I meant paying the modders to fix it for them or even paying them to incorporate the mod into their branch. And tbh if anything has the tendency to break on updates, it is the official game. Fyi: I only play on official servers.
Thanks Tascha for the reply. Great insight into the game revealed and sounds like a fun experiment to try.
I wear it as a badge of honour
5 Likes
Fickle Elf! You said that to me last week!!
Those are excellent, constructive suggestions. Particularly a Mod Think Tank: this was an idea @Enyo and I discussed a while ago, and it still makes sense.
8 Likes
If anyone has played Ark here they (ue4) employ the same static world, when you come into render and the dinos start moving around like they say to themselves “someone is coming act normal!” I find it quite comical.
Modders are already heavily involved in the background.
I agree in the essential with OP, but is unfair to blame devs for that mess. Devs are doing what they can, and Conan Exiles have one of the better art team that I have ever seen on a game.
That’s a business matter. Conan Exiles have saved company but is unable to save itself. Also, CE have saved Funcom for the moment, but the image of the company that will remain on players mind is a greedy one, unable to develope a well performed game after a year and a half of EA and several months after fake full launching.
Personaly, If things doesn’t change soon and fast, I will leave CE and I doubt I’m going to buy another Funcom game in the future.
4 Likes
I agree the makers have passion for the game I am sure of that. Wether it be managment desitions or whatever, a game that could be #1 on top lists, is struggling to be even playable. Was it not for the people and rp I find enjoyable, I would certainly be playing something else rather than ce. Heck, even old AoC performs better…
2 Likes
At least that is what they think, if they do not start investing money that we paid for this game back into it, they are doomed. Consumers/gamers are not stupid, they have limits too. The sales success becomes a failure success that was witnessed by 1.6Million gamers…
Alright, now for the hardware portion of this talk!
I hope you are prepared for some serious behind the scenes facts and info.
OFFICIAL SERVER HARDWARE:
The majority of our official servers are running
- Intel Xeon E3-1270 V6
- 3.80GHz Quad Core.
- 64 GB of memory
There is some slight variation, but this is the standard we use. But keep these numbers in mind
QUICK OVERVIEW OF WHAT AFFECTS A GAME SERVER’S PERFORMANCE:
- CPU usage
- memory usage
- bandwidth/internet connection usage.
Bandwidth/internet connection:
If the servers are trying to receive or send 1 TB of data per second, when the connection only allows 1 MB, it’s obvious that things would go wrong.
Or if there’s a point between the server and client where packets are dropped, it would force the server or client to have to re-send data. Which would make the gaming experience bad.
But we are pretty safe here, we haven’t had issues with this at all.
Memory usage:
For memory usage we just have to make sure that the servers have enough memory to not need to swap out to disk. And with 64GB it just depends on how many servers are running on the machine. See conclusion.
CPU usage:
On this front it’s all about how many instructions can be done in the shortest amount of time. And there are two ways of doing it:
- Clockrate of the core. (Newer generations of CPUs can also do more instructions per cycle, so that also has an effect.)
- Multiple cores
CPU clock rate is not increasing as fast as it used to, and you can read up on your own as to why that is. But the fastest clock rates you generally get nowadays is around 4.00 GHz.
So the new tactic has been to increase the number of cores a CPU. That’s why we have CPUs like the AMD Threadripper with 16 cores(hyperthreaded).
But multiple cores only helps with work that can be multithreaded. (Multi-threaded means that parts of the program can run on different cores in parallel)
HOW THIS AFFECTS CONAN EXILES AND OTHER UE4 GAMES:
Unreal Engine 4 does utilize some multithreading, but it runs the vast majority of its gameplay logic on a single thread called the GameThread.
This means that it has to run on a single core. And it is the costliest thread to run for the game, which affects the player experience the most.
Since the engine is designed around this principle of having all of the gameplay run in the GameThread, it means that having many cores doesn’t really improve the server performance.
Since there is some actions (like loading of assets) that are multithreaded, it makes sense to have some cores, but having server hardware with 32 cores wouldn’t help.
Small note on topic of GameThread cpu utilization:
If anyone has ever inspected the CPU usage of a Conan Exiles server that is not running at 30 SFPS, and thought it odd that none of the cores are maxed out at 100%:
It’s because the GameThread is only ever executed by one core at the time, but it can swap which core is executing it quite frequently. So across a Quad Core machine, the GameThread will be executed at 100% of the time.
But in turns by the 4 cores, so you are unlikely to see any of them report 100% usage. This does not mean that the GameThread could be running any faster. If it were to be locked to a single core, you’d be seeing 100% utilization.
CONCLUSION:
What does this mean for our selection of server hardware?
It means we need to go for CPUs with high single-thread performance scores, that are compatible with the infrastructure provided by GPortal, and within reasonable cost.
Which is why we are going for the Intel Xeon E3-1270 V6, which has a high clockrate of 3.80 GHz, and only 4 cores. Since multi-cores do not help our performance issues.
How many servers do we run per machine?
Three. Because that means it can run 3 GameThreads at 100% utilization, and still have one core to deal with the actual multithreaded parts of the engine that these 3 servers are using.
This way they do not impede each other. The moment we were to run a 4th server, we would see them start to slow down the other servers running on the machine.
And since the machines have 64 GB of ram, it means we have 64/3 = ~21.3 GB per server. So we don’t end up with any swapping of memory to disk. So no performance loss because of memory either.
We’ve basically gone for the best hardware we could get that fits what Conan Exiles needs. It is not possible to get server hardware that is five times more powerful.
**We’re continuously looking into improving performance, and moving things out of the GameThread and making them multithreaded is a good way of doing so. **
But it’s not done with a snap of your fingers, and a lot of things are just not feasible to move out of it in UE4.
12 Likes
This is why I’ve got serious doubts about Ashes of Creation being able to run servers with several hundred players, and think that they’ve doomed themselves by choosing UE4. It will be easier for Exiles to optimize performance, since there is a player cap. I’m sure your programmers have been working studiously to shift as much of the data load to the user end as possible, to try and not overload the Game Thread. I look forward to seeing the progress made. 
1 Like
What about the game breaking bug that exploiters used to break my base in PVE server 1040?? I lost 700 hours of gameplay. You think i want to start all over again after this ? and the person that did it brags about it and has built all around my base so i cant expand and do anything. I was building this place for months i lost more than 300chests of items 2 of my 4 vaults and all the t4 thralls that i had all of them in chests. Whats going to be done? why theres nothing done for that ? you think me or my clan will farm all this loot again ? You said you fixed it but my last vault broke 2days ago. i lost around 5k starmetal, 3-4.000 gold and 6-7.000 silver. the bug is still on. Do something about it. We played pve for the safety of the base. the time invested was alot and i am disappointed. Wheres the Punishement for cheating and exploiting ? wheres the help to us affected?? its your bug and your fault this happened and it completely ruined my experience.
1 Like
Thank you for the deep explanation, I’m curious if this is the same way for pc and consoles, do servers work the same way regardless the platform?
1 Like
UE4 is perfectly capable of multithreading, Funcom made it so that the vast majority of the logic is on a single thread = bad design. So basically what you say is that when you fix the multithread design flaw you will run into problems with your hardware as the hardware choice was influenced by the bad design, not very reassuring imo.