Major Long term Systemic issue found to the game.db file

Conan Exiles PC Discussion
1

Game mode: [Online | Singleplayer]
Problem: Bug | Performance
Region: [US]

I have noticed and have direct evidence and correlation using charts of a major issue concerning the loading of the main game.db file of the game itself.

Here is what I have learned and it will require some major thinking to solve this problem.

Hint #1
Originally on server US 1941 Our gaming community noticed that once the servers went down that it took anywhere from 5-8 hours for them to come back online. As we continued to play, week after week, that time only increased.

Notes: After dealing with the constant daily server down time, far prior to you guys fixing the recent repeated server crashes on the last server update, we inevitably couldn’t decide who to blame. We originally thought that it was gportal not intervening and restarting the server in a timely fashion. Some of us also assumed that it may in fact be Funcom related. Nothing made sense so we as a group decided to purchase hosting a server using Hosted Havok. In our bundle we purchased 20 slots, CPU priority at 8GB of ram and from that point it was a dream to experience. No lag, no rubber banding and hardly ever crashing, that is, for the first week at least. That didn’t last.

Hint #2
The second week on the private server, the game.db file would crash approximately every 16-24 hours requiring a forced reboot. During that first week it took 10-20 seconds for the game to be loaded and to be displayed as listed withing the game when refreshing. The game.db file was approximately 2MB.

Hint #3
The third week things got busy, new blood of 5 people joined us along with our 15, those from the start have already become level 40 and began building or have completed minor structures. Server stability continued to last between 16-24 hours after which, still requiring a server restart. During the 3rd week, the server became available to log into online after 45-50 seconds after the server restart. The game.db file at this time was approximately 15MB.

Hint #4
The 4th week the same continued with more members joining, many numerous builds completed, numerous level 60’s and this time a server restart brought the game online after approximately 1-2 min when refreshing the list. At that point the game.db file was approximately 25MB.

Hint #5
The 5th week which is now the same continues but now the server struggles to stay online for more than 12 hours straight until wide spread lag, rubber banding, and also thralls become non responsive until attacked but more importantly it took around 3 min for the server to come back online and the game .db database file is now approximately 30MB.

IN CONCLUSION:
The point I am trying to make here is, given the experience with gportal servers that have been online and in use for a very long time since the last full wipe and having far more interior customization made to the map by players it is safe to assume that the game file for US server 1941 as an example is far greater then ours. This leads to a simple logical reasoning that the larger and more intricate the game.db file becomes the longer it takes to reload to become available to log into.

If you learn after investigating that this is the case and can see this correlation yourselves, it then begs the question, how are you guys going to fix this ever growing and very major systemic issue? Given that downtime is the single defacto killer for an MMO sandbox title, this would seem to be a major issue and concern and it leaves me pondering, where will our server be in 6 months from now? Will it also take our server to come online for hours at a time just like your official servers? Our server settings mirror your official PVE-C server settings. Hosted Havok’s hardware (non VPS) is on par if not better then what gportal is currently using to host conan exiles official servers. We even purchased CPU priority which gives us a slight added advantage. This issue has many of us funcom fans very concerned, especially since many of them donated to and continue to donate to keep the server up long term. 15 of us already left the official server having spent so many hours building to our hearts desires and obtaining rare items to continue exploring and enjoying the game only to have us all start over again due to our frustrations, frustrations enough to warrant spending a lot of money on hosting and I being the admin am taking it upon myself to administrate it, no small feat mind you. I would appreciate a firm investigation on this concerning matter. If this issue is fact then it could very well spell disaster over the next year as servers continue to grow. Despite DLC’s, giveaways and the amazing game that you have built, no one will stick around to play a game that is down 1/2 the time. If I can help in anyway it would be my pleasure to assist.

I await your response fellow Developers/Management.

Thank you for you time.
Ninja_Havok

3 Likes
2

Hey

I run HyborianAge RP, our save file is over 200mb after a 6week rp seasons with over 300 people in and out of the server all day.

I can’t say we have experienced these crashes my server on either one of the dedicated boxes we run be it the one with Nitrado or the our own personal box, can stay up for 24 hrs without a crash.

we reboot one time daily and then unless there is a patch or a mod update it stays up all the way round till the next days restart.

i would suggest taking a closer look at the server specs as 8gb of ram imo is not enough, we avg a use of 12gb out of the 64gb we have allocated on our boxes, using ssd’s and 7700k cpu’s and our restart time takes 8 to 10 minutes at this point.

So i’d say up your hardware on your server and see how it performs instead of using the bare minimum to run your test.

Just my two cents

4 Likes
3

You didn’t read that correctly, this isn’t being hosted by myself, it is being hosted by a professional company that hosts non vps, Conan and other title servers situated on 2 backbones with 5 separate redundancies for any circumstances, power included. The specs you stated are literally identical to our servers setup minus the ram which at this point never surpasses 6GB so at 8 GB its currently a non issue but I agree an upgrade will be be needed in a week or two. We too have the occasional days when the server doesn’t crash for 24 hours but they are rare which is why a server reboot is needed/required. Despite the reality of both of our situations, an MMO sandbox of this caliber shouldn’t need to be restarted at all, lets be frank here. Aside the patches they release no restarts should ever be required let alone enough for us as admins to intervene and create a restart schedule ourselves to combat this issue. Furthermore, Gamers who choose and enjoy to play on a Sandbox MMO or MMORPG literally don’t and should never experience daily restarts of the game they play, nor should they experience downtime’s of between 5-8 hours every single day on official server. I am trying to assist the developers to localize the issue here and to come up with a way to eliminate the memory leaks/other issues they find to quash this issue once and for all for the long term and the only way that will happen is if we do this together, official and hosted.

This issue with the game is systemic, plain and simple and the larger it grows, the longer it takes to come back online which is the entire point of this thread to begin with. Thank you for your thoughts, suggestions and opinions.

1 Like
4

Official server 1040 is less than 2 minutes to make a restart after a force crash.

5

I believe what you are seeing is noting out of the ordinary.

Your database will grow exponentially as more and more people play. everything, and i mean everything is saved, including every item ever found by a play and specifically every building piece placed.
The engine has to process all the extra information on each restart and will take longer to do so every day.

Ideally you need a minimum 16GB of ram to comfortably run a server with some spare overhead.

Database on my current server is about 130mb and takes about 4 min to fully restart right now.

6

If you run private servers, remember to maintain your DBs optimal performance.

3 Likes
7

What a great link, the scripts and info work a treat. Many thanks.

2 Likes
8

I just wanted to add some info and stats about our private server, it might be of some help to the dev’s and also a benchmark for other hosts.

Our Server : The Isle of Pit (PvE-Conflict UK Server)
Server Stats Page : PitMonk Conan Exiles Server Stats
Other Stats : Top Conan Servers Stats
Mods : None Used
Date Since Last Map/Database Wipe : 12th May 2018 (9 Weeks Online)
Individual Players Who Have Connected in 9 Wks : 565
Avg. Players Online : 11 - Peak Players Online : 35
The Server Is Online : 24/7

Server Spec - Game server is running on a dedicated box:
Windows Server 2012 Essentials
Xeon E3-1240 v2 @3.4ghz
24GB RAM
Game files on a Seagate IronWolf 4 TB HDD

As detailed in previous comments here, our server would also crash once per day on average, 5-6 times a day during a 2 week period until this was fixed by the devs.

We solve rubber banding and lag issues by rebooting the server on a daily bases or it will run away with our RAM.

After 9 weeks of use, after 24 hrs from a reboot, the RAM usage will have increased to 12GB - 13GB and remains there even after all the players have logged off.

Our game.db file was at 145mb, reduced to 130mb since running the mysql tool and script above.
RAM usage on boot was at 8.5GB, reduced to 7GB since running the mysql tool and script above.

We solved our regular server crashes by:
After a crash, the game.db–wal and game.db-shm files would remain. We deleted the main game.db file along with the game.db–wal and game.db-shm files and renamed the newest game_backup.db file to be our new game.db file. After doing this, our server stopped crashing on a regular basis.

We run the supplied database maintenance tool after each crash and before and after each update.

In comparison, we also host a RUST server, at it’s peak with 80 players online, the server would use 5GB of RAM.

9

you’re wrong, a server is a pc like any other, and need be restarted from time to time.
If you have lot people and action on your server, i would recommand to restart your server twice the day. Also a restart before the most people come online, some raids, or other heavier acitivities, is still good and can only help your performances.

If you want the big machine for your game, yes it’s not cheap, and you may experience a big difference between the standart offers, and a big server, and some admin knowing what they do.

10

@Vattende
Name me a non funcom MMO Sandbox that requires a daily restart. This simply doesn’t and shouldn’t exist and if it does its simply wrong. Server’s never need a reboot unless there is a serious problem or a requirement for an upgrade. You never want to stop a hardrive, or shut down a power supply unit, it decreases its lifespan. Vattende please keep personal comment to yourself and ask anyone who’s and IT professional that you know before making such statements.

@ PitMonk Thank you for your information.

@ Silentangelz Out of the ordinary? The entire situation is out of the ordinary on every single official and hosted server.

@ ShintaiDK Thank you, I will take a glance at it shortly and let you know.

11

CE is a survival sandbox game, not a MMO at first.

If you want play a MMO, play WOW or else, CE is not, and the servers still needs restart, not only with CE. I played thousand of hours ARK, wich is also a survival game, i had my own server, and there to, one or two restart the day was simply a norm.

12

This isn’t quite true. A server OS is far more stable and the RAM is better for crunching data.
When he says about the server needing to be restarted, he is referring to the game’s server service, not the Server OS itself.

When we say our “server crashed and needs rebooting” The actual server OS and hardware haven’t crashed, just the game process has.

The issue is more with the database becoming unstable in some way and crashing unexpectedly or the service will start to run away with your rigs RAM, a restart of the game service resolves this temporally.

I have run Minecraft servers in the past and rarely needed to reboot the game, for months at a time.
My RUST servers only need rebooting once per week as updates are released.

I only need to reboot my actual server after a Windows update forces me to, my server itself hasn’t been restarted in over 3 months.

13

There are a lot of scripts in the link, but the basic maintenance script will do me just fine.

Copy the sqlite3.exe from your Saved folder file into a new folder then created a .bat file with the follow inside:

@echo off
for %%a in (game.db) do (
echo "%%a"
sqlite3.exe %%a "VACUUM;REINDEX;ANALYZE;pragma integrity_check"
)
pause

After you have closed the server down, copy the game.db file into this new folder and ran the .bat file. Once complete, move your game.db file back. ( Obviously saving a backup of the game.db beforehand )

When I did this, the file reduced from 145mb to 130mb, and after rebooting, the server is using 1.5-2gb less ram.

14

sure, it’s not the entire server crashing, this would not only make crash mostly one world, but lot of. One machine hosting serveral game-worlds mostly, at least when you rent at hosting-compagnies.

Cleaning data would be a way, but not realy still doable in a viable manner, so mostly a wipe after a while is necessary i agree.
A fact also is that realy good servers cost money, and less you want spend, less service and lifetime for your world you will have.

1 Like
15

Guys I just took a glance at these sqlite command and noticed that they are all from pre wipe times. Is there anyone to know if this will effect the game today with all of the work put into it thus far? Feb 2017, I am reluctant to proceed. I did a test on a copy and got some odd results as well.

16

What did you test and what were the odd results you had?

17

It was possible a missing packet, I downloaded a fresh copy and the result was ok and that’s it. I did notice a slight drop in file size however it is still taking the same amount of time to come back up. Seem even longer actually.

18

8 minutes and counting, server still hasn’t come online.

19

Does anyone know what the files game.db-shm and game.db-wal are?

20

They are temporary “cache” files which hold on to and pass data back and forth the the main database. They should delete themselves as you shut down the game server.

If the server crashes, they sometimes remain. I delete them before re-launching the server.