Server Crashed when using Explosive Jar

MarcBomber · February 6, 2019, 2:08pm

My dedicated server just crashed as I lighted up some explosive Jars.
No Mods and I was the only player.

The jar next to me, rolled to to that point from the wall. Some seconds after I was disconnected.

Server logfile:

    [2019.02.06-13.40.23:107][485]Network:Warning: Data: BasePlayerChar_C_2 (server) received 1271.5 bytes per second > 1024 (sample window size: 2.00 seconds)
[2019.02.06-13.40.29:814][269]LogCharacterMovement: BasePlayerChar_C_2 is stuck and failed to move! Velocity: X=-105.73 Y=106.40 Z=0.00 Location: X=15870.52 Y=137407.95 Z=-8378.20 Normal: X=-0.00 Y=0.00 Z=-1.00 PenetrationDepth:5.648 Actor:BP_BuildWall_T3_Japan_C_3 Component:InstancedBuildingComponent_55 BoneName:None (6 other events since notify)
[2019.02.06-13.40.30:829][386]LogCharacterMovement: BasePlayerChar_C_2 is stuck and failed to move! Velocity: X=20.00 Y=-143.00 Z=0.00 Location: X=15870.52 Y=137407.95 Z=-8378.20 Normal: X=-0.00 Y=0.00 Z=-1.00 PenetrationDepth:5.648 Actor:BP_BuildWall_T3_Japan_C_3 Component:InstancedBuildingComponent_55 BoneName:None (35 other events since notify)
[2019.02.06-13.40.35:649][944]LogServerStats: Sending report: exiles-stats?players=1&=1.83%3A8.65%3A323.85&uptime=31131&memory=6436990976%3A12883869696%3A3917238272%3A3943555072&cpu_time=12.379391%3A24.758781&npcailods=1%3A2%3A12%3A5860&buildingailods=14%3A0%3A0%3A0&placeableailods=0%3A8%3A3%3A5&ipv4=213.239.192.230&sport=7777
[2019.02.06-13.41.35:645][896]LogServerStats: Sending report: exiles-stats?players=1&=1.66%3A8.63%3A333.71&uptime=31191&memory=6453026816%3A12883869696%3A3913199616%3A3943555072&cpu_time=12.550173%3A25.100346&npcailods=1%3A2%3A12%3A5860&buildingailods=14%3A0%3A0%3A0&placeableailods=0%3A8%3A3%3A5&ipv4=213.239.192.230&sport=7777
[2019.02.06-13.41.43:651][837]Performance:Display: [ItemDistributorController BaseGameMode_C_0] Finished item distributor processing; 0 handled this frame.
[2019.02.06-13.42.29:211][167]LogCharacterMovement: BasePlayerChar_C_2 is stuck and failed to move! Velocity: X=148.39 Y=-21.88 Z=0.00 Location: X=15874.67 Y=137405.91 Z=-8378.20 Normal: X=-0.00 Y=0.00 Z=-1.00 PenetrationDepth:5.644 Actor:BP_BuildWall_T3_Japan_C_3 Component:InstancedBuildingComponent_55 BoneName:None (25 other events since notify)
[2019.02.06-13.42.30:218][285]LogCharacterMovement: BasePlayerChar_C_2 is stuck and failed to move! Velocity: X=-63.03 Y=-136.12 Z=0.00 Location: X=15874.67 Y=137405.91 Z=-8378.20 Normal: X=-0.00 Y=0.00 Z=-1.00 PenetrationDepth:5.644 Actor:BP_BuildWall_T3_Japan_C_3 Component:InstancedBuildingComponent_55 BoneName:None (36 other events since notify)
[2019.02.06-13.42.31:109][391]Network:Warning: Data: BasePlayerChar_C_2 (server) received 1162.0 bytes per second > 1024 (sample window size: 2.00 seconds)
[2019.02.06-13.42.31:249][408]LogCharacterMovement: BasePlayerChar_C_2 is stuck and failed to move! Velocity: X=-55.51 Y=-139.35 Z=0.00 Location: X=15874.67 Y=137405.91 Z=-8378.20 Normal: X=-0.00 Y=0.00 Z=-1.00 PenetrationDepth:5.644 Actor:BP_BuildWall_T3_Japan_C_3 Component:InstancedBuildingComponent_55 BoneName:None (47 other events since notify)
[2019.02.06-13.42.32:862][596]LogCharacterMovement: BasePlayerChar_C_2 is stuck and failed to move! Velocity: X=-46.50 Y=0.14 Z=0.00 Location: X=15874.67 Y=137405.91 Z=-8378.20 Normal: X=-0.00 Y=0.00 Z=-1.00 PenetrationDepth:5.644 Actor:BP_BuildWall_T3_Japan_C_3 Component:InstancedBuildingComponent_55 BoneName:None (36 other events since notify)
[2019.02.06-13.42.35:648][924]LogServerStats: Sending report: exiles-stats?players=1&=2.14%3A8.54%3A297.77&uptime=31252&memory=6456000512%3A12883869696%3A3914448896%3A3943555072&cpu_time=15.095675%3A30.191349&npcailods=1%3A3%3A11%3A5860&buildingailods=14%3A0%3A0%3A0&placeableailods=0%3A8%3A3%3A5&ipv4=213.239.192.230&sport=7777
[2019.02.06-13.43.35:647][955]LogServerStats: Sending report: exiles-stats?players=1&=2.34%3A8.53%3A314.25&uptime=31312&memory=6456872960%3A12883869696%3A3913412608%3A3943555072&cpu_time=6.225634%3A12.451267&npcailods=1%3A2%3A10%3A5862&buildingailods=14%3A0%3A0%3A0&placeableailods=0%3A4%3A7%3A5&ipv4=213.239.192.230&sport=7777
[2019.02.06-13.43.40:343][506]LogNavOctree:Display: Purged data in 55 NavOctree elements in 0.98 ms, worth 2875.8 kb
[2019.02.06-13.43.47:347][326]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.47:347][326]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.47:788][378]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.47:788][378]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:062][411]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:062][411]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:308][438]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:308][438]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:407][450]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:407][450]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:870][505]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.43.48:870][505]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-13.44.10:362][965]LogWindows: Windows GetLastError: Der Vorgang wurde erfolgreich beendet. (0)
[2019.02.06-13.54.10:354][965]LogWindows:Error: === Critical error: ===
[2019.02.06-13.54.10:354][965]LogWindows:Error: 
[2019.02.06-13.54.10:354][965]LogWindows:Error: Fatal error: [File:C:\UE4\CSAND\Engine\Source\Runtime\Core\Private\HAL\MallocBinned2.cpp] [Line: 122] 
[2019.02.06-13.54.10:354][965]LogWindows:Error: MallocBinned2 Corruption Canary was 0x3941, should be 0xf317
[2019.02.06-13.54.10:354][965]LogWindows:Error: 
[2019.02.06-13.54.10:354][965]LogWindows:Error: 
[2019.02.06-13.54.10:354][965]LogWindows:Error: 
[2019.02.06-13.54.10:365][965]LogExit: Executing StaticShutdownAfterError
[2019.02.06-13.54.10:829][965]LogWindows: FPlatformMisc::RequestExit(1)
[2019.02.06-13.54.10:829][965]Log file closed, 02/06/19 14:54:10

Ignasi · February 6, 2019, 2:19pm

Hey @MarcBomber

Thanks for your detailed report. We’re sending it to our team and they’ll look into it

MarcBomber · February 6, 2019, 2:23pm

@Ignasi thx for the formating.

@Toolguy The DedicatedServerLauncher does not restart the server itself. Can you add a healthcheck and automated restart?

Toolguy · February 6, 2019, 2:30pm

Strange, I just checked:

Started the DedicatedServerLauncher
Made sure the “Start Server If Not Running” check box is enabled (and saved)
Started the server
Killed the server process using the task manager
Saw the patching/starting process start again

Can you confirm that the server process was actually gone?

At the moment I do not actually check if the server is frozen/stuck. I guess I could add such a feature, but I need to have a foolproof way to check the health of the server, and then also check if I can kill the process with non admin rights.

MarcBomber · February 6, 2019, 2:42pm

@Toolguy the application is frozen, had to kill it. Then it restarted. As you started the app you could have the rights to kill the process.

I could reproduce the crash a second time. Want the game.db?

[2019.02.06-14.26.19:574][3> 14]LogDataTable:Warning: UDataTable::FindRow :

 'GETTHRALLTIER' requested row '15' not in DataTable  
 '/Game/Systems/Thrall/ThrallTierDataTable.ThrallTierDataTable'.
 [2019.02.06-14.26.19:576][314]SpawnTable:Error: Data:  
USpawnTableComponent::GetThrallTier - could not find SpawnTable with ID: 15
[2019.02.06-14.26.19:586][315]LogDataTable:Warning: UDataTable::FindRow : 'GETTHRALLTIER' requested row '15' not in DataTable '/Game/Systems/Thrall/ThrallTierDataTable.ThrallTierDataTable'.
 [2019.02.06-14.26.19:586][315]SpawnTable:Error: Data: USpawnTableComponent::GetThrallTier - could not find SpawnTable with ID: 15
[2019.02.06-14.26.33:395][914]LogServerStats: Sending report: exiles-stats?players=1&=1.18%3A8.67%3A252.51&uptime=120&memory=6533668864%3A12883869696%3A3796504576%3A3797618688&cpu_time=64.592583%3A129.185165&npcailods=2%3A2%3A10%3A5780&buildingailods=11%3A0%3A0%3A0&placeableailods=6%3A8%3A3%3A5&ipv4=213.239.192.230&sport=7777
[2019.02.06-14.26.38:351][465]LogAsyncObjectFinder:Error: FAsyncObjectFinder: Asset "/Game/Systems/Progression/DmgTypeHealth_BP.DmgTypeHealth_BP_C" (used by "/Game/Maps/ConanSandbox/ConanSandbox.ConanSandbox:PersistentLevel.BP_BuildFoundation_T3_Japan_C_3") should have been loaded at this point! Please fix! Note: Will try to syncload the asset to continue execution.
[2019.02.06-14.26.38:351][465]LogAsyncObjectFinder:Error: FAsyncObjectFinder: Asset "/Game/Systems/Progression/DmgTypeHealth_BP.DmgTypeHealth_BP_C" (used by "/Game/Maps/ConanSandbox/ConanSandbox.ConanSandbox:PersistentLevel.BP_Demolition_Jar_C_4") should have been loaded at this point! Please fix! Note: Will try to syncload the asset to continue execution.
[2019.02.06-14.26.38:500][482]LogWindows: Windows GetLastError: Der Vorgang wurde erfolgreich beendet. (0)
[2019.02.06-14.36.38:493][482]LogWindows:Error: === Critical error: ===
[2019.02.06-14.36.38:493][482]LogWindows:Error: 
[2019.02.06-14.36.38:493][482]LogWindows:Error: Fatal error: [File:C:\UE4\CSAND\Engine\Source\Runtime\Core\Private\HAL\MallocBinned2.cpp] [Line: 122] 
[2019.02.06-14.36.38:493][482]LogWindows:Error: MallocBinned2 Corruption Canary was 0x3941, should be 0xf317
[2019.02.06-14.36.38:493][482]LogWindows:Error: 
[2019.02.06-14.36.38:493][482]LogWindows:Error: 
[2019.02.06-14.36.38:493][482]LogWindows:Error: 
[2019.02.06-14.36.38:504][482]LogExit: Executing StaticShutdownAfterError
[2019.02.06-14.36.38:994][482]LogWindows: FPlatformMisc::RequestExit(1)
[2019.02.06-14.36.38:994][482]Log file closed, 02/06/19 15:36:38

Toolguy · February 6, 2019, 2:47pm

Ok, so it at least works correctly, the only part missing is to find a fool proof (and easy to implement) way to detect if the game server is running fine.

I can think of a number of ways: Last time the log file was updated, last time the game.db was touched, etc… not sure what criteria people generally use for this type of thing.

I could reproduce the crash a second time. Want the game.db?

That’s for the game team, I think @Ignasi is the one to talk with for that part

Ignasi · February 6, 2019, 2:55pm

Hey @MarcBomber

If you could send me the db privately it would be interesting for our team when trying to reproduce this issue internally

MarcBomber · February 6, 2019, 3:32pm

The game.db still works for reproducing the error, just get more jars and fight through the wall. Get 5-7 at the middle end let it burn. repeat 2x

Ignasi · February 6, 2019, 3:33pm

What do you mean with “fight through the wall”?

MarcBomber · February 6, 2019, 3:39pm

look at the screenshot. There is the wall. The server crashes when I let jars explode in front or in that wall with the fences.
Guess there is one element buggy in that wall, or the server needs to be running for at last some minutes.

Modify the game.db to get (or to be) my character, cheat yourself some explosive jars, place them like in the screenshots, and after 2-3 runs the server is down.

[2019.02.06-15.26.03:896][350]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.03:896][350]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.03:972][359]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.03:972][359]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.04:220][389]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.04:221][389]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.20:933][386]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.21:027][397]Network:Warning: Data: BasePlayerChar_C_0 (server) sent 1139.5 bytes per second > 1024 (sample window size: 2.00 seconds)
[2019.02.06-15.26.21:514][437]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.52:241][ 70]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.52:241][ 70]CODEConanGameState:Warning: No guild found with id = 11. Number of registered guilds = 1.
[2019.02.06-15.26.53:024][158]Network:Warning: Data: BasePlayerChar_C_0 (server) sent 1591.0 bytes per second > 1024 (sample window size: 2.00 seconds)
[2019.02.06-15.26.58:256][743]LogWindows: Windows GetLastError: Der Vorgang wurde erfolgreich beendet. (0)
[2019.02.06-15.36.58:247][743]LogWindows:Error: === Critical error: ===
[2019.02.06-15.36.58:247][743]LogWindows:Error: 
[2019.02.06-15.36.58:247][743]LogWindows:Error: Fatal error: [File:C:\UE4\CSAND\Engine\Source\Runtime\Core\Private\HAL\MallocBinned2.cpp] [Line: 122] 
[2019.02.06-15.36.58:247][743]LogWindows:Error: MallocBinned2 Corruption Canary was 0x3941, should be 0xf317
[2019.02.06-15.36.58:247][743]LogWindows:Error: 
[2019.02.06-15.36.58:247][743]LogWindows:Error: 
[2019.02.06-15.36.58:247][743]LogWindows:Error: 
[2019.02.06-15.36.58:254][743]LogExit: Executing StaticShutdownAfterError
[2019.02.06-15.36.58:697][743]LogWindows: FPlatformMisc::RequestExit(1)
[2019.02.06-15.36.58:697][743]Log file closed, 02/06/19 16:36:58

MarcBomber · February 6, 2019, 3:57pm

no crash when using an avatar to kill the wall. So it is more likely to be related to the explosive jars.

After a client reconnect to show the damage

MarcBomber · February 6, 2019, 4:03pm

seconds after posting, the server crashed again. Then maybe it is the structure.

sirvink · February 6, 2019, 6:18pm

I wonder if this is something like a combination lock. Each number you add (explosive jar) multiplies the possible combinations of damage exponentially. Since each jar is affecting multiple building pieces, you are setting multiple chain reactions, all impacting each other. I’d love to see what the CPU spikes to, trying to manage it all. I have not used avatars yet, but I imagine each attack it does is just a single attack. So if you did one explosive jar at a time, I would guess it does not lock up the server?

system · February 13, 2019, 6:18pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.