Bug fixing process

[pageslinger]

@Multigun

I prefer hard data over player eyeballs.

Multigun made this comment in a previous post regarding the visual changes on the EL map and after I made a snarky comment poking fun at the patch process I started thinking and maybe it is all starting to make sense. At least to me anyway.

I had a car one time that had a problem, I took it to a few dealerships and told them what I thought the problem was. They put it on the computer and promptly told me that there was no way that what I thought was the problem could be the problem because the computer didn’t show it. So after the last dealership I stopped at the parts store, got the part I thought I needed and installed it and the problem was fixed.

I say that because I wonder how many bugs that get reported get filed under the “there is no way that could happen because my code is solid and I cant recreate it” Most bugs that get on here the first question is “do you have mods” (and yes, a lot of issues are caused by mods no dispute on that) and if the answer is no in some instances I have seen the next question to be “are you sure”. I wonder how many bugs that reach the devs from testlive and after a patch are looked at from that perspective and filed under “I cant recreate it and my code is solid so it probably is user error” Whether that’s wrong or right isn’t the question, I just wonder based on how many things fly under the radar or go unfixed that maybe that is part or all of the reason why?

[Bodin]

Your observations are true, but to be fair, many of the times it really is a mod and/or user error. That’s why its the “go to” default response. That doesn’t account for all of them, to be sure… but it has and will probably remain the ‘default’ for most of them since it has happened too frequently.

That’s why its always a good idea to “answer” those default questions in the original post with something like, “I am not using mods” and “yes, I’m sure I’m not using mods.” Then we can skip that part and go straight to solving the issue.

[CodeMage]

@pageslinger

I wonder how many bugs that get reported get filed under the “there is no way that could happen because my code is solid and I cant recreate it”

Every programmer worth their salt has learned that “my code is solid” is a feeling they can’t trust. So unless we’re going to assume that the majority of the dev team is composed of either extremely junior or simply incompetent coders, we can safely ignore the first part of that statement and focus on the second half.

“I can’t repro it” is the worst blocker you can have when trying to fix a bug. Nailing down a consistent repro for a bug is often more than half the work. This is especially true in highly complex software that has lots of moving parts. Like video games

@pageslinger

Most bugs that get on here the first question is “do you have mods” (and yes, a lot of issues are caused by mods no dispute on that) and if the answer is no in some instances I have seen the next question to be “are you sure”

That’s just normal troubleshooting checklist. You always try to eliminate the most frequent causes first:

• “It keeps saying my password is wrong.”
  • “Is your caps lock on?”
• “The computer won’t turn on when I press the button.”
  • “Is it plugged in?”
• “I put fresh batteries in my remote control and it’s still not working.”
  • “Did you put the batteries in correctly?”

And yes, sometimes people ask you to double-check you’re not running mods, because it really does sound like a problem you’ve already heard about, a problem that was confirmed to be caused by mods. And guess what? Sometimes the answer is “oops, you’re right, I was running mods”

@pageslinger

I wonder how many bugs that reach the devs from testlive and after a patch are looked at from that perspective and filed under “I cant recreate it and my code is solid so it probably is user error”

Again, let’s not conflate “no repro” with “my code is perfect”. There is absolutely no reason to insist those two go hand in hand, or to imply that Conan Exiles devs are inexperienced, incompetent, or unprofessional.

@pageslinger

I just wonder based on how many things fly under the radar or go unfixed that maybe that is part or all of the reason why?

Unlikely. But the question about why things slip through TestLive into live is a very interesting one. I would really love to be a fly on Funcom’s wall and take a peek at how they do certain things. One thing I would love to take a peek at is their internal bug tracker.

See, it’s not only the question of bugs not being fixed in TestLive and going to live, because there can be all sorts of reasons for that, ranging from perfectly reasonable to downright unsavory. It’s also about known bugs not being communicated in patch notes.

I can understand not being able to fix bug X because the suits have decreed that you have to meet the deadline Y, for example. The decent thing to do in that case is to include it in the patch notes under “Known Bugs” or whatever you want to call it. But I’ve seen examples of bugs that have been reported in TestLive and acknowledged by the community managers, then reported again in live and acknowledged by the community managers again. That’s not only discouraging to people who do us all (and Funcom) the huge favor of spending their free time on TestLive, but it’s also a sign that some bugs simply slip through the cracks.

In the end, my bet is that there’s something seriously wrong with Funcom’s QA practices. Not the QA team, not the devs, not the people, but the process. And, well, the process does ultimately come from the people, but it’s too often the case that the process isn’t under control of the people who do the work and who get blamed for it.

Just my 2 cents.

[pageslinger]

I all ways like the way you break things down.

@CodeMage

or to imply that Conan Exiles devs are inexperienced, incompetent, or unprofessional.

I wasn’t trying to imply that at all and I didn’t mean for it so sound that way.

[CodeMage]

@pageslinger

I wasn’t trying to imply that at all and I didn’t mean for it so sound that way.

No worries, just making sure nobody reading this jumps to that conclusion

[Narelle]

@pageslinger

Most bugs that get on here the first question is “do you have mods” (and yes, a lot of issues are caused by mods no dispute on that) and if the answer is no in some instances I have seen the next question to be “are you sure”.

As someone who spends a lot of time helping people, you cannot imagine how many people think or swear that they are not running any mods just to send me their log for me to see that they have a bunch of mods loaded. Sometimes I even get reports with Multigun’s mod X broke my server, it crashes. When I get the log I see that none of his mods is even loaded. Then there are the cases, when you open the log files and see a clear mod reference pointing to the issue. Sometimes people load a bunch of mods and then remove them but those mods leave a trace in the database and could continue causing issues depending on how they are coded.

The least people could do is try to eliminate the most frequent causes. I would feel stupid if I call the internet company about my internet and they figure out that I have not plugged in the cable. Furthermore, most of the issues turn out to be indeed caused by one or another mod. I played on a server once whose owner was instructing his players to report issue X to Funcom on their forums and to tell them that they are not using any mods even though it was quite clear that the issue was caused by a mod and it is server related as it was not something you experience on other servers.

The devs are not eliminating the possibility that the report could be a base game issue but if they have 10 bugs reports from officials (assuming players didn’t lie about that too) and 1 bug report with 40 mods, it is perfectly reasonable for them to put priority on the 10 reports where it is 100% clear that the issue is caused by something on their end.

@pageslinger

I wonder how many bugs that reach the devs from testlive and after a patch are looked at from that perspective and filed under “I cant recreate it and my code is solid so it probably is user error”

If you can’t reproduce the issue, it is very difficult if not impossible to fix it because it could be anything causing it. Again, it comes down to priorities, if you have 10 issues that you can reproduce reliably, it is to be expected that those will be prioritized. If you have 1 issue that cannot be reproduced on your end, you would have to see whether it is something that happened once to someone or reported by multiple users. If it is something affecting a massive amount of users with various setups, the odds are that there might be something base game related.

To give you an insight. Me and my team found an issue on TestLive which we reported directly to Funcom (not via the forum), the issue was pretty critical (continuous client crashes). Even though all of us experienced it we couldn’t exactly say what was causing it because there were so many variables and it was happening inconsistently. However, since we are playing on TestLive without mods, we were 100% sure that it was not a mod issue and Funcom were aware of that too. It took us almost a day of troubleshooting with 6 people testing and Funcom helping to finally figure out what could potentially be the reason. Once we could reproduce the issue reliably, Funcom could pin point exactly where the problem was. The issue was fixed before going live. I can’t expect 6 people people spending a day trying to reproduce an issue that affects one-two players when they can spend the same time helping many others or focus on troubleshooting another difficult to reproduce case which affects a huge amount of players.

[pageslinger]

@CodeMage

In the end, my bet is that there’s something seriously wrong with Funcom’s QA practices. Not the QA team, not the devs, not the people , but the process. And, well, the process does ultimately come from the people, but it’s too often the case that the process isn’t under control of the people who do the work and who get blamed for it.

And there in lies a large part of the problem doesn’t it?

Nobody likes comparing this game with that game, but because Codemage and Narelle have done a good job of explaining why the process is so difficult I cannot help but do just that to see what the possible insights and answers could be. The answers to what I am getting at are both pointed out in Codemage’s answer and this article interviewing Joel Bylos back in 2018.https://www.polygon.com/2018/5/14/17352346/conan-exiles-ark-survival-evolved-funcom-controversy

In the last 30 days according to steam charts Conan Exiles has 14,036 current players and had a 30 day average of 8,747.3 players with a -3.39% loss, Ark has 72,513 current players and had a 30 day average of 53,559.4 players with a loss of -13.31% loss.

Both are heavily modded games, both run on the unreal engine and both are the same genre of games and both have the same survival mechanics…but yet they are vastly different in player base.

Why is that? I firmly believe that it is due to what I was talking about when I mentioned “poorly thought out and implemented game systems and mechanics” even Joel alluded to that in the article I linked. Even a new expansion couldn’t change the slow downhill slide due to…you guessed it…“poorly thought out and implemented game systems and mechanics” If you add into that the poor QA system and what seems to be an impossible task of squashing bugs brought on from said poor mechanics I think you have, as near as I can tell a pretty clear reason for the decline.

[Multigun]

Ah the ole site statistics and claim that whatever issue you personally have with the game is the very reason for the player population differences between Ark, and Conan…

Ya, that’s never been said before.

[pageslinger]

@Multigun

Ya, that’s never been said before.

Well first off I don’t have any personal problems with it…it was just a discussion that we were having. As a seemingly fairly smart active member of the modding community do you have anything other than that to say? I mean after all you like hard data…that is hard data after all.

[CodeMage]

@pageslinger

In the last 30 days according to steam charts Conan Exiles has 14,036 current players and had a 30 day average of 8,747.3 players with a -3.39% loss, Ark has 72,513 current players and had a 30 day average of 53,559.4 players with a loss of -13.31% loss.

To echo Multigun’s sentiment, but worded differently, I ain’t touching statistics with a 10 foot pole. I still remember learning statistics in high school and at the univesity, and if there’s one thing I truly learned from all of that, it’s that statistics are notoriously easy to misunderstand and misuse. You can take the same set of observations and use statistics on it in 10 different ways to to convince people to accept 15 different conclusions.

And you can see that happening on these forums a lot. Different people have interpreted the same Steam Charts data for Conan Exiles to conclude whatever they want, ranging from doom-and-gloom “Conan Exiles is dead” to “Conan Exiles is perfect and doing better than never”.

That said, if I wanted to fiddle with statistics – which I most definitely don’t – I wouldn’t take last 30 days as the basis for any serious comparison

[pageslinger]

Even if you take the “statistics” out of it the rest of my sentiment still rings true.

Everything that has been said is nothing new, just fun to point it out now and then and I don’t think anything will come of it so it is what it is. The article is a good read tho…fun to see all the stuff said in 2018 that we all know now was just a bit…off shall we say?

[CodeMage]

@pageslinger

Even if you take the “statistics” out of it the rest of my sentiment still rings true.

If we take the statistics out of what you wrote, then we’re taking away the underlying argument that Conan Exiles is experiencing a decline. In my opinion, taking that out makes your post better, not worse

@pageslinger

The article is a good read tho…fun to see all the stuff said in 2018 that we all know now was just a bit…off shall we say?

It’s a very good read. I especially liked the bit where he said he wished they hadn’t done human NPCs. That bit gives a whole new perspective to the decision to make Siptah without human NPC enemies. Ultimately, they couldn’t get away with it and they had to add NPC camps.

[Starkwolf]

I have no idea what does comparing player bases have to do with QA process, but I find it hilarious that you present some of the hard data that was requested and get resented for it.

Yeah, I get it that people can play for just the functionalities and whatever, and not give a damn about lighting details, but at least I trust my eyes over any patch notes. Patch notes, after all, only cover things that are intended to be changed and published.

The changes are, depening on weather and time of day, subtle, yet significant if you look around. And especially so in the extreme conditions. Day-time, the world is just more often covered in a more of the white hazy mist than it used to, the nights are just dimmed (no longer actually dark), and sandstorms are more saturated (in lightness, not in color) and less “sandy”.

[Taemien]

@Multigun

Ah the ole site statistics and claim that whatever issue you personally have with the game is the very reason for the player population differences between Ark, and Conan…

Ya, that’s never been said before.

Don’t get me started on that lol.

Its no wonder Ark/Rust do so well. Their combat system is quite literally move your mouse and click it. And yet we still have people here that think PVE is too hard still. Easier games always do better than harder ones.

Or I should clarify, games that put less responsibility on the player do well. You can have a harder game than Conan do well. But only if the player can point to something other than themselves as the reason for failure. Little harder to do that in Conan.

[Dzonatas]

I tried Ark, and it looked like a port. My kids played it for a little bit back in EA. I hit “randomize” on character creation, and it gave me a fat boy. Isn’t there a randomize button that works within normal limits? Eh, stopped playing after the initial spot killed me in complete darkness.

Anyways… there are two main ways to tackle bugs: through development or through maintenance.

[pageslinger]

@drachenfeles

what ark has is decently designed survival struggle loop while conan has none, that’s why ark is more appealing to actual survival game playerbase then conan and I mean it in a long run

That is pretty much it…

@Taemien

Its no wonder Ark/Rust do so well. Their combat system is quite literally move your mouse and click it.

That’s exactly what Conan combat is…Conan just has a extra layer of poor animations, flashy weapon trails and combos that don’t really do much other than lock you in an animation. But at the end of the day it is all the same.

[Maedhros]

Contrary opinion just for fun, Ark survival loop sucks. The hunger/thirst system is ridiculous, the temperature system is beyond idiotic, and the building system in Ark makes me want to commit terrible crimes.

Ark has the same issues with the playerbase that Conan has, if anything in Ark the playerbase is worse and has a greater diversity of options for being a**holes. Conan has the climbing mechanic and the building allows for more passive defense options.

But yea, Ark is a much smoother game. Most of the time.