Extracting map name and db name from map mods

Conan Exiles General Discussion Players Helping Players
1

I need some help from the modding community. I don’t want to make a map mod, but I would like to be able to recognize one without running it in the game.

I know how to unpack the .pak files (unless they’re encrypted like DLCs) and I’m confident that I can write code that can work with the data formats of the files inside these packs to extract data from them.

What I don’t know is where to look inside the .pak files to extract the data that I’m interested in, which is the following:

  • does this mod add a new map to the game? (true/false)
  • what is the display name of this map? (e.g. “Savage Wilds”)
  • what is the filename for the game database in single-player mode? (e.g. savagewilds_game.db)

If there’s anyone on the forums who has at least dabbled in making a map mod, can you point me in the right direction? What do you, as a mod creator, have to do to make a map mod and how do you define the display name and the database name for your map?

1 Like
2

Try the mod community on discord. Really smart group.

1 Like
3

You not supposed to do that :frowning: Evil!!!
No rummaging in other people’s stuffs without permission!! :smiley:

(On a completely unrelated note, if you were to be making your own map mod for example… that information in the devkit is stored in the MapDataTable. I haven’t made a map mod (yet), but I assume any mod adding to that would have their own table with whatever name and merge into it on load like we do with other tables.)

3 Likes
4

Thanks, I will if this thread produces no useful results. I’m not a huge fan of using synchronous communication for everything, so I thought I would try async first :wink:

I’m assuming that’s a joke, right? Sorry, Poe’s Law :slight_smile:

That sounds interesting. I’ll try to use that to start figuring out which .uasset and/or .uexp file(s) to look at.

How do you specify how tables are merged?

5

There is a mod controller built in devkit. You link your mod with the table to “overwrite” on load up. Doesnt jave to have sam name. I mainly did tables, but blue prints etc work the same. For tables you could have it named differently, usually originalname_moddedname link to originalname is how i did it to remember wjat i was editing.

I believe it basically attaches your fnod info to the ingame and used any added info, removing javing to find every instancw of the original and editing those calls.

2 Likes
6

Well, not necessarily :slight_smile: Depends on how you look at it I guess.
Here’s a section from the rules of the discord @biggcane55 linked, although it’s more aimed towards the intention of reusing said things - still, I for one don’t particularly appreciate when people poke around too much.
For example I had cases where I made a mod for a game and due to the nature of the game the pak files can be extracted with any unpacker and the contents are mainly xml based so after I figured out how to do a thing, someone just took my mod, changed some values around and reuploaded it :slight_smile:
Now I didn’t make a fuss about it being xml based and all, it’s not like you can copyright that and I definitely wouldn’t mind if it was outdated or deprecated in any way, but that wasn’t the case so I was a bit like… “Sure… good job…” seeing as they just made a competing mod with it…

image

In any case, may I ask what you wanna do with the maps? :smiley:

7

Having read those rules, I can see how my use of the word “extract” might have been alarming. I probably should’ve used “inspect” instead :smiley:

Basically, I’ve been writing a replacement for the Funcom launcher in my spare time. I’m calling it BUGLE, the Butt-Ugly Game Launcher for Exiles :stuck_out_tongue:

So far, I’ve only done work on the server browser for it:

image
image1280×790 54.6 KB

(now you know why the BU in BUGLE)

Anyway, I’m starting to work on the single-player section of it, and I want to give people the ability to have multiple saves, kinda like Mikey’s Toolbox used to do it. For that, I would like to be able to recognize which .db files in the Conan Exiles/ConanSandbox/Saved directory are used by the game when you want to continue the singleplayer session.

I know that game.db is for Exiled Lands, and dlc_siptah.db is for Isle of Siptah, but I don’t want to maintain a hardcoded list of map names and their corresponding database filenames. So I have three options:

  1. Make the UI completely agnostic to which .db file represents a game in progress for which map. Treat every .db file the same and leave it to the user to treat a particular file as a game in progress or a backup.
  2. Hard-code only Exiled Lands and Isle of Siptah, and let users manually designate a particular .db file as a game-in-progress file for a particular map.
  3. Inspect all install mods and extract the metadata I need (map display name, database filename, and map name used in the map column in the actor_position table in the game database).

Naturally, my ambition is to implement approach 3, but I’ll fall back to approach 2 or even 1 if it proves to be too much of a pain in the ass for a hobbyist side project like this :wink:

TL;DR: I don’t intend to “steal” anything or compete with any modder. I just want to be able to query all installed mods for a few chosen bits of metadata.

2 Likes
8

I can respect a person who delivers what they promised.

2 Likes
9

That’s pretty awesome :slight_smile:

Maybe you can look for the keys for the values, this is what the table looks like

image
image1909×937 97.7 KB

What you might want to consider though is that some people use up to 80 mods and some of them might be over 1GB :smiley: So parsing through them all just to look for map names every time might be a bit of an overkill

If that presents any issue, I’d probably try to mix options 1 and 2 from your list and look at all db files, strip the _backup_# from their names (since that’s hardcoded anyway) and maintain a possible db list based on that.

A potentially easier option if you want to get the map filename back is to actually open them and do a quick query on those databases
image

Then you can even name them accordingly without having to parse through a million mods. It’s not the “pretty name” that you have in the table, but it’s a start and it’s the exact filename of the map itself

1 Like
10

Yep, it’s exactly those keys that I was looking for: MapName, MapWorld, and DBName. I’m not bothering with localization yet :wink:

Fortunately, I don’t have to parse the whole mod, but just look for specific keys in specific files. The .pak format has been pretty well analyzed, so I don’t have to do a lot of my own research there, and it should be relatively easy to locate and read any specific file I need within the pack. It’s the .uasset and .uexp formats that aren’t well understood, so I’m faffing about with those.

And if the parsing/scanning process ends up taking too long, I can probably make a cache for these mappings and scan again only when the mod files change, and only those that changed.

Oh, I’m not worried about the backups made by the game itself, those are easily identifiable. I’m trying to account for other tools and for users who know how to copy these files on their own. My own directory, for example, has stuff like citadel_v4.db and hand_of_derketo.db, which hold separate game worlds where I designed my builds.

Yeah, that query is how I’m determining what map the database uses. The problem I was trying to solve is different: if I have foo.db and bar.db, and the query returns FooBarMap for both of those, how do I know which one of them is the database that will be used when you continue the single-player game and which one is just some backup that I didn’t make?

Thanks to the info your supplied earlier, I’m making headway. I see the names for those keys inside the .uasset file for the data table, and I’m working on properly parsing the exports section of the .uasset and its .uexp. I also need to figure out how to ascertain exactly what file to look for, e.g. Mods/Savage_Wilds/DataTables/SW_MapDataTable.uasset and its .uexp companion. I don’t want to assume it’s just a naming convention, but I haven’t found references to it in any other file (except SW_Preload). I’m wondering if they don’t need to reference it at all and simply recognize it based on its imports section. That would be a pain in the ass.

Anyway, reverse engineering is fun :smiley:

1 Like
11

The name of the database is the mod authors whim; could be anything. The name of the map comes from the main map level (also chosen but not necessarily the same). Would it be just to tally the list for the drop down so people can easily select because conceivably, someone has multiple map mods in the modlist and uses amunets to transfer between them, so just because a map mod is in the list does not mean a given server would use it as the active map.

12

Yea because those can be almost anything ranging from blueprints to datatables or textures… basically it’s what the extension implies… whenever you create or import ANY asset to unreal, it’s going to end up as a .uasset file, (maps are .umap) and .uexp and .ubulk (exports and bulk data) are parts of the corresponding .uasset file that are simply stored separately for convenience (fast parsing etc) and to avoid large filesizes

For this I would put up some character information. If you’re using the save system then you’re either single-player or co-op anyway, but most of the time single-player so that makes it easy to select the active character and display whatever additional data you want with it.
For example you could present the user with a list and let them load up the save that has “George Shrub, level 60 on Exiled Lands”

The thing about this though is that while modders like to keep some form of legible standard… that table could be named “basdkjackvjaushc” and it would still work :stuck_out_tongue: So no, there’s no naming requirement other than the ones we subject ourselves to and yes… the reference you found to it SW_Preload that is the name of their mod controller blueprint most likely that’s in their Preload folder (map table needs to be merged on preload), but yet again that modcontroller can be called ANYTHING, the only fixed requirement on that one is that it’s in the preload folder inside their mod folder :slight_smile:

1 Like
13

Yep, that much I understood so far. What I’m saying is that the serialization logic for a .uasset file isn’t documented anywhere. Epic doesn’t wanna document it, probably so they can introduce any changes to it without having to maintain docs, and although other people’s reverse engineering efforts have produced some documentation, it’s not very clear. But I’m slowly getting a hang of it :slight_smile:

Fortunately, I don’t need to worry about supporting several different versions of UE, so I can cut some corners there :smiley:

“Let them load up” is precisely the thing :smiley:

Here’s how you launch Conan Exiles straight into single-player mode, so that the player doesn’t have to click on anything in Conan Exiles main menu:

  • In the [SavedCoopData] section of the Game.ini file, set LastMap to the correct map name. For Exiled Lands, that’s /Game/Maps/ConanSandbox/ConanSandbox, for Siptah it’s /Game/DLC_EXT/DLC_Siptah/Maps/DLC_Isle_of_Siptah.
  • In the same section, set the StartedListenServerSession to True.
  • Launch Conan Exiles with the -continuesession command line argument.

If I want my users to be able to say “I wanna continue my current Savage Wilds single-player session”, I need to be able to set the LastMap key to its correct value. If I want them to be able to say “I wanna play in this particular saved Savage Wilds single-player session”, I need to be able to copy the .db file that represents that saved session over the savagewilds_game.db file, for which I need to know that savagewilds_game.db is the name of the “current” session.

That’s why I’m trying to determine this stuff by inspecting the mod .pak files. I mean, I figured out how to measure a server’s “ping” the way Funcom does it and a bunch of other stuff like that, how hard could this be? :crazy_face: (that’s the “famous last words” joke right there)

But it has to be somehow linked to the original MapDataTable, right? I’m guessing that can be identified by inspecting the .uasset imports section. Which would, unfortunately, require me to scan every .uasset file if I can’t find one that follows a reasonable naming convention. Still, it sounds doable. I’ll worry about getting it to work first, then I can check whether it’s too slow and needs to be cached or something like that.

Shіt, this is much more fun when I have someone to talk to about it :laughing:

14

The only link to that is the MapDataTable being one of the parameters in a function call in the preload blueprint uasset :smiley: Imagine the bottom one is a mapdatatable even thou it isn’t, I just swapped the top one so it’s less confusing

image
image764×325 33.6 KB

1 Like
15

image
image769×536 25.3 KB

So now we’re wading into the territory of actually parsing the blueprints, and that’s not where I’m prepared to go with this. That being said, I’m still going to try to figure out if there’s a half-assed way of doing this that “works well enough”. For example, if I can do what I want to by scanning .uasset files to find a file that has certain imports and exports, and that heuristic works for most mods, then that’s good enough.

If not, I’ll use one of my fallback strategies.

Thanks for all the help! I really didn’t want to go through all the trouble of learning how to make Conan Exile mods in general just to figure out these details, and your pointers and tips have been extremely valuable. I’m really grateful that you took the time to help me muddle my way through this.

1 Like
16

I don’t know if this helps or not. But the way the game determines which map you load up is determined by a server setting. This applies to singleplayer as well, just in a weird way that I don’t exactly understand.

So a server has to be loaded with a mod that adds a map. It then has to be set to RUN that map in its settings. A server can potentially (not recommended, potential for crashes is there) run a mod adding a new map and play on Exiled Lands or Siptah. I think… you can even run multiple maps on the same server but the server setting will determine which map is actually being ran.

When you connect to the server your mod list needs to match the servers. The server will then tell your client on login which map to load into.

You can kind of see this yourself, if you run Conan with Savage Wilds for example, when loading into single player it will ask you if you want to load Exiled Lands, Siptah, or Savage Wilds. The savage wilds database will be unavailable to you if you run without the mod. But again singleplayer handles it differently because the client and server are the same program in that case.

17

I understand that part well enough. See one of my posts above where I explain how you can jump into single-player with the map-and-database combo you want. Of course, that approach will fail if you’re trying to load into a map provided by a mod and that mod isn’t in your modlist, but I’m aware of that and will manage modlists for saved sessions.

Oh, yeah, I already tested what happens if you copy the Exiled Lands database over the Siptah database and run single-player, and it blew my mind: the game is perfectly capable of storing stuff for both maps in the same database and keeping them separate!

That’s why I can’t rely 100% on the SQL query @Xevyr and I talked about to figure out what map the database is for, because the query could theoretically return more than 1 row. But I’m just not going to support multi-map databases in the first version :man_shrugging: :smiley:

Yeah, unfortunately the modlist info isn’t stored in the database itself, which I think is an unfortunate oversight on Funcom’s part. The way things are right now, it’s perfectly possible to load up a single-player database with an incorrect modlist, and potentially corrupt it.

Even worse, when you load the database into the game, it removes any table that isn’t one of the tables it uses, so I can’t even extend the database by adding my own tables that hold extra info. It’s okay, though, I’ll just keep the extra info in a different file. :man_shrugging:

1 Like
18

Ikr :joy: I usually type random questions and then monologue answers to them in that modding discord if nobody replies… In fact I just typed a few giant paragraphs there an hour ago or so…
@Taemien told me off that I ask too many questions though :frowning:

2 Likes
19

I was joking, I always enjoyed the questions you asked :stuck_out_tongue:

You ask questions sometimes I didn’t even know to ask.

2 Likes
20

So, I have a small, inconsequential update on this.

I’ve spent most of the weekend reading UE 4.15 source code and untangling the parts of the (de)serialization process that interest me, and I’ve gained a whole new level of respect for the people who wrote the XeNTaXWiki page for .uasset format.

I mean, holy crap, look at how much they figured out without looking at the source!

At any rate, I started my journey on that page, but I ended up having too many questions I couldn’t answer and then I remembered that Epic allows people to view UE source code and clone it from GitHub.

I think I’ll soon be able to start writing the code that inspects unpacked .uasset/.uexp files to find map information. The only gotcha is that it might end up reporting a false positive if there’s ever a mod that defines a row for the MapDataTable but doesn’t merge it in the preload blueprint, and I can live with that :stuck_out_tongue:

So why am I even writing this? Because I have a bad tendency to stop once I feel like I’ve made a breakthrough, instead of following through and finishing the work. I feel like blabbing about it here will motivate me to go on and actually try to write that code :laughing:

1 Like