I faced the same issue with the owner of the post. One time at march 2022 and for second time on the beginning of October 2024.
My castle disappeared upon log out from the game at single player’s at the map exile lands, location mountains of the dead few clicks away at the opposite side of the chimmerian village. I noticed this on the next day after restoring the back up I did on the end of the previous day.