Even on Official PVP servers, there is a raid time set, so build damage is limited to within that time.
usually between 6pm to 1am server time.
And explosives are the only way to break someones buildings, but you have a choice, there are explosive jars, orbs, and arrows, any combination of these will work.
Toss a few grease orbs, then a fire orb, and more grease orbs occasionally to keep the fire going.
Explosive jars or Arrows, are harder to get because you make them, jars need dragon powder made in a firebowl cauldron an alchemist helps here, arrows are made on a carpenters bench, and the last tier arrow you need to find the recipe. Orbs, Mobs will drop orbs, but an alchemist will help you make those as well.
But if these are abandoned buildings, all you need to do is wait, the decay time will make it so the building will either be destructible with the activate button, or will crumble to dust at your approach.
If the Crumble happens, run up quickly because any loot will fall into loot bags, and will only be retrievable for a short time.
Building damage with weapons was possible in early access, but was removed because of the player base asking for it to be nerfed.
Welcome to Conan Exiles, now go build conquer and dominate.