I believe you need to have PvP building damage on too, since non-following thralls are bound by that rule rather than strictly the PvP one.

2 Likes