It’s not just them. I’ve had passive wildlife attack me after being damage by NPCs. Such as shalebacks that wandered too close to an Exile camp. They killed the NPCs and then came after me. (It was green shalebacks, no aggressive red ones.)
Also, thralls will attack nearby passive creatures after they finish a fight with hostiles. Perhaps these issues are related.
Not sure about the wildlife issue but I just found the Relic hunter city today for the first time. I had no wildlife following me and was in the tavern talking too Conan when a random tanner started attacking me. 3 others joined in and killed me.
When I went back, I was able to walk into the city, past many people but every now and again one or two random people would turn and start attacking me. Everyone else would ignore me.
It’s not even a race thing. Stygian tanner and fighters initially. Yet the guards at the entrance are also Stygian and they let me walk right past.
Only 2 groups are not set to passive in this city; 1. the group with a dancer/bearer/tanner
2. the group around the taskmaster/exiles at the edge of the
city where there is a greater wheel of pain
Once you know that, just be prepared, everyone else should not attack you unless YOU start the agression or if they have been aggro (by some wildlife for ex) and you are standing in their aggro range. So just move away from the aggro range before the fight against wildlife ends or simply kill everything while they are still being busy.