After playing a while longer it seems that it is only the Dragons and Crocodiles that cause this. It looks like the follower is trying to move underneath the model to hit it and when a lunging attack happens, that’s when the follower gets knocked into the terrain.