In any case, the mystery is solved and I literally just finished posting the findings in a bug report, which you can read here:
@Tephra helped a lot with brainstorming and testing this one and also made the initial video of the bug that’s also in the report
The TLDR is that this mainly affects DLC mounts and it only happens if you don’t sit on them after they begin following you, so temporarily mounting / dismounting after asking them to follow will minimize the issue like @droch-aon said above
