Passive exp was a sort of bonus for surviving the lands themselves. When the game was in early access, the environment was much harsher and harder to survive in.
There is a slider for this exp amount as well.