We’re aware of this issue and we issued a patch on Testlive (PC only for now) that should alleviate the loading issues when nearby player-made buildings. You can check the list of included fixes in this post:
We estimate it should be available on all platforms (including PS4) by the end of this month/start of February, once we have ironed out the feedback we received from our community.