There’s a couple workarounds for this and there’s been a few threads about it. Current active thread is in the testlive section.
Anyway, pull out a left-handed item and keep it in your hand while building. (Torch, bindings, shield, etc.) After the 1st punch, you’ll be fine as long as it stays in your hand. It’s still annoying but it works…