I think the game looks at it as your bench loosing stability. I have seen this issue on some pvp videos where things like gods altars get destroyed with couple bombs, because player build them on the edge of the carpet. Because game thinks this is a collapse and not removal of the structure it does not give you your materials back.