That has been my experience as well.
For me, it appears to be be associated with an overabundance of golems.
An approach I tried was arming my archer defenders with star metal arrows to take out the golems faster.