Are you at max limit? Have you been doing purges and collecting new thralls? This will contribute heavily to increasing thrall numbers and result in losses.