Correct.
You need to gather Skeleton Keys from the world bosses (they don’t drop it, you need to use a pick to mine their corpses for skeleton keys) and then use them on all these chests in the dungeons.