You are correct.
Perhaps it would have been better to say that the specific outfit looks very different depending upon whether it is worn by a man or a woman.