And you shouldn’t fall from them if you are out of stamina.
As far as I can tell, ladders have the same stamina cost as free climbing.
Out of shape me can climb a ladder easily enough but there is a big difference between rungs and a rock cliff. If you get tired on a ladder, you can lean into it and stand as easily as if you were standing on the ground. Not so if its only your grip preventing a long fall.