The only time I like the skin on my potatoes is when they are baked. When I am served a potato salad with the skin still on the potatoes, I just let it sit there or just break off the front of the piece of potato and leave the skin behind. Make sure your peeler is really sharp and don't be lazy. Peel them.
Yeah, yeah, I know red taters are the preferred one for salad. And to peel them can take what seems forever since they are so small. Then find some very large Yukon Gold ones. They make great potato salad and can be peeled quickly.