If there's anything to the theory (and I readily admit there probably isn't), my guess would be that most spices have a larger molecular structure that can't penetrate the potatoes (or at least, not as well)....though I have no idea how to verify or repudiate that.
Personally, I'm with you...