In my opinion, not having actually watched you cooking, I'd have to say that you are either overcrowding the pan and/or not heating the oil enough. Oh, and I'd skip the butter altogether. A mixture of olive oil and butter is great for adding flavour while sautéing, but for frying potatoes you need an oil that can take the heat, like canola or peanut oil, and you need to get it really hot, like close to the smoking point, before you add the potatoes. Oh, and you need to add the potatoes a few at a time because every time you add them, the temperature of the oil drops.
Life is much more manageable when thought of as a scavenger hunt as opposed to a surprise party - Jimmy Buffett