I think the buttermilk will just make it even better
Think buttermilk pancakes I also have a corn bread recipe that calls for buttermilk and what about coffee cakes or kugel hopf using sour milk or sour cream kind of the same thing.Buttermilk and whole milk is completely different things. I don't know if it will work or not, but it just doesn't sound like it would.
I agree - it should be even better with the buttermilk. The buttermilk will activate the baking soda, and the baking soda with alkalize the acid in the buttermilk. You should have an nice, tender, result.