In cakes, vanilla compliments the flavor of the flour. I agree though that it isn't necessary in every cake. I personally don't care for vanilla in fruity cakes, especially those with citrus, or berry flavors. I, like you, want the primary flavor to be the citrus, or berry, or apple, or whatever.
Think of cakes as giant pancakes. If you want vanilla in your pancake batter, with a bit of extra sugar, then go for it. But if you're going for blueberry flavor, omit the vanilla altogether, and maybe even crush some of the blueberries and swirl into the batter.
The vanilla is there to add richness to the cake flavor. It's essential in a yellow cake, or angel food cake, but not so in a strawberry cake.
Seeeeeeya; Chief Longwind of the North