I'm not sure what's going wrong with that recipe, but perhaps you'll have better luck with a different one. Here's one that I've used in the past with good results. I've never tried it with a bread machine, but it's a similar size to the recipe you used, so it should fit.
2 tablespoons (1 ounce) orange juice
1 cup (8 ounces) lukewarm water
4 tablespoons (½ stick, 2 ounces) unsalted butter; cut into 6 pieces
3 cups (12 ounces) traditional whole wheat flour
3 tablespoons (1 ¼ ounces) sugar
Heaping ½ cup (1 ¼ ounces) dried potato flakes or 3 tablespoons (1 ¼ ounces) potato flour
¼ cup (1 ounce) nonfat dry milk
1 ¼ teaspoon salt
2 ½ teaspoons instant yeast
From here, I would just do it the same way your other recipe advised or follow your bread machine's instructions for whole wheat bread if it has any.
This recipe is from King Arthur’s Whole Grain Baking
, but I learned it at 100% Whole Wheat Bread
, a great site for those new to bread baking.