i believe that the oil and sugar might be getting in your way. my best cracker crust comes from the most basic recipe (i've tried many and keep coming back to this one) --- made in the bread machine, it's ready in less than 2 hrs for rolling out. i get three to four crusts out of the recipe, depending on thinness ---
2 c. bread flour
1 c. warm water
1 tsp. salt
2 tsp. yeast
now, i typically add things like herbs and sesame seeds that would not affect consistency. if you use this as the base recipe, then add water or flour as needed for a smooth, elastic dough ball, you should be fine. i know that adding oil/fat and/or sugar always changes consistency for mine to a more chewy crust.