Those starter pots are designed to be soaked first even before you use them.
Their purpose is to hold water.
If you don't soak them, they will dry out to fast and the seedling will die.
I agree less water is best with mature rosemary plants. But at this stage and in peat seed starter pots, they must be soaked and allowed to drain and almost dry before soaking again.
Seeds and seedlings do not do well under stressful conditions. Under watering a peat seed pod is a mistake.
Read the directions on a seed starter pod. It says to embed the seed and to soak the whole pot until saturated.
Then continue to keep the pod moist until time to plant.
Not only is this how its supposed to be done, its common sense.