I just found this and I must admit I wonder about it..?
Why shouldn't someone be allowed to something for herself? Let them talk, if they ask about the time just reply how they find time to chatter at the fence or just let them know that is nothing they should care about..
as long as you get everything done... what do they think you do in your spare time? polish the family silver you never need??? Just to have soemthing doing?
You cannot spend a whole day in the house..
Or why not ask them if they would like to join you?
It's more fun to walk with a group, over here in Germany there are lots of walking groups organized, here at the Kronsberg, where I live, I know at least three of them... and nobody wonders about it...I think they rather appreciate it as a way to stay healthy...