I have the exact same problem, Andy, and was struggling with it this morning, as a matter of fact. I had used the self-cleaning function last night, and I always have to wipe out some residue. But increasingly, there is more and more "grime" on the window that is resistant to cleaning. I find that I can get most of it off with a microfiber cloth and/or a nylon scrubbie thing and some elbow grease, but there are two or three stubborn spots that remain.
BTW, I tried Bon Ami and Softscrub, to no avail. I do not want to use anything stronger for fear of scratching the glass. I've ordered some microfiber sponges that a scrubber side, so I will try one of those when they arrive.