I have the exact same set. The plastic that is inside the cap is glued in place. It's not meant to come off. You will have to get some glue, such as super glue gel, and make sure everything is dry and clean. Apply the glue to the metal rim of the window, then press the plastic disk in place with the pack end of a spoon, to avoid gluing yourself to the container lid. Once that is done, let dry for an hour. Then your cap is as good as new. I checked my lids after reading your post, and after a few years of use, there is no accumulation of debris in the cap.
Seeeeeeeya; Chief Longwind of the North
“No amount of success outside the home can compensate for failure within the home…
Check out my blog for the friendliest cooking instruction on the net. Go ahead. You know you want to.