I'm going to say detergent build up. I noticed some 10 years ago, I think, when i stopped using detergent, and started using homemade laundry soap, I could wash the towels and clothes and get 'soapy water' in the washer without even using detergent. Watch your spin rinse cycle. It was because the detergent was building up and and using too much and I have hard water. Now, and then, I softened my water, so that wasn't the problem.
Now I use homemade laundry detergent and my poly/cotton towels are nice and soft and my 100% cotton towels are soft and I use no fabric softener.
I think the same thing can happen with soap building up, I say this because I have a washcloth in the shower and I get it all lathered up with homemade soap and if I don't rinse it really good, it will get very stiff and there is soap lather left in it when I wet it down and take a look.