As a hortculturist and retired retail greenhouse grower and owner, this is my area of specialty.
Good windowsill herbs are parsley (moss-curled in 4" pot, Italian Flat-leaf in 6" pot), thyme, chives, tarragon, sage, oregano, rosemary and winter savory...any perennial herb, really.
Basil is an annual, but if you can give it a warm spot and plenty of sun, you can keep some going. The sun part is what's tricky up her in Illinois. About mid-winter it tends to poop out. The small leaf varieties do best in pots.
Another thing that may surprise you is that peppers are actually perennial plants in warmer climates. A pot of small chili peppers will do great in a sunny window in the winter. Mine are outside now, pinched back and fed and going for a second season.
We have a 30x12' passive solar greenhouse across the south side of our house, adjoining the living area with 3 pairs of sliding glass doors, so I have a good place to grow herbs in the winter. It's nice to be able to step out and pick fresh herbs for the evening's dinner.
It's also pretty cool to see snow on the ground and orchids blooming in the greenhouse.