I'm not certain, not having seen your skillet directly, but it is most likely carbon ash from grease. With non-cast iron skillets, it has always amazed me no matter how careful I am, the sides and bottom develop a bronze or black patina from grease that refuses to wash off with regular detergent. If you check closely, you will most likely find that "grease or oil condensation" has collected on the side of your skillet and gotten beneath it as well. Direct flame from your stove will further oxidize it into a carbon ash-flake. Periodoc scrubbing on the outside with hot water and salt (used as an abrasive paste) should keep that kind of flaking to a minimum as long as you re-season it afterward. You said you rub it down with olive oil after you wipe it down - this will also develop into a carbon ash when heated. I suggest you not intentionally use oil on the outside bottom.
Here is a link to more information than I can give you and I hope it is helpful to you:
using a cast iron skillet ain't so hard!
Do enjoy your cast iron!