You definitely have my sympathy!
I always keep the other side covered, but over the past month (he's 14 months now) my son has been plunging his hand down my shirt and fidgeting around. I remove his hand immediately and cover the breast with my hand. So far he has given up pretty easily, but I'm a little worried that it could escalate. I physically cannot nurse him if he twiddles. I absolutely cannot stand it.