I'm one to say "you leave if it bothers you" BUT...there's always a but...If there was a certain person that had a staring problem it would make me uncomfortable and I would go somewhere else. I'm lucky in the fact that DH supports me wholeheartedly and would tell Uncle Ike to go home if he couldn't stop staring at LO's ni-ni's.
I wouldn't debate it with your DH though. I would just do what I was going to do and smile about it. Who knows...he may not have a problem when the time comes and he sure isn't going to say "I told you I didn't want you to nurse in front of people" while your company is there. KWIM? If he is too embarrased for you to nurse in front of people he will be more embarrased to admit he asked you to hide nursing.