It sounds like you don't think you will stay married. But are scared of what it will mean to be single. Don't let your fear put you in a position where lose your power in relationship to how you parent. Meaning, would you wean because you think if you don't he will leave? Because if you let yourself be bullied into weaning out of fear, what else will he be able to get you to do or not do with the same fear?
I don't know enough about your relationship to know whether or not it can be saved. But Your relationship with your child you will have forever. And while he may have memories of his parents fighting, (Which I personally don't equate with damage but the a personal thing I guess.) but his relationship with you will be the backbone of the person he becomes. He will have a relationship with his father too. And it will be important. But if you stop meeting his needs because his father "makes you".....IMO I think there will be repercussions of that as well. Especially with a child who is as obviously intelligent and intuitive as your son seems to be.
I would focus on your son. We have people in my son's world who try to shame him for breastfeeding this old. He's three and His father's mother is really adamant and vocal about the fact that he's too big and she speaks directly to him about it. About the fact that he shouldn't. But my son takes his ques about whether or not to feel shame from me. And since I don't have any neither does he. I tell him what to say to her when she gives him a hard time. And I have no need to please her in regards to this issue. So when it comes up with me I speak facts and she can't argue. If I was ashamed or even ambievilent about it around her, than maybe my son would react differently. But he stands tall and proud in regards to his nursing relationship with me. We don't flaunt it and I don't go out of my way to do it in front of her, but if he needs it when we're there and I know it's a NEED, I comply.
So maybe if it's inevitable that your DH is going to try to shame your son, you could be proactive in talking to him in regards to examining his feelings about it ahead of time (or right after such a conversation) and talk to him about how to express how HE feels about nursing to his father when it comes up.
Because ultimately the relationship is between you and your son. Your DH has the ability to make both you and your child feel pressure, but short of staying home from work to harass you, he really can't stop you. Remember that. And make sure your son understands it too.