I'm no expert but I do A LOT of reading since my son and I have had quite the roller coaster when it comes to nursing. I partially agree with the other mamas that it may be due to teething...nursing is sometimes uncomfortable to little babes with aching gums so he may get frustrated that he wants to nurse but is in pain. I also think it may be the two following problems:
1-food allergies...this is not super common but I definitely find that if I eat to much of a particular food my son gets very fussy during his nursing sessions (arching back and crying) other signs of food sensitivities include mucus in the diaper, unexplained rashes (they don't have to be full blown, my sons is very faint on his torso), a ring of redness around the anus, and increased fussiness and sleeping issues (I noticed you said he has issues with his naps, but this could also be due to teething discomfort)
2-He is overtired. My son went through a nap transition recently where he started taking less naps but longer and he did this weird nursing behavior around noon because he wanted to sleep but not eat an he would get overtired so my letdown would just frustrate him when he just wanted to9 drift to sleep so i started putting him for naps in his stroller...
Anyway I hope I didnt overload you with different suggestions...I guess I just like to know all the possibilities so I can deal with whatever the situation may be. good luck and hopefully things will resolve soon! Remember, with babies, EVERYTHING is "just a phase" :-)