You may want to give the dream feed a try then...it might be much easier & less stressful than pumping. If it disturbs him too much you can always go back to pumping at that time. I did the same as you for a long time (pumping before bed) - it was exhausting! Then my DD started eating less at her bedtime feeding so I started doing a dream feed instead. SO much easier every baby is different but it's worth a try.
Also want to add that stress can definitely work against you in terms of supply. I know it's easier said than done but try to relax & don't get too worked up about the numbers