with PPs. Four or five nursing sessions a day is a lot less than you need for that age. If you want to increase your milk supply, you will have to increase the number of times you're nursing him. Otherwise, every time you're introducing formula right now, you're undermining the supply you have left, and you will continue to see a decrease.
In order to increase your supply, you will have to nurse or pump "dry" a bit. It's the nipple stimulation that encourages your body to make more milk. So it takes a few days of nursing or pumping even when there is not a lot of milk for your body to get that message. If your baby is hungry and needs formula, try to give as little as possible to satiate him, and ALSO PUMP.
One final thought: Make sure you're not pregnant. Because that often results in an immediate loss of supply that cannot be regained through nursing more (it's hormonal). It sounds more like you've lost supply by oversupplementing (which can totally happen ), but just in case.