Yeah, if he's not stimulating the breast, your supply will drop quickly without pumping. Try eating oatmeal, lactation cookies, drinking Mother's Milk Tea or even taking fenugreek. Pump diligently. Maybe try power pumping once a day for a few days. (Pump 10 minutes rest 10 minutes pump 10 minutes rest 10 minutes repeat. For a full hour.)
I actually seen an increase in my supply when I went back to work because I went from feeding on demand about every 1.5 to 2.5 hours where he'd empty one breast and maybe have a little snack from the other to pumping every 2.5 hours and emptying both breasts.