ok, so yeah, a very good place to start is, how much weight has baby lost? expected is up to 10% but more than that may be normal and acceptable if you were very swollen before birth and then were given a lot of IV fluids, which would have passed to baby and artificially inflated his weight--so once he started peeing, he would lose weight fast.
Weight gain from lowest weight should be 1/2-1 oz per day or 4-7 oz a week. if he lost 10% of weight it would naturally take a while to regain birth weight, up to 3 weeks of age is within the realm of ok after ruling out milk transfer issues.
2nd question is how often is baby nursing and for about how long? often the first and best thing you can do for weight loss is to nurse more often, every 2 hours around the clock for 24 hours, then every 2-3 hours during the day and no more tha 3-4 hours between feedings at night. focus on baby nursing well on side #1, using nagging techiques to keep them actively nursing longer, sitting baby up, change diaper, cool washcloth on feet and back of neck, no swaddling. use breast compressions and massage to get baby nursing longer. offer side #2 as dessert.
3rd a weigh in on a very accurate scale before and after a feeding will tell you how much milk baby is getting at the breast. an IBCLC lactation consultant can do this, but drs office can as well (a nurse visit which is usually free), LLL meeting, and yes I know moms who have used the self serve scale at the post office, fed ex, and have tried it at the grocery store produce section with usually poor results and strange looks ;)
If you know how much baby is taking in at the breast, you know if baby needs supplement. for such a young baby, they will not need the whole 25 oz a day that a bit older baby is taking in. but that is your goal, to get to 25 oz a day. so if baby nurses 12 times a day, and you do a weigh in and baby gets 1.5 oz, then baby needs an addition 1/2 ounce ONLY. use a breastfeeding friendly device to give this supplement --a supplemental nursing system, syringe, cup, spoon, eyedropper, finger feeder are all options. sure you can use a bottle with a premie nipple, but with a newborn just figuring out how to latch at the breast, sometimes they learn bad habits with the bottle that can make nursing harder.
after giving supplement, pump for 5-15 minutes after the milk stops flowing to signal a higher supply. if you are pumping more than 1-2 ounces, 12 times a day AFTER nursing, then there is nothing wrong with your supply. so that means baby needs help getting the milk from the breast, or baby needs time and will regain birth weight if we are patient.
from what you've said I don't see a need for panic , if you give we all your weights, I can evaluate and give better more specific advice.