In my house, parents are off limits. No matter how intrusive or opinionated they may be at any given moment, they are off limits when we are in the throws of a disagreement. I would strongly advise that you take that stance in your own house.
Now, with the baby, I do not understand why your man is doing what he is doing. I rush home from work every night to make sure I get as much time as I can with my family. I am hard-wired that way, so I really don't even think about it. But rarely do I go out on my own, weekday or weekend. I met an old boss of mine this past Monday after work, and it was the first night I was not with my family since Jacob was born. And that was only for a little over two hours.
Some men are not wired that way. However, there is a level of unfairness going on here that I would address, if I were you. He is obviously blind to the fact that you are home all day caring for your child. I am sure there are days that go by where you do not shower, barely get a bite to eat, and are punch drunk from lack of sleep. It seems to me that is the trap you have fallen into here. You need to stand up for yourself and make your man see that his role during this stage is to support you and what you have to do each and everyday. That means being there to take some of the pressure off of you. Let you get a chance to shower, to eat, to be an adult for a little bit. He owes you that, and you need to make him aware of that fact. This is not just your job, it is a job both of you have to be involved in for it to be successful.
But you have to stay off the mom. Going there would ruin an leverage you have as the mother of his child. This has to be a conversation about the two of you, and your roles as parents for your child. That is really what this is about.