I would let the brother stay, as annoying as it is, he is family. But I would put them I their place and ask them to clean up after themselves.
As for his friend tell dh they an visit but stay in a hotel it's way to much on you. Or they come a different time.
We went to see our 10 day old nephew last Christmas... My hubby's sister... We stayed in a hotel. We were very happy to do so because for one, we knew they were in no shape to entertain and two, we wanted some privacy of our own as a childless couple at the time. It drives me batty that there are people out there who assume it is okay to stay with family when they are having children for the first time. This isn't the 1600s where you probably lived in a multi-family community with a whole bunch of women pooling together to help the new mom. Nowadays, people expect you to be Suzy Homemaker: adjust to having a newborn, keep your house sparkling and wait on guests. Sorry. I am obviously having a raging hormone moment. Because this is a fantasy.
You need to ask your DHs friends where they plan on staying. Ask it innocently. Like, "oh, which hotel did you book? Is it close by?" I can understand if immediate family is hard to relocate to a hotel, but friends of should not expect the same service.
And you need to make it clear to your BIL and his GF that if they plan on staying over, they will be asked to help out like family, and not be treated as guests. Good luck to you. Stories like this make me thankful that all my out of town in laws are planning other accommodations or are waiting a few months to come see our LO.