With DS#1, my DH came to all appointments and was never asked to step out of the room. They did give me some questionnaires to fill out in the waiting room about my safety, abuse, etc., so if I needed to, I probably could have made a comment there.
It actually was helpful to have him there, because even though I was an established patient in the practice (routine GYN stuff), the doc had some questions for my husband about his family history to make sure we wouldn't need any special screening or genetic counseling.
I would call back and ask if DH will be able to join you later in the appointment- I think it's really nice for both parents to be able to hear the info, ask questions, etc. and he'd probably want to be there for the 1st ultrasound.