Yes you are being totally paranoid. Nurses are required to either get the flu shot or wear a mask at ALL TIMES while at work if they opt not to get the shot. Yes there is a chance that they could be sick, but there is actually more risk that a fellow mom that is in the unit will come in unvaccinated and sick than the nursing staff. (Think about it-if you have the flu and go into labor the L&D unit can't turn you away, so there is a chance that moms will be sick and in labor and there's just not much that can be done about that.)
As for the L&D nurses "hanging out with the ER docs"-that really doesn't happen. For the most part units stick together and hang out/stay in their own areas. The only exception I can think to this is when I worked at a TINY hospital, if L&D had an emergency the ER crew would go up to help, but otherwise L&D and ER don't tend to mingle much.
Ultimately if you choose to give birth in a hospital you are taking a risk that infection could reach you/your baby. It's where sick people go, that's just how it is. For the most part L&D units (and post partum) are really good about keeping a "clean' environment (infectious patients are not admitted to those units), but with the amount of people that go in and out (between patients and MANY visitors) there is a chance you or your baby will catch something. The only way I see to avoid it is a home birth.
Just remember hand hygiene (washing your hands). Even above wearing a mask or getting a flu shot hand hygiene is the number one thing that people can do to prevent spreading illnesses.