I understand why DH would be upset, but for you to get sick is really scary. I know it means that the holidays arent going to go as planned, but your health is much more important than anything else. I dont think its crazy for you to leave. Especially, if someone actually has the flu (and not only just getting over it, which is also still scary for you). You wouldnt take a newborn baby to a house with people who have the flu and are just getting over the flu, so then you shouldnt be there either.
And I LOVE the nb baby analogy.
T&Ps you get healthy soon!