Are you breastfeeding? If so, you're giving him the best weapon against any flu you would have.
Otherwise, just continue to be vigorous about hand washing and not kissing him on the mouth or coughing in his face (I wouldn't wear a face mask, but that's just me).
Babies who get the flu or RSV need to be watched closely for dehydration and breathing issues. If there mucous becomes too thick for them to handle and if they are not able to eat and breathe at the same time, they will need to be hospitalized. We see lots of babies this time of year (in fact, when I stopped by my work - a PICU - to send a fax, 7 of our 9 patients were flu/RSV babies! needless to say, I ran my baby out of there as fast as I could!) and they all need IV fluids and vigorous suctioning with the machine.
Keep up what you're doing mama, he'll probably be fine, truly, so take care of yourself first and foremost, so you can take care of him too. Good luck!