There's no one age when it's safe. Do it when you feel well enough to have guests. I don't think there's anything wrong with wearing baby to keep him/her away from lots of hands, but I kind of think "What's the point of me going to this party if I don't actually get to hold the baby?" Just politely ask people to wash their hands and/or leave out hand sanitizer. You can't keep the baby in a bubble forever. When you feel well enough and comfortable with others holding the baby, have the party.
Also, with the way this board is going today, I wouldn't bring up showers for 3rd kids. :)