I know what you mean--I was nervous to take LO out by myself mostly because it's hard enough to deal with her at home when she starts to cry, let alone out! But it gets easier. One thing I did--I started really small, and even just took a couple of drives without going anywhere to get used to maneuvering the carseat and having her with me in the car. Now that I'm used to it, I really enjoy going out and getting the change of pace. She was a week on our first solo trip (to the vet...geez!) and she's now six weeks.
As to worrying about her catching something--I can understand the worry, but think of it this way--what is she coming into contact with that she doesn't already? Do you worry about DH bringing something home, or you giving her a cold? Probably not--yet she's not going to be touching anyone or getting breathed on by anyone but you while out. I know people worry that a stranger will touch or come close, but this has yet to happen to me.