Take him out and enjoy him. I've honestly never had a problem with people coming up and touching my LO.
The only reason I don't go out now is because LO is more aware and has trouble napping. Then she becomes a cranky butt. When she was just a NB, we were out all.the.time.