We've been out for walks since day 1 - the day DS came home from the hospital! We usually go for 30-40 mins on our walks. During his first week of life, we've also taken him to the grocery store, and out to dinner (I was craving raw fish). As long as you are not taking your LO to a super-crowded place, you will be okay!
We live just outside of boston so temps are in the low 40s. I just bundle DS up with 2 layers of blankets plus a hat. That is warm enough for him.