Skip the peroxide. Maybe stuff a cotton ball in there with a band aid over it so you don't irritate it further.
I would definitelt NOT go against the nurse's directions. And I think a cotton ball would actually irritate it further, not to mention possibly causing bacteria to get caught in a moist, warm, dark area like that if a cotton ball were taped over it.
I use peroxide on my LO's belly buttons, before the stump falls off after birth, and let them air dry as much as possible. I would think the same may apply to your belly button, peroxide it and uncover it as much as possible, especially after a shower, or after using the peroxide.
I don't think you should be too concerned if they're not, but also don't be afraid to call them back and ask more questions if it's really bothering you. However, definitely show your doc at your next appointment, because chances are they'd have to look at it to really tell you anything other than a "best guess" anyway.