If you are comfortable, and your doctor and pediatrician don't see an issue, go for it. It's your house and your baby- and if it's a concern as to if the paramedics will find the baby, put a sign on the door that says "Beware: Baby in Closet". I know I'm sort of making light, but seriously, there are so many people that sleep with their baby in their bed (suffocation risk), put a bumper in the crib (strangulation risk), put baby to sleep on stomach (SIDS risk), leave the baby to sleep in a swing or carrier (skull shape issues), take to daycare, leave with nanny, **gasp** formula feed by choice, etc. etc. etc.
What my personal choice is has nothing to do with what's best for your family- now if you were asking if it were ok to prop your baby on a pillow wrapped in a queen size down comforter I might speak up, but kids are resilient little things- Best of luck, unfortunately there's always someone on here that has an opinion that's other than yours, and I get frustrated when I see the whole "it isn't worth it to ....(fill in the blank)" because we all have different priorities and opinions.
Sorry it seemed that some people chewed you out- you're an adult and have the right to make your own decisions.
I digress... :)