DD is doing this too, ever since she switched to a toddler bed. Naps like a champ at daycare but refuses to at home. We do the same "quiet time" and usually discover she's rearranged her books and diaper stacks when it's over. I don't think DD is ready to give up her nap - she's cranky without it and she does so well at daycare with it. I think she's going through a phase where she thinks she's missing out by being in her room.
Glad DD isn't the only one and keep doing what you're doing! If nothing else,the quiet time allows him to decompress.