I was stressed about the same thing - so I'm glad you posted this - it is good to hear from others about this.
My LO pretty much naps in his swing all day but then at night he is in his crib and I was afraid that he would one day not want to sleep in his crib because he would be so use to sleeping in his swing but every night he falls asleep just fine in his crib - so - so far so good.
But I've heard all the same things - he is too young to create habits and he needs his sleep so do what you can to make sure he gets it.