Yeah, I end up making a lot of empty "yes, we'd love to come over! just let us know when it works for you" promises and then not following through with getting a date out of people. I figure if they don't offer a date and time that works, I'm not going to go out of my way to add another appointment to our schedule.
For the most part, people are pretty good about understanding that we don't want to do a lot of traveling with a toddler (next time with a toddler and an infant), so they're usually ok with it if I say that I'd rather they come to us.