Why is it that people who can afford the time and money to take vacations always tell people who can't afford it, "you should go on vacation"? Especially since we are not exactly a culture that encourages or promotes time off -- the number of vacation days for most workers in the U.S. is far behind that of most other developed countries. And even if we're just talking about academics, you know that most of those people aren't really going to be taking time off, just schlepping their work with them to the beach or the mountains or whatever. So that the question just becomes some other version of competitive consumption and competitive work ethic. (i.e., "oh yes we go to our mountain cottage every year for two months -- it's so ideal for writing)
In my fantasy life, I'd love to go on vacation, a real vacation where I would get to see a new place, do the things I like to do, and not work. But my real life doesn't really allow for that sort of thing.
And deep down I'm bascially a hermit, so just having time off to stay home, play on the internet and watch movies is fun too. So once I'm done with grading etc, I'll designate a couple of days as days "off" -- and try not to get too caught up in household chores and projects.
So what I ask people when I'm chatting in the halls is "what are your plans for the summer" -- it's much less awkward to respond to for those of us without a lot of extra money or time than "are you going anywhere" which presumes that you will be or will be wanting to travel.