There's a pretty common metaphor in psychological/self-help circles that labels certain people as "toxic" or "unhealthy," which is very helpful in understanding why you're left feeling awful after dealing with those people.
Since nutrition and good health are important to me in the physical realm, I've been thinking a lot about how similar metaphors might be extended. There are people and foods which are toxic; there are people and foods which are nourishing; and then there are an awful lot of people and foods that are somewhere in the middle. And too much of the food or people that are in that middle spectrum, over the long haul, is unhealthy too.
One lunch or coffee or meeting with people who are emotional equivalent of white bread won't kill you, any more than a slice of white bread would. On occasion it might even seem comforting or tasty (if we're thinking French baguette and not Wonder bread). But eating baguette more than once every few months would have me feeling lousy.
Luckily, I have a few people in my life who are like the sprouted whole grain bread I eat -- hearty, textured, complicated, and nourishing. And the rest? I'm cleaning out my emotional cupboards.