Maintaining healthy boundaries with difficult people can be, well, difficult.

That’s because they don’t want you to have boundaries in the first place, said Julie de Azevedo Hanks, LCSW, founder and executive director of Wasatch Family Therapy, a private practice in Utah.

It may not be a conscious decision. “It’s often the only relationship strategy they know.” But regardless of whether it’s intentional, the result is the same: Your boundary has been violated.

How can you stand your ground? Here are five suggestions.

1. Realize that your needs are important.

“When you doubt your own importance, you’re allowing the manipulations of difficult people to gain a foothold,” said Ryan Howes, Ph.D, a clinical psychologist in Pasadena, Calif. However, when you understand that your time, money, dignity and needs are vital to your well-being, it’s easier to tune out people who want to break your boundaries, he said.

If you doubt your importance, he suggested the following:

Be with people who value you. “Your social group is like a mirror, reflecting your value back to you.” You can surround yourself with selfish, difficult people who reflect you have little self-worth, which you eventually start to believe. Or you can surround yourself with caring, loving people and start believing that you’re also worthy of love and care, he said.

See a therapist. Therapy helps you build self-worth and pinpoint the obstacles that prevent you from valuing yourself.

Be objective. Create a list of the ways you make the world a better place, he said. For instance, you’re someone’s good friend, you make your spouse smile on a regular basis, and you’re committed to recycling. “Just being human means you deserve fundamental rights and respect, but if you look a little deeper you might find unique qualities you can appreciate about yourself.”

Be fair. “If you believe all people deserve respect, this includes you. If you allow others to treat you like dirt, and you believe they’re entitled to do so, you’re not being fair.”

2. Be firm and kind.

Being firm doesn’t mean being callous, belittling or hurting another person, said Hanks, author of The Burnout Cure: An Emotional Survival Guide for Overwhelmed Women. “You can be firm and loving, firm and validating.”

For instance, you’ve gone on several dates with the same person, but you just don’t click. You let the person know, but they keep persisting and want to continue the relationship. According to Hanks, you might say: “I really enjoyed our time but I’m not interested in pursuing a relationship. Please don’t contact me. I wish you the best.”

3. Have realistic expectations.

“If you know the person is difficult for you to have a relationship with and doesn’t respect your boundaries, limit the amount of time, or the place of your interaction so you can have healthy boundaries,” said Hanks, who also pens the Psych Central blog Private Practice Toolbox.

4. Walk away.

“Many times it is important to confront difficult people to have a voice, stand up for yourself, and maybe even put them in their place,” said Howes, also author of the blog In Therapy. But sometimes walking away is a better approach.

He likened it to a tornado coming your way: Rather than face it, the best response is to retreat. Some people are simply too toxic to confront, he said.

If you’re talking on the phone, the equivalent is to end the conversation. In her clinical practice, Hanks often sees boundary violations play out with ex-spouses. For instance, your ex-husband calls to talk about your child. However, the conversation shifts, and he starts making derogatory remarks about your new boyfriend. You explain that your relationship isn’t up for discussion, but he continues to pry. That’s when you decide to hang up, Hanks said.

5. Remind yourself you’re in charge.

Remember that how you approach boundaries is really up to you. Difficult people want you to believe that you’re over-reacting, said Jan Black, author of Better Boundaries: Owning and Treasuring Your Life. Take the example of your brother regularly ridiculing your spiritual beliefs at family gatherings. When you ask him to stop, he says that you just don’t know how to take a joke.

“Do you grin and bear it? Stop going to family events if he’s there? Lash back at him about his lazy-ass efforts at getting a job? Invite him to breakfast to find out what it is about your spiritual beliefs that concern him? Write him a letter asking him to stop? Work out an agreement that signals him when he’s on the edge of going too far?”

Again, this is your decision – not his or the person who’s trying to cross your boundary, she said. Assess the situation and figure out how you’d like to enforce your limits.

Ultimately, when difficult people violate your boundaries, you can use it as an opportunity to better understand who you are and what’s important to you, and to “develop the voice to claim [your] territory and declare [your] value.”

5 Ways to Maintain Boundaries with Difficult People