Back in 2009 when I first set foot in Ho Chi Minh City, I couldn’t stop raving about it.

I loved its people and its food, and not even its motorbikes could dampen my strong feelings for the city. I went back there three more times and only had positive experiences, so I couldn’t understand why a lot of travelers hated Vietnam.

The fourth time that I returned to Saigon, in December last year, I finally understood why.

Popular travel blogger Matt Kepnes says that he will never return to Vietnam. He said that “no one ever wants to return to a place where they felt they were treated poorly… [he was] constantly hassled, overcharged, ripped off, and treated badly by the locals.”

I witnessed this personally last December when I went with an American friend to Ben Thanh Market. I had been there several times before but I was never harassed. The salesladies would call out to me, I would just shake my head, and that was it.

When I was with my friend, however, things got a bit uglier. I was looking at a pair of jeans while he just waited off to the side. I had paid for my purchase and we were about to leave, when suddenly, two salesladies got hold of his shirt and tried to physically stop him from leaving. They were pulling at him in opposite directions, demanding that he buy something.

I was aghast. While I had read that vendors there could get aggressive, I had never seen it firsthand. My friend, understandably, was very pissed off, and it wasn’t the last time he was harassed in Vietnam. Like Matt, he, too, was overcharged several times (from Saigon to Hanoi), was ripped off by xe om (motorcycle) and cab drivers, and generally had an overall negative experience in Vietnam.

There was another incident that affected my feelings for Saigon. I had been staying for several days in a guesthouse in De Tham St. and I had felt very comfortable there. I had a dorm bed for $6 (free breakfast and wifi!) and I loved it.

One day, a German friend waited for me outside the hostel and a xe om driver came up to him and chatted up with him a bit. When my friend told the driver that he was waiting for me (“a friend”), the driver said (non-verbatim), “Ah, the Filipina? You should be careful with her. She had been here several times and had scammed a lot of foreigners. She even invited an Australian to go with her to the Philippines and robbed him there!”

I was amused at first when I heard the story from my friend. It was only later on that I got upset. I knew that it didn’t mean anything, and of course, my friend never took him seriously. However, the incident made me feel bad. According to my friend, the driver was really trying to convince him that he knew me to be a scammer personally.

I didn’t (and I still don’t) understand what the driver was trying to gain by discrediting me. Did he want my friend to go with him somewhere? Did he want money for his “tip”? Whatever the intention was, the incident left a bad taste in my mouth, and I know I will never see Vietnam the same way again.

Looking back on it, and on my previous trips to Vietnam, I think I could identify the reason why my first three trips there were uneventful while the last one was not.

When I went to Saigon in 2009 and 2010, I stayed in District 3, in a guesthouse within a residential area. My landlady was the nicest person ever, and the only tourists I saw were the ones in the Ben Thanh Market.

Last year, however, I stayed for more than a week in District 1, which is HCMC’s backpacker district. It’s chock full of tourists, which are a major source of income for local businesses, and maybe it has affected (some of) the locals and made them the way they are in relation to the visitors in their country.

Maybe if–and when–I go back to Vietnam, I should stick to less touristy places. Maybe by being in contact with the “real” Saigonites, I could learn to love Saigon again.

Read this post to know more about Vietnam safety. You can also read this Ho Chi Minh City Travel Guide.