New Delhi, India (CNN) The call to prayer rings out through the crowded, bustling streets of Old Delhi -- a maze of chaotic streets full of honking scooters, shouting rickshaw wallas, vendors selling jewelery and eateries laden with sweet delicacies.

Set away from the din of the street through narrow alleys and stone stairs flanked by old buildings with ornate facades is the house of 65-year-old Alauddin, who goes by one name, his four sons and their families. They can trace their relatives back to the Mughal Empire, which ruled India between 1526 and 1857.

The time ticks past 7 p.m. and the family is settling down on the floor to break their fast with iftar, the first meal eaten after sunset during the Muslim holy month of Ramadan, or Ramzan in India.

Dates are eaten first followed by deep fried cheese, potatoes and a spicy dish of curried apple, melon, and lentils washed down with iced strawberry milk.

It's a fun and loving family atmosphere, with some of Alauddin's five grandchildren running about and giggling as their pet birds chirp in their cages, and the adults discuss their fourth-generation family sweet shop.

"We eat, all of us together," said Alauddin's oldest son Adnan Qureshi, 39. "We feel good when they all come, everyone sits and has a laugh."

Mateen's family sit down to break their fast with iftar in Old Delhi.

But conversation soon turns to politics.

On May 23, Indian Prime Minister Narendra Modi won another landslide victory in the country's mammoth general elections. He'll be sworn in as Prime Minister again on Thursday, ushering in another five years of Bharatiya Janata Party (BJP) rule.

While much of the country celebrated the stunning victory of a man who has promised economic reform and development, others, especially minorities and liberals, have grown increasingly concerned about the impact of the BJP's Hindu nationalist background on the country's secular fabric.

in a country of 1.3 billion people. The BJP has its roots in the right wing-Hindu group Rashtriya Swayamsevak Sangh (RSS) -- of which Modi is a member -- and many of its members are adherents of the Hindutva ideology that promotes a Hindu-first India. It's a stance that worries liberals and minorities, including more than 170 million Indian Muslimsin a country of 1.3 billion people.

"There are a lot of effects (from nationalism), majorly on Muslims and it's going to get worse," Alauddin said.

The family, who live in Old Delhi's Muslim majority neighborhood, said they have not experienced any communal tension here. The community is strong, with Hindus and Muslims living and working together for generations, they said.

But they are concerned that the social fabric may change with the BJP in power for five more years.

"There is no violence in Delhi, but there is a possibility it might happen now," said Mateen, 35, the youngest son who, like his father, uses just his first name.

Some members of India's Muslim community say they don't feel safe traveling to other towns and villages.

Attacks on Muslims and minorities

Attacks under the name of "cow protection" have risen since Modi came to power, according to a Human Rights Watch report. The group said that between May 2015 and December 2018, 44 people suspected of killing or transporting cows for slaughter, or even just eating beef, were killed in vigilante attacks. That number included 36 Muslims.

Human Rights Watch said many of the murders went unpunished in part due to delayed police investigations and "rhetoric" from ruling party politicians, which may have incited mob violence.

"Muslims are scared, very scared," said Alauddin. "The cow protectors, what they have done in all these places. Muslims are affected."

In Old Delhi, Mateen said goats and buffalo used to be slaughtered in the neighborhood, but no longer.

"Everything has to go to the slaughterhouse and then the meat is transported here. They are shifting the slaughter house further away," Mateen said.

It's not just cow vigilantes that are cause for concern, according to activists. Human Rights Watch South Asia director Meenakshi Ganguly points to a larger theme of right wing nationalists targeting anyone they disagree with, saying many Indians -- not just Muslims -- now fear a "culture of mob violence."

"BJP's supporters have attacked people whether it is to oppose an inter-community relationship, or because they claim to be protecting cows, or simply for their religious identity. They have also disrupted meetings, book readings or film screenings, and threatened activists, because they are 'offended,' and declared that opposing views are 'anti-national,'" she said.

In August, Modi condemned the vigilante attacks and has called on the states to prevent mob violence. "I want to make it clear that mob lynching is a crime, no matter the motive," Modi said. "No person can, under any circumstances, take the law into his own hand and commit violence."

Yet reports of mob attacks continue.

Minorities and liberals have grown increasingly concerned about the impact of the BJP's Hindu nationalist background on India's secular fabric.

The violence has cast a pall over many communities and the family said, though they haven't been impacted themselves, they will not take what they see as a risk and travel.

"We are not safe going to other towns or villages," said Mateen. "We are not safe. We see in the news, it's very scary actually. That's why we won't go."

Yusuf Qureshi, president of the Muslim All India Jamiatul Quresh Action Committee, which provides legal aid and support to India's Muslims, said the problems faced by minorities under Modi run deep.

"They are closing all opportunities for us -- education, employment -- all the doors are being shut." He repeated Modi's motto used during campaigning, "Sabka Saath, Sabka Vikas," which means "everyone together, development together."

"If you want us together with you, then give us development also," he said.

Rise of right-wing groups

In 2014, Modi was elected with a massive mandate to reduce corruption and create jobs. He also promised to be a champion of minorities.

But the appointment of hardline nationalists to key posts during his first term had observers questioning these promises.

BJP President and Modi's right hand man Amit Shah called Muslim migrants from Bangladesh " infiltrators " and "termites" and promised to "remove every single infiltrator from the country, except Buddha, Hindus and Sikhs."

He promised to do so by implementing the National Register of Citizens nationwide. The NRC is a hugely controversial policy mooted last year in Assam, a region of India which shares a porous border with Bangladesh.

Meanwhile, the BJP-picked Pragya Singh Thakur , who was elected to Parliament in recent elections, and is currently facing terrorism charges connected to a bomb attack on Muslims several years ago. Thakur denies the charges.

The BJP has portrayed the case against her as a conspiracy by its opponents to tar the country's Hindu community. However, as campaigning ended in the 2019 election, Thakur made headlines again when local media quoted her as calling the hardline Hindu who murdered independence leader Mahatma Gandhi a patriot.

The party censured her and initiated disciplinary action, she apologized and Modi, speaking to a local television network, said he would never be able to forgive her. But she remains one of the BJP's flag-bearers.

"They are all very dangerous people are running India," said Alauddin.

Modi's own track record with the Muslim community has come under intense scrutiny. A few months after Modi assumed office in Gujarat in late 2001, the state was rocked by riots , in which more than 1,000 people, mostly Muslims, were killed.

Modi was criticized for not doing enough to halt the violence, but was not charged with a crime. The US State Department denied Modi a visa in 2005 over the issue.

There are fears among minorities and activists that another five years of Modi will embolden right wing Hindu groups, which observers say have become more vocal during Modi's first term.

Alauddin fears the right wing will grow. "When they come to power, nobody is going to move them. They can do anything -- whatever they like."

Human Rights Watch's Ganguly said the old Delhi family is not alone in its fears.

"There is great concern that Hindu extremists engage in violence because they believe they enjoy political patronage," she said. "It is for the state to uphold rule of law, including to take action against those that might back the ruling party's political ideology."

Speaking to members of the BJP-led National Democratic Alliance in parliament's Central Hall this week, Modi promised to win the trust of minorities.

"Vote bank politics created this imagined fear, this imagined atmosphere and an environment of dread was created," he said. "In 2019, I am coming to you responsibly with a certain expectation; I am standing in front of the constitution with my head bowed and making this plea to you. We need to break this deception."

But Yusuf Qureshi questioned whether Modi has the will, or even the power, to halt the right-wing or extremist elements of his support base.

"He has said these things but the organizations associated with the BJP -- which harass us -- they are not under his control it seems. Every day we see incidents circulating on social media where minorities are being beaten and abused, he should be able to control them and punish them," said Yusuf Qureshi.

"Based on the past five years, I think there is no point in trusting unless he does something substantial -- gives us educational opportunities, gives us employment."

Ultimately, the family is concerned about what kind of India their children will grow up in.

"They are not secure," said Adnan Qureshi, of the Old Delhi family. "We are worried about our next generation and their next generation. They are not at all secure in any means. If Hindutva comes, then we have no means to live. No power, nothing."