Matthew Gibson (pictured), 44, has been charged with the sexual abuse of his own daughter, over a period of six years at a New York Church

The pastor of a New York City baptist church has been charged with the sexual abuse and rape of his own young daughter over six years.

Son of the bishop at the Progressive Baptist Church of Brownsville in Brooklyn, Matthew Gibson was arrested on Thursday and accused of repeatedly sexually assaulting his now 14-year-old daughter.

The abuse is alleged to have been taking place on the church's grounds since 2012, shortly after the victim's mother died.

'He should rot in hell,' said the victim's grandmother, Josephine Maples. 'How can a father do this?'

Myia Maples, the victim's sister-in-law, says the victim confided in her about the abuse last week, blurting it out in a moment of emotion.

Gibson (pictured with his children) allegedly raped his daughter at the New York church where he works as a pastor. His son says he was unknowingly present at the church during the various alleged assaults but says his father would 'distract him' with something to do

The Progressive Baptist Church of Brownsville in Brooklyn (above) has been temporarily closed as a crime scene investigation is carried out. The victim's family claim sexual abuse has been a problem at the church for some time

Maples and her husband have had full custody of the victim and her twin brother for seven years, with Gibson only having bi-weekly weekend visitation rights.

The victim's twin brother says he was present at the church when the assaults were taking place, but had no idea what was going on.

'He would make me play on the drums, and when it was happening, I wouldn't be nowhere around,' he said in an interview with NBC New York.

'I feel like, in the moment, I let my sister down.'

Gibson, 44, was arrested on January 10 and charged with criminal sexual abuse and forcible touching of a female victim, aged between nine and 14.

The defendant (left) only had bi-weekly visitation rights to see his two children, family have said. It's been alleged that his father, Ben Gibson (right), was meant to be supervising the visits at the church

Myia Maple, who has had full custody of the children after their mother died in 2012, says she has been to court many times attempting to revoke Gibson's visitation rights, after the children said he was physically and verbally abusive

The girl's family say they've been let down the legal system, as they claim to have appeared in court a number of times attempting to end Gibson's visitation rights after allegations he was physically and verbally abusive to his children.

They also claim the sexual abuse of young girls at the Baptist church has been a systemic problem for a number of years, involving a number of other assailants.

Gibson's father, Bishop Ben T. Gibson, was meant to be supervising his son's visits and clearly failed, the family say.

Gibson was arrested on January 10 and charged with criminal sexual abuse and forcible touching of a female victim, aged between nine and 14

Led by Maples, protesters gathered outside the church on Sunday. They are asking for the permanent closure of the facility

Maples and several other protesters gathered outside the church on Sunday demanding the facility's closure.

Speaking to PIX 11, Maples said: 'We want to speak to the district attorney’s office because you failed us as well.

'We just found out he got a $10,000 bail, which is unheard of for a case like this.

'We’re really disappointed with the district attorney for setting his bail so low. It’s a shame.'

The church has temporarily closed down while a crime scene investigation takes place.