(CNN) -- A Philadelphia-area day care center said Thursday that members of a private swim club made racist comments about the center's children, and the club then canceled their swimming privileges.

Some kids from the Creative Steps Day Care center say club members made racial remarks.

The Creative Steps Day Care children -- ages kindergarten through seventh grade -- went to the Valley Swim Club in Huntingdon Valley, Pennsylvania, on June 29.

The day center's director, Alethea Wright, had contracted to use the club once a week. During their first visit, some children said they heard club members asking why African-American children were there.

One of the boys told the Philadelphia Inquirer that a woman at the club said she feared the children "might do something" to her child.

Days later, the day care center's $1,950 check was returned without explanation, Wright said.

The stepfather of one of the children was filing a complaint against the club with the Pennsylvania Human Relations Commission, the panel's chairman, Stephen Glassman, said Thursday. Watch one of the children say what he heard at the swim club »

The commission is investigating the allegations, spokeswoman Shannon Powers said, and will attempt to resolve the conflict out of court.

"Protests and public outcry won't do much to help these families, but the Human Relations Commission can determine the facts and hold people accountable for any illegal acts of discrimination," State NAACP President J. Whyatt Mondesire, who serves as a commission member, said in a press release.

Sen. Arlen Specter's office said the Pennsylvania Democrat has sent a letter to the Valley Swim Club president asking him to reinstate the contract with Creative Steps, saying, "I think that you would agree that there is no place for racism in America today."

On its Web site, the Valley Club disputed the allegations of racism.

"We had originally agreed to invite the camps to use our facility, knowing full well that the children from the camps were from multi-ethnic backgrounds," the statement said.

"Unfortunately, we quickly learned that we underestimated the capacity of our facilities and realized that we could not accommodate the number of children from these camps. All funds were returned to the camps and we will re-evaluate the issue at a later date to determine whether it can be feasible in the future."

CNN attempted to speak with club President John Duesler on Thursday. He said he had no comment and asked CNN to leave the club property.

But Duesler told two Philadelphia television stations that the children had changed "the complexion" and "atmosphere" of the club.

A small group of protesters gathered in front of the club Thursday and said they think Duesler's statements prove that the club's action was racially motivated.

"I was hoping it wouldn't be something that was so blatantly apparent. I was hoping that it was overcrowding," one protester said. "When I read the statement by the club president, it made it blatant."

Jim Flynn, who said he was one of the club members who made a complaint against the children, said it was not racially motivated.

"There were a lot of children in the pool and not enough lifeguards," he said. "As general members, we were not told that they were coming. If we knew, we could decide to not come when the pool was crowded or come anyway. We could have had an option."

He said two other day-care centers, neither of which included minority children, had previously been similarly disinvited.

A private Philadelphia boarding school said Thursday that it plans to help the day care center.

"We thought it was appalling what happened," said Girard College Admissions Director Tamara Leclair.

She is meeting with the director of Creative Steps to sign a contract allowing the children to use Girard's facilities this summer, as early as Monday.

"We've had a wonderful response. I even received an e-mail from a woman in Florida telling us we did the right thing. "

All About Philadelphia