Attorneys in the trial of George Zimmerman for the murder of Trayvon Martin struggled to put together a jury on Tuesday, as it was hard to find people who hadn't heard something about the case.

One juror told attorneys they could only hope they find residents who could keep an open mind.

"I haven't lived under a rock for the past year," juror B-51, a white, female retiree, said. "It's pretty hard for people not to have gotten some information."

By the end of Tuesday, the attorneys had questioned 14 potential jurors in person, and more than 40 jury candidates had been dismissed after filling out a questionnaire.

Only four candidates from a pool of hundreds were questioned in court on Monday.

Court officials in Sanford have summoned 500 residents of Seminole County to appear before the judge, prosecution and defence and will choose a panel of six, with four alternates. In Florida, 12 jurors are required only for criminal trials involving cases when the death penalty is being considered. Zimmerman is charged with second degree murder, for the shooting of the unarmed black teenager in February 2012. He has pleaded not guilty.

While court sources told the Guardian that judge Debra Nelson was hoping to have a jury in place by early next week, Mark O'Mara, Zimmerman's lawyer, has warned he expects the process to take two to three weeks.

The biggest challenge facing both sides is finding jury members who have no preconceived ideas or opinions about the night Zimmerman, a neighbourhood watch volunteer, shot and killed Martin during a confrontation at Sanford's Retreat, at Twin Lakes gated community. Zimmerman, 29, denies murder on the grounds of self-defence. Prosecutors will seek to portray him as an unstable vigilante who pursued and attacked the 17-year-old as he walked through the development.

On Monday, the first 100 prospective jurors filled out a questionnaire, which sought to establish what they knew of the high-profile case. Martin's shooting sparked racial tensions and civil rights protests when Zimmerman, who is of mixed white and Hispanic parentage, was not arrested for six weeks.

A shortlist of 21 jurors was picked and the first four, whose names are not disclosed, appeared in the courtroom later in the day. Jurors B12, B29 and B76 are women who all said they were aware of the case to varying degrees but were not swayed by what they had seen. The only man interviewed, juror B30, said he never watched the news but said he felt there was fault on both sides.

A woman interviewed when court resumed on Tuesday, juror B37, told prosecutor Bernie de la Rionda that she had not read into the case in the newspapers because, she said: "I just don't think [newspapers are]

truthful. They're a lot better use in the parrot cage." The married woman, a volunteer at an animal shelter who said she looked after "one parrot, one crow with one wing, three dogs, four cats, and a couple of lizards", added that she was sure that there had been riots in Sanford over the case. Last year's protests in the central Florida city passed off without incident and there were no arrests.

Barbara Jackson protests outside the Seminole County courthouse during the George Zimmerman trial. Photograph: Phelan M Ebenhack/AP

The selection process was slowed on Monday by discussions between the legal teams and the judge, who has indicated that the case will last four to six weeks. That pattern continued on Tuesday, with only two more prospective jurors having been questioned in the first two hours. A further 100 candidates will be summoned to the court each day this week, until all 500 have reported.

Court officials announced at lunchtime on Tuesday that 41 potential jurors had been dismissed on Monday, 40 after completing the questionnaire and one after examination by lawyers in the courtroom.

Robert Zimmerman spoke to reporters on Monday to say he believed his older brother was "in good hands" with his legal representation. "We understand the state has a very high burden to meet," he said, "and we don't think they will be able to do so. We believe George will be acquitted when the evidence is presented."

Zimmerman faces a likely sentence of 25 years to life if he is convicted.

Tracy Martin and Sybrina Fulton, Trayvon Martin's parents, also spoke alongside their lawyer, Benjamin Crump, to ask for calm during the trial. "We ask that the community continue to stay peaceful as we place our faith in the justice system," Martin said.

Extra security has been laid on at the courthouse, where areas outside have been set aside for protesters. Additionally, pastors from a dozen local churches will be in the courtroom each day to monitor proceedings and report back to their communities.