CHARLOTTE AMALIE, U.S. Virgin Islands (AP) — Ask about Jeffrey Epstein on St. Thomas and rooms go quiet. Some people leave. Those who share stories speak in barely audible tones.

The 66-year-old billionaire bought Little St. James Island off this U.S. Caribbean territory more than two decades ago and began to transform it — clearing the native vegetation, ringing the property with towering palm trees and planting two massive U.S. flags on either end. When guides took scuba divers to spots near the island, security guards would walk to the water's edge.

It was off-putting to residents of St. Thomas — a lush tropical island east of Puerto Rico with winding roads through mountains dotted with dainty Danish colonial-era homes. Then, when Epstein pleaded guilty in a 2008 to soliciting and procuring a minor for prostitution, his need for privacy began to appear more sinister.

"Everybody called it 'Pedophile Island,'" said Kevin Goodrich, who is from St. Thomas and operates boat charters. "It's our dark corner."

Many people who worked for Epstein told The Associated Press this week that they had signed long non-disclosure agreements, and refused to talk. One former employee who declined to be identified said Epstein once had five boats, including a large ferry in which he transported up to 200 workers from St. Thomas to his island every day for construction work.

The man said he saw a handful of young women when he was on Epstein's property but he believed they were older than 18.

"When he was there, it was keep to yourself and do your thing," the man recalled, adding that Epstein paid well and would give away older machinery and surplus including lumber to his employees.

Epstein built a stone mansion with cream-colored walls and a bright turquoise roof surrounded by several other structures including the maids' quarters and a massive, square-shaped white building on one end of the island. Workers told each other it was a music room fitted with a grand piano and acoustic walls. Its gold dome flew off during the deadly 2017 hurricane season. Locals recalled seeing Epstein's black helicopter flying back and forth from the tiny international airport in St. Thomas to his helipad on Little St. James Island, a roughly 75-acre retreat a little over a mile (about 2 kilometers) southeast of St. Thomas.

Epstein later bought neighboring Great St. James Island, which once was popular with locals and tourists for its main attraction, Christmas Cove, a place where you could hang out and order pizza and have it delivered via boat.

18 PHOTOS The Jeffrey Epstein case See Gallery The Jeffrey Epstein case NEW YORK, NY - JULY 08: US Attorney for the Southern District of New York Geoffrey Berman announces charges against Jeffery Epstein on July 8, 2019 in New York City. Epstein will be charged with one count of sex trafficking of minors and one count of conspiracy to engage in sex trafficking of minors. (Photo by Stephanie Keith/Getty Images) NEW YORK, NY - JULY 08: US Attorney for the Southern District of New York Geoffrey Berman announces charges against billionaire financier Jeffery Epstein on July 8, 2019 in New York City. Epstein will be charged with one count of sex trafficking of minors and one count of conspiracy to engage in sex trafficking of minors. (Photo by Stephanie Keith/Getty Images) NEW YORK, NEW YORK - JULY 08: Two of the purported victims of multi-millionaire Jeffrey Epstein, Michelle Licata (L) and Courtney Wild leave a Manhattan court house after a hearing on sex trafficking charges for financier Jeffrey Epstein on July 08, 2019 in New York City. Epstein is charged with having operated a sex trafficking ring in which he sexually abused dozens of underage girls. (Photo by Spencer Platt/Getty Images) NEW YORK, NY - JULY 08: Two of Jeffrey Epstein's alleged victims, Michelle Licata (L) and Courtney Wild (R), exit the courthouse after the billionaire financier appeared for a hearing on July 8, 2019 in New York City. According to reports, Epstein will be charged with one count of sex trafficking of minors and one count of conspiracy to engage in sex trafficking of minors. (Photo by Stephanie Keith/Getty Images) In this courtroom artist's sketch, defendant Jeffrey Epstein, center, sits with attorneys Martin Weinberg, left, and Marc Fernich during his arraignment in New York federal court, Monday, July 8, 2019. Epstein pleaded not guilty to federal sex trafficking charges. The 66-year-old is accused of creating and maintaining a network that allowed him to sexually exploit and abuse dozens of underage girls from 2002 to 2005. (Elizabeth Williams via AP) NEW YORK, US - JULY 08: David Boies, attorney for the alleged sex victims of the US financier Jeffreey Epstein case, delivers a speech to the media outside the United States Federal Court on July 08, 2019 in New York, United States. (Photo by Atilgan Ozdil/Anadolu Agency/Getty Images) NEW YORK, NY - JULY 08: A residence belonging to Jeffrey Epstein at East 71st street is seen on the Upper East Side of Manhattan on July 8, 2019 in New York City. According to reports, Epstein is charged with running a sex-trafficking operation out of his opulent mansion. (Photo by Kevin Hagen/Getty Images) NEW YORK, NY - JULY 08: Prosecutors exit the room after US Attorney for the Southern District of New York Geoffrey Berman announces charges against Jeffery Epstein on July 8, 2019 in New York City. Epstein will be charged with one count of sex trafficking of minors and one count of conspiracy to engage in sex trafficking of minors. (Photo by Stephanie Keith/Getty Images) NEW YORK, NY - JULY 08: Member of the press listen as US Attorney for the Southern District of New York Geoffrey Berman announces charges against Jeffery Epstein on July 8, 2019 in New York City. Epstein will be charged with one count of sex trafficking of minors and one count of conspiracy to engage in sex trafficking of minors. (Photo by Stephanie Keith/Getty Images) NEW YORK, NY - JULY 08: A protest group called "Hot Mess" hold up signs of Jeffrey Epstein and President Donald Trump in front of the Federal courthouse on July 8, 2019 in New York City. According to reports, Epstein will be charged with one count of sex trafficking of minors and one count of conspiracy to engage in sex trafficking of minors. (Photo by Stephanie Keith/Getty Images) United States Attorney for the Southern District of New York Geoffrey Berman speaks during a news conference, in New York, Monday, July 8, 2019. Federal prosecutors announced sex trafficking and conspiracy charges against wealthy financier Jeffrey Epstein. Court documents unsealed Monday show Epstein is charged with creating and maintaining a network that allowed him to sexually exploit and abuse dozens of underage girls.(AP Photo/Richard Drew) In this courtroom sketch, defendant Jeffrey Epstein, second from right, listens along with defense attorneys, from left, Marc Fernich, Michael Miller, and Martin Weinberg as Judge Richard M. Berman denies him bail during a hearing in federal court, Thursday, July 18, 2019 in New York. Judge Berman denied bail for the jailed financier on sex trafficking charges, saying the danger to the community that would result if the jet-setting defendant was free formed the "heart of this decision." (Aggie Kenny via AP) FILE - This March 28, 2017, file photo, provided by the New York State Sex Offender Registry shows Jeffrey Epstein. A judge denied bail for jailed financier Jeffrey Epstein on sex trafficking charges Thursday, July 18, 2019, saying the danger to the community that would result if the jet-setting defendant was free formed the "heart of this decision." (New York State Sex Offender Registry via AP) In this courtroom sketch, Judge Richard M. Berman speaking as he denies Jeffrey Epstein bail during a hearing in federal court, Thursday, July 18, 2019 in New York. Judge Berman denied bail for the jailed financier on sex trafficking charges, saying the danger to the community that would result if the jet-setting defendant was free formed the "heart of this decision." (Aggie Kenny via AP) In this courtroom artist's sketch, defendant Jeffrey Epstein, left, and his attorney Martin Weinberg listen during a bail hearing in federal court, Monday, July 15, 2019 in New York. Epstein's lawyers want him released on house arrest to his Manhattan home while he awaits trial. (Elizabeth Williams via AP) This courtroom sketch shows Judge Richard Berman as he speaks during the Jeffrey Epstein bail hearing in federal court, Monday July 15, 2019. (Elizabeth Williams via AP) In this courtroom artist's sketch, defendant Jeffrey Epstein, left, listens as accuser Annie Farmer, second from right, speaks during a bail hearing in federal court, Monday, July 15, 2019 in New York. Farmer says she was 16 when she "had the misfortune" of meeting Epstein and later went to spend time with him in New Mexico. Accuser Courtney Wild, right, said in the hearing that she was abused by the wealthy financier in Palm Beach, Florida, starting at age 14. Epstein's lawyers want him released on house arrest to his Manhattan home while he awaits trial. (Elizabeth Williams via AP) In this courtroom artist's sketch, defendant Jeffrey Epstein, left, and attorney Reid Weingarten, second from right, listen as attorney Martin Weinberg, right, speaks during a bail hearing in federal court, Monday, July 15, 2019 in New York. Epstein's lawyers have insisted he will not run. They want him released on house arrest to his Manhattan home while he awaits trial. Courtney Wild, third from left, said in the hearing that she was abused by the wealthy financier in Palm Beach, Florida, starting at age 14. She called him a "scary person" and urged detention "for the safety of any other girls" out there. (Elizabeth Williams via AP) Up Next See Gallery Discover More Like This HIDE CAPTION SHOW CAPTION of SEE ALL BACK TO SLIDE

"He wasn't well received," recalled Spencer Consolvo, a St. Thomas native who runs a tourist shop near a large marina. "People think he's too rich to be policed properly."

Federal authorities consider the smaller of the two islands to be Epstein's primary residence in the United States, a place where at least one alleged victim said in a court affidavit that she participated in an orgy, as well as had sex with Epstein and other people. She said she saw former U.S. President Bill Clinton on the island, but that she never saw him having sex with anyone. A Clinton spokesman issued a statement saying he never visited there.

A day after he pleaded not guilty in a New York courtroom to charges of sexually abusing dozens of underage girls, there was scant movement on the Caribbean island. Hurricane shutters covered the windows, locals hadn't seen any lights at night and a lone worker drove a bright blue golf cart around the property.

At a nearby office that locals say Epstein owns in a seaside strip mall, a man in a T-shirt and sunglasses on his head opened the door a crack, shook his head vehemently when asked about Epstein and locked the door. The firm, Southern Trust Company Inc., hired Cecile de Jongh, wife of former Gov. John de Jongh, as its office manager, according to records with the U.S. Virgin Islands Economic Development Authority.

Meanwhile, Epstein's arrest also prompted the U.S. Virgin Islands representative in Congress, Stacey Plaskett, to announce she would give the money Epstein had donated to her campaigns to charitable groups.

Now that Epstein has been arrested a second time, locals say tourists are increasingly asking about his islands when they visit St. Thomas. A woman who did not want to be identified for fear of losing her job running a charter company said she was elated when Epstein got arrested but is now vexed at tourists' curiosity, saying she reluctantly shares whispered details of his case to prying adults if children are around.

Some of that fascination aggravates Vernon Morgan, a taxi driver and St. Thomas native.

"It brought some kind of notoriety to the Virgin Islands," he said. "We would much rather that the Virgin Islands be seen in a different light."

More from :

Tourist says she was raped, thrown off resort balcony in Dominican Republic

New Jeffrey Epstein accuser: He raped me when I was 15

Small dog set on fire, later dies at Connecticut beach