Story highlights Source says women charging Saudi attaché kept their passports, didn't pay them

Authorities investigate possible human trafficking

Official says probe is at home of Saudi military attache

Two women allegedly claim they were mistreated

Federal authorities are investigating a potential case of human trafficking at the suburban Washington home of a Saudi diplomat, officials said Wednesday.

Authorities were called Tuesday afternoon to an upscale neighborhood in McLean, Virginia, where they "removed two potential trafficking victims," U.S. Immigration and Customs Enforcement spokesman Brandon Montgomery told CNN.

A State Department official said the McLean house is the home of the Saudi military attaché.

The case is under investigation by Homeland Security Investigations, a unit within ICE.

The two women, who are from the Philippines and currently work at the Saudi Embassy in Washington, claim they were mistreated, according to the State Department official.

JUST WATCHED Two women removed from diplomat's home Replay More Videos ... MUST WATCH Two women removed from diplomat's home 03:02

The women are charging, according to a different State Department official, that the Saudi attaché kept their passports, made them work extremely long hours and did not pay them. The source said they had not seen anything to indicate the women were physically harmed.

The official also pointed out these allegations are similar to several other cases they heard from domestic workers who work for diplomats from the Persian Gulf.

According to ICE, cases like this are very "victim-centric," meaning that once there is "confirmation of suspicions," authorities immediately go in and remove the victim and then begin the investigation.

There was no word on the identity of the Saudi diplomat who lived there.

The Saudi Embassy in Washington did not respond to requests for comment.