The man who Miller was arguing with in the elevator apparently left the banker's apartment late Sunday or early Monday, sources. Miller called down to the lobby and asked the doorman not to allow the man back into the building, according to sources.