WASHINGTON — Only seven days after Donald J. Trump was sworn in as president, James B. Comey has told associates, the F.B.I. director was summoned to the White House for a one-on-one dinner with the new commander in chief.

The conversation that night in January, Mr. Comey now believes, was a harbinger of his downfall this week as head of the F.B.I., according to two people who have heard his account of the dinner.

As they ate, the president and Mr. Comey made small talk about the election and the crowd sizes at Mr. Trump’s rallies. The president then turned the conversation to whether Mr. Comey would pledge his loyalty to him.