(CNN) On a beautiful January day in 1961, three Marines walked out of the American Embassy building in Havana and lowered the flag that had been flying proudly outside.

The act itself was routine, but the implications were enormous. The Embassy's flag was never raised again.

"I didn't have any particular thoughts," recalled James Tracy, then a master gunnery sergeant in the Embassy's security detachment. "We were just taking the flag down."

But it was also a rolling up of the U.S.-Cuba relationship. Only a day earlier, President Dwight D. Eisenhower had announced that diplomatic relations between the two countries would cease -- the coup de grace to what had been an increasingly frosty relationship.

The three Marines didn't realize that flag would be the last to fly over the U.S. Embassy in Havana for 54 years. But after a historic thaw in U.S.-Cuba relations this summer, they returned to Havana for the reopening of the U.S. diplomatic post Friday.

Eisenhower's move followed an order from the relatively new Cuban president, Fidel Castro, to remove all but 11 U.S. diplomats from the country.

JUST WATCHED Cuba celebrates US embassy re-opening Replay More Videos ... MUST WATCH Cuba celebrates US embassy re-opening 01:59

"There is a limit to what the United States in self-respect can endure," said Eisenhower at the time. "That limit has now been reached."

But the Marines who took down the flag on that Jan. 4 were surprised by the suddenness of the Embassy closure and its longevity.

"Three years, I figured," Tracy told CNN this week. That's how long he expected the freeze to last.

Kerry acknowledged the long wait in remarks at the flag-raising Friday. He said the three Marines had made "a bold promise that one day they would return to Havana and raise the flag again. At the time, no one could have imagined how distant that day would be."

"We knew we were closing it up, but we had no idea as to all the particulars behind it," said then-Gunnery Sergeant Francis East, another Marine who participated in the flag lowering. "As far as I was concerned, it was a sad day."

East was particularly sorry to leave his friends behind. He had been at the Embassy for nearly two years and had grown to love Cuba.

"I thought it was a paradise," he recalled.

The third Marine who participated in the flag removal, Cpl. Larry Morris, also remembers Havana fondly.

"I was impressed with the city," he told CNN. "And the people were wonderful people, friendly people."

As the Marines gathered up the flag, a large crowd of Cubans hoping to get a last-minute visa to the U.S. quietly watched.

"We looked at the [flag] pole, we looked at the people, we looked at the pole," Tracy said of that moment. "And they just spread off the sidewalk."

"They knew we were going to go get the flag," he added. "They just got out of our way."

It was a solemn moment, but a hectic one as well.

Tracy, East and Morris said the days before the Embassy closure were very busy. There were classified documents to destroy, equipment to remove, personal effects to pack up -- all while protecting the Embassy and its personnel.

After the flag was lowered, the Marines went about their packing. Later that day, they got on a ship and left the country.

Morris had to leave a treasured possession behind.

"It was a '49 DeSoto," he told CNN. "And I'll spend most of my time [in Havana] looking at the old cars go by to see if I can find it!"

Locating one DeSoto in a city known for its ubiquitous classic cars will be a tough order, but Morris finally has the opportunity to try, courtesy of the same government that ordered him to fold up the flag on that fateful January day.

East, for one, is excited to be back at the U.S. Embassy to see the stars and stripes finally raised over the building.

"Just to see Old Glory go back up over Cuba," he said, "It's going to be great!"

JUST WATCHED U.S. & Cuba's complicated history in 2 mins Replay More Videos ... MUST WATCH U.S. & Cuba's complicated history in 2 mins 02:02

The trio didn't expect to actually raise the flag this time around -- something done by current Marines. Tracy said that's how it should be.

"That's their job now," he said. "They'll be protecting [the flag] just like we did in our time. It's going to be there. It's going to stay there. I hope."

Protecting the flag is a duty he understands and takes to heart.

"I protected that flag for 30 years," the Vietnam veteran, who's now retired and living in Alabama, told CNN, "and sometimes I didn't think I was going to be able to continue, but I did."

Tracy served two tours in Vietnam and held another embassy security post in Rabat, Morocco, where he met his wife Mary, then a foreign service administrator. At one point he also served in Secretary of State Dean Rusk's personal detail.

East also served two tours in Vietnam, then went on to become a Marine Corps recruiter. After retiring, he became a postal clerk and now lives in Memphis.

He was one of only two African-Americans in his class of 50 at the Marine Corps' "Embassy School" in 1959 and one of the first African-American Marines to attend boot camp at Parris Island.

Morris did one more tour -- in India -- before leaving the service. He went on to work as a machinist in his home state of West Virginia.

Raising a flag, like lowering it, is a routine act for the Marines that carry it out. But just like 54 years ago, the implications Friday were enormous.

"It's one step more for peace," Tracy said.