In Washington, however, a Soviet diplomat, Vladimir M. Kulagin, issued a statement saying the American officer had been caught ''red-handed'' photographing Soviet military equipment in a restricted area near Schwerin.

The Soviet diplomat said the American, wearing a camouflage uniform, failed to heed the warning shot and was killed while trying to flee. Mr. Kulagin said the American's driver was apprehended in their vehicle nearby.

''The Soviet side launched a resolute protest in this connection,'' Mr. Kulagin said.

American diplomats disputed the Soviet account, saying Major Nicholson was shot without warning. ''If you hit someone with a warning shot, they have to be pretty close,'' said an American envoy involved in the case.

Russians Said to Keep Body

According to another American, the Russians kept Major Nicholson's body for a day, turning it over at 4:30 P.M. today to the white stucco American Mission in Potsdam. This afternoon an American military ambulance brought the body across the Glienicke bridge to nearby West Berlin.

A United States Army sergeant who was with Major Nicholson, who was identified as Sgt. Jessie G. Schatz, was held for a few hours by the Russians near Ludwigslust and then released, according to the American sources. The town lies about 25 miles from the West German frontier and 70 miles northwest of East Berlin.

The Americans of the elite Potsdam mission speak Russian and some speak German as well. Their commanding officer is Col. Roland Lajoie, a former Army attache in Moscow.

The missions were established in 1947 to coordinate activities in the four occupation zones in Germany.