An American Federal court may soon have to slog through the Antarctic jurisdictional quagmire. In 1979, an Air New Zealand DC-10 on a tourist flight, crashed against Mt. Erebus, the volcano that towers over the McMurdo station. Last November, the families of the 17 crew members filed civil damage claims for $1 million each against the United States Navy, contending that radar operators here were negligent in not warning the plane it was heading for the mountain in a storm. It is unclear if any United States court will accept jurisidiction.

What worries the State Department and the National Science Foundation, which is responsible for the United States Antarctic presence, is the possibility of criminal behavior. So far as is known, there has never been a murder or other major crime in the Antarctic. But the influx of civilian workers in recent years makes a crime-free future unlikely. Already drug abuse, theft and other minor crime is rife. A military policeman with a billy club stands regular duty at the station store to deter shoplifting. There was at least one case, possibly two, of arson in the last few months.

''We've got a disaster just waiting to happen, with all sorts of international political overtones,'' said Joseph Bennett, head of polar coordination and information for the science foundation. Where Treaties Fail

Legislation and case law have extended American jurisdiction to ships, planes and other extraterritorial places. Some years ago, a Federal court took jurisdiction in the case of a murder committed on a floating ice island in the Arctic, T-3 Island. But the Antarctic poses special problems because of the unusual, and in many ways inadequate, treaty that governs operations here. Navy and other military people at the American station come under military justice, and presumably a New Zealand subject on the New Zealand sector would come under that country's law. But the Antarctic Treaty, of which the United States is a signatory, specifies only that official observers and scientists on official exchanges shall come under the jurisdiction of their own countries, and says nothing about others. That means that people who come to the Antarctic unofficially exist in a legal twilight zone. And even official American personnel probably aren't covered by any law because Congress has not explicitly extended United States jurisdiction to the Antarctic.

Richard B. Bilder, a professor of law at the University of Wisconsin, worried about this problem for years when he was with the State Department's legal office. ''I was just telling my class the other day that Antarctica is the place to murder your mother-inlaw,'' he said. ''The challenge is to avoid a jurisdictional vacuum, so that if something happens it is a violation of somebody's law. More than one country could have potential jurisdiction.''