Photo

The hedge fund SAC Capital Advisors is moving closer to a plea deal with prosecutors that would force it to wind down its business of managing money for outside investors, punctuating its decline from the envy of Wall Street to a firm caught in the government’s cross hairs.

An agreement to stop operating as an investment adviser is one feature of a larger agreement SAC is negotiating as it seeks to resolve insider trading charges, according to people briefed on the case. The plea deal, these people said, would also require SAC to plead guilty to criminal misconduct and pay more than $1 billion in penalties, a record for an insider trading prosecution.

The winding-down of the investment adviser business at the company, owned by the billionaire investor Steven A. Cohen, would be a symbolic move for the hedge fund, which has already returned billions of dollars to investors who fled as its legal problems escalated. It will probably continue to operate in a different form as a so-called family office, allowing Mr. Cohen to manage his personal fortune. Under the proposed terms of the tentative deal with the government, however, he will be prohibited from managing outside money for some period of time.

The people briefed on the matter, who spoke on the condition of anonymity, cautioned that the deal could still fall apart and an agreement was not imminent.

Related Links Documents | Graphic | Video

A spokesman for SAC declined to comment. Previously, the spokesman said that “SAC has never encouraged, promoted or tolerated insider trading.” Representatives for the government also declined to comment.

As the negotiations unfolded in recent weeks, the people briefed on the matter said, the government demanded $1.8 billion in penalties from SAC. The government was asking for $900 million in fines and $900 million in forfeited profits.

SAC’s lawyers were urging the government to deduct from that amount the $616 million the fund has already paid to the Securities and Exchange Commission, which previously issued civil charges against the firm’s subsidiaries.

The government seems prepared to strike that compromise. If SAC agrees to the deal, one of the people said, it would pay $900 million in fines and forfeit an additional $284 million to the government, reflecting the $616 million credit.

SAC’s payments, if approved, would eclipse those of earlier noteworthy Wall Street prosecutions.

Raj Rajaratnam, the hedge fund manager who was convicted of insider trading in 2011, was ordered to pay $156.6 million. Michael R. Milken, the junk bond pioneer who pleaded guilty to securities fraud charges more than two decades ago, paid about $1 billion, adjusted for inflation.

While SAC would be agreeing to a record penalty, it could have been steeper for the company. If it balked at a plea deal, the people briefed on the matter said, the government threatened to pursue a much larger fine against the fund.

And with next month’s insider trading trial of Michael S. Steinberg, one of Mr. Cohen’s former employees who is central to the indictment of the firm, SAC faces the prospect of an even stiffer penalty. The government, according to one of the people briefed on the matter, was contemplating doubling its demand to more than $3 billion if SAC refused to settle and Mr. Steinberg was subsequently convicted.

The tentative plea deal is emerging just months after federal prosecutors in Manhattan and the F.B.I. announced the indictment of SAC, calling the fund “a magnet for market cheaters.” It was a rare criminal action against such a large corporation.

The indictment, announced in July, cited the many prosecutions of SAC’s former employees as proof that the firm and its units permitted a “systematic” insider trading scheme to unfold from 1999 to 2010. Six former SAC employees have pleaded guilty to insider trading while at the fund. Two others, Mr. Steinberg and Mathew Martoma, are fighting the charges and have trials scheduled.

Preet Bharara , the United States attorney for the Southern District of New York, called SAC’s scheme “substantial, pervasive and on a scale without known precedent in the history of hedge funds.”

SAC was left with little to do but cooperate. Under the law of corporate criminal liability, an entity like SAC would be held responsible for the acts of its employees. And the employees who have admitted insider trading charges are cooperating with prosecutors and would most likely testify at a trial, bolstering the government’s case.

In the aftermath of the indictment, investors continued to flee SAC, but the fund stayed afloat as banks and other trading partners continued to do business with it. SAC managed about $15 billion at the beginning of the year, but virtually all of its outside investors have asked for their money back. After returning that money in installments over the coming months, SAC would be left managing about $9 billion of Mr. Cohen’s own money.

In addition to his investments, Mr. Cohen owns one of the world’s largest private art collections, and he plans to sell three prominent paintings at a Sotheby’s auction next month.

The group — two works by Andy Warhol and one by Gerhard Richter — could raise as much as $60 million, according to Sotheby’s estimates.