To demonstrate the stunning capability of its new Amiga computer, Commodore often displays on its screen a surprisingly realistic picture of a colorful bouncing ball. So it was a deliberate challenge when, at a recent trade show, rival Atari placed its new 520 ST computer beside the Amiga and displayed on its screen a virtually identical bouncing ball.

But there was a major difference: ''Amiga, $1,795,'' read the sign under the Commodore. ''Atari ST, $999,'' read the Atari sign.

''A picture is worth a thousand words,'' said Jack Tramiel, chairman and chief executive of Atari.

The biggest battle between these two companies, and the newest skirmish in a jaded computer market, has begun. Commodore International Ltd. and the Atari Corp., fierce competitors for years, are challenging each other--as well as industry giants International Business Machines Corp. and Apple Computer Inc.--with these two flashy machines aimed at business and home users.

Success with these products is vital if both companies are to remain in business. Yet, there is a feeling in the industry that only one product will make it.

The two computers are the major new ones for this year`s holiday buyers. And both are selling well, according to the companies and their retailers.

However, analysts say both computers arrived on the market too late and with too little software to make a big dent in the Christmas market.

In addition, production volumes and distribution of the two newcomers are limited, and many machines are being bought primarily by computer enthusiasts. Hence, the battle will be decided in 1986, when production volumes rise.

The main attraction of the Amiga and the ST is their color graphics capability at a relatively low price, especially for the business user. Many in the industry say that the Amiga, which sells for $1,300 without a monitor or $1,795 with one, is more technologically impressive than the Atari.

''Video and music people have freaked over it,'' said Jim O`Flaherty, Dallas manager for Computer Age Inc., a computer store chain. But the Atari machine, at $999 with the color monitor or $799 with a monochromatic one, has the price advantage.

Atari, which reached the market in mid-summer, has sold about 50,000 machines to dealers and expected to ship 50,000 more this month, Tramiel said. However, more than 70 percent of Atari`s sales have been to dealers in Europe, where the machine has been accepted more readily than in the U.S., Tramiel said in an interview.

''I`m not happy in the United States,'' he said.

Commodore began shipping Amigas at the end of September, but supply has been limited and delivery of monitors has been slow. Company officials and some analysts estimate that the company will ship 40,000 to 60,000 by the end of the year, but other expectations are lower.

Charles Wolf, an analyst with the First Boston Corp., predicts that Commodore`s 1985 shipments might total 20,000 to 30,000. Because of limited supplies, the computers are selling as fast as Commodore can ship them.

To sustain early momentum, both computer makers are eager to provide more software and wider distribution. Thus, in addition to battling for consumer purchases, they are competing for the support of software writers and dealers. ''It reminds me of a presidential election,'' said Trip Hawkins, president of Electronic Arts, a home software producer. ''There is a lot of innuendo floating around about both machines.''

Atari has an early lead in the software campaign. Many industry experts were surprised to see more than 30 companies displaying programs at a huge Atari booth at Comdex, the recent personal computer industry trade show in Las Vegas.

For the Amiga, however, there is virtually no software, and this threatens to slow sales.

''There`s nothing to do with it,'' said David Lau, president of Island Micro Systems, in Minneapolis, which sells to large businesses.

Still, many companies have committed to developing programs for the Amiga, the choice of the software-industry establishment. Though Atari software developers are generally small, little-known companies, some from Europe, several leading home computer software companies, such as Electronic Arts and Activision, are coming out with programs for the Amiga soon.

But the leading business software companies, such as Lotus Development, Microsoft and Ashton-Tate, have not committed to Amiga or Atari. This could be the biggest problem for the Amiga and the ST, because, as machines destined for business, they face stiff competition from IBM and Apple. Furthermore, it is not clear that the business user really cares about colorful graphics.

In terms of distribution, neither company has been able to attract the top chains, such as Computerland or Businessland, relying instead on independent stores. Commodore says that its product is carried in more than 700 stores and that there is a long waiting list of dealers.

Atari Vice President Michael Katz said the ST is carried in 1,000 outlets, but industry analysts say they think the number is smaller and say that Commodore seems to have better distribution.

One thing deterring dealers from carrying the Atari is the reputation its chief, Tramiel, gained in his Commodore days. As head of the company that he founded more than 25 years ago, he suddenly would drop prices and move a product into mass distribution, undercutting the dealers.

Many in the industry say it is virtually certain that after Christmas, Atari will move the 520 ST to mass distribution and cut its price.