The difficulty of studying Order 10 squares is staggering. Such a square has 111 columns and 111 rows, in each of which 11 of the numbers used in the square must be selected and arranged so that the array fits together according to the rules. Dr. Graham said that the number of ways of choosing just one of the 111 rows is more than 470 trillion.

No ordinary computer can deal with such monster problems, but the Montreal group enlisted the help of an American supercomputer - a Cray 1S normally used for Defense Department projects by the Institute for Defense Analysis at Princeton, N.J. The institute provided Dr. Lam's group with time on the computer during periods when it was not at work on military problems.

Even with the help of the Cray, computation took several thousand hours spread over three years, ''possibly a world record'' for lengthy computation time, Dr. Lam said in a telephone interview.

He explained that he and his Concordia University collaborators, Dr. John McKay, Dr. Larry Thiel and Dr. Stanley Swiercz, had attacked the problem in somewhat the way a computer chess program tries to defeat an opponent.

''The computer chess player must examine all its possible moves and all the countermoves its opponent could make in response to each one,'' he said. ''The farther ahead the computer tries to look, the more immense its task becomes. Our search of the possibilities for an Order 10 finite projective plane faced the same kind of obstacle - gigantic complexity.''

The computation that would be needed to solve higher orders of the problem than Order 10 would take too long using present-day equipment, he said. For example, mathematicians would like to determine whether a finite projective plane of Order 12 could exist, but Dr. Lam said that only a computer 10 billion times faster than today's supercomputers could answer the question in a human lifetime.

Dr. Lam said that even using a supercomputer, his study depended heavily on time-saving strategies derived from the theoretical work of Dr. Jessie MacWilliams and Dr. Neil J. A. Sloane, both of A.T.&T. Bell Laboratories, and Dr. John Thompson of Cambridge and Rutgers universities. Dr. Thompson was the winner of a 1970 Fields Medal, an international award for mathematicians comparable in prestige among mathematicians to the Nobel Prize.