One of the rules mathematicians soon learn is never to answer a possible crank. The mail often comes with cover letters assuring that the work has been ''endorsed'' by a certain Harvard or Stanford professor, say, or even a member of the United States Senate. The attached letters are usually no more than polite brushoffs: ''Thank you for your interesting letter. . . .'' A mathematician who makes a perfunctory reply can count on its being stapled to the next round of mailings.

Dr. Hersh replied to a letter from an amateur mathematician in India. ''This guy wrote very well,'' he said, ''in a good expository style. His penmanship was fine. What he said made enough sense that I thought I would try to explain and straighten him out.''

Over several years, Dr. Hersh realized that his correspondent had come to believe that the whole edifice of mathematics was about to crumble because of a dreadful mistake made centuries ago. Dr. Hersh finally wrote back in exasperation: ''I've done all I can for you. I can't do anymore.'' The writer answered that he did not consider himself Dr. Hersh's student but rather his opponent in a debate.

Surprisingly, the rise of the Internet has not increased the amount of mathematical crank mail. Most of the letters still come typewritten, often on what appear to be manual typewriters. ''Cranks are always about one level of technology behind,'' Dr. Dudley said. Dr. Casti said he imagined some of his correspondents as ''penniless guys in cold-water flats.'' They save paper and postage by single-spacing and often type on both sides of the page.

Sometimes, just sometimes, a mathematician finds that it pays to answer an unsolicited letter, one that does not have what Dr. Stewart calls ''the strange fairy dusting of lunacy.'' He once received a clearly written letter from a man in China who believed he had trisected the angle. ''I knew it had to be a fallacy, but it gave me a spur to try and see where it was wrong,'' Dr. Stewart said. ''There were three points in the diagram that looked as if they were on a straight line, but actually were not.'' Dr. Stewart wrote back and received a reply thanking him for pointing out the error. ''It is the only time I've had any success convincing someone like this that they were wrong,'' he said.

Several years ago Dr. Stewart heard from a man -- in India again -- who had found a new, simpler proof for an obscure, pointless theorem in number theory written by Ramanujan and a collaborator. According to the Ramanujan-Nagell theorem, the only numbers one can square and add 7 to, yielding an answer that is a power of 2, are 1, 3, 5, 11 and 181. For example, squaring 3 and adding 7 gives 16, which is the fourth power (the square of the square) of 2.

Dr. Stewart was surprised to realize that the proof was correct, but it was badly typed on strange paper and cast in an idiosyncratic style that would have given any journal editor the impression that the writer was a crank. Dr. Stewart advised the writer to find an Indian number theorist who could teach him how to present a proper paper. Several years later the result was published, and soon after came another publication from the same man. ''It is worth reading these things occasionally,'' Dr. Stewart said.

But only occasionally, Dr. Dudley advised. ''There is always that chance of success, but it is so small,'' he said. ''I've gone through enough reams of crank stuff to know that the probability is close to zero.''