The Rangers and the St. Louis Blues have opened talks to settle their differences over compensation for the Blues' signing of Coach Mike Keenan, according to a hockey executive familiar with the discussions.

The talks come even as Commissioner Gary Bettman is reviewing documents from the dispute in advance of a hearing he has scheduled Monday in Manhattan. The Rangers say Keenan violated his five-year contract with them when he walked out on July 15. Keenan has charged that the Rangers breached his agreement by not paying a $620,000 bonus check by July 14. Keenan then quickly signed with St. Louis for five years last Sunday.

Barry Watkins, a spokesman for the Rangers, said he had no knowledge of any talks between the two clubs, and Blues officials did not return calls seeking comment. But WFAN Radio reported yesterday morning that the teams were discussing a deal, and a hockey official who insisted on anonymity confirmed that. Tampering Could Become Issue

Although Bettman is not discouraging settlement talks, he will probably still hold the hearing. If he approves of a settlement, he may then focus on whether the Blues or the Detroit Red Wings contacted Keenan during the season in violation of N.H.L. tampering rules.