Prime Minister Mir Hussein Moussavi of Iran has announced that Iran will boycott the 1984 Olympic Games to be held in Los Angeles, the official Iranian press agency reported.

The agency, monitored here, reported that Mr. Moussavi had said the decision was reached unanimously at a Cabinet meeting Sunday.

It quoted Mr. Moussavi as saying the boycott was decided upon ''in view of United States interference in the Middle East, its support for the regime occupying Jerusalem and the crimes being committed by the U.S.A. in Latin America, especially in El Salvador.''