WASHINGTON—Congress should address the spending cuts that are part of the so-called fiscal cliff to avoid painful cuts to the Pentagon budget and staffing, Sen. Lindsey Graham (R., S.C.) said Sunday.

Mr. Graham is concerned that in reaching a deal to avoid the tax increases of the fiscal cliff, lawmakers will do nothing to avert major cuts in defense spending. Mr. Graham said on “Fox News Sunday” that he recently got a call from Defense Secretary Leon Panetta who warned that he would need to send out layoff notices unless Congress canceled defense spending cuts set to kick in.