For middle class Americans, there will be no escaping higher taxes in 2013.

Congress was able to pass legislation to keep broader middle class income taxes from rising. But workers will still have to pay at least 2% more in payroll taxes.

That's because the government had temporarily lowered the payroll tax rate in 2011 to 4.2% from 6.2%, in an effort to keep more cash in the pockets of Americans and provide a boost to the economy. The tax cut, which applies on the first $113,700 in annual earnings, expired on Monday.

Now most of the country's 160 million workers will see smaller paychecks. No one is expecting the payroll tax cut to be extended.

Monthly paychecks will have $50 less for those earning $30,000 annually, and will shave off $189.50 for those with incomes totaling $113,700.

The amounts could be large enough to deal a blow to middle class Americans' spending budgets, especially at a time when the economic recovery is still struggling to gain a foothold.

Related: Forget the rich, fiscal cliff would hit everyone else too

Already Americans are feeling jittery. The latest reports show that consumer confidence took a dive in December as fiscal cliff fears weighed on the country. A closely watched consumer confidence index, which measures the American public's sentiments every month, sank six points. It was major turnaround from October, when the index reached a four-year high.

It was also the largest drop in consumer confidence since the nation's credit rating was lowered in August of last year.

Concern over these tax hikes are already holding shoppers back.

During the holiday shopping season, between Oct. 30 to Dec. 24, shoppers spent just 0.7% more than they did last year, according to a MasterCard Advistors SpendingPulse report released last week. That's compared to the 2% sales growth in the same period in 2011.

President Obama's top economic advisers have also warned that if Congress didn't prevent tax hikes on middle class families, consumer confidence could decline, retailers could be hurt, and jobs may be cut.

"As we approach the holiday season, which accounts for close to one-fifth of industry sales, retailers can't afford the threat of tax increases on middle class families," the report said.

Payroll taxes are used to fund Social Security, and the tax cut has cost about $120 billion each year. The Treasury Department has made up for the difference with money from general funds.