Steven Puetzer / Getty Images

Being unemployed is rough enough, but it makes the annual ordeal of paying taxes even more challenging. The biggest problem is that many people who were unemployed in the previous year simply neglect their taxes. “Many taxpayers don’t realize if they were unemployed in 2011, they still need to file taxes because federal unemployment benefits are taxable,” says Amy Miller, director of the Tax Institute at H&R Block.

Although you can opt to have taxes withheld when you file for unemployment, many people skip over it because the amount they’re getting is already much less than their old take-home pay. Tax experts urge people collecting unemployment to have those taxes withheld on the front end, as challenging as that might be to your budget.

“I know they’re not getting a lot of money, but you are required to pay taxes on it, so you are just putting it off until later,” says German Tejeda, senior director of income policy at the Food Bank For New York City, which helps 40,000 lower-income New Yorkers file their taxes for free every year. Tejeda says some 15% of Americans have at least some unemployment in their income history for the year — a figure that reflects the struggles many are still having long after the official end of the recession.

(MORE: Romney’s Tax Returns Show He Paid More in Taxes Than He Owed)

Others are surprised to learn that unemployment benefits are taxable at all, says Chad Stone, economist at the Center on Budget and Policy Priorities. “People look at it as a government benefit, but it’s really a replacement for wages.” Unfortunately, unemployment benefits are not considered earned income, which means they don’t count toward your eligibility for the Earned Income Tax Credit or the portion of the child tax credit that is based on your wages. Tejeda says taxpayers should look into whether or not they qualify for the Earned Income Tax Credit if they were employed for part of the year.

Whether or not state unemployment benefits are taxable depends on the state, according to Matthew Gardner, executive director of the Institute on Taxation and Economic Policy. Of the states that levy income taxes, seven fully exempt unemployment benefits. (They are: Alabama, Arkansas, California, Montana, New Jersey, Pennsylvania and Virginia.) Indiana and Wisconsin partially exempt unemployment benefits. All other states that levy income taxes tax unemployment benefits in full.

(MORE: Almost Half of Americans Live in a Household Receiving Government Benefits)

Being unemployed does make you eligible for a couple of additional deductions if you file an itemized return, however. Expenses directly related to a job search are deductible, such as travel expenses incurred while going to interviews. These expenses are considered miscellaneous itemized deductions; in order to claim them, they must add up to 2% or more of your adjusted gross income. (Of course, this category can contain other expenses besides money spent job-searching.)

If you find a new job that requires you to move, you can deduct moving expenses if the commute to that new job from your old home is 50 miles farther than your old commute. You also must hold that job for at least 39 weeks, although you don’t have to wait until those 39 weeks are up to deduct moving expenses. If you quit before 39 weeks, you’ll have to file an amended tax return without the deductions; this doesn’t apply if you’re laid off before the 39-week period is up, though.

Miller says one benefit of the deduction for moving expenses is it’s “above the line,” meaning you don’t need to file an itemized tax return in order to claim it.