That unplanned stop at the ATM might be costing you more than you think.

The average fee of using an out-of-network ATM is now $4.57, according to Bankrate. This is the tenth straight year of increases.

That means that you'll pay 9% in fees for withdrawing $50 from an out-of-network ATM.

"Paying the fee every once in a great while isn't going to make or break you, but it's the habit of routinely paying these types of fees that is going to put you in the poor house," said Greg McBride, chief financial analyst at Bankrate.

Related: How to make a budget

Fewer people are using ATMs, which sounds like good news, but it's also what's driving up the cost of getting cash from an out-of-network machine.

"The cost of maintaining and upgrading ATM networks gets spread over fewer transactions each year," explained McBride.

There are two parts that make up ATMs fees: the surcharge from the ATM owner and the fee charged by the customer's bank for going out of network. The ATM charge increased to $2.90 from $2.88 while the bank's fee rose almost 2% to $1.67.

While the higher costs of accessing their own money isn't likely to sit well with customers, the fees are usually avoidable.

Using a debit card to make a purchase usually offers a chance to get cash back, and many banks' apps have maps to help find nearby in-network ATMs.

Plus, a few banks even have certain accounts that don't charge for using another bank's machine.

"There's plenty of competition in banking," said McBride. "So if there are fee policies you don't like at your current bank, there are plenty of other banks both large and small and credit unions that will be eager to have your business."

Related: Alternatives to traditional banks

ATM fees also differ by region.

Here are the cities with the lowest average ATM fees:

San Francisco $3.90

Cincinnati $3.92

Dallas $4.22

Los Angeles $4.28

Boston $4.33

Philadelphia $4.33

And the cities with the highest:

Phoenix $5.07

Atlanta $5.05

Cleveland $4.98

Miami $4.94

Denver $4.88