ROXOBEL, N.C.—Danielle Baker wanted a $324,000 loan last year to expand the peanut-processing business she ran from the family farm. She had a longstanding relationship with the Roxobel branch of Southern Bank, and she thought Southern would help fund the peanut operation she had spun off, too.

But that branch—the town’s only bank—closed in 2014. A Southern banker based in Ahoskie, 19 miles away, said Bakers’ Southern Traditions Peanuts Inc. was too small and specialized, she says. A PNC bank branch also turned her down.

“If you are not a big company with tons of assets and a big bank account,” Ms. Baker says, “they just overlook you.”

She finally got a loan from a nonprofit in Raleigh two hours away that provides financing to small businesses but not other traditional banking services. She must drive 19 miles every afternoon to make cash deposits or get change for her cash register, and expects to make a two-hour trip when she wants to refinance. Without a local branch close to her business, she says, “it’s very aggravating on a day-to-day basis.”

The financial fabric of rural America is fraying. Even as lending revives around cities, it is drying up in small communities. In-person banking, crucial to many small businesses, is disappearing as banks consolidate and close rural branches. Bigger banks have been swallowing community banks and gravitating toward the business of making larger loans.