Sam Brownback, the Kansas governor whose tax cuts brought him political turmoil, recurring budget holes and sparse evidence of economic success, has a message for President-elect Donald Trump: Do what I did.

In 2013, Mr. Brownback set out to create a lean, business-friendly government in his state that other Republicans could replicate. He now faces a $350 million deficit when the Kansas legislature convenes in January and projections of a larger one in 2018. The state’s economy is flat and his party is fractured.

Still, Mr. Brownback views his signature idea—eliminating the 4.6% state individual income tax for partnerships, limited liability corporations and similar businesses—as a national model.

This system benefits any firm where income doesn’t face the corporate income tax and instead passes through to its owners’ individual tax returns. The owners still pay federal income taxes on those profits, but they now pay Kansas nothing.

“My critics, which are many, they only want to look at the budget,” Mr. Brownback said in an interview. “They won’t look with any depth or detail at the impact on small-business growth or private-sector job growth. That’s the target, that’s what we’re after.”

Many of the businesses affected by the tax plan fit the mom-and-pop archetype, including car dealers, insurance brokers and manufacturers who are influential leaders in lawmakers’ districts. But pass-throughs also include private-equity firms, hedge funds and global law firms.

About half of pass-through business income in the U.S. is earned by the top 1% of households, according to the Tax Policy Center, a joint venture between the Urban Institute and the Brookings Institution.

Mr. Brownback’s policy gave businesses an incentive to come to Kansas. But it also gave existing businesses a big tax break—or a reason to restructure to avoid paying taxes—without creating any new jobs.

Bill Self, the basketball coach at the University of Kansas and one of the highest-paid state employees, got a big tax break on $2.75 million he received for “professional services” through an LLC. He pays state income tax on his $230,000 university salary.

In Washington, the Kansas idea—preferential business tax rates—is part of the Republican plan for the biggest federal tax overhaul in 30 years. The Sunflower state’s hard lessons pose economic and legal challenges for Mr. Trump and his party.

The Republican plans in Washington wouldn’t scrap pass-through taxes, and they have other components designed to boost growth, including lower corporate tax rates and immediate write-offs for capital investment. But both Mr. Trump and the House GOP would open a larger gap between top tax rates on wages and business income than Mr. Brownback did.

Mr. Trump would drop the business income rate to 15%, below his proposed 33% maximum tax rate on wages. Mr. Trump’s advisers left some details fuzzy, and a second layer of tax may apply. House Republicans would tax wages at a maximum of 33% while pass-through income would get a 25% rate, with no tax on later dividends as Mr. Trump envisions.

To tax-policy watchers, that looks a lot like Kansas.

“There was just a windfall for law firm partners and others, who just happen to be organized as pass-throughs,” said tax lawyer Steve Rosenthal, a senior fellow at the Tax Policy Center. “They had many more owners of small businesses than they expected, because it doesn’t take much to reorganize yourself.”

At least 330,000 entities used the zero rate, more than 70% above projections. Mr. Brownback’s office points to different data showing steady increases in pass-through entities, not a 2013 spike.

The cuts have done little to jump-start Kansas’ economy overall—with growth for this year projected to be flat, compared with 2% GDP growth nationally. Ratings firm Standard & Poor’s twice downgraded Kansas’ bond ratings.

“The experiment clearly has failed if it was intended to show that cutting taxes would help the Kansas economy,” said Michael Mazerov, who follows state tax policy at the Center on Budget and Policy Priorities, a left-leaning Washington think tank.

Mr. Brownback said a number of states face budget problems and said Kansas has “never had more private-sector jobs.”

While private-sector jobs have rebounded to prerecession levels, the pace of job creation in Kansas broadly followed the national trend until mid-2014, when hiring in the state began to fall off. Private nonfarm payrolls in Kansas were down 0.4% in November from a year earlier, versus 1.7% growth for the U.S. as a whole, according to the Labor Department.

The tax-cut plan was paired with other policy changes, and the top individual income-tax rate dropped to 4.6% from 6.45%. Kansas also reduced growth in state spending to an average of 1.5% a year under Gov. Brownback, compared with 6.6% under his predecessor, due to cuts in higher education, Medicaid and other programs.

To fill budget holes, Mr. Brownback also won legislative approval to take about $2 billion from highway funding, delaying more than 30 transportation projects, according to the state transportation department.

“The broad effect of that tax action has been to dramatically destabilize the Kansas budget,” said Duane Goossen, senior fellow at the nonpartisan Kansas Center for Economic Growth and a former state budget director.

Those cuts contributed to dismal approval ratings for Mr. Brownback. He’s the country’s least popular governor, according to a 2016 MorningConsult survey. Mr. Brownback blamed his ratings primarily on his elimination of teacher tenure. “Change is a difficult thing in the body politic,” he said.

Stephen Moore, an advocate of supply-side economic policies who advised Messrs. Brownback and Trump, said the Kansas plan’s designers didn’t anticipate income shifting into pass-through business form, and he took that lesson into Mr. Trump’s proposal.

“Sometimes it was legitimate, and sometimes it was a gaming of the tax system to pay the zero rate, so that loophole has to be closed,” he said. “Unless you have some rules about this, people really will shift income and they’ll find ways to legally avoid paying tax, and that was never the intention.”

Nationally, the idea is to measure “reasonable compensation,” the portion of a business’s income attributable to the owner’s labor, which would be taxed as wages—with the rest taxed at the lower pass-through rate.

Any rules written by Congress or the Treasury Department would be complex, and court cases would follow as business owners challenge audits. “Divining the difference would be really challenging,” Mr. Rosenthal said.

Write to Richard Rubin at richard.rubin@wsj.com and Will Connors at william.connors@wsj.com