Welfare firms are involved in widespread "gaming" of the Work Programme, with the most vulnerable jobseekers often ignored because they are too costly to help, according to new research into how the government's flagship employment initiative is working in practice.

Providers privately admit they are focusing resources on the "easy customers" who are more likely to generate a fee, and sidelining jobless clients who require more time and investment to become ready for work, a process known as "creaming and parking," the study says.

It concludes that the quality of services offered to jobseekers is being undermined because the design of the Work Programme, in which companies are not paid until customers have been in work for two years, creates such huge financial stresses that many providers have little option but to cut corners.

The study, based on a series of often frank interviews with welfare providers of all sizes, both private companies and charities, reveals a widespread concern that the "payment by results" approach has been undermining ministers' stated aim of getting "hard to reach" customers, such as those with disabilities, off benefits and into work.

Gaming the system was a rational response to the financial demands, the study heard. One anonymous provider said: "There's a level of parking [of vulnerable customers] going on which we are not particularly comfortable with but we also need to achieve what we need to achieve."

A series of interviews and focus groups by the Third Sector Research Centre at Birmingham University reveals how the biggest Work Programme providers – predominantly private companies – transfer financial risks on to smaller subcontractor firms and charities.

The study cites a small private-sector provider which complained that big corporate providers, known as "primes", would keep "job-ready" customers for themselves while passing on more difficult cases to subcontractors. "It's not being PC but I'll just say it as it is … you tend to get left with the rubbish; people who aren't going to get a job … If the [prime] thought they could get them a job, they wouldn't [refer them to] someone else to get a job."

The design of the system meant that providers offering specialist services targeted niche groups, such as homeless people and ex-offenders. The suggestion is that the people in this category are regarded as such a risk that the primes simply "park" their cases and do not pass them down the chain at all. So charities providing specialist services for these candidates are often getting no referrals, putting them under financial pressure. One private sector interviewee described the market for customised interventions as "a washout": "Certainly for third sector providers that have a specialist niche, it's a killing zone."

The coalition launched the Work Programme in April 2011, with ministers declaring it to be a "boost for the 'big society'". It has been dogged since then by concerns that charity providers, often small, community-based operations offering bespoke services, were being sidelined and failing to get contracts or referrals.

However, the study concluded that the way the system worked meant that small private firms low in the supply chain were just as likely to be adversely affected.

Ministers argued that by offering providers higher fees for getting harder-to-reach customers into a job, they would remove the incentive for providers to game the system. But the study says gaming is still likely to happen, and "all the more so in a system under immense commercial pressure in a low-resource environment".

A co-author of the study, James Rees, told the Guardian that providers were likely to be gaming the system as a way of dealing with the acute financial pressures inherent in the payment by results approach, and he had come across no evidence that gaming had taken place to excessively inflate profits.

Kirsty McHugh, chief executive of the Employment Related Services Association, which represents Work Programme providers, accepted that the programme operated on a tight financial model. But she added: "Most Work Programme advisers, regardless of sector, are motivated by helping those furthest from the labour market into a job."

Despite concerns over the Work Programme, the government has announced plans to use a similar payment by results system as it prepares to introduce market reforms to the probation service.

The Department for Work and Pensions said: 'The Work Programme has already got more than 207,000 of the hardest-to-help unemployed people into work, and many specialist charities are playing an important role.

"We've given providers the freedom to find the right support for each participant, and we pay them more money for getting those with more complex needs into work, so there's a clear incentive to help every participant."