The CBC’s Fifth Estate recently produced an investigation on the quality of hospitals in Canada — Rate My Hospital — which has been enormously popular and has set off discussions across the country about the need to improve our hospital services.

Clearly, getting a better picture about how our hospitals perform is of interest to Canadians, and the wish to exceed status-quo health service delivery resonates with patients, policy makers and healthcare providers alike.

So what’s the problem?

Rate My Hospital is based on various pieces of evidence collected from patients, hospital workers and hospitals themselves (where they cooperated), along with data from the Canadian Institute for Health Information (CIHI). The general goal of the initiative is to make all of the information relevant to patients at the individual hospital level: Anyone can go on the program’s web page, pick a hospital and learn about how well it performs.

Each hospital receives a letter-grade built on five standard indicators of hospital performance: mortality after surgery; re-admission after surgery and after medical treatment; and adverse events after surgery and after medical treatment. CBC used indicators taken from CIHI annual hospital data, ‘standardized’ them within four peer-groups (teaching hospitals, large, medium and small community) and then assigned each hospital a grade.

So, instead of saying that St. Joseph’s Healthcare (Hamilton) has a 5.34/1,000 mortality rate compared to 8 at the national level, Rate My Hospital determines that it is one standard deviation below the average in its peer-group, and thus gives it a letter grade A.

The goal of making data accessible and in a formula most of us can understand is laudable. Unfortunately, it is also misleading.

For starters, the formula assumes that each indicator contributes equally to the overall ranking. However, how can we decide that re-admission matters as much as mortality?

But the main concern is that the ranking system is relative, not absolute. In any system, excellent or terrible, some hospitals will be one standard deviation away from the mean, by the very definition of the standard deviation. A standard deviation is the average distance to the mean. It necessarily takes some units to be distant by more than the average to get that average distance.

The goal of making data accessible and in a formula most of us can understand is laudable. Unfortunately, it is also misleading.

Does this mean that the information in Rate My Hospital is valueless?

Certainly not — but it should be of more interest to hospital CEOs than to patients or relatives. Patients do not really need an overall score for each hospital, because individual patients are admitted for a specific diagnostic. As a result, they should be much more interested to know how a given hospital performs on a specific treatment.

And what are the untold consequences of publishing evidence on adverse events at the hospital level and having patients use that evidence to decide where to be admitted?

It really depends on the origin of adverse events. If they result from overwhelmed providers in facilities being used beyond capacity, publication of adverse events may have a welcome balancing effect. That is, patients may move to facilities with lower occupancy rates and lower adverse events rates — a good thing.

But if they result from caseload characteristics — such as more frail patients at one hospital than at another — and if better informed, potentially less-frail patients are the most likely to use that information to select their hospital, some hospitals will end up with ever-more-complex cases and more frail patients, while others will end up cherry-picking the easy ones. In other words, it could lead to imbalance and an ever-greater disparity in outcomes between facilities.

Other data used by Rate My Hospital are similarly problematic, such as the survey in which hospital nurses were asked whether they would recommend their hospital to relatives. The results made headlines because, worryingly, 25 per cent of nurses reported they would not recommend their own institutions. This was not, however, a truly representative survey and it is quite likely that dissatisfied nurses were more willing to answer the survey in the negative.

There is no doubt that having more information available on hospitals is good thing — and patients should make healthcare decisions based on good evidence. But patients do not need rankings based on assumptions and standardizations that paint only a general portrait. Patients need more refined measurements of how a given hospital performs on a menu of relevant items specific to them.

We can’t rely on the media to provide this. Perhaps it’s time governments — and hospitals themselves — stepped forward and worked together with the media to make this happen.

Michel Grignon is an expert advisor with EvidenceNetwork.ca, an associate professor with the departments of Economics and Health, Aging & Society at McMaster University and director of the Centre for Health Economics and Policy Analysis (CHEPA).

The views, opinions and positions expressed by all iPolitics columnists and contributors are the author’s alone. They do not inherently or expressly reflect the views, opinions and/or positions of iPolitics.