I recently came across a “State Fragility Index Map” from 2010 that made me doubt the usefulness of the index. As a predictive measurement, its failures are obvious. Note that Syria was depicted at the time as a relatively stable state, while Libya was portrayed as quite secure. In the 2010 Index , Syria ranked 48from the bottom (out of 177 countries), placing it in a much more stable position than Iran (32). Libya, on the other hand, was put in the 111position from the bottom, indicating that it was less fragile than Malaysia (110), Mexico (96), India (79), or China (62).

The Fragile States Index, which was until recently called the Failed States Index, is produced by the Fund for Peace and the periodical Foreign Policy. According to the Wikipedia, the indicators used in the index “are not designed to forecast when states may experience violence or collapse. Instead, they are meant to measure a state’s vulnerability to collapse or conflict.” But do they actually measure such vulnerability accurately? If they did, would not the 2010 index have indicated rather more vulnerability on the parts of Syria and Libya? According to the Fund for Peace, the index is supposed to “spur conversations, encourage debate, and most of all help guide strategies for sustainable security.” In that spirit, I would argue that makers of the index might want to rethink their methodology in order to more accurately gauge the fragility of states.

My most serious doubts about the index do not actually stem from its failure to note the fragility of Syria in 2010. I certainly did not suspect in 2010 that Syria would soon implode, nor did many others. (As Yogi Berra supposedly said, “It’s hard to make predictions, especially about the future.”) My problem is rather with the failure of the index to capture current conditions. The 2014 iteration places only five countries in the “very high alert category”: South Sudan, Somalia, Central African Republic, DR Congo, and Sudan. Syria, Iraq, and Afghanistan, in contrast, only rank in the less-fragile “high alert category,” alongside Pakistan, Haiti, and several other countries. But can one seriously argue that Sudan is currently more fragile, as a state, than Syria or Iraq? Libya is even more problematic, as it is currently categorized as more stable than any of these countries, ranked at virtually the same level as Iran and as less fragile than Uganda or Rwanda.

In actuality, Libya is not merely an extraordinarily fragile state, but rather it has virtually ceased to be a state at all in any real sense of the term. A recent Los Angeles Times article describes Libya as “teetering on the brink of disintegration,” and as “a security vacuum that was quickly filled by armed men belonging to a dizzying array of militant factions.” But that may be something of an understatement. As a Guardian story from September 9th explains:

A Greek car ferry has been hired as last-minute accommodation for Libya’s embattled parliament, which has fled the country’s civil war to the small eastern town of Tobruk. The 17,000-ton Elyros liner has been deployed, complete with its Greek crew, as a floating hotel for a legislature clinging to power in the Libyan city that is last stop before the Egyptian border. … Islamists and their allies have captured the capital, Tripoli, and most of Benghazi, the country’s second city. Derna, the next town up the coast, has been declared an Islamic caliphate and the front line begins at Tobruk airport, where pickup trucks mounting anti-aircraft guns face out into the shimmering empty desert. The small port is home to what remains of Libya’s sovereign power.

It is possible, of course, that Libya could remerge as a coherent state, whether under the power of its nominal, internationally recognized government or under that of one of its many militias. I very much doubt, however, that this will happen any time soon, and I would not count on it ever happening. Somalia, after all has been essentially a non-state for a quarter century, yet our basic political maps and other sources of official geographical information continue to regard it as at sovereign state in good standing in the “international community.” In reality, it is little more than a figment of the geopolitical imagination.

How does the Fragile State Index miss the fact that Libya is so fragile that it has essentially ceased to exist? It does so by relying on proxy measurements. In particular, it takes into account: demographic pressures; massive movements of refugees; legacy of vengeance-seeking group grievance; chronic and sustained human flight; uneven economic development along group lines; sharp and/or severe economic decline; criminalization and/or delegitimization of the state: progressive deterioration of public services; widespread violation of human rights; security apparatus as “state within a state”; rise of factionalized elites; and intervention of other states or external factors. Such indicators, to the extent that they are accurate, tell us significant things about social, political, and economic conditions, but as a whole they evidently do not accurately reflect the fragility of the states that they purport to assess.

One problem with the index is the fact that it downplays the more subtle ideational aspects of state fragility and stability. Of signal importance here is the issue of legitimacy. Although “delegitimization of the state” is incorporated into the index, it is placed in the same category as “criminalization of the state,” which is a very distinct matter. Yet if most people (and especially, most elites) in a country deny the legitimacy of its government regardless of any “criminal” aspects, fragility is enhanced. If large numbers of them deny the very legitimacy of the country itself, fragility will necessarily be pronounced. Many experts argue that such a loss of support contributed powerfully to the downfall of the Soviet Union. By the same token, it is of crucial importance that large numbers of local people—ranging from Kurdish nationalists to Islamist supporters of a renewed caliphate—do not accept the legitimacy of the states of Iraq and Syria.

I like to use such composite indexes in my teaching, as they tend to generate interesting and provocative world maps. But whenever I examine any particular index in detail, it tends to fall apart. It is evidently not so easy to shoehorn the vast complexity of the world into neat statistical categories.