Is the Economist Intelligence Unit’s Democracy Index little more than a joke indicator?

The public and the media increasingly use Democratic Global Performance Indicators (GPIs). But how good are these indicators at measuring democratic health? Nicolás Palomo Hernández looks at the Economist Intelligence Unit's Democracy Index as an example. He argues that the Index has methodological and ideological biases, yet significant impact nonetheless

A biased measurement of democracy?

‘This must be a joke.' So reacted well-known political scientist Adam Przeworski to the findings of the 2021 Economist Intelligence Unit’s Democracy Index.

But why?

The literature on global performance indicators (GPIs) shows how the Democracy Index promotes a gradual, partially substantive and inherently liberal understanding of democracy. It wields agenda-setting power through socially driven demands.

Kevin E. Davis, Benedict Kingsbury, and Sally Engle Merry define GPIs as

a named collection of rank-ordered data that purports to represent the past or projected performance of different units generated through a process that simplifies raw data about a complex social phenomenon

Davis, Kingsbury, and Merry, Global Governance by Indicators, OUP, 2012

As David Nelken puts it, in their aim to simplify, quantify, and standardise social phenomena, GPIs present themselves as credible, legitimate, objective and neutral. For this reason, with the aim of delivering scientific rigour, the Democracy Index devotes several pages of its annual report to developing the methodology used to construct it. The Index assesses sixty indicators in five categories:

  • electoral process and pluralism
  • functioning of government
  • political participation
  • political culture
  • civil liberties.

Some indicators are evaluated in a dichotomous way: 0 or 1; others in a three-point scoring system: 0, 0.5 or 1. Building on the sum of the respective indicators, the Index constructs a 0 to 10 scale for each category. The final Index is simply the average score of the five categories. Finally, the Index groups states into four different regime types according to their final score. It then ranks these in descending order according to their democratic quality: full democracy, flawed democracy, hybrid regime, and authoritarian regime.

However, there is only one reference to the process of assessing such indicators: 'in addition to experts' assessments, we use, where available, public-opinion surveys'. This begs the question: how many experts do these surveys consult? What are their backgrounds, and how do their political perceptions influence the assessment of democracy? This is Peter Tasker's main objection. He criticises the use of 'anonymous experts' and defines the methodology of the Index as 'flawed'.

Democratic Global Performance Indicators as ‘political creatures’

Nelken and Merry argue that GPIs are 'political creatures'. They encourage depoliticising the phenomena they seek to measure, replacing value judgements with high quantification. In this respect, Debbie Collier and Paul Benjamin, Alexander Cooley and Gaby Umbach theorise that GPIs promote a neoliberal agenda or demands to increase rationality and accountability in the public sector by promoting new public management techniques. In the case of the Democracy Index, such a thesis is not hard to believe. The origin and editorial stance of The Economist is openly liberal and, more specifically, advocates for free trade.

According to Debora Malito et al, the ‘embeddedness of power relations’ in GPIs' construction is clear from the biased selection of their indicators. In the case of the Democracy Index, for example, no indicators concern social protection and social welfare or outcome-based economic equality.

The Democracy Index lacks any indicators concerning social protection and social welfare or outcome-based economic equality

The annual report acknowledges that the Index 'does not include other aspects which some authors argue are also crucial components of democracy, such as levels of economic and social wellbeing'. However, it does consider indicators typical of a broadly liberal view of democracy. These include 'Perception of democracy and the economic system; proportion of the population that believes that democracy benefits economic performance' or 'Extent to which private property rights are protected and private business is free from undue government influence'.

Furthermore, in its annual report, the Democracy Index outlines the ongoing debate regarding the lack of academic consensus on how to measure democracy. The report also accepts that its methodology leads to a particular understanding of democracy. Following from this, the Index promotes a rather gradual (non-dichotomous) and partially substantive methodology.

Democracy Index: influencing perceptions of democracy

Judith G. Kelley and Beth A. Simmons argue that GPIs shape global governance without the need for specific directives or legislation. GPIs can promote voluntary compliance with certain requirements with minimum external enforcement. Similarly, Cooley argues that GPIs exert normative pressure on states by promoting specific outcomes. Two perspectives can explain states' response to GPIs:

  • rationalist – fear of possible material costs
  • socially driven – fear of social pressure from hierarchical positions or change in state status.

Political science scholars do not often use the Democracy Index. However, it features prominently in the media and is widely consulted by the general public. It shapes global governance by exerting pressure on states through socially driven demands. Thus, it conditions states' reputation in the media and social spheres.

Despite being little used by political scientists, the Democracy Index shapes global governance by exerting pressure on states through socially driven demand

According to Nelken, organisations use GPIs to boost their image by publicising them in the media. The Democracy Index, for example, is made by a private media company whose main product is The Economist magazine. This suggests that diffusion and agenda-setting – rather than triggering direct state response – are its main purpose. Indeed, Cooley emphasises GPIs' importance as powerful agenda-setters in a global and transnational fashion.

The main outcome of the Democracy Index is the creation of a ranking structure among states on the basis of their democratic quality. According to Kelley and Simmons, such rankings trigger mechanisms of competition between states and activate transnational institutional and social pressures. Similarly, Cooley argues that GPIs reshape political relations at national and at transnational level.

In the case of the Democracy Index, national and international media cover major changes in regime type. This sets the agenda and shapes the national political debate. Thus, despite its flawed methodology and ideological bias, the Democracy Index can indeed shape perceptions of democracy around the world.

This article presents the views of the author(s) and not necessarily those of the ECPR or the Editors of The Loop.

Author

photograph of Nicolás Palomo Hernández
Nicolás Palomo Hernández
MA Candidate in Transnational Governance, European University Institute, Florence

Nicolás holds a dual bachelor’s degree in political science and international studies from Universidad Carlos III de Madrid.

His research interests focus on democratic theory and practice, democratic innovations, European integration and institutional behaviour.

Nicolás is now researching on the democratic legitimacy of the European Union’s political system and decision-making process.

He has experience as a research assistant in European governance topics, as a project officer in international cooperation, and as a columnist for digital media.

He tweets @nikopolita

Read more articles by this author

Share Article

Republish Article

We believe in the free flow of information Republish our articles for free, online or in print, under a Creative Commons license.

Creative Commons License

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

The Loop

Cutting-edge analysis showcasing the work of the political science discipline at its best.
Read more
THE EUROPEAN CONSORTIUM FOR POLITICAL RESEARCH
Advancing Political Science
© 2020 European Consortium for Political Research. The ECPR is a charitable incorporated organisation (CIO) number 1167403 ECPR, Harbour House, 6-8 Hythe Quay, Colchester, CO2 8JF, United Kingdom.
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram