Several liberal democracies view the People’s Republic of China no longer as a strategic partner, but as a systemic rival. Yet how intense is China’s influence? The China Index seeks to measure this influence across different domains. This is a welcome first step, but it is not without far-reaching flaws, write Lars Pelke and Katrin Kinzelbach
In December 2022, the Taiwanese civil society organisation Doublethink Lab published an attempt to quantify the influence of the People’s Republic of China (PRC) across nine dimensions. These are academia, domestic politics, economy, foreign policy, law enforcement, media, military, society, and technology. The quantifications are based on expert assessments and factual information. Regional think tank partners quantified said influence in 82 countries, covering the period March 2021 to March 2022.
The immense effort to construct an index on this topic is laudable and indeed much needed. The China Index may be one of those initiatives that generate political and academic attention via quantification. But the politics of numbers is not without risk, especially when the data quality is debatable.
The China Index project defines PRC influence as 'the power to affect the decisions of other countries so that they align with the cultural, diplomatic, economic, military, political, or social interests of the PRC'. In our view, this definition requires further clarification. We wonder what 'the power to affect the decisions of other countries' means, and how to measure it with indicators.
The China Index project has a normative, pro-democracy orientation, seeking to foster a global network 'connecting academics, democracy movements, digital communities, like-minded CSOs, and experts'. Given this, we recommend that the specific concerns related to the PRC’s autocratic influence be made even clearer. We prefer to use the term 'autocratic influence' instead of 'foreign influence'. It is true that the latter term is not an invention of the China Index team and is by now broadly established. But we caution that any interaction across borders involves some form of 'foreign influence'.
What does the China Index project mean by the power to affect the decisions of other countries, and how should this be measured?
The power to influence should not be equated with the actual use or success of normatively undesirable influence. Indeed, the China Index project distinguishes three areas of concern. The first is the exposure of a country to the PRC. The second is pressure 'overtly applied through these mechanisms of PRC influence'. Third is the observable effect resulting from exposure to PRC influence that comes into play via pressure, though the definition of 'exposure' remains rather too vague. Also, causality is often unclear, For example, the fact that politicians 'have publicly expressed positive views of the PRC government' does not in and of itself indicate autocratic influence.
The China Index relies on different questions and factual data to construct indices on several dimensions, such as academia, or the economy. It provides an overall index indicating China’s autocratic influence. As such, it is important to understand how it aggregates data.
Doublethink Lab uses the sum of all responses and divides this sum by the maximum scores that can be achieved. But why do all questions weigh the same on the aggregate index? The questions ask about different intensities of influence. For example, incidents of censorship carry the same weight as the mere availability of PRC state media. It would be more intuitive to discriminate between questions and related information. Before quantification comes conceptualisation.
While the idea of an overall index is laudable, we have concerns regarding question weighting, comparability, and theoretical justification
Furthermore, we note that the maximum scores vary across countries because 'some indicators can be skipped where data is unavailable'. As a result, despite the Index's claim of comparability, countries' relative positions are not actually comparable.
Last but not least, the indices conflate different ordinal scales: yes/no questions, and more evaluative questions with a four-point scale. They also mix factual and evaluative indicators, which should be treated separately. Overall, from our point of view, the data aggregation lacks theoretical justification and adequate techniques to deal with missing data and different question types and scales.
The China Index relies on country experts who assess a host of questions. These experts provide ratings in line with the options given by the ordinal scales. We wonder whether the pool of coders is adequately qualified, and the protocol sufficiently robust, to ensure comparability. Czech-based Sinopsis, for example, provides ratings for Germany, Poland, and the Czech Republic, across all nine dimensions in the Index. It is unlikely, however, that Sinopsis has sufficient insight into the specifics of, say, the German academic system, to provide reliable assessments.
Even if all involved experts have the necessary expertise, individuals always have biases
Even if all involved experts have the necessary expertise, individuals always have biases. A challenge for any index that rests on expert assessments is, therefore, modelling coders’ differing views and coding behaviour. If subjective biases are not taken into account during this process, the China Index’s data risks producing biased data, even when more than one expert contributes assessments. Expert deliberation, as practiced in the China Index project, can produce a joint score but does not automatically resolve biases. Overall, the process of expert data aggregation – in our view – remains non-transparent, and unintended bias likely arises.
In addition, the interpretation of percentage scores and their implications remains open to interpretation. For example, what do 50% exposure, 60% pressure, and 21% effect regarding the PRC’s influence in Germany actually mean? How are these scores related?
Some scores and rankings do not stand up to a simple check of face validity: the United States ranked first with a score of 37/44 in the academia dimensions, succeeded by Germany with a score of 36/44. The United Kingdom ranked eighth with a score of 23/40. This ranking suggests that PRC influence poses a greater risk to academic freedom in Germany compared with academic freedom in the United Kingdom, Pakistan (6), and Kyrgyzstan (7). Given all we know about academic freedom in these countries, their financial dependency on foreign student fees, and the issue of gifts and related reputation laundering in the academic sector, that is a highly misleading message.
Overall, we remain concerned that the aggregation of different factual information and subjective expert assessments of 'PRC foreign influence' and the process of expert deliberation behind closed doors is prone to unobserved biases, especially when expert qualifications and data quality protocols remain undisclosed.
Some shortcomings of the China Index can be solved relatively easily. These include transparency and data quality protocols. However, others, such as conceptualisation and theoretical embeddedness, would require extensive revision. For now, unfortunately, we still lack a rigorous measure of China’s transnational, autocratic influence.
