Benchmarking critical technologies

Building an evidence base for an informed critical technologies strategy

Kitsch Liao, Dr Samantha Hoffman and Karly Winkler, with Baani Grewal, Cheryl Yu, Saki Kikuchi, Tilla Hoja, Matthew Page and Jackson Schultz.

What’s the problem?

Technology policy formulation has recently gained a renewed importance for governments in the era of strategic competition, but contextual understanding and expertise in deciding where to focus efforts are lacking. As a result, decision-makers might not understand their own national strengths and weaknesses. It’s difficult to judge whether a country’s R&D outputs, no matter how advanced, and its development of production capacity, no matter how significant, align with the country’s intended strategic objectives or can be used effectively to achieve them.

The ability to measure the relative strengths and weaknesses of a country by weighing specific strategic objectives against technical achievements is of paramount importance for countries.

This is especially true as nations seek to resolve supply-chain resilience problems underscored by the Covid-19 pandemic. China’s rejection of the Quad’s vision of a free and open Indo-Pacific, willingness to use economic coercion and the resulting strategic competition, call further attention to multiple technology sectors’ heavy reliance on a single source. A solution must be found that can exploit synergy across multiple technology sectors among collaborating countries while ensuring supply-chain resilience.

What’s the solution?

Governments’ ability to ensure that strategic objectives pertaining to critical technologies are both well articulated and achievable, and researchers’ and industry’s ability to collaborate in meeting those objectives, would be greatly enabled by the development of an objective and repeatable methodology for measuring technical achievements against clearly defined strategic goals for the critical technology sector. The most pressing challenge should be a relatively straightforward one to resolve: standardise metadata about national objectives and R&D efforts to enable business analysis.

The Quad Critical and Emerging Technology Working Group is an important step towards building collaboration in the research, development and production of critical technologies among like-minded governments. While in nascent stages, the group is gathering momentum and working towards addressing the September 2021 objective to monitor trends in critical and emerging technologies for cooperation, with an initial focus on biotechnology. We recommend as follows:

• Conduct detailed analysis to understand current and emerging gaps in critical and emerging technologies, starting with biotechnology, among like-minded countries.
• Develop a partnership between like-minded countries with advanced technological capabilities to deliver a secure technology supply chain for critical tech. This should include a commitment to a set of core principles for technology development and delivery, including ‘baking in’ democratic principles to the technology and agreeing to share any civilian advances on market terms and refrain from coercion.
• Establish a Quad or Quad Plus critical technologies fund to which participating states pledge investment funds that are then disbursed to address current and emerging critical technologies gaps.

Introduction to the Benchmarking Critical Technologies Project

Benchmarking Critical Technologies is a pilot project at ASPI ICPC that examines the development of a handful of critical technologies in the context of strategic partnership and strategic competition.

‘Critical technologies’ broadly refers to strategically important technology areas.¹ Australia, for example, defines ‘critical technology’ as ‘technology that can significantly enhance or pose risks to Australia’s national interests, including our prosperity, social cohesion and national security’.² For this pilot study, we focus on the biotechnology and energy technology sectors in China and in the Quad— the quadrilateral Indo-Pacific diplomatic network consisting of Australia, India, Japan and the US.

This project will be expanded over the course of 2022 to include more technology areas and countries.

During the Quad Leaders’ Summit in March 2021, the Quad Critical and Emerging Technology Working Group was announced. The communiqué from the summit said that the working group was intended to ‘ensure the way in which technology is designed, developed, governed and used is shaped by the Quad countries’ shared values and respect for universal human rights’.³ The communiqué didn’t directly name China, but China was clearly implied in its pledge to recommit to ‘promoting the free, open, rules-based order, rooted in international law and undaunted by coercion, to bolster security and prosperity in the Indo-Pacific and beyond’.⁴

It’s clear that China is the key strategic competitor that the Quad countries are hedging against. They’re technology and manufacturing powerhouses with strong geopolitical influence in the region, which makes the competition both more important and more difficult. As the Quad works to develop capabilities in a range of critical sectors, the Quad members will need to also understand how to leverage each other’s strengths and overcome collective weaknesses to guarantee supply-chain resilience, among other strategic objectives.⁵ They will also need to triangulate the effects of each nation’s digital enmeshment in Chinese supply chains and the net effects of that in particular sectors.

There’s a lack of empirical data to ground decision-makers’ advice on everything from capability gaps to priority investment areas. This project is an attempt to begin to bring additional empirical data to the decision-making process. Our intent is to offer improved clarity on each country’s strengths and weaknesses in each critical technology. After consultation with the Australian Government, we decided to focus on hydrogen energy and solar photovoltaic (solar PV) technologies from the energy sector, and genetic engineering and vaccines and medical countermeasures in the biotechnology sector.

The broader technology areas that these specific technologies sit within are of clear strategic importance. The Quad Leaders’ Summit communiqué established that biotechnology would be the starting point for the Critical and Emerging Technology Working Group’s collaboration. It also highlighted, in the context of the recent COP26 conference, that the Quad would coordinate to ‘establish responsible and resilient clean-energy supply chains’.⁶

To assess national capabilities, we measured each country’s R&D and infrastructure development efforts using patent and patent impact data and academic impact data, and compared those results against the country’s technology-specific policy goals. For patents, we collected two measures for each critical technology: the quantity and quality of the patents. IP Australia provided ASPI ICPC with patent data to analyse the quantity of patents for each critical technology. Additionally, using the commercial product PatentSight developed by LexisNexis, we assessed patent quality with the Patent Asset Index (PAI).⁷ The tool assesses patent quality across various measures in the overall ecosystem of a technology field. Those measures are technology relevance (TR), indicating how much future patents in the field depended upon the patent; market coverage (MC), indicating how much of the global market the patent offers protection of; and competitive impact (CI), the aggregate of TR and MC indicating the economic value of the patent. The aggregate economic value of all patents in the field then constitutes the field’s PAI. For academic impact factors, we used the CiteScore (CS) methodology for measuring impact factors embedded within Elsevier’s Scopus commercial database product.

We also drew on background interviews with industry specialists and senior officials in relevant government departments. Budget data was more challenging to collect, normalise and assess.

Consequently, it isn’t treated as a separate metric, but included with general policy analysis. (For more on our methodology, see the Appendix.)

We recognise that both the policies and technologies on which we base our assessments are evolving. Technology development doesn’t always move in a linear trajectory, and current capabilities aren’t the only indicator of future outcomes. Moreover, the strategic interests and desired policy outcomes one country seeks might not align simply or easily with those of another. Therefore, it isn’t possible to directly compare countries against each other. Rather than arbitrarily rating each country’s progress against the others, we’ve rated each country’s progress in achieving the strategic objectives that it has outlined for each technology area (Figure 1). The progress indicator’s location should be interpreted as being dynamic, given that both policies and technologies will evolve.

Figure 1: Rating scale—country progress in meeting national policy objectives

Rating scale legend

1. Some high-level policy objectives specific to the technology area have been set, but there’s little evidence of efforts making progress towards meeting those objectives.
2. Despite the articulation of some policy objectives pertaining to the technology area, those are still relatively unclear. The country’s R&D and production capabilities don’t appear to be sufficient to contribute to realising the country’s stated policy objectives.
3. There’s some evidence that the country is developing actionable policy in the technology area. There’s clear progress in the country’s ability to contribute to the R&D of the technology, or production capacity. It isn’t clear, however, whether this progress aligns with the country’s stated policy objectives.
4. There’s evidence that stated policy objectives, research and investment are beginning to translate into aligning capabilities.
5. There’s strong evidence that stated policy objectives, research and investment have already translated into aligning capabilities.

Source: Image produced by ASPI.

Overall assessment

• Quantity doesn’t mean quality, at least in terms of the way patents and research shift global knowledge and capabilities in the overall ecosystem of a technology field. Our findings on patent impact—measured by how often a patent is cited or purchased—highlighted that China, with the highest number of patent applications filed, didn’t have a correspondingly high impact factor.
  Australia and India, and to a lesser extent Japan, filed far fewer patents, but those few patents had impact more on par with US patents, which were high in both number and impact. One patent can significantly influence the evolution of a technology; others might incrementally advance knowledge or create offshoot fields. Impact factors in these types of analysis can be an objective measure for determining scientific advances or commercial success but aren’t necessarily useful in indicating whether national capabilities support policy objectives. If the point of benchmarking critical technologies capabilities at a national level is to understand what makes a country capable of meeting national policy objectives, competitive in a strategic competition and well placed to work with like-minded partners, then the ability of individual researchers or organisations to advance a technology field doesn’t tell us how competitive a country is in translating concepts to capabilities that align with its strategic objectives. For example, ASPI ICPC believes that in China, the disproportionately large number of patents filed internally is most likely attributable to companies patenting specific applications of technology. In the Quad, countries such as Australia and India have been more impactful for a fewer number of patent applications filed and research papers significantly advance the field.
• Success in connecting policy objectives to outcomes isn’t yet entirely measurable. Our comparison of national policies pertaining to each critical technology we research shows that China, followed by the US, tends to have more clarity about what it seeks to achieve by investing in R&D and production capabilities, and following that up with actions that will achieve those objectives. India, Japan and Australia don’t lack policy development or innovative capacity, but we believe they have been less effective at connecting concepts to capability. This assessment is no doubt at least partially because the development of policy objectives postdates most of our data.
• Metrics don’t explain the context in which innovation is taking place, including incentive structures, and how that affects a country’s ability to meet specific objectives. In China, the incentive structure is designed so that researchers are working to meet specific policy objectives. In fact, companies closely collaborate with the state in technical standards development. According to the revised 2017 Standardisation Law,⁸ the Standardisation Administration of China (an agency under the State Administration for Market Regulation) is required to oversee standards initiation and implementation, and in practice technical committees for standards setting under the Standardisation Administration tend to consist of both companies and research institutes.
  We believe the knock-on effect of the incentive structure in China is that the R&D base is disadvantaged, while companies and researchers focus on implementing specific applications of technology that meet policy needs. China’s National Patent Development Strategy (2011–2020) was designed as a ‘long-term and comprehensive plan to use the patent system and patent resources to enhance the country’s core competitiveness’.⁹ The strategy document prioritises ‘encourag[ing] and supporting[ing] enterprises to upgrade the core technologies and key technologies with patent rights in China’s advantageous fields to national and international standards’.¹⁰ We believe companies are seeking to achieve those objectives by owning the market first, and patents support that approach. They’re adding economic value by increasing the quantity of applications, and owning the market comes before efforts to refine the product. Many PRC-originated technologies are being exported globally (see ASPI ICPC’s Mapping China’s Tech Giants project), no matter what the overall quality of the product in comparison to competitors, and that proliferation is probably achieving some market power and incumbency. It’s a cumulative and individual challenge for the Quad nations to move more rapidly from concept to capability in order to avoid the PRC leading in meeting strategic objectives with that technology

Download

Readers are warmly encouraged to download the full report to access the detailed sector by sector analysis.

The Patent Search Strategy used in the formulation of the report is available for review here.

---

Acknowledgements

We acknowledge the assistance we have received from IP Australia, the Department of the Prime Minister and Cabinet, the Department of Industry, Science, Energy and Resources, and numerous interviewees and peer reviewers in policy and industry roles across the Quad.

Thank you to ASPI ICPC researcher Albert Zhang for assistance. We are grateful for the valuable comments and assistance provided by Fergus Hanson, Michael Shoebridge and Jocelinn Kang.

This project was supported through a $150,000 grant from the Department of the Prime Minister and Cabinet.

What is ASPI?

The Australian Strategic Policy Institute was formed in 2001 as an independent, non‑partisan think tank. Its core aim is to provide the Australian Government with fresh ideas on Australia’s defence, security and strategic policy choices. ASPI is responsible for informing the public on a range of strategic issues, generating new thinking for government and harnessing strategic thinking internationally.

ASPI’s sources of funding are identified in our annual report, online at www.aspi.org.au and in the acknowledgements section of individual publications. ASPI remains independent in the content of the research and in all editorial judgements.

ASPI International Cyber Policy Centre

ASPI’s International Cyber Policy Centre (ICPC) is a leading voice in global debates on cyber, emerging and critical technologies, issues related to information and foreign interference and focuses on the impact these issues have on broader strategic policy. The centre has a growing mixture of expertise and skills with teams of researchers who concentrate on policy, technical analysis, information operations and disinformation, critical and emerging technologies, cyber capacity building, satellite analysis, surveillance and China-related issues.

The ICPC informs public debate in the Indo-Pacific region and supports public policy development by producing original, empirical, data-driven research. The ICPC enriches regional debates by collaborating with research institutes from around the world and by bringing leading global experts to Australia, including through fellowships. To develop capability in Australia and across the Indo-Pacific region, the ICPC has a capacity building team that conducts workshops, training programs and large-scale exercises for the public and private sectors.

We would like to thank all of those who support and contribute to the ICPC with their time, intellect and passion for the topics we work on. If you would like to support the work of the centre please contact: icpc@aspi.org.au

Important disclaimer

This publication is designed to provide accurate and authoritative information in relation to the subject matter covered. It is provided with the understanding that the publisher is not engaged in rendering any form of professional or other advice or services. No person should rely on the contents of this publication without first obtaining advice from a qualified professional. 

© The Australian Strategic Policy Institute Limited 2021

This publication is subject to copyright. Except as permitted under the Copyright Act 1968, no part of it may in any form or by any means (electronic, mechanical, microcopying, photocopying, recording or otherwise) be reproduced, stored in a retrieval system or transmitted without prior written permission. Enquiries should be addressed to the publishers. Notwithstanding the above, educational institutions (including schools, independent colleges, universities and TAFEs) are granted permission to make copies of copyrighted works strictly for educational purposes without explicit permission from ASPI and free of charge.

First published November 2021. ISSN 2209-9689 (online). ISSN 2209-9670 (print).
Cover image: Leslie Sharpe.

Funding Statement: Funding support for this publication was provided by the Department of the Prime Minister and Cabinet.

1. ‘What is a “critical technology”?’, in ‘Appendix A: National Critical Technologies List’, Clinton White House archives, online. ↩︎
2. Marise Payne, ‘Launch of the International Cyber and Critical Technology Engagement Strategy’, speech, 21 April 2021, online. ↩︎
3. Scott Morrison, ‘Quad Leaders’ Summit communique’, 24 September 2021, online. ↩︎
4. Morrison, ‘Quad Leaders’ Summit communique’. ↩︎
5. See, for instance, the Biden administration’s memorandum, ‘Multi-agency research and development priorities for the FY 2023 Budget’, August 2021, online; and the former Trump administration’s October 2020 list of critical and emerging technologies, online. ↩︎
6. Morrison, ‘Quad Leaders’ Summit communique’. ↩︎
7. For a more comprehensive definition and explanation of the methodology behind the Patent Asset Index, consult the product website, online. ↩︎
8. Standardisation Law of the People’s Republic of China, Standardisation Administration of China, 23 March 2018, online. The Standardisation Law was revised and adopted at a meeting of the Standing Committee of the National People’s Congress in November 2017 and came into force on 1 January 2018. ↩︎
9. ‘National Patent Development Strategy (2011–2020)’ [‘全国专利事业发展战略(2011—2020年)’], China National Intellectual Property Administration, 18 November 2010, online. ↩︎
10. ‘National Patent Development Strategy (2011–2020)’. ↩︎