The PBRF challenge

It all started at Parliament’s education select committee.

During the committee’s annual review of the TEC, chief executive, Tim Fowler, said that the PBRF had led to a lift in research performance, but that there were now only “marginal” benefits from the system. He suggested that the value of the PBRF had become questionable; marginal returns but very heavy compliance costs, both for the TEC (in managing the system) and for institutions (in compiling the detailed evidence portfolios the system feeds on). Compliance costs that Tim dubbed “back-breaking”.

in the light of Tim’s comments, Newsroom ran two articles looking at the PBRF. If you are reading this, you would almost certainly have seen the Newsroom stories.

The research DVC at Victoria University of Wellington, Margaret Hyland, in a measured and thoughtful comment, talked to Newsroom about the importance of the PBRF in funding university system research, but agreed that compliance costs were very high – for individual academics, as well as for institution management.

Jonathan Boston, one of those involved in the design and development of the PBRF, raised an additional problem – in the PBRF research quality evaluation (QE), each individual academic’s research is assessed and graded. Many academics question the basis of the assessment made in the PBRF and so, are contemptuous of that level of scrutiny and judgement. Jonathan told Newsroom that he had been described as “the most hated academic in the country” for his role in the design of the PBRF. The fact that individuals’ research is assessed is a major point of contention – especially with the TEU. (I have heard of one academic whose unopened PBRF assessment notices were on display in his university office – I am told he has never opened them, his contempt for the system trumping any curiosity).

Tertiary education expert and long-time commentator Dave Guerin endorsed Tim’s comments. The University of Auckland’s Simon Holdaway raised the vexed issue of whether the assessment should be of individual academics, as opposed to research groups. And academics commenting on Newsroom’s stories were adamant – the PBRF has had its day[1].

Margaret Hyland questioned whether the QE “was still answering the right question”. She saw it as “timely to look at it” but cautioned that any review needs to be done carefully ….

Review, review, review, review … and even more review?

In 2004, an evaluation of the implementation of the first PBRF QE round. Three years later, after the second QE, a “strategic review” of the system by an independent, international expert, looking at benefits of the system, the effects on participants and possible improvements[2]. After the third (2012) QE, an evaluative review, led by officials, which led to simplification of the processes and of the policy. And in 2020, yet another comprehensive review resulted in further refinements. That’s one review every four years and nine months. And remember that the QE process works on a six-year cycle. The QE is reviewed more frequently than it is used!

But here’s the interesting thing … each of those reviews has broadly confirmed the overall underlying policy architecture. Including peer review as the basis of the assessment of researchers’ portfolios and including assessing each individual academic’s research – the two factors that drive the very real compliance burden imposed by the PBRF. These reviews have confirmed the architecture, despite the costs. Despite the earnest pleas of the TEU and of the many academics who made submissions. Despite …. Despite everything.

How come? Let’s check out why. Then let’s check out if there are realistic alternatives.

The lift in performance

Most everyone – even those who call for change – appears to accept that the PBRF has led to an improvement in research performance, that it led to changes in institutional culture in universities by increasing the focus on research and by the incentives it creates. The panel that conducted the most recent review concluded that the PBRF had “… increased the overall quality and quantity of research in New Zealand, research is more central to tertiary education, we have much more information about research activity and the system is producing more people with excellent research skills than ever before…”[3]

The 2012/2013 review of the PBRF found clear evidence of improved research performance. Looking at bibliometric data, the review found that, following the introduction of the PBRF, New Zealand higher education’s share of world indexed publications increased (from 0.40% in 2000-2004 to 0.46% in 2007-2011). Even more significant was the increase in the share of world citations – from 0.34% in 2001-2005 to 0.49% in 2007-2011[4].

Bibliometric analysis shows that the citation rate of the research papers produced by NZ universities rose between 2006 and 2021. Papers published by NZ universities between 2006 and 2009 were cited at around 98% of the average for all universities in the world, in the same fields of research. For papers published between 2018 and 2021, that had risen to 102% of the world average[5].

And, despite the fears of some critics of the PBRF, research collaboration has increased over the life of the PBRF[6]. Of papers published by NZ universities in 2006-2009, around 66% involved collaboration with a researcher at another university[7]. But in the period between 2018 and 2021, that had risen to over 80%. Each of the seven universities represented in Leiden University’s CWTS database saw its collaborations increase over that time.

See the appendix for the data.

… but there are also problems …

But if it is clear that research performance has increased, then it is equally clear that there are problems. For one, there is the problem, noted by Tim Fowler and recognised by nearly everyone – of compliance burden[8].

Plus, the panels set up to undertake the peer assessment of academics’ research are thought by many to “… devalue certain types of research”[9].

And the PBRF has the effect of incentivising academics to devote more of their energy to research than to other roles, especially teaching[10].

During the second round, one university was guilty of grossly gaming the system, but gaming has either reduced or become less visible[11] while there was some “excessive striving” in the early rounds (I am told that, as the staff at one university worked on their portfolios for the 2006 QE, the DVC Research’s guidance on the Contribution to the research environment section was something like: “Put in everything. Put in so much that it makes you feel sick. Then double it”. Is it any wonder that academics were disaffected!)

… so why has the PBRF been so powerful in changing higher education culture?

The usual answer is “follow the money”; funding always creates incentives for institutions. But with the PBRF, institutional prestige also plays a part.

But that’s not all that’s at play here. In fact, the funding passes directly to the institution’s management for allocation according to institutional priorities; the individual academic won’t directly see the money. I suspect that the really powerful driver of the intense focus that some academics now place on research is reputation. The prestige and satisfaction that comes from being seen as excellent. Plus, the opportunities that an A or B grade in the PBRF creates for career progress.

Research needs to be at the heart of our higher education system ….

In the University of Berlin, in 1810, Wilhelm von Humboldt created an ideal of higher education that wove research and inquiry with learning and knowledge. That led to a higher education model that saw the emergence of academic freedom and autonomy for institutions, a contrast to the previous rigidly controlled curricula that drew on dogma. This ideal spread throughout much of Europe and to the United States. It created a learning culture that questioned received wisdom and pushed the boundaries of knowledge and thought. It underpinned the flowering of scientific and scholarly innovation that has shaped much of our scientific and cultural development since the early 20^th century[12].

The Humboldt ideal provides the rationale for the statutory protection of institutional autonomy and academic freedom[13] and for the requirement that university “… research and teaching are closely interdependent and most of their teaching is done by people who are active in advancing knowledge[14]”.

We need high-quality research in our higher education institutions because:

Research improves teaching and learning. The interplay between teaching and research, between learning and enquiry, enriches teaching and learning, especially (but not only) at higher levels. Engaging with the research helps ensure learning is current and up to date. And more exciting. Using enquiry as a teaching method and an assessment approach stretches learners and helps build transferable skills (as well as disciplinary skills).
Research degrees build human capital. People with research-based degrees have highly developed critical and enquiry skills, as well as the capacity to organise and manage large projects. Analysis shows that research degree graduates help boost innovation in the workforce; employees with research degrees help firms build absorptive capacity[15].
Higher education research contributes to our culture, society and economy. Research from higher education institutions contributes to social development, community welfare, public health, and to cultural development[16]. Universities capture a high proportion of government public good contestable research funds. In the 2023 MBIE Endeavour Fund round, 68% of the funded applications were from universities[17] while 84% of the funding applications in the latest Health Research Council (HRC) funding round were to university-based researchers[18]. Some higher education research is undertaken at the request of/under contract to firms to improve their products and services. Government ministries, NGOs, and philanthropists fund research that helps them understand social dynamics.

A good research funding system will reflect, honour and encourage those three values.

The last – the impact or contribution to society of research – is especially important. It’s hard to justify public funding for research in higher education unless there is evidence, visibility of public benefit[19]. More of that below ….

Designing a research funding system

Let’s suppose we were asked to design a new research funding system for Aotearoa New Zealand’s higher education system. A system to replace the PBRF. Let’s suppose we could start with a clean sheet. What are the questions we need to ask. What are the elements we need to include. What are the challenges and trade-offs.

But first ….

… there needs to be research funding …

If research is a requirement of institutions and if higher education research contributes to public, cultural, social and economic good, then it should be funded. And I mean that there needs to be untagged, bulk funding for research, funding that the institution can deploy as it sees fit, how it sees fit, to foster research, across the institution, across its complement of staff, in whatever disciplines it sees fit. Blues skies research. Applied research – whatever it sees fit.

Of course, institutions will seek revenue for research elsewhere – from public funding bodies like the HRC and MBIE’s research funds, from business …. But that’s usually funding for a specific designed project, it’s more akin to purchase than to funding. What I am talking about here is the need for funding that reflects and reinforces institutional autonomy and academic freedom. In alignment with the Humboldt principles. Funding that is spent according to the institution’s strategy.

… and I mean designated research funding …

Between the disestablishment of the old University Grants Committee in 1989[20] and the launch of the PBRF in 2003, all core research funding for New Zealand universities was embedded within funds that were driven by enrolments of students. Therefore institutions that invested little in research were advantaged relative to institutions that devoted a large proportion of their spending to research. That became especially important as polytechnics moved into degree-teaching, competing head on with universities that were relatively research-intensive.

The PBRF was the first substantial allocation of core funding specifically tagged for research. Whatever happens to the PBRF, whatever any future review comes up with, the system needs an allocation of designated research funding, driven by the scale and costs of research, that allows institutions to spend that funding as they judge fit on fostering research and researchers.

We should never again allow a system where research funding is embedded in funding for teaching and learning. The drivers are different. The result is distortion and the slow, progressive but inevitable erosion of research effort.

The challenge, the trade-offs

The first and most important challenge is: How do we apportion that funding to institutions, in a way that acknowledges the scale of each institution’s research activity and that is “fair” and justifiable. That recognises costs, that creates the appropriate incentives, that is complex and fine enough to treat institutions and fields of research in an even-handed way, but simple enough to avoid an excessive compliance burden.

A research funding system must reflect scale …

In any funding system, there needs to be a scale factor – an institution with a greater number of researchers should receive more funding (all other things being equal). This is so obvious, it’s hardly necessary to say.

… and performance …

In theory, it would be possible to provide research funding as a capitation grant, based on the number of research active academics, a universal funding rate, either a flat rate or rates differentiated by field of research or by type of research[21]. That’s as simple as it gets.

But it’s hard to see that as fair to institutions that invest heavily in pursuing higher research performance. Plus, we need to build on the improvements we have seen in the last 20 years. Because we need to provide an environment that encourages our best researchers to remain here and that acts as a draw for researchers from other countries’ systems. And because, in a system that doesn’t incentivise improvement, institutions that coast along will be better off than institutions that strive for the highest performance (and, in doing so, incur greater costs and hence, need to crib resourcing from other activities).

… which requires complicated measurement …

However, if we use performance to shape funding, then we need to measure performance. Here we get to the complicated bit. Complex activities rarely have widely accepted, simple unambiguous measures of performance. Research is no exception. We usually know poor research when we see it and we can often readily identify outstanding research. Pretty much all the research we want to assess fits between those extremes. Meaning we need first to identify what outcomes we want to assess against and then we need to find or construct measures that enable us to work out whether instance X is “better or worse” than instance Y.

Now, measures are broadly of three types …

metrics that precisely quantify or measure what it is we are assessing
where there are no clear metrics, we rely on assessment by independent experts who are qualified to make judgements about the relative merits of X and Y
proxy measures where we choose something that is measurable and that gives us an approximation to a measure of quality, something “close enough” that guides our judgement as to whether X is better than Y.

There are very few, if any, precise metrics that fully characterise the quality of research. So the challenge we face is that we have to use expert judgement and/or proxies. Expert judgement is time-consuming and costly. And independent experts, however independent, however expert, may have blind-spots. And proxies may be low cost, but, by definition, they aren’t exact.

So this is the really key trade-off – the simplicity (but performance risks) of capitation versus the incentives and the fairness (but the complexity) of a performance dimension.

For now, let’s stick with a performance-based system.

Research funding should recognise …

… the three values of higher education research …

The three important features we discussed above are:

research that builds human capital
research that contributes to our society – to the culture, to community development, to the economy
research that informs and shapes learning across the institution.

All three are important. We need a funding system that reflects and rewards them all. Which means making decisions on how to assess each. And then, we need to make a judgement on how to weight them.

… and the breadth of types of research and knowledge creation

In the PBRF. research is defined broadly and that has become broader still following the 2020 review. It’s not only about scientific papers that appear in research journals. Or books that explore knotty problems in depth. It includes the range of fields that are offered across our higher education system, applied disciplines (such as agriculture and engineering), vocational fields (such as accounting, law, nursing, acting, fine arts, musical performance and creative writing), as well as natural sciences, health sciences, the humanities and social sciences. The ways in which knowledge creation is evidenced differs between fields.

We need a funding system that can respect that breadth and hence, an assessment system that can cope with the range of ways that research is evidenced.

And then there is the perennial problem of assessing applied research. How will the system deal with rating a leading paper in an international journal (say Nature Neuroscience) which might draw a hundred citations in its first year against a clinical research paper in published in the journal Practical Neurology that is rarely cited but that is widely read by clinicians )who then incorporate the findings into their practice)? What about a paper in family law that appears in a NZ journal read by family court judges and that helps judges make sound decisions; how would that compare with a theoretical paper in jurisprudence that is published in the Columbia Law Journal?

Linda Tuhiwai Smith and her PBRF review panel in 2020 addressed that problem by proposing an expansion of the (already broad) definition of what qualifies as research under the PBRF[22].

It’s easy to see why we ended up where we are

When you look at it like that, it’s easy to see why we ended up with the PBRF in the shape we now have.

For human capital, the PBRF has a reasonable proxy measure – research degree completions (RDCs).

For contribution to society, the proxy is external research income (ERI). In theory, that is a reasonable kind of proxy; funding bodies, firms, agencies, NGOs grant funding that directs institutions to produce research outputs that are of service to their business, to communities, to society. Public good funds – like those administered by MBIE, the Royal Society and the HRC – make awards on the basis of recommendations of panels of experts who have assessed the quality of the research bids. ERI is a fair (and efficient) proxy for social and economic impact[23] – it’s just that someone else has made the assessment, not the TEC, not the institution.

For overall research quality, the PBRF has the QE, where the portfolios of individual researchers will still be assessed by panels of experts. Where compliance costs will remain.

But the PBRF relies on the QE, which is complex and back-breaking, for government and institutions. And the QE is disliked by many academics, intensely by some.

Is there a realistic alternative?

Dave Guerin told his subscribers:

A simpler system should be found to maintain the positive changes that the PBRF created

A simpler system that addresses the three outcomes of research in higher education, that meets the principles laid out above – that sounds like a challenge to me … What might that look like? Watch out for Part 2 of this series!

Endnotes

[1] It’s worth noting that one of the academics who commented, Professor Peter Davis, a distinguished researcher, criticised not the PBRF system but the way that universities allocate the money they are awarded under the PBRF – an entirely separate question.

[2] See Adams (2008)

[3] Smith et al (2020) Toward the Tertiary Research Excellence Evaluation (TREE): the report of the PBRF Review Panel

[4] Smart (2013) Analysis of the impact of the PBRF – interim findings

[5] This analysis is of data from Leiden University’s CWTS bibliometric database. See the appendix for the full data.

[6] Smith et al (2020) noted that many people believe that the PBRF reduces research collaborations. It is conceivable (if unlikely) that that is true of creative work and artistic performances, possibly even of some types of applied research. However, it is entirely incorrect in relation to the kind of research recorded in bibliometric databases.

[7] See the CWTS bibliometric data.

[8] This matter was explicitly dealt with in the first (2004) evaluation of the PBRF, conducted by WEB Research. The report noted the very high burden of the QE for institutions but pointed out that the QE was on a six-year cycle; if the cost were to be accrued over the six-year interval, it appeared less anomalous.

[9] Smith et al, op cit

[10] ibid

[11] Even if one university in the 2018 round managed to have more funded research profiles than it had staff! See this analysis.

[12] See for instance, Marginson (2024) Higher education faces a more fragile and contested future

[13] See Education and Training Act 2020, sections 252, and 266-268

[14] Education and Training Act 2020, s 268

[15] Arara et al (23023) The effect of public science on corporate R&D

[16] See Finegold (1999) and Drabenstott (2008) for just two examples

[17] See this page

[18] See this page

[19] See Mark Reed’s essay Why research evaluation needs to include real-world impact in Elsevier (2024) Back to earth: landing real world impact in research evaluation

[20] Under the old UGC quinquennial system and under the University of New Zealand that redated the UGC, most research-related funds were allocated by a competitive bid process, meaning that student-led funding likely accounted for most research spending by universities. See this note for an account of the history.

[21] See Reed, op cit.

[22] Note, however, that sceptics may question whether peer assessment panels, facing thousands of pages of research portfolios, might (possibly subconsciously) apply conventional values (such as journal ranking, journal prestige, citation counts).

[23] The 2020 review report (Smith et al (2020) – see page 11) described ERI as an input, missing the point that ERI is an index of the quality of an institution’s research impact. The group was, however, correct in noting that their proposals for changes to the QE would have increased the existing overlap between the QE and ERI. The government rejected the group’s proposal to eliminate ERI altogether and instead, altered the weightings., devaluing the impact of the public good and social impact research contract work. The weighting changes had the effect of devaluing public good contract research (such as epidemiological research funded by the HRC) and policy-related social science research conducted on behalf of government agencies.

Bibliography

References

Adams J (2008) Strategic review of the Performance-Based Research FundEvidence Ltd

Arara A, Belenzon S, Cioaca L, Sheer L, Zhang H (2023) The effect of public science on corporate R&DWorking Paper 31899, National Bureau of Economic Research

Arbo P and Benneworth P (2007) Understanding the regional contribution of higher education institutions: a literature review OECD Education Working Papers, No. 9, OECD Publishing.

Drabenstott M (2008) Universities, innovation and regional development: a view from the United States Higher Education Management and Policy Volume 20, No 2, OECD Publishing

Finegold D (1999) Creating self-sustaining high skill ecosystems Oxford Review of Economic Policy 15(1):60-81

Marginson S (2024) Higher education faces a more fragile and contested future University World News

Reed M (2023) Why research evaluation needs to include real-world impact in: Elsevier (2023) Back to earth: landing real world impact in research evaluation Elsevier

Smart W (2013) Analysis of the impact of the PBRF – interim findings Ministry of Education

Smith L, Larner W, Phipps D, Town I, Underhill-Sem Y, Williams M and Mischewski B (2020) Toward the Tertiary Research Excellence Evaluation (TREE): the report of the PBRF Review Panel Ministry of Education

Smyth R (2019) A quick look at the 2018 PBRF results Strategy, Policy, Analysis, Tertiary Education