A submission in response to Number 4 Part 2 of the University Advisory Group’s questions
| Executive summary Research makes three important contributions to the university and thus to society: – research underpins high-quality university teaching – research degrees build human capital university – research contributes to our culture, society and economy – to social development, to community welfare, to business development and innovation, to public health, to cultural development. The UAG should recommend that the government retain the existing statutory expectation that universities should be institutions that conduct teaching and research, that they weave together their teaching and their research and that they meet international standards in each. Analysis shows that the New Zealand universities currently do meet (but don’t exceed) international standards in research. One of the main reasons for the relatively high performance in research is that there are two research-focused funding streams that incentivise excellent research. The CoREs funding mechanism has built on the research strengths of the system, built critical mass of expertise in areas of research strength and delivered good results. The UAG should recommend to government that the CoREs mechanism should be retained in its current form. The PBRF funding mechanism is especially important as it is untagged, meaning that the funding can be used throughout the institution, in line with the university’s strategy, to foster research, to lift research capability and to reward research excellence. While the PBRF has lifted research performance and effected positive change to the culture of the universities, there is momentum (and persuasive arguments) to modify the mechanism: – the peer review that underpins the PBRF quality evaluation (QE) carries very high costs for institutions and for government, is intrusive and is a source of division and resentment in the university community – it is likely that future rounds of the QE will lead to only minor shifts in the shares of the research funding pool, meaning that there will be diminishing returns from a very high-cost process – there are doubts as to whether the peer review process necessarily delivers its results in the way intended – ie, as an independent, expert, first-principles assessment of research quality. There are alternatives to the PBRF QE (outlined in the discussion below) that would deliver a similar distribution of funds with a much lower transaction (and financial) cost. The UAG should recommend to the government that it investigate alternatives to the PBRF QE, seeking a mechanism that would: – allocate research funds to universities, fairly, simply and transparently – recognise the scale of each institution’s research and reward and encourage excellent performance – recognise performance across the three primary values of university research listed above. |
The context – universities active in teaching and research
The culture of our university system is derived from the ideals pioneered by Wilhelm von Humboldt in the establishment of the University of Berlin in 1810[1]. Von Humboldt developed an ideal of higher education that wove research and inquiry with learning and knowledge, leading to a learning culture that questioned received wisdom and pushed the boundaries of knowledge and thought. That model led to the now commonly understood and accepted principles of academic freedom and university autonomy. It has become the dominant model for universities in most countries in Europe, in the United States and in much of the developed world.
The von Humboldt model underpins the Education and Training Act which requires that universities’ “…research and teaching are closely interdependent and most of their teaching is done by people who are active in advancing knowledge [and] they meet international standards of research and teaching”[2].
As a result, all eight of our universities are institutions that conduct research alongside their teaching.
The statutory requirement for the interdependence of each university’s teaching and research and the expectation that “most” of university teaching is done by people who are active in research derives from the commonly held view that research-active academics make better teachers (a view, incidentally, that is challenged by research[3]).
The word most … the legislation allows universities to appoint academics – a minority – who are scholars (but not researchers) on the basis of their excellence as teachers, just as they now appoint people to roles that are research-only and have no teaching obligations at all.
As to whether our universities meet international standards, … of the seven NZ universities whose bibliometric data is included in Leiden University’s annual analysis[4]:
- Two (UA and UW) have citation rates that exceed the international average
- The citation rate for the seven universities, as a whole, is 2% above the international mean[5]
- All seven have citation rates of more than 90% of the world average
- Each of the seven has an above average citation rate in at least one broad field of research, while one university (UA) is above the average in all five of the broad fields reported by Leiden and one (UW) is above the average in four of the five.
An alternative bibliometric indicator of the international standing of NZ universities’ research is to look at the proportion of indexed papers that are among the 50% most cited papers in their field. That indicator gives a view of the influence that a research paper has on peer researchers’ work. In fact 49.5% of NZ university papers appear among the top 50% most cited, a small fraction less than one might expect, all else being equal[6].
So the system does meet (if not exceed) international standards in research. Whether the universities meet international standards in their teaching is, obviously, impossible to gauge.
Whether the standard of teaching and research meets international norms is obviously a vital matter for each university; each must strive to hire as many excellent teachers and as many outstanding researchers as they can afford, and each needs to support its staff to develop and improve in their research and their teaching.
The universities need to ensure they comply with the requirement that most teaching is done by people who are active in research, but they need to be flexible, appointing research-only staff, as well as scholars who are excellent teachers.
On balance, there is no argument for a shift from the current statutory expectations that:
- our universities will be institutions focused on teaching and research
- most of the teaching is done by research-active academics
- universities should meet international standards.
Nor is there any suggestion that the system is failing to comply with those requirements. However, those statutory requirements set a floor, a base. The objective of this review is to examine how best to improve, how to exceed that base to the greatest extent possible, given the inevitable constraints of resources, remoteness, lack of population density and small scale that we face.
What university research contributes to New Zealand
To understand the importance and the value of research in a university, to assess effectiveness of the systems for improving the quality of university research, we need to measure the quality of the research against the role and function of university research and to ask what it delivers to New Zealand. Broadly speaking, the value of university research is:
- Research that informs teaching and learning. The statutory requirement for the interdependence of research and teaching means that there is an opportunity for research to enrich students’ experience and hence, their learning, especially (but not only) at higher levels. Using enquiry as a teaching method and an assessment approach can stretch learners and helps build transferable skills (as well as disciplinary skills). Analysis suggests that when university teaching[7] is conducted by those active in research, the teaching is more likely to be at the frontier of knowledge in the field.
- Research degrees build human capital. People with research-based degrees have highly developed critical and enquiry skills, as well as the capacity to organise and manage large projects. Research shows that research degree graduates help boost innovation in the workforce; employees with research degrees help firms build absorptive capacity[8].
- Higher education research contributes to our culture, society and economy. Research from higher education institutions contributes to social development, national economic progress, community welfare, public health, and to cultural development[9]. Universities are a primary mechanism for transmission (and critique) of our culture; the research of universities is an essential component of that role.
Funding research
If research is a requirement of institutions and if higher education research contributes to public, cultural, social and economic good, then research needs to be funded. By that, I mean untagged, bulk funding for research, funding that the institution can use as it sees fit, how it sees fit, to foster research, across the institution, across its complement of staff, in whatever disciplines it sees fit. Blue skies research, applied research – whatever the university decides to prioritise. And, in particular, building the research capability and experience of early-career academics.
Universities also bid for research contracts and can earn research revenue from that source. But that is research purchase, not funding. The contract will come with specific deliverables and outputs that are determined by the purchaser of the research, the contractee. That research purchase is an important source of research revenue, derived from the expertise and capital base that the university has built, is obviously important, but it’s a supplement to the core research funding. The statutory requirement for research places an obligation on the Crown to provide funding.
The drivers of cost in research are different from the drivers of cost in teaching. When the Learning for Life reforms were launched in 1991, research funding was embedded in tuition funding[10]. That meant that the most effective way for a university to increase its research funding was to recruit more students (which would lead to a greater teaching load, risking displacing research work). So, the research funding structure created an incentive to be excellent in activities like recruitment and teaching, but not in research. Therefore, in the competition for students, institutions that invested little in research were advantaged relative to institutions that devoted a large proportion of their spending to research. That became especially important as polytechnics, wānanga and private providers built the scale and reputation of their degree-teaching, competing head on with universities that were relatively more research-intensive and whose research was costly.
So the 2003 decision to introduce designated funding streams for research, following the recommendation of the Tertiary Education Advisory Commission, was a good one.
How has that played out?
It is well-known that the introduction of the Performance-based Research Fund (PBRF) and the Centres of Research Excellence (CoREs) was associated with a significant lift in the research performance of New Zealand universities[11]. New Zealand universities’ share of indexed research publications and of research citations – two proxy measures of performance – both show a measurable lift around the time of the introduction of these two new funds[12].
The PBRF was especially important. The size of the PBRF as a share of total government funding obliged university leaders to reorient their personnel practices in favour of research and to improve the way research was organised institutionally.
But the importance of the PBRF in relation to the CoREs was more than just a matter of scale. The PBRF was also the more important because the PBRF generated funding was entirely undesignated, whereas the CoREs funding was tagged to particular research groups focused on particular research fields. And the PBRF was a fund that was driven by the performance of every research-active academic – individuals motivated, in many cases by the prestige (and the career-enhancement) that a high PBRF grade confers[13]. The focus brought by the PBRF quality evaluation on the work of each individual academic created powerful incentives on individuals (not just managers) to lift performance.
That led to a change in culture of the universities in the early years of the century. The PBRF may not have been the sole cause of that shift to a greater focus on research; the CoREs certainly played a part[14] and a greater government and public attention on the value of university research may also have contributed. However, the PBRF was probably the most important factor in lifting university research performance.
How does the PBRF serve the purposes of university research?
The three important values of university research discussed above are:
- informing teaching and shaping learning
- building human capital
- contributing to our society – to the culture, to community development, to the economy.
The PBRF serves all three values.
For quality research that can inform teaching and shape learning (and thus, increasing the chances of the teaching being at the frontier of knowledge[15]), the PBRF quality evaluation (QE) is an assessment of the quality (and also the scale) of the research performed by academics across the university, meaning that most university teachers have their research assessed and have an incentive to lift the standard of their research. This component of the PBRF creates a quality-weighted measure of scale or volume in the funding formula.
For human capital, the PBRF includes a reasonable proxy measure – research degree completions (RDCs). Research shows that graduates with research degrees boost firms’ innovation and improve firms’ absorptive capacity[16] .
For contribution to society, the proxy is the amount of external research income (ERI). Funding bodies, firms, agencies, NGOs grant funding for research outputs that are of service to their business, to communities, to society. Public good funds – like those administered by MBIE, the Royal Society and the Health Research Council – make awards on the basis of recommendations of panels of experts who have assessed the quality of the research bids and their fit to national strategic priorities. ERI is a reasonable (and efficient) proxy for social and economic impact[17] – an assessment of the social, economic or cultural contribution of the research made by the funder, someone with a vested interest in the value of the research.
It’s far from a perfect proxy; much research transfer occurs informally, through professional associations and journals, through environmental groups, through social services agencies, through arts organisations and events, through advocacy by researchers and academics, through news media …. But it’s not possible to quantify that sort of research transfer, meaning ERI has to stand as a proxy. (This matter is elaborated in the Improving Knowledge Transfer section of this submission).
Do we need change?
The first thing that most everyone knows about the PBRF is that it is disliked by many in the academic community. The PBRF QE is focused on the individual academic, rather than the research group or the academic department; it rates individuals’ research (and hence, is seen by some as creating a stratification of the academic workforce). That is seen by some academics (and especially, the academic trade union[18]) as antithetical to the tradition of academic collegiality, notwithstanding clear quantitative evidence[19] that the PBRF was associated with increased, not diminished, collaborative research). The second consensus view about the PBRF QE is that it is a very high-cost exercise. The QE – requiring each academic to compile an evidence portfolio – carries high compliance burden[20].
In addition, there is a risk that the field of research differentials could create perverse incentives for institutions[21]. And some international research[22] has questioned the effectiveness of peer review, the assessment process on which the QE is built.
And, of course, the 2025 QE has been deferred, pending the UAG review, with the chief executive of the TEC telling the Education and Science Select Committee in February 2024 that the PBRF had led to a lift in research performance, but that there were now only marginal benefits from the system. He suggested that the PBRF has low returns for very heavy compliance costs, both for the TEC (in managing the system) and for institutions (in compiling evidence portfolios)[23].
It’s important here to note that there have been four reviews of the PBRF in its short life, one review every five years. And every one of those reviews has noted the high cost but has broadly accepted the original policy parameters, including its most contentious feature – a QE using the individual researcher as the unit of analysis. And most people – including those who wrote the four reviews – would agree that the PBRF has been good for the research productivity of the system.
Yet the case for change is compelling and the momentum for change is strong. Tim Fowler’s statement to the select committee picked up on a common view – that the PBRF has led to a positive change of culture and an improvement in performance, that the cultural change and the focus on research performance in the universities mean that future gains will be marginal – and make it hard to justify the challenging, costly, intrusive evaluation of each individual’s research performance.
What would a new approach to core research funding look like?
The primary purpose of the PBRF[24] is:
- to allocate research funds to universities, fairly, with as much simplicity and transparency as possible
- to do so in a way that recognises and rewards excellent performance, as well as the scale of each institution’s research
- to recognise performance across the three key values of university research – quality research that informs teaching, builds human capital and contributes to our society, our economy and culture.
It’s an allocation mechanism. We need a fair transparent allocation mechanism. A simple mechanism. That needs measures, measures that reflect excellent performance, scale, the capacity to inform teaching, to build human capital and to contribute to our society.
The measures will never be exact. All measures that reflect the quality of research will be proxies. Perhaps one of the flaws of the PBRF design was that it sought (and claimed) a level of precision that is unattainable and in doing so, it may have compromised the values of simplicity and transparency. There is an unrealistic expectation that it will be equally sensitive to every aspect of a university’s creative activity – no funding system will ever be so fine.
What we need are indices that allow us to allocate money in ways that broadly reflect the range – both in scale and in quality.
Creating an alternative to the PBRF
In my previous paper on this topic[25], I suggested an adjustment to the PBRF quality evaluation. I proposed a mechanism comprising four components, two of them from the PBRF, two of them new:
- A bibliometric measure, a count of indexed publications
- A simple impact assessment
- The research degree completions measure – as now
- The external research income (ERI) measure, almost as now.
Replacing the QE with a bibliometric measure: In my earlier paper, I analysed the impact of replacing the QE with a count of all indexed publications over the relevant time period, a measure that reflects both the scale and the quality of a university’s research programme. There are a number of important points to note about this suggestion:
- A change from the QE to a count of research publications actually wouldn’t change the distribution much[26]. That is, a simpler approach, a count, ends up delivering something broadly similar to the costly, intrusive QE. The correlation between indexed publication counts and QE scores is around 99%. Had the most recent quality evaluation been cancelled and this proposal used in its place, around 1.5% of the funding (about $4 million in aggregate) would have shifted. There would have been winners (UA, up 1.6%) and losers (UO and AUT, down 5% and 6% respectively, which would be painful but …).
- Many people are sceptical of the value of bibliometrics. I am too. I am conscious of the limitations of bibliometrics (for instance, it can’t read creative arts performance scale, quality or value. But what we are looking for here is something that will do what the QE does but without the pain. Something that will provide a method of allocation without forcing people through the QE. The quantitative analysis in my earlier work[27] shows that it is possible to get something of the same split as the QE out of bibliometrics.
- And, as noted above, there are doubts about the effectiveness of peer review, the foundation on which QE has been constructed. Critics have questioned whether the QE panels, faced with so many evidence portfolios, are able to do justice to the expectation that they apply a “first principles” assessment of the research outputs in the evidence portfolios they are expected to read, without resorting to proxies like journal status and bibliometric measures[28].
But a count of indexed publications carries a serious risk. In attaching a financial reward to a pure quantitative measure with a relatively low-quality threshold (acceptance by an indexed journal), this would almost certainly incentivise more and more publications of a minimum standard, just good enough to get into any indexed journal. It would reward quantity over quality … I would therefore suggest counting instead, the number of indexed publications among the 50% most cited papers in their field. The correlation between the total number of papers and the number in the top 50% most cited is very high – more than 99% across all the NZ universities, over time and across fields of research. This would inject a stronger quality dimension; a paper will attract citations at that level only if it is a moderate/high impact paper.
So, I would modify my earlier suggestion – replace the QE with a count of the number of indexed publications among the 50% most cited papers in their field.
An impact assessment: For instance, each university could be required to submit a small number (say, three to eight depending on scale) of short case studies that demonstrate the impact of a research group or research programme. Case studies presented in a standard form, similar to the sort of template used in the UK research impact assessment[29]. A small number of cases, with no pretence to comprehensiveness. Discussed by expert assessors but without any scoring[30]; rather, submission of the required number of case studies would be a threshold condition, a prerequisite, a mandatory requirement for research funding[31], an opportunity for the university system to showcase the value of their work.
Research Degree Completions: No change from the current PBRF.
External Research Income: I said above that this should be retained almost as now.
The latest review of the PBRF wanted it eliminated altogether, arguing that ERI is an input rather than an outcome, echoing Hazledine and Kurniawan’s (2006) critique of this dimension of the PBRF[32]. Yes, an input of sorts[33]. But it’s also (and primarily) a proxy measure of the social, cultural and/or economic value of research[34]. Businesses only purchase research outputs if they add to the firm’s commercial performance, a proxy for the economic contribution of those outputs. MBIE and the HRC only fund projects through their schemes if they contribute to government’s health or strategic objectives for NZ. The Royal Society uses the Marsden Fund to reward research that is socially or culturally valuable. ERI reflects independent, expert assessment of social, cultural or economic value. Sure, it’s a proxy. And sure, it may be a bit crude, (given the peer review processes used in the public good research funds are imperfect and given that much research transfer is unfunded and hence, excluded from the ERI measure). But what is needed is a measure of how the universities compare in developing research that society values. ERI is intended to contribute to the distribution of funding in a way that acknowledges the importance of this value of research[35].
But … following the 2012/13 review, the government decided to weight non-governmental ERI higher than ERI from NZ government sources – MBIE research funds, HRC, Marsden and research contracts for government agencies. That decision, by implication, devalued socially important, non-commercial research in, for instance, health, culture, environment, society. That was a bad decision and should be reversed. (See also the Improving Knowledge Transfer section of this submission).
Other possible elements
In the UK, a recent review of their Research Excellence Framework[36] (REF) saw a proposal (from an international advisory group (IAG)) to de-emphasise the weighting of research outputs in the REF and instead, increase the weighting applied to impact, engagement and the “people, culture and environment” (PCE) characteristics of the university. This caused some concern in the academic community, with some complaining that PCE is an input and that If the research environment is strong, that will be reflected in outputs[37].
The IAG however noted that it is the PCE elements of a university’s research system that build research performance. Creating and nurturing PCE is an institutional responsibility, so a greater focus in the funding system on PCE weights the focus of the assessment more on the performance of the institution and its leadership, as opposed to individual academic and research group performance. They argued, too, that focusing on outputs is retrospective whereas the funds generated in the REF are to be used prospectively; the quality of PCE creates the conditions that will shape the future outcomes of a university’s research system.
The IAG acknowledged that part of the PCE assessment must “…to a significant extent, be subjective and require some form of panel approach, but submissions must be supported by evidence and some elements can clearly be metricised[38].
Panel assessment of narrative submissions, of course, raise some of the questions raised in the analysis of the 2021 REF environment statements by Inglis et al (2024) and thus, reopens some of the concerns about the process of the QE. That means that the need for “… evidence and … elements that can … be metricised” is especially important.
New Zealand should follow the hunt for metrics in the UK currently underway with interest.
No mechanism will ever be perfect
No measure will be perfect, exact, precise. The common complaint is that the PBRF is too onerous. But other people complain that simplifying the system means that insufficient account is taken of this kind of research or that kind of researcher. The debate on the structure, components, adequacy, accuracy, … of the PBRF has been endless. It’s a debate that often misses the point – what we need is to allocate a significant amount of funding in a way that is transparent, that reflects research performance, that reflects scale, that honours the outcomes and values we want from university research … and that is as close to fair as is reasonable to expect.
But no system removes the responsibility of the management of the university to use the funding, however calculated, in a way that fosters research in the university, that reflects the university’s research strengths, that develops research capability and ensures a growing pipeline of excellent researchers.
But we can do better at lower cost …
What I suggest here is certainly not ideal. Adding some metrics relating to the research environment and culture may (or may not) help. While foregoing expert peer assessment entirely might be controversial, a metrics only approach is feasible, as long as we keep focused on the purposes of the fund – to get a sense of the relative scale and performance of each university’s research programme in ways that are transparent and fair, and in keeping with the outcomes we need from the universities research system[39].
But either way, if nothing else, this shows it is possible to address some of the problems with the QE while maintaining incentives on universities to lift their research performance and to uphold the values of university research identified above.
There is a need to investigate alternatives to the PBRF QE, seeking a mechanism that would:
- allocate research funds to universities, fairly, simply and transparently
- recognise the scale of each institution’s research and reward and encourage excellent performance
- recognise performance across the three primary values of university research discussed above.
[1] Much of this part is drawn from three recent comments on university research – Smyth (2024a, b and c) – published on Strategy, Policy, Analysis, Tertiary Education.
[2] Section 268(2)(d). This statutory definition was largely drawn from the Hawke report that laid the foundation for much of the system as we have it today – see Hawke (1988) Recommendation 5(i), pp 12, 13.
[3] Hattie and Marsh (1996 and 2004) and Marsh and Hattie (2002). These analyses show that there is no correlation between the quality of teaching and research productivity. There are academics excellent in both spheres, some who are excellent in one, not the other and academics who are poor in both. Hiring excellent researchers can raise, reduce or be neutral to the quality of teaching. That suggests, they argue, that a requirement for university teachers to be active in research is a statement of “mission” – a choice that governors of an institution or of a system may make, rather than a requirement for good teaching. Notwithstanding the findings of Hattie and Marsh, Biasi and Ma (2022) point out that teaching conducted by those active in research is more likely to be at the “knowledge frontier” of the field.
[4] This analysis is drawn from Leiden University’s Centre for Science and Technology’s annual analysis of data from the Clarivate Webb of Science. The data was for the years 2019-2022 and uses fractional counting. These findings use the Mean Normalised Citation Scores (MNCS) as an indicator of citation rate. The MNCS counts the number of citations attracted by each paper and compares that with the world mean for each field of research. Note that Lincln University doesn’t meet Leiden’s size criteria and hence, is excluded from their analysis.
[5] The corresponding figure for the Australian universities is 18% above the world mean. Of the 35 Australian universities whose data is reported by Leiden, two are at less than 90% of the world mean and three others between 90% and 99.8%, with 30 of the 35 above the world mean. The Australian system certainly exceeds international standards in research!
[6] This analysis also draws from Leiden’s curation of the Web of Science. While 49.5% of NZ papers are among the top 50%, the figure for Australia is 55.4%. One of the seven NZ universities analysed by Leiden had a rate over 50%. For Australia, the figure was 31 out of the 35 institutions with a proportion of > 50%. NZ’s relative international significance would likely account for part of the discrepancy.
[7] Biasi and Ma (2022)
[8] Arara et al (2023)
[9] See Finegold (1999) and Drabenstott (2008) for just two examples of commercial development. Dalziel et al (2018) and Browne Gott (2020) look at the relationship between culture and the economy, education and social development.
[10] Smyth (2012)
[11] See Smart (2009) and Smart (2013)
[12] Smart (2013) page 31
[13] See Hicks (2012)
[14] Smyth et al (2013). That evaluative study also suggests that the CoRE model is sound. It has the effect of creating critical mass of expertise in strategically important areas of research in a country with a widely distributed population. There is little to be gained from tinkering with the design of the CoRE model.
[15] Biasi and Ma (2022)
[16] Arara et al (2023), Finegold (1999)
[17] The 2020 review report (Smith et al (2020) – see page 11) described ERI as an “input”, missing the point that ERI is an index, a proxy of the quality of an institution’s research impact. The government rejected the group’s proposal to eliminate ERI altogether.
[18] See for example this page and this one.
[19] Smart, Smyth and Hendy (2013)
[20] The implementation review of the PBRF (in 2004) estimated it as a bit less than 2% of the value of the fund (Hazledine and Kurniawan 2006). However, that the 2012/13 review of the PBRF made extensive changes to the specifications for the evidence portfolio, reducing the burden (and, hence, perhaps reducing the 2% figure).
[21] See Hazledine and Kurniawan (2006). While that paper lays out an impressive array of potential perverse incentives, in practice, most of those risks appear not to have been realised in the two decades since.
[22] See, for instance, Azoulay and Li (2022), Gillies (2014), Bromham et al (2016), Lee et al (2012), Woods and Wilsdon (2021). The risk is that QE panellists may fall back on traditional bibliometric measures, rather than do a (time-consuming) first principles assessment of the research outputs in the evidence portfolios.
[23] Smyth (2024a). See also this video of the Select Committee hearing.
[24] The argument here is a summary of the case made in Smyth (2024b and 2024c)
[25] Smyth (2024b)
[26] See Smyth (2024c) and this appendix for the modelling.
[27] Smyth (2024c)
[28] See, for instance, Gillies (2014), Bromham et al (2016), Lee et al (2012), Woods and Wilsdon (2021), which focus mainly on peer review leading to grant allocations. The same criticism applies to the present quality evaluation component of the PBRF.
[29] See this page for examples. In the UK, cases are presented on a standard template that limits the narrative (in three sections: summary of impact, underlying research and details of impact) to no more than 1,350 words and limits the number of references to 16. No graphs, no photos, no illustrations. Case studies end up at no more than four A4 pages.
[30] Inglis et al (2024) discusses the assessment of Research Environment statements in the UK REF – the UK equivalent of the PBRF. Modelling the ~18 million words that made up nearly 1,900 research environment statements, they found that factors relating to choice of topic and choice of words explain 69% of the score awarded by the panels; that is not to say that choice of words determine the assessment. But, reviewing the literature on decision-making, they state that the correlations they have established “… may well be, in part at least, causal.” Questions remain over the validity of granular assessment of narratives.
[31] In Smyth (2024b), there is a fuller description of the project and its differences from (as well as its similarities to) the UK model. This is intended as a systematic way of highlighting some examples of the value created in university research but without the excesses and artificiality of the approach taken in the UK.
[32] See Smith et al (2020) page 83 and Hazledine and Kurniawan (2006) page 273.
[33] Note however that most NZ universities would argue that research contracts make a contribution to institutional financial overheads but that there is a close to zero margin on research contracts – the researchers spend up to the budget.
[34] See Smyth (2019a)
[35] It was noted above that ERI will never capture all research transfer; research is disseminated beyond the academic and research community via multiple mechanisms – professional networks, performances, the media, the political process. Instead ERI stands as a proxy.
[36] International Advisory Group (2023), See also this summary of the decisions made in response to the International Advisory Group report,
[37] See for instance, the comments in this piece.
[38] International Advisory Group (2023) para 108.
[39] Sivertson (2017) discusses the risks and strengths of a metrics only assessment in national research assessment systems.
Strategy, Policy, Analysis | Tertiary Education 