Jun 4, 2018 - Alex Hulkes at the Economic and Social Research Council and Steven Hill's team at Research England are three impressive examples.
Research Professional - The value of evaluating
05/06/2018, 06*02
The value of evaluating Why we need a What Works Centre for Meta-Research. Comment on this article Evaluation frameworks are rarely a page-turner, but for me the most compelling aspect of UK Research and Innovation’s strategic prospectus, published last month, is its pledge to create an evidence-informed “culture of evaluation” at the heart of the organisation. A dedicated team headed by Jo Peacock, deputy director for data and analysis, will lead this work through a “UKRI Data Hub”. This is long overdue. For a country that channels in excess of £6 billion a year in public funds through UKRI—soon rising to £8bn and far more by 2027, if we believe the prime minister’s renewed commitment to investing 2.4 per cent of GDP in R&D—we spend an infinitesimally small sum on analysing how effectively our research system is working, testing different approaches and learning from innovations elsewhere. Peanuts doesn’t come close. This is not to say that no effort has been made. The individual research councils have all grappled with these issues, and some have built serious in-house capacity. Ian Viney at the Medical Research Council, Alex Hulkes at the Economic and Social Research Council and Steven Hill’s team at Research England are three impressive examples. Outside of government, Nesta’s deep pockets and platoon of policy wonks (now commandeered by Kirsten Bound) have transformed our ability to measure and make sense of the UK’s innovation landscape. And the Research Excellence Framework is, of course, a large and resource-intensive process of evaluation, although it has many other purposes and mostly operates at a micro scale, with limited application to more systemic questions. To use an example close to home, the REF will tell you lots about the relative qualities and impacts of politics research at the University of Sheffield, but far less about the overall health, shape and direction of politics as a discipline. Fragmented field The REF aside, systematic analysis of the research system is an underfunded and fragmented field in the https://www.researchprofessional.com/0/rr/news/uk/views-of-the-u…ource=rpMailing&utm_campaign=personalNewsDailyUpdate_2018-06-04
Page 2 of 5
Research Professional - The value of evaluating
05/06/2018, 06*02
UK, overly reliant on quick, cheap and thinly researched reports by consultants, or formal government reviews that lean on the clout and authority of their chairperson in lieu of actual data and evidence (the Nurse review being a standout example of the genre). We’ve never come close to a UK equivalent of the Science of Science and Innovation Policy programme in the United States, set up after a famous 2005 speech by the late John Marburger, president George W Bush’s science adviser. Marburger admitted with refreshing candour “how primitive the framework is that we use to evaluate policies and assess strength in science and technology”, and argued that “the nascent field of the social science of science policy needs to grow up, and quickly”. The resulting programme, launched by the National Science Foundation in 2006, has since funded well over 100 projects to inform and strengthen US science policy decision-making. The big promise of UKRI has always been the greater strategic coherence it will bring to policies and priorities across our funding system. This will require more analytical firepower and brains within UKRI itself—so it’s good to hear that its Data Hub is recruiting hard—but also more distributed, independent and sustainable capacity across the research system. Expanding potential Worldwide, the field of research on research, or meta-research, is advancing rapidly. There is now expanding potential to combine metrics, analytics and machine learning with a mix of qualitative methods, expert judgement and horizon scanning to provide real-time intelligence on how research systems and institutions are performing and on the changing dynamics of disciplines, impacts, diversity and concentration within them. Pioneered 50 years ago in the UK by groups including the Science Policy Research Unit at the University of Sussex and the Manchester Institute of Innovation Research, the field of meta-research has seen its real energy flow over the past decade from the US and the fantastic work of scholars such as John Ioannidis and his colleagues at the Meta-Research Innovation Center at Stanford. We need to regain the initiative. Over the next decade, UKRI and other UK funders, policymakers and universities will be desperate for more capacity in meta-research in order to surf the next wave of explosive growth in the global scientific enterprise. We need to deal with a greater emphasis on interdisciplinary, mission-oriented and challenge-directed research, a premium on smart collaboration after Brexit, and persistent problems with reproducibility, incentives and career pathways. For now, meta-research remains, in the words of one recent overview, “a hot but fragmented scientific discipline”. More effort is required to develop theoretical frameworks, standardise methods, strengthen networks and test the transferability of evidence and approaches from one area to another. So here’s an idea to put us on an evaluative front foot: UKRI and others should commission a What Works Centre for Meta-Research. Collective intelligence The What Works network recently celebrated its fifth birthday. During its five years, it has expanded to include seven independent centres, two affiliate members and several others on the road to centre status. Across diverse areas, these all contribute to better decision-making by collating existing evidence on effective policies and practices, producing high-quality synthesis reports and reviews, sharing findings in an accessible way, and encouraging practitioners and policymakers to make better use of evidence. https://www.researchprofessional.com/0/rr/news/uk/views-of-the-…ource=rpMailing&utm_campaign=personalNewsDailyUpdate_2018-06-04
Page 3 of 5
Research Professional - The value of evaluating
05/06/2018, 06*02
Meta-research is a field crying out for an initiative of this kind. And we shouldn’t stop there. If we’re serious about the 2.4 per cent GDP target and about the ambitions laid out in UKRI’s prospectus, we can’t afford not to spend more on meta-research. I could write another 1,000 words listing questions in research and innovation policy to which we have no, or at best very patchy, answers. Should we replace peer review with a lottery-based system for awarding grants? How many PhD graduates are still working in academia five, ten or fifteen years after completion—and is this too few or too many? How can we give place serious weight in a funding system that’s driven by “excellence”? What even is research excellence, and how is our understanding of it changing? More research, as well as synthesis of existing evidence on a What Works basis, is clearly required. In a positive move, the Wellcome Trust recently issued a call for pilot projects in this area. If UKRI did the same, perhaps through its Strategic Priorities Fund, it could be game-changing for UK capacity and leadership in meta-research. And it would provide UKRI with a rich seam of real-time, collective intelligence on how the research and innovation system is performing. There are exciting opportunities here for researchers from a range of disciplines, using diverse methods and approaches. And the best meta-research will often be co-produced with research managers, information specialists and other practitioners—hundreds of whom will be gathered in Edinburgh this coming week for the International Network of Research Management Societies’ conference. Personally, I sometimes dream of making a modest contribution to this field by responding to the birth of UKRI through USCREAM, the University of Sheffield Centre for Research Evaluation and Management. What will you do when the meta-research revolution hits? James Wilsdon is professor of research policy at the University of Sheffield. He will be speaking this coming week at the biennial congress of the International Network of Research Management Societies in Edinburgh. He is on Twitter: @jameswilsdon. Image: cosma, via Shutterstock 03 Jun 18, 09:01 By James Wilsdon
Comments Add a comment You are signed in as Professor James Wilsdon, University of Sheffield Type your message here Send
Sign in to add a comment
https://www.researchprofessional.com/0/rr/news/uk/views-of-the-…ource=rpMailing&utm_campaign=personalNewsDailyUpdate_2018-06-04
Page 4 of 5
