Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Fool's avatar
  2. Santa Monica's avatar
  3. Charles Bakker's avatar
  4. Matty Silverstein's avatar
  5. Jason's avatar
  6. Nathan Meyvis's avatar
  7. Stefan Sciaraffa's avatar

    The McMaster Department of Philosophy has now put together the following notice commemorating Barry: Barry Allen: A Philosophical Life Barry…

REF 2021 results for philosophy

Brian Leiter

The periodic Research Excellence Framework in the UK is significant because it affects the allocation of funding to the universities.   While in its origins it focused exclusively on research output, it has since incorporated criteria like "public" impact, which does not mean the impact of one's scholarship on one's field, but rather, e.g., how often one gets media coverage.   The "top ten" in the UK overall are, perhaps unsurprisingly, somewhat bizarre: 

1.  University of Birmingham

2.  University of York

3.  University College London

3.  London School of Economics

5.  University of Kent

6.  University of Southampton

7.  University of Manchester

8.  University of Warwick

9.  University of Glasgow

10. Cambridge University

Oxford, in case you're wondering, was 19th.  Less bizarre (and more like the PGR results) was the top ten based on "research power":

1.  Oxford University

2.  Cambridge University

3.  University College London

4.  University of Edinburgh

5.  University of Leeds

6.  King's College, London

7.  University of Durham

8.  University of Bristol

9.  University of Warwick

10. University of York

(You can see the full panel for philosophy here.)

Comments from UK readers who are informed about the process are welcome.  Submit comments only once; they may take awhile to appear.

Leave a Reply to Simon Kirchin Cancel reply

Your email address will not be published. Required fields are marked *

21 responses to “REF 2021 results for philosophy”

  1. Samir Okasha

    Brian:

    You are right that the REF used to focus exclusively on research output, but now includes "impact" and "research environment".
    But your suggestion that it is because of this that the 2021 results are, as you put it, "bizarre", is wide of the mark.
    (I don't agree that the results are particularly bizarre, but that is not my main point here.)

    The current system, from which the ranking you link to is constructed, is weighted 60% outputs, 25% impact and 15% environment.
    But for each department, the disaggregated data (on output, impact, environment) is available too.
    So one could perfectly easily construct a ranking based on output quality alone.

    If one focuses on "outputs" alone, and calculates a GPA for each department, then the ranking that emerges is (I think): 1. St. Andrews, 2. Kent, 3. Birmingham, 4. LSE, 5. York. 6. Kingston. 7. Edinburgh, 8. Glasgow, 9. Warwick, 10. Bristol.
    Note that none of Oxford, Cambridge, UCL nor KCL appears in the top ten here. So this ranking is still "bizarre" by your criterion.
    (I emphasize that these GPA calculations are my own, based on the raw output profiles on the REF website. Others may check them.)

    Research power is not a particularly interesting metric, since it takes into account no. of staff in a department, so is biased towards larger departments.

    The main difference between REF and PGR is that the latter is a reputational survey, while the REF is based on painstaking reading of the submitted work by a panel of experts.

    The main reason they produce different results, I suspect, is that reputations are slow to build up and are rather "sticky".
    At any rate, the reason doesn't have anything to do with the inclusion of "impact" and "environment" in REF 2021, as explained above.

    Reply
  2. Brian Leiter

    Thanks for the clarification about the "research power" metric and for pointing out that the criteria can be disaggregated.

    There are other differences between REF and PGR worth noting: (1) REF counts only publications by academic staff during the specified period; an excellent philosopher who didn't publish much or anything during that period counts for nought; (2) the "painstaking reading" of the submitted work in the RAF has also been described by others as "speed reading"; the reading is only as good as the readers, of course; which brings me to (3) the evaluation of the research output of each department in the REF is done by only a handful of people (two or three? or one or two?), rather than the dozens (or many doznes) that evaluate programs in the PGR.

    It does seem to me an evaluation of philosophy programs that puts the best program in the UK at 19th is, indeed, bizarre. That Bristol (Prof. Okasha's department) is not in the ten is also bizarre (it is in the top 10 in the PGR and in the "research power" metric).

    Reply
  3. Matthew Kramer

    I am puzzled by Samir Okasha's assertion that "[r]esearch power is not a particularly interesting metric." Research power is precisely the parameter that is the main focus of Cambridge, Oxford, and UCL precisely because it is largely determinative of the share of funding that will be apportioned by the government. A university (or department) which does concentrate on that parameter and which therefore submits a much larger number of staff will almost inevitably end up with a somewhat lower GPA. That drawback is deemed to be worth the advantage of gaining a considerably larger share of funding.

    Reply
  4. Samir Okasha

    I am puzzled in turn by your comment, Matt.

    Research power is simply GPA x no. of staff.

    To compare different institutions' research quality, the obvious thing to do is to look at GPA, (as per the ranking Brian pointed to), not research power.
    Just as, to compare different countries' standard of living, we typically look at GDP per capita, not total GDP,
    The fact that total GDP is greater in (say) India than in Norway, doesn't tell us much about the relative standard of living in each country.

    Now, since Oxford submitted over 90 staff to the Philosophy REF, but some other institutions only 15 or less, research power is hardly a meaningful way to compare them.

    The REF 2021 rules require that all staff who conduct research at an institution be submitted (in some "unit of assessment" or other).
    The only way an institution can "concentrate" on research power, is to try to raise their GPA, or else to hire more staff.
    There is no freedom as to how many staff to submit.
    This was one of the (welcome) changes to REF2021,

    Reply
  5. Chris Timpson

    Perhaps it’s worth observing that the REF explicitly *does not* and is *not intended to*, produce a quality ranking of departments. It’s a device to divide-up government research funds, informed by a painstaking process of reading lots of philosophy. (One may, and people should, have very different views about that exercise of dividing state money in the first place, both in terms of aims and in terms of process. But we are where we are.) It’s also worth observing that it is not known what the funding formula will yet be.

    Any rankings one sees are not the product of any official channel, but are more by way of interesting perspectives on an enormous range of activity. Many such interesting perspectives may (and do, and should) exist.

    At Oxford we’re very happy to have submitted a lot (really a lot) of research deemed top rate. I personally am impressed by the commitment and fortitude of the assessment panels. My instinct is that the results overall across all departments in the UK speak to the health of philosophy in the UK in general (*despite* dwindling government funding). I’m delighted to see so much of UK philosophy getting its proper recognition by peers. And I’m delighted that there is so much to celebrate across the board. I think the general message is well done all, not least the assessors.

    Chris Timpson
    Chair of the Philosophy Faculty Board,
    Oxford

    Reply
  6. Erstwhile admin

    "A university (or department) which does concentrate on that parameter and which therefore submits a much larger number of staff will almost inevitably end up with a somewhat lower GPA. That drawback is deemed to be worth the advantage of gaining a considerably larger share of funding."

    I thought that used to be the trade-off, but that in this iteration of the game, this move was blocked by requiring that anyone with a research contract be submitted with at least one output, i.e. the deadbeat can no longer be omitted, that no one could be entered with more than five, i.e. the superstar can't compensate, and that the mean number of submissions per FTE staff member with significant responsibility for research had to be at least 2.5.

    Am I simply wrong about this? I do seem to recall that a (perhaps unintended) consequence of this rule was that researchers whose research was not (thought to be) up to much risked being moved on to teaching-only contracts, and therefore exempt from submission.

    And so the game continues…

    Reply
  7. Matthew Kramer

    Samir, the latitude pertains to the nature of the contracts under which people are employed (as well as to the sheer number of employees). Oxford and to a lesser extent Cambridge can employ large numbers of staff under contracts that will have the effect of requiring them to be submitted to the REF. In so doing, Oxonians and Cantabrigians know that some of those members of staff (especially the younger members) will probably lower the GPA somewhat — in contrast with what the GPA would be if only the core members of the department were put forward. However, because of the sizeableness of the core department, that effect will be more than offset (for the purposes of gaining funding) by the increase in research power that is attained.

    Chris Timpson is of course correct in saying that the funding formula has not yet been specified. That was also true after the 2014 REF, and there is little or no basis for thinking that the ultimate dispensation now will differ significantly from what it was then.

    Reply
  8. Simon Kirchin

    Hi all, I just want to echo what Chris Timpson said. What this exercise shows is that philosophy in the UK is in a healthy state with lots of good work found all over. It takes a huge amount to submit – I say this as an ex-Dean who oversaw submissions to eight sub-panels – and everyone who contributed, not least those who were the REF leads (and impact leads, etc.), should be proud of their work.
    Simon Kirchin
    Director of the British Philosophical Association

    Reply
  9. Erstwhile admin

    Tangentially: "Jo Grady, general secretary of the University and College Union, said the REF was a drain on staff time and resources, describing it as “emblematic of a research culture obsessed with arbitrarily designating institutions or departments as winners or losers”."

    I think that this was probably the prevailing view in British academia a decade ago, say, but that now it's largely accepted that if the REF didn't exist, it, or something like it, would have to be invented.

    Reply
  10. Birchlse

    I see these GPA rankings as a welcome form of recognition for smaller departments that are not going to top any league table based on size (or anything correlated with size) but which are generally regarded by peers as having very high quality standards.

    In effect, they make public something that the philosophical community already knows, namely that the research quality distribution is very flat in the UK, with a big group of departments producing work of very high average quality. Oxford and Cambridge dominate on wealth and size, but in no way dominate on average quality.

    Reply
  11. Samir Okasha

    Matt, I can't comment on whether Oxford and Cambridge really did what you suggest, that is, to strategically increase the number of people on REF-eligible contracts in order to increase their research power.

    But even if they did, it's not at all obvious that this would depress the GPA.
    For each additional staff member submitted to REF 2021, 2.5 additional outputs were needed, but only 1 of these need be attributed to the additional staff member in question. The other 1.5 outputs could come from existing staff members ("core department" as you put it).

    So the net effect on GPA could just as easily be positive.

    Reply
  12. Jennifer Hornsby

    Understandably “research power”, being so much a determinant of research funding, is something of a focus for all universities. It’s a good question whether research power should have come to be of so much general concern — at least on the assumption that there are qualities conducive to a good university (not to mention good research which may impinge on good teaching as well as good publications) which have nothing to do with research power.
    The PGR is of most interest to students considering graduate work. I don’t know who ought to be interested in the REF results, although the subject breakdowns are bound to be of much interest to competitive Departments. Yet being so widely reported, I suspect that the REF is treated by many as an indicator of universities’ overall reputation. I’m waiting for rightwing UK newspapers to tell us how much Oxford and Cambridge and the like have been diminished by widening participation.
    However that may be, the readers of the Leiter Report will be interested in how UK philosophy is faring. The BPA Director is right to say that it’s in a healthy state.

    Reply
  13. Huw Price

    Samir, I'm puzzled by the claim that the net effect on GPA 'could just as easily be positive'. Imagine a UoA with say 40 FTE. It needs to submit 100 outputs. These will presumably be the highest-ranked 100 from its internally-ranked pool of eligible outputs, so far as the rules allow. If a particular researcher has no output in the top 100, then the UoA will need to go further down the list, to get that researcher to the minimum 1 output. If a researcher has more than 5 outputs in the top 100 then those in excess of 5 cannot be submitted, and again the UoA needs to go further down the list. Either way, everything at the top end of the ranked pool that can be submitted, will be submitted. So there's no reserve at that end, into which the UoA could dip, if another FTE comes along. This means that unless that person's outputs are themselves above average, compared to the top 100, they can't raise the GPA. Am I missing something? (I'm assuming that UoAs will do internal rankings of outputs, of course, and that these bear some relation to the Panel's rankings.)

    Reply
  14. Robert Stern

    On the question of the health of UK philosophy, there will be more discussion of this in the Philosophy sub-panel report which is due out early next week, which will support the positive views expressed above. This report will also put some of the other matters mentioned in a wider context. So I suggest people who are interested should keep an eye out for that.
    And just to quash a few REF myths while I can: impact is emphatically not about getting media coverage, as that counts as just dissemination, not the kind of substantive change required for impact. And as someone who has spent many hours of his life recently carefully reading outputs, alongside my other colleagues on the sub-panel, I rather resent our work being characterised as 'speed reading' – if only that were so!
    Robert Stern
    Chair of the Philosophy sub-panel, REF2021

    Reply
  15. Brian Leiter

    Perhaps it is unfair to refer to the process as "speed reading," but I've heard more than one British philosopher describe it that way. I have no experience either way, obviously.

    On "impact," point taken. What "impact" does not mean, however, is scholarly impact.

    Reply
  16. Huw Price

    Bob, warm thanks to you and your sub-panel colleagues for all your hard work!

    Reply
  17. Ian Rumfitt

    Indeed, thanks to Bob and his fellow panellists.

    Having served on a REF panel once (more than enough for a lifetime), I know that it is quite unfair to describe its work as 'speed reading'. Such service is, alas, a paradigm case of 'the thankless task'. Those departments which have done well don't thank the Panel because they think (usually rightly) that it simply gave them their due. Those whose scores are disappointing also don't thank it because they think (usually wrongly) that it was blind to their merits of their submission. But such, of course, is human nature.

    Reply
  18. Martin David Kelly

    Samir is right that hiring additional 'non-core' members could improve the GPA, but only in fairly unusual circumstances. Take Huw's example of a UoA with 40 staff (all FT), and imagine that 20 of them are superstars (each with at least five outputs at 4*) and 20 are mediocrities (who've produced only 1* outputs). The GPA is 3.4 (80 @ 4* + 20 @ 1* = 340; divide by 100 outputs). Now add two new full-time staff, each of whom has at least one output at 3* (but no 4* outputs). The GPA is now 3.41 (83 @ 4* + 2 @ 3* + 20 @ 1* = 358; divide by 105 outputs). The pool at the top end is the 20 outputs at 4* that the UoA can't submit (because it has to submit at least one output for each member of staff). However, all this talk of GPA is a bit of a red herring – because (according to best information) the funding will not be allocated by GPA, but rather by the number of 3*+ outputs (with 4*s given disproportionate weight). So even if the UoA in our example hired two new FT mediocrities (thus lowering the GPA to 3.37) this could have a net cash benefit, as it allows the UoA to submit three more 4* outputs. In fact, it might be in its best financial interests to hire 12 mediocrities, allowing it to submit an additional 18 outputs at 4*, even though this would lower its GPA to 3.26 (98 @ 4* + 32 @ 1* = 424; divide by 130 outputs).

    Reply
  19. Dan

    Martin David Kelly says ‘(according to best information) the funding will not be allocated by GPA, but rather by the number of 3*+ outputs (with 4*s given disproportionate weight)’

    I notice that all the research submitted to the REF by staff at the University of Kent produced research rated 3* or 4*. And all but 1% at the LSE. And all but 4% at the Universities of Glasgow and Birmingham.

    Regardless of funding implications, these are impressive achievements so congratulations to the staff there.

    Reply
  20. Thom Brooks

    Huge thanks to Bob and the panel for a job very well done – and seems clear Philosophy in the UK is in excellent shape.

    I agree with Matt and Chris that the exercise is about funding, but I would note an additional reason why research power matters: it is commonly used in UK-based league tables (and not gpa). So research power is important for multiple reasons.

    Congrats to everyone involved in REF submissions and assessors for a huge job done well! (Now to prepare for the next REF…)

    Reply
  21. Simon Kirchin

    Colleagues may be interested to learn that the main Panel D – Arts and Humanities – which incorporates the sub-Panel Philosophy report (and the reports of other sub-Panels) is now out. Many thanks, again, to Bob and the team for all their hard work and for their report.

    https://www.ref.ac.uk/media/1855/mp-d-overview-report-final.pdf

    To Thom. The *next* REF? The current REF is always with us. 😉

    Simon Kirchin
    BPA Director

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress