Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. J.P. Loo's avatar
  2. Sebastian Sunday Grève's avatar
  3. Giovanni Molteni Tagliabue's avatar
  4. Fabien Muller's avatar
  7. Saul Smilansky's avatar

2008 Research Assessment Exercise in UK is Out

Brian Leiter

The philosophy results are here.  I confess I still don’t know what they mean, though the matter is under review!  Bear in mind, of course, that RAE results are always backwards-looking:  so, e.g., St. Andrews was getting full credit for Crispin Wright (who is now mostly at NYU).  Comments from UK philosophers (and others) welcome.  Signed comments only.

ADDENDUM:  I should confess to being a bit facetious, above.  Here’s a quick primer about what the chart means:  4* is the highest mark for submitted research, and it’s down from there.  The number under each column is the percentage of submitted work by each faculty that received the score in question.  So, e.g., UCL submitted a higher percentage of work getting the highest score than Oxford, but Oxford submitted a lot more such work because, of course, it’s a bigger faculty.  As a measure of per capita quality of research output, this is probably quite instructive (putting aside certain weirdnesses:  e.g., how could Middlesex rate above Southampton, both programs that emphasize post-Kantian Continental philosophy?  It makes no sense).  But what it means for a student choosing graduate programs?  Harder to say.   The list of philosophy evaluators:  Alexander Bird (Bristol), Ruth Chadwick (Cardiff), Roger Crisp (Oxford), Jonathan Dancy (Reading), Nicholas Davey (Dundee), R.A. Duff (Stirling), Katherine Hawley (St. Andrews), Joanna Hodge (Manchester Metropolitan), Christopher Hookway (Sheffield), Stephen Houlgate (Warwick), Peter Lamarque (York), Robin Le Poidevin (Leeds), E.J. Lowe (Durham), Mike Martin (UCL), Suzanne Stern-Gillet (Bolton), Alan Weir (Glasgow).  Lots of good philosophers here, though far fewer than participate in PGR surveys of course.  But the real surprise is the weak representation for Continental philosophy, except for Houlgate in 19th-century.  Where are Poellner, Janaway, Finlayson, Martin, Han-Pile, Geuss, S. Gardner, Stern?

UPDATE (Dec. 17, 7 am Chicago time):  A U.K. philosopher writes:

One thing that should be borne in mind is that the GPA average [see Professor Otsuka’s comment, below, containing the average scores] doesn’t take account of the overall proportion of staff eligible for inclusion who were actually included. So Department A can get a higher GPA than Department B by submitting fewer eligible staff as "research active". Previous RAEs have included data on this, but this time there was a big mix up and they eventually decided not to include it for legal reasons I can’t quite understand. I suspect this may account for some of the stranger looking results on the GPA list.

It appears, for example, that Oxford submitted all their staff this time around–including, I take it, overworked tutorial fellows who probably have little opportunity to publish.  I’m also told that St. Andrews included the quarter-time staff at the Arche Research Center there, i.e., philosophers like Jason Stanley, Graham Priest, and Stewart Shapiro, among others.

Leave a Reply

Your email address will not be published. Required fields are marked *

35 responses to “2008 Research Assessment Exercise in UK is Out”

  1. Mike Otsuka

    Here, for what it's worth, are the top twenty departments, ordered by 'GPA' (average point score):

    St Andrews 3.15 (1st)
    University College London 3.15 (1st)

    King's College London 3.05 (3rd)
    Reading 3.05 (3rd)
    Sheffield 3.05 (3rd)

    Cambridge HPS 2.95 (6th)
    LSE 2.95 (6th)
    Oxford 2.95 (6th)
    Stirling 2.95 (6th)

    Bristol 2.90 (10th)
    Essex 2.90 (10th)

    Birkbeck 2.85 (12th)
    Cambridge Philosophy 2.85 (12th)

    Edinburgh 2.80 (14th)
    Leeds 2.80 (14th)
    Middlesex 2.80 (14th)
    Nottingham 2.80 (14th)

    Sussex 2.70 (18th)

    Warwick 2.65 (19th)
    York 2.65 (19th)

    Reply
  2. S L Chao

    Follow the link to get a detailed account of the RAE philosophy league table:
    http://www.guardian.co.uk/education/table/2008/dec/18/rae-2008-philosophy

    Reply
  3. Neil

    Here's how Oxford is reading the results: "in Philosophy Oxford had more than twice as much research activity at the highest quality level as any other department in the UK."

    Reply
  4. John Dupre

    Another (self-serving) thing to note is that it is possible to be entirely invisible in this process if one should be so rash as to be part of an interdisciplinary program. The University of Exeter department of sociology and philosophy, having been returned to the sociology panel, appears not to have a philosophy program. For what it's worth, it achieved a 2.7 gpa, wholly incommensurable to the list above, but worth eighth place in the sociology charts.

    Reply
  5. Simon Kirchin

    An extra bit of information for those who haven't been living with this for the past seven years!

    The evaluations are reached roughly thus. Each full-time permanent member of staff submitted four pieces – articles, books, etc. – for assessment. (There are other categories of staff; but this will probably account for most if not all staff submitted by each department/unit.) The evaluation of these outputs constitutes 80 per cent of the overall evaluation. Each department/unit also had to submit a description of its research activities and organisation, and descriptions of its staff. An evaluation of 'Research Environment' constitutes 15 per cent of the overall evaluation, whilst 'Esteem' of individuals accounts for the final 5 per cent.

    If you can bear to read it, full details here, 'Panel Working Methods':

    http://www.rae.ac.uk/panels/main/n/philos/

    Reply
  6. Jo Wolff

    Even more meaningless, but I can't help myself from pointing it out, is that with a GPA of 3.15 UCL and St Andrews came an equal 64th of all subject areas in all subjects in the UK, out of 2367 submissions. Ahead of us are, apparently, are 5 Economics Departments, 7 Music Departments and 8 Drama, Dance and Performing Arts Departments, among others.

    And with 45% of 4* UCL came equal 25th of all departments in the country (6 music depts did better), and St Andrews with 40%, 72nd.

    Reply
  7. Chris Bertram

    More meaningless stats … Assume that the funding will reward 4* work three times as highly as 3* work, and that 2* and below get nothing, then multiply this weighted sum by the total staff submitted. You get the following rank order:

    1. University of Oxford
    2. University of Cambridge (HPS)
    3 King's College London
    4. University of St Andrews
    5. University of Leeds
    6. University College London
    7. University of Sheffield
    8. University of Cambridge (Phil)
    9. University of Bristol
    10. University of Warwick

    Reply
  8. Mike Otsuka

    My understanding is that UCL included everyone in its submission whom it was possible to include.

    Reply
  9. Mohan Matthen

    Here are some (apparent) facts that strike an outsider who doesn't know how the system works.

    1. There are in the UK thirteen departments with 15 or more faculty in the research stream, i.e., equal to a small-to-middle size department in N. America. (Birkbeck comes close with 14.5, but Cambridge HPS's 35 certainly includes many, including many extremely distinguished, non-philosophers and possibly some non-teaching researchers.)

    2. Oxford University has nearly 80 research-stream philosophers (but that possibly includes some non-teaching junior research fellows who are not available to supervise graduate students). That puts to rest the persistent twittering among my colleagues that U of Toronto is somewhere in the same range as far as numbers are concerned. (We have 50+, and five or six post-docs.)

    3. Let's look at some higher median scores (probably more significant than means for these sample sizes) for the baker's dozen. I'll give a median as x-, where the x-1 rating is significant compared to the x.
    a. There is only one unequivocal 4, UCL.
    b. Cambridge HPS (but see point 1 above), Oxford (though with that huge sample size, and nearly 30 faculty members rated 4), Sheffield (4=3), and St. Andrews are 4-.
    c. Bristol and KCL are 3+.
    d. Cambridge Philosophy, Leeds, Nottingham (where both 4 and 2 are relatively numerous), and Edinburgh are 3.
    e. Durham, Warwick, York are 3- (3=2 for Warwick, York).
    That's more or less in line with Mike Otsuka's GPA ordering, but with some interesting differences. (It pulls UCL and St. Andrew's apart, recognizes Oxford's awesomeness, pushes KCL a bit lower.)

    Reply
  10. Andrew McGonigal

    Median and GPA scores seem to strongly disfavour strong, large departments. (Nobody thinks that UCL is stronger than Oxford, or that Oxford ranks with Stirling). It's probably most sensible to look for patterns across a range of weighted sums that take into account the number of staff submitted. For example, there is a good prima facie case to be made that what matters is quantity of research rated 3* or 4* (i.e. no of staff submitted multiplied by percentage output in either 3* or 4*). On that measure, the top seven are:

    (1) Oxford
    (2) Cambridge (combining Philosophy and HPS)
    (3) Leeds
    (4) KCL
    (5) St Andrews
    (6) Sheffield
    (7) UCL

    For purposes of comparison, here are three more related measures. Let N4 be the number of 4* people, N3 the number of 3* people etc. in a department.

    The weighted sum (4.N4 + 3.N3 + 2.N2 + N1) brings Leeds out third too.

    Suppose we ignore 2* and lower.

    Taking the weighted sum 2.N4 + N3 Leeds comes out 4th after Oxford and Cambridge and just after KCL

    As Chris noted, taking the weighted sum 3.N4 + N3 Leeds comes out 5th after these and St.Andrews.

    Reply
  11. S L Chao

    If "nobody thinks UCL is stronger than Oxford," nobody thinks Leeds is stronger than UCL, St. Andrew, or KCL, either.

    Reply
  12. Duncan Pritchard

    It's also worth thinking about these results from a University administrator's point of view (i.e., by thinking about the bottom line), since that's the economic reality that we'll be dealing with within our Universities over the coming years. With that in mind, it isn't so much performance as such that's important (however that eventually feeds into income), but performance relative to salary costs. On this measure, incidentally, somewhere like Leeds has clearly performed very well since it has comparatively few Profs compared with similar ranked Departments, and yet has made a strong showing over a large submission. Conversely, some Departments may well find that their strong showing doesn't quite suffice to cover the large salary costs they've taken on in recent years in preparation for this RAE round (this will be especially true, of course, of those Departments who didn't submit everyone). It'll be interesting to see how this all pans out!

    Reply
  13. Andrew McGonigal

    Brian Weatherson has a careful and interesting discussion of the figures over at TAR:

    http://tar.weatherson.org/2008/12/18/st-andrews-ftw/

    (His headline news seems to be the disjunctive claim that either a Scottish department or the University of Leeds has done remarkably well).

    Reply
  14. Antti Kauppinen

    Median and GPA scores, unsurprisingly, disfavour unevenly good departments (in terms of RAE submissions). It is also unsurprising that larger departments are more uneven than smaller ones. Nor is it particularly shocking that if you focus on the quantity of good research, good big departments look better. However, leaving out 2s and 1s would amount to entirely ignoring 35% of submissions from Oxford or Leeds, say. If you're looking at the overall quality of research at the department, average measures that take everyone's contribution into account make a lot of sense. But at the end of the day, it all depends on the purpose of the ranking – it would be silly to give too much weight to any particular method.

    On a side note, though I don't know if St Andrews actually submitted work by quarter-time staff, it may be worth emphasizing that the long-term visitors are a strong presence at Arché and a huge bonus to grad students, many of whom have them as advisers.

    Reply
  15. Simon Prosser

    This will already be completely clear to most of you, but just to prevent any possible misunderstandings: where a department has part-time staff their contributions are weighted accordingly. That's why the numbers of staff submitted often aren't integers. Consequently no one gains any advantage just by submitting staff that are employed part-time. Their (pro rata) wages have to be paid, so of course any department with part-time research-active staff will submit them (and as Duncan Pritchard pointed out above, this is really all about how much money departments will have over the next few years relative to their wage bill).

    Reply
  16. Jo Wolff

    Sorry to complicate things, but my understanding is quite different to Simon Prosser's. As I understand the rules, the University's profile looks only at the work submitted, not whether the person who submitted it is full time or part time. That becomes relevant only when it is translated into FTEs submitted. I could give a worked example, but having just been to UCL's RAE celebration party I'm not quite capable of generating enough concentration.

    If anyone has an appetite for more of this stuff, I wrote a (genuine) before and after the results blog for the Guardian:
    http://www.guardian.co.uk/education/2008/dec/18/higher-education-rae-tables-jonathan-wolff

    Reply
  17. Jose Zalabardo

    Andrew and others have tried to explain the disparity between RAE results and our intuitions about the quality of departments — the phenomenon that he helpfully illustrates with the statement that "nobody thinks that UCL is stronger than Oxford".

    His hypothesis is that this disparity is due to the fact that the RAE doesn't take size into account. I have two points to make about this.

    The first is that given the amount of effort that has gone into producing these figures, it seems perverse to dilute the information they provide by factoring it with a figure as readily available as the size of the department.

    The second is that it can't be right. It may put Oxford in the right place, but the results further down the list are frankly bizarre. I could illustrate my point with statements of the form "nobody thinks that Leeds is stronger than X", but I'll resist the temptation.

    Furthermore, I don't think size gives the right explanation of our intuitions about the quality of a place like Oxford. The reason why many of us think that Oxford is the best department in the country has little to do with its size. Those of us with no direct connection with Oxford don't usually realize that it is so big. The fact that it employs around 80 philosophers comes to many of us as a surprise, and I'm sure many would struggle to name more than, say, 20% of them.

    I have an alternative hypothesis. I think that our intuitions about the quality of Oxford are due to its strength at the very top, to the fact that it employs five or six of the best philosophers in the world.

    I think that this factor has a disproportionate weight on our intuitions about quality. If I see a faculty with, say, ten members, and two of them are among the best philosopers in the world, I'm likely to think it's an excellent place. I may not have heard of the others, but I will guess that they have to be pretty good, to have those stars as colleagues.

    The problem for Oxford is that the RAE can't reflect this feature of our intuitions. Leaving the 5% esteem factor to one side, the most that the best philosopher in the world can do for his/her department is to produce four 4* pieces, the same as some of his/her less elevated colleagues.

    Finally, there wouldn't be much point in producing or discussing these surveys if we weren't prepared to contemplate a third hypothesis–that the reason why the results don't track our intuitions is that our intuitions are wrong. I realize, though, that I'm not the right person to explore this possibility.

    Reply
  18. Simon Prosser

    Sorry to create a misunderstanding by trying to clear one up – my point was only that part-timers are not simply treated as full-timers right through the process.

    Reply
  19. Robert Williams

    I think we're going to all get a bit flustered if we try to discern the One True measure of research quality from RAE results.

    Clearly GPA is a significant metric—average research quality. GPA*numbers also seems significant—roughly, quantity of quality research. Relying on either would be perverse (it seems to me): On the first, a single 4* rated philosopher all on their lonesome would constitute an optimal department; on the second, a notional department of 250 philosophers none of whom produce international quality research would top the table. (This ignores the "environment" contribution—but that only accounts for 20% of the total rank).

    So it seems to me that the responsible thing to do is to look at the figures in the round, play around with a bunch of different ways of extracting orderings, and concentrate more on what they actually tell you than on "who wins".

    I do agree with Jose Zalabardo's point that prestige will probably be influenced by a scattering of really top-notch philosophers at a department. The RAE results don't break down by individuals, so there's no real way to pick that out exactly, but you can look at things like: quantity of world class (4* rated) research activity, which might be a good surrogate.

    1. University of Oxford 27.6465
    2. University of Cambridge (HPS) 12.25
    3. University of St Andrews 7.672
    4. King’s College London 7.3605
    5. University College London 7.326
    6. University of Sheffield 6.65
    7. University of Leeds 5.4
    8. University of Bristol 4.8
    9. University of Cambridge (Philosophy) 4.75
    10. LSE 4.7425
    11. University of Nottingham 3.875
    12. Birkbeck College 3.625
    13. University of Reading 3.6
    14. University of Edinburgh 3.4
    15. University of Warwick 3.3

    It's interesting to compare this to the most recent gourmet report UK rankings. Here are the 2006 figures (ordered by local mean to match the locally determined RAE rankings; note that the RAE census date was 2007 so some faculty moves will have occurred in the interim):

    1. Oxford (4.7)
    2=. Cambridge (4)
    2=. St Andrews (4)
    4=. Sheffield (3.9)
    4=. UCL (3.9)
    6. Birkbeck (3.7)
    7. KCL (3.6)
    8=. Leeds (3.3)
    8=. LSE (3.3)
    10. Bristol (3.2)
    11=. Nottingham (3.1)
    11=. Reading (3.1)
    13=. Edinburgh (3.0)
    13=. Warwick (3.0)

    There's some unavoidable disparities here (e.g. Cambridge being fractured into two depts in the RAE). But the two seem to fit relatively nicely (with only one really big discripancy) —and much closer than the fit between bare GPA or GPA*numbers and the gourmet rankings.

    But again, this is just one ordering among many that the RAE data can provide—and it's very natural to think that a measure that completely ignores "internationally excellent" research isn't the most revealing.

    Reply
  20. Antony Eagle

    These results will be used eventually to allocate funding. The way they will be used will involve something like (though not as crude as) multiplying the GPA by the faculty size. Because of this, departments have an incentive to submit as many people as possible, so long as they think they can maintain a minimally acceptable quality standard. Oxford's 78.99 staff includes many researchers who are not holders of 'ordinary' jobs: that is, many people who do not have permanent lectureships/professorships, the standard kind of academic post. The people in charge of the RAE submission here (and I stress I was not involved at all) obviously decided that including non-permanent and non-standard members of staff was overall likely to be advantageous, even though (one supposes) non-permanency and non-standardness may well correlate with reduced research performance. I assume other large departments, with similarly diverse ranks of affiliated philosophers, would have made the same calculation.

    As to what this means for the somewhat strange question as to which department is best: who knows? As Jose Zalabardo rightly points out (and Robbie Williams more or less points out on Brian Weatherson's blog), our intuitions about reputation are often dictated by our opinion of the brightest stars. Jose suggests he'd struggle to name more than 20% of Oxford's faculty, and I don't think he's confessing a special ignorance: I probably couldn't name more than 20% of any department's faculty. What guides my off the cuff judgments is my recollection of the big names, and it is no wonder that an exercise like the RAE might come apart from that. (The sheer volume of work that comes out of Oxford is surely going to influence our off-the-cuff reputational judgments too; we're just more likely to have seen a good paper written by an Oxford person than one written by a member of any other department.)

    However, we do know that size matters in some ways. Oxford is big enough to have more moral philosophers (metaphysicians, historians of philosophy, etc.) than most departments have members, and this kind of community brings benefits of scale and concentration to those who are part of them, including graduate students, that no smaller group can provide no matter how high quality its members.

    Moreover, Oxford is big enough to have submitted more pieces of 4* work than any other department in the country submitted pieces of work of any quality (excepting Cambridge HPS, a somewhat special case). Put it another way: the RAE numbers suggest that there were 502.5 pieces of 4* work produced in the UK over the RAE period. Oxford produced 110.6 of these: more than 1 in 5 of the best pieces of work were written by people here. Who, graduate student or faculty member, wouldn't want to be in a place where so much good work is done, even if some poor work is done too?

    Out of interest, if we rank departments by the proportion they contributed of all the 4* work there is, we get the following 'top 10', which may track reputational intuitions more closely:

    1. Oxford 22.0%
    2. Cambridge HPS 9.8%
    3. St Andrews 6.1%
    4. KCL 5.9%
    5. UCL 5.8%
    6. Sheffield 5.3%
    7. Leeds 4.3%
    8=. Bristol 3.8%
    8=. Cambridge Phil 3.8%
    8=. LSE 3.8%

    Of course this prompts the question: were there really 502.5 pieces of 4* work—work 'which is or ought to be a primary point of reference in its field, ie, a contribution of which every serious worker in that field is or ought to be aware', according to the descriptors—produced in the UK over the last seven years? I better get reading then…

    Reply
  21. Jo Wolff

    Well, I don't know. In the good old days there were departments that were ranked 5*, those ranked 5, etc etc, and no-one tried to concoct tables adjusted for numbers of members of staff. So why do so now?

    The following, I think, are unquestionable inferences from the data. When the formula for cash is announced, Oxford will get the most, but UCL will get the most per capita staff member submitted. More contentious but I think reasonable, Oxford have the most people who have produced some 'world class' research, but if a graduate student wants to be supervised by someone who has produced work of world class quality, as judged by the panel, there are several other universities that offer a greater chance, other things being equal, with UCL top of that list.

    Reply
  22. Brian Leiter

    I am certainly pleased for my friends at UCL (and at other departments whose excellent work was recognized in the RAE), but I do not think Jo Wolff's interpretation of the results in the last posting is warranted. I think the fairest reading of the results is something like this: during the period 2001-2007, the average member of staff at UCL produced work rated slightly higher by a panel of 16 philosophers than the average member of staff at Oxford. This is not irrelevant to the question of which philosophy faculty is stronger or more attractive for a prospective postgraduate, but it significantly undetermines the conclusion one should draw.

    I also do wonder whether there aren't gradations of difference that might be striking among works in the 4* category if we knew what precisely they were.

    Reply
  23. Chris Bertram

    Jo asks

    Well, I don't know. In the good old days there were departments that were ranked 5*, those ranked 5, etc etc, and no-one tried to concoct tables adjusted for numbers of members of staff. So why do so now?

    The only real interest in doing so it to work out who will get largest sum of money. That interests VCs of course. Apart from that someone from a large department which hasn't done so well on average GPA or on proportion of 4* work might favour such a calculation.

    Reply
  24. Thom Brooks

    I have compiled a different weighted ranking of the results on my blog here:

    http://the-brooks-blog.blogspot.com/2008/12/rae-results-in-philosophy.html

    Reply
  25. Tom Hurka

    "World-leading" does seem a stretch as a label for the 4* category if there were over 500 pieces of that quality published from 2001-07. Like Brian I think there must be important gradations of quality within those 500, and I suspect Gourmet Report evaluators are very sensitive to those gradations, i.e. being much more impressed, in however indirect a way, by knowing of publications that would be in the top 50 of those 500 than by ones in the bottom 100.

    Reply
  26. Andrew McGonigal

    It surely isn't right to think that the only reasons that one might have for thinking in absolute rather than relative terms are financial, and then that only administrators should care. If I was weighing up competing job offers, or grad schools, it might well matter to me that there were (for example) 5 really top class people rather than 3 to interact with, get letters from, and so on. That might continue to be true even if the 3 were at a smaller department, where the 3 out of 9 were doing better in relative terms than, say, the 5 out of 16. (After all, it's usually considerably more difficult to get to hang out with great interlocuters than it is to avoid hanging out with unhelpful ones.) Similarly, if I was weighing up job offers or grad schools, I'd also be interested in what the distal and proximal financial prospects of the relevant departments were, how favoured within their institutional structure they were likely to be, what scope for further strategic investment they had in the near future, given recent appointment decisions, etc. These aren't the only things wort taking into account, obviously – no one wants to be stuck with a bunch of colleagues that one regards as borderline imbecilic, even if they are a minority in a large department – but it's surely one relevant set of considerations.

    Reply
  27. Jimmy Lenman

    1. I think too many people are rather too obsessed by how many superstars a department has. Not that superstars are not nice. Andy McG’s preference for doing a PhD somewhere with absolutely more but relatively fewer masters of the universe makes sense in a world where all graduate students get their first choice of supervisor but we do not live in that world. It surely matters how good the strongest people are; it surely also rather matters how good – and how numerous – the weakest people are. (St Andrews and Reading, by that showing, do especially well.)
    2. I don’t think there’s much mileage in trying to explain away any discrepancies between the results and our intuitions about what they should have been. What do we really suppose the latter are mostly worth, especially outwith the comfort zone of our own specialisms? The RAE panel didn’t cook the results up by having such intuitions but by looking exhaustively at very detailed evidence in the form of submitted published work in a way that nobody else (certainly not excepting the PGR's reviewers) has done. So departments that have done strikingly well – such as Reading and Stirling – should continue to feel unequivocally chuffed with themselves.

    Reply
  28. Robert Williams

    "no-one tried to concoct tables adjusted for numbers of members of staff. So why do so now?"

    …at least one person did!

    http://crookedtimber.org/2004/04/30/ranking-uk-philosophy-departments-rae-versus-leiter/

    It does seem more reasonable to look at a range of ways of parsing the data this time around. An obvious reason for this is that the results are so much more fine-grained. The old rankings just gave you a single number between 1 and 5*, purporting to reflect proportions of individuals working at a certain level. This time there's a lot more data, and so there's just more significant info to be extracted.

    I can see why averages are a really significant indicator of research quality. I think that UCL, St Andrews and others who do well on this measure can and should be proud of what they've achieved (disclosure: I'm a St Andrews PhD, now working at Leeds).

    What I guess I don't see, is why we should refrain from looking at other info implicit in the data. As Ant Eagle emphasized, there are obvious ways to figure out from RAE data e.g. how much top notch research is done in place X (in the view of the panel). Why wouldn't this be something we're interested in? Certainly it'd be a mistake to look at it in isolation; but that's no reason to insist that we look at averages in isolation!

    What significance the various kinds of info one can extract from the RAE data have for grad students and others is another matter entirely (see the exchange between Andrew and Jimmy). But the boring-but-obvious thought (I hope) is that we should at least take a look at the full range of info that the data provides.

    Reply
  29. James Ladyman

    Appeals to intuition are less epistemically unsound in disputes about the best philosophy department than they are in disputes about the fundamental nature of reality but that is not saying much. The RAE ought not to be taken very seriously as a measure of quality since (a) it says nothing about specialisms, (b) percentages are averaged to the nearest 5 making the measures very blunt, and most importantly, (c) the score of a particular department depends on the views of a couple of members of a panel of individuals whose opinions about what is world-leading as opposed to merely internationally excellent or whatever are indeed opinio and not scientia (and this is not to criticize or dispute the credentials of the panel). The only things really worth caring about with respect to the RAE is how much money your department will get and whether it will be enough to florish or at least survive until next time, and whether or not your score makes you vulnerable in the context of the idiosyncracies of your institution' politics.

    Reply
  30. Jo Wolff

    My thanks to Robbie for drawing attention to the 2004 discussion on Crooked Timber. I have to say that while UCL's showing is obviously very good for us internally and in publicity terms it hasn't changed my general rather negative view of the RAE which I expressed in the comments thread then. The 2008 exercise has been conducted more sensibly than previously ones – the British Philosophical Association successfully lobbied for a larger panel and co-ordinated the profession's response to the consultation concerning the panel members, so that the administrators were not drowned out with too many conflicting suggestions. We ended up with a bigger panel than before, with greater credibility. But nevertheless there are many reasons to doubt that the rankings show very much, despite my milking of UCL's showing in previous comments. Here are some:

    1. There has been little mention of the 20% available for 'research environment and esteem'. I don't know how the panel dealt with this, but there are various ways, and some ways of doing it could create huge effects. Of course research environment is a relevant factor for graduate students but as far as I can see all statements being made by universities ignore it as a factor, and say 'x' percent of work submitted was of 4* quality. However, unless you know how the panel took research environment and esteem into account, this is unsafe inference.

    2. As before, the RAE reflects those people submitted on the day. If they leave a few months, or even days, after this will not be reflected in future adjustments to the score, or the total quantity of cash flowing to the department.

    3. To be included in the profile a person has to be employed 20% or more. But their impact on the profile is not adjusted to reflect their part-time status. It is theoretically possible, therefore, for all the top ranking work for a submission to have been produced by people who are on site only 20% of the time, and there is no way of knowing whether this is so from the available data.

    4. Although the panel members have to commended for doing an exceptionally demanding job in incredibly difficult circumstances it is unclear how much reliance we can place on their judgement. I believe everything was read by at least two reviewers, but it must have been at immense speed, and immediate judgements have to be made. In my own case it takes me about three years to decide whether a work of philosophy is any good.

    5. There is likely to be a correlation between the seniority of people and the result. We have seen this at UCL. In 1996 we got a 4 (out of 5), in 2001 a 5 (out of 5*) and this time the equivalent to a 5*. Yet we have not had much staff turnover during this time, but now have a department full of professors rather than lecturers. LSE has had the opposite effect – in 2001 they were mostly professors and got 5*, now they have a higher proportion of more junior staff and fell slightly in the table. My conjecture is that within the top 15 or 20 there will be a reasonable correlation between age of staff submitted (or at least the absence of staff beyond 'early career' but not yet fully established) and the final result. By this standard, as others, Leeds has done very well indeed, I think.

    Finally, to return to my earlier competitive mischief-making, I would ask Philosophers from St Andrews for their opinion on how the university is presenting their result here:

    http://www.st-andrews.ac.uk/research/ResultsofRAE2008/

    Reply
  31. Ralph Wedgwood

    James Ladyman's comments are absolutely correct, it seems to me. As he says, the only thing really worth caring about with respect to the RAE is how much money your department will get.

    This was the view that we took at Oxford. So we decided that we should aim ruthlessly to maximize the amount of money that we would get, and completely ignore any reputational effects of our RAE performance. This is why we submitted absolutely everyone whom we possibly could. So, in addition to the 55 or so philosophers who are directly employed by the Philosophy Faculty, our submission also included philosophers in the following categories as well:

    1. Philosophers who are employed by an Oxford college but not by the Philosophy Faculty, including:
    * Senior Research Fellows (e.g. Derek Parfit at All Souls and R. M. Adams at Mansfield College)
    * Junior Research Fellows (e.g. Michael Blome-Tillmann and James Morauta at University College, Hilary Greaves and Maria Lasonen at Merton, Ian Phillips at All Souls, etc.)
    * Philosophers with permanent Tutorial Fellowships at Colleges (e.g. Paolo Crivelli at New College, Stephen Blamey at St Edmund Hall, Tim Mawson at St Peter's)

    2. Philosophers who are attached to one of the research institutes in Oxford, including:
    * The Oxford Uehiro Centre for Practical Ethics (e.g. Julian Savulescu, Guy Kahane)
    * The Program on Ethics of the New Biosciences (e.g. Stephen Clarke, Neil Levy, Matthew Liao, Mark Sheehan)
    * The Future of Humanity Institute (e.g. Nick Bostrom, Rebecca Roache)

    It remains to be seen whether our strategy will pay off. But we're pretty optimistic in Oxford that it was the right strategy to take….

    Reply
  32. Mike Otsuka

    Ralph,

    Just to set the record straight, I know that Oxford didn't, in the end, submit everyone whom they could possibly submit, since I'm aware of a temporary lecturer in your Philosophy Faculty who was originally told that he would be submitted to the RAE on the basis of an internet 'publication' on Oxford's server of a piece of his that was still under review at a journal (and which was, by the way, eventually accepted for publication in a very good journal, but too late for him, alas) but was later told that they had decided against submitting him on grounds that they wanted to minimize the number of internet-only publications in their submissions. He could have been submitted, but wasn't. And I doubt he was the only person in that category.

    Reply
  33. Ralph Wedgwood

    I'm sure that Mike's information on this point is correct. Although I was on a committee that decided on the general strategy that the Oxford Philosophy Faculty would adopt, I wasn't in charge of our precise tactics in every detail.

    So, yes, it's true, there were some philosophers whom we could possibly have submitted but didn't. The one category where we were very slightly less than maximally inclusive was with "temporary college lecturers". To explain: These are philosophers, mostly young people (doubtless including many who will go on to have very distinguished careers in philosophy), who have temporary teaching positions at colleges, but are not Fellows of any college and are not employed by the Philosophy Faculty. (I believe that the lecturer whom Mike is referring to is a member of, but not actually employed by, our Philosophy Faculty.)

    In my defence, I must say that my comment on this blog was not misleading, except perhaps to those who are as well informed of all the details of Oxford's system as Mike is. It *is* true, I believe, that we included everyone whom we could possibly have included in the categories that I explicitly enumerated. Moreover, by the standards of British universities, our submission was extremely inclusive. (I am sure that there are many other UK philosophy departments that also opted not to submit philosophers in temporary teaching positions!)

    Reply
  34. Peter Smith

    One thing I'd be rather interested to know is how much the need to make a show in RAE returns (and so get promotion) constrains — an even distorts — the intellectual life of younger philosophers in the UK. Here's a scenario. Dr A writes a superb PhD thesis on topic X, gets a junior research fellowship, and turns the thesis after a few more years into a very impressive book on X, getting a permanent job at a good department on the basis of it. Understandably, after seven years intensive work, Dr A now wants to move on to thinking about something else. "Ah, no …" say the newly appointing department, "your growing reputation is as a star thinker about X, so for the next RAE we do really need you to keep writing another few papers about that, because those papers are bound to be very well ranked. If you start working in another area, you might well not have publications as good in the needed time-frame." And so, not entirely happily, Dr A knuckles down to grinding out the needed papers …

    Just how frequently does this sort of scenario occur, I wonder? (This isn't a fanciful question, for I do have reason to suspect that this sort of thing happens.) The ever-increasing professionalization and specialization of philosophers does seem to be deeply at odds with that kind of wide-ranging intellectual curiosity, that liking for making connections and seeing "how things in the broadest possible sense of the term hang together in the broadest possible sense of the term", which gets people into philosophy in the first place.

    Reply
  35. Colin Farrelly

    I think Peter Smith raises some interesting concerns about the ever-increasing professionalization and specialization of philosophers.

    It would be interesting to compare how the RAE system might impact intellectual curiosity differently than say the tenure process in North America. My own sense (having spent almost 5 years in both systems) is that the RAE might create a greater incentive for specialization than the tenure system (though the requirements for the latter obviously vary from institution to institution).

    My understanding (someone please correct me if I'm wrong about this)is that RAE process doesn't care if your 4 top submissions are in the same specialized area or not. Whereas in the tenure system evidence of intellectual growth and breadth of expertise can be a requirement for tenure. This creates an incentive to go beyond the topics and work covered in say one's PhD dissertation or first book. I suspect these differences might impact the intellectual curiosity of scholars (especially in the early stages of their career).

    Cheers,
    Colin

    Reply

Designed with WordPress