Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Jeremy Goodman's avatar
  2. David Wallace's avatar
  3. Mark's avatar
  4. Valerie Dawson's avatar

    Exactly! The question is not whether AI is conscious or forms human-like intentions. The question is whether AI–in carrying out…

  5. Valerie Dawson's avatar
  6. David Wallace's avatar

    This is a very strange comment. Searle’s Chinese Room, and his general position on intentionality, were controversial, and criticized (correctly,…

  7. Nat Hansen's avatar

SAT scores and socioeconomic class

Brian Leiter

MOVING TO FRONT (ORIGINALLY POSTED MAY 27)–AN INFORMATIVE DISCUSSION, BELOW

I do not know this literature, but this author claims that SAT together with GPA is a very good predictor of college performance, and that the impact of socioeconomic status on SAT scores is minimal.  Is he correct?  Is he correctly representing the literature?  I'd be interested to hear from readers more knowledgeable about these topics.

UPDATE:  There's a useful summary of the state of the discussion in comment #26, below, by philospoher Chandra Sripada (Michigan).

Leave a Reply

Your email address will not be published. Required fields are marked *

38 responses to “SAT scores and socioeconomic class”

  1. Gary Lerner

    1) Not related per se to performance, Professor Caroline Hoxby has argued that elite universities have often overlooked low-income, high-scoring students through (unintentionally?) bad policy choices, so part of the issue about the inequalities that result from the SAT may have to do with the ineptness of the end-users rather than the exam itself.

    2) With respect to economic class, the explicit advantages of class are relatively modest because test prep only improves test scores modestly. https://slate.com/technology/2019/04/sat-prep-courses-do-they-work-bias.html Class's ability to buy educational opportunities generally that translate into high scores isn't really disputed, but it is certainly not determinative. One of the poorest high schools in America, Stuyvesant High School, for instance, always outscores high-income public and private schools in the NY/CT/NJ area.

    3) With respect to social class (most saliently race), African-Americans tend to score 1 standard deviation lower than their white or Asian peers, according to Professor Alex Johnson.

    The tests are often more academically predictive for African-Americans than for white or Asians, so they are not biased in the sense that they are less accurate for marginalized student performance than for non-marginalized student performance.

    So, on the whole, the economic class argument is narrowly true, although it's unclear how those claims don't also preclude considering grades (i.e., tutoring access) or whether eliminating the ability to compare low-income and high-income students on a strictly equal playing field would result in more low-income students being admitted to elite universities. Hoxby seems to argue that colleges can use the latter effect to attract and admit low-income students more easily. The social class argument does not seem true.

    Reply
  2. Peter Tan

    I'll let others who know more about the literature on these subjects chime in more, but I do think that it's worth noting that the author of that Substack piece seems to me to be engaged in some disingenuous misrepresentation of the numbers.

    The author alleges that in one of the pieces he refers to, Zwick and Green (2007), the best-fit value (r-squared) for the regression between SAT scores and income is .137 for the SAT Verbal and .128 for the SAT Math. He cites this as an objectionably low r-squared value. But for the social sciences, low r-squared values are no surprise at all, and a common rule of thumb in social science contexts is that r-squared values above .1 are perfectly acceptable.

    Also, the author of the Substack piece fails to mention is that there is in fact a well-corroborated correlation between SAT scores and income even in the Zwick and Green piece he refers to. In that piece the r value is above .35 for SAT and income (.371 for the SAT Verbal, .358 for Math, pg15 linked here: https://eportfolios.macaulay.cuny.edu/liufall2013/files/2013/10/New_Perspectives.pdf). This is corroborated by Sackett et al's (2009, linked here: https://files.eric.ed.gov/fulltext/ED562860.pdf) findings that correlation between income and SAT scores rises to .42 when considering the entire SAT-taking population, or Sacket et al (2012, link here: https://www.jstor.org/stable/23260359) which provides an r value of .35. Obviously values of roughly .3-.4 aren't the most meaningful, but there still is some measurable positive correlation. Anyway, it seems to me that the author of the Substack piece is misrepresenting the data somewhat.

    Reply
  3. p a

    To be fair the author isn't claiming r2 in Zwick and Green is so small as to be statistically insignificant, but that it is small enough that even if the correlation is significant the % of variation explained is low… whereas the way some advocates talk about the SAT as an "income test" you'd expect to say a r2 of .7 or .8

    Also everything you say about the higher correlation could be true, and parental education could still confound (which to me seems like the most likely explanation).

    Reply
  4. Doug Stalker

    My old friend and former colleague, the social psychologist James Crouse, showed in his book The SAT and College Admissions (U of Chicago press, 70s or 80s) that you will admit pretty much the same freshman class using GPA alone, thus no reason to pay for the SAT in addition. The GPA predicted freshman grade points just about as well as SAT plus GPA did. His work was attacked by the SAT people (see Harvard Educational Review) and he responded (same journal, I think). Perhaps Crouse has other work more directly relevant to the issue of your link, but I thought the claim that both GPA and SAT were the way to admit was errant in itself.

    Reply
  5. Peter Tan

    Yes, I suppose that's right. I still think, upon rereading the article, that he misrepresents the data as suggesting no positive correlation whatsoever.

    Also, a worthwhile note to my previous comment: as someone has privately pointed out to me, the author of the Substack piece surely knows that the r must be .37 if the r2 is .137. So it would be inaccurate to say that the author omits this information. Yes, this is right, since the author clearly knows how to calculate the r2 to begin with. My complaint about the Substack piece is that the author *presents* the information in such a way as to suggest that the r2 is objectionably low, and one part of this presentation is the "omission" of the r value. (The piece seems to be written to target popular audiences who might not know how r2 values are used or calculated – see the author's own hyperlink to a brief explainer about r2 values). If that's the case, then including the r2 but "omitting" the r value (even though it can be easily calculated from the r value) seems to me to be a bit devious. Plus, again, my main issue is that the author's complaint that the r2 value is ~.14 seems to flout the rule of thumb that r2 > .1 are acceptable.

    Reply
  6. MBW

    Freshman GPA is a poor metric because students take a big range of courses even as freshmen. Comparisons should be made using a collection of individual course effects. The tendency of lower-scoring students to take easier courses will tend to hide the actual predictive effect.

    Reply
  7. Gary Lerner

    Wouldn't that mean that the SAT is NOT a driver of inequality because it would have no impact on outcomes? Inefficiency isn't per se inequity.

    Just browsing the table of contents for his book, it seems like his argument is fairly different from yours. https://press.uchicago.edu/ucp/books/book/chicago/C/bo3628187.html He seems to argue that not relying on the SAT would change the demographics of admitted classes.

    I don't buy his argument's relevancy to the current SAT. The previous versions of the SAT were fairly obviously white supremacist,
    privileging knowledge of WASP-y sports (https://www.clearchoiceprep.com/sat-act-prep-blog/the-most-infamous-example-of-cultural-bias-on-the-sat) and doing no differential item functioning analysis to fight against cultural bias.

    However, the current SAT has many more psychometric safeguards and studies that show increased predictive power of various kinds. Moreover, a lot of the recent analysis of the SAT's predictive power has found that its inaccuracy is fairly randomly distributed across socioeconomic classes, signaling it does not have any particular bias (https://www.insidehighered.com/news/2016/01/26/new-research-suggests-sat-under-or-overpredicts-first-year-grades-hundreds-thousands).

    Reply
  8. Mark van Roojen

    In the old days there were gender-relative discrepancies insofar as scores are used to predict subsequent academic success, IIRC. Women with lower scores did as well as men with higher scores in their first years of grad school . I don't know if that is true with newer versions of the test. ETS was aware of this and in fact put this in one of their publications about how to interpret the tests. So they may have tried to fix that. (I would have to use Google to find support for this claim beyond my – failing – memory, but since you can Google as well as I can . . .)

    While ETS does a better job coming up with this sort of question than their competitors (in my opinion based on having written such questions for them as a summer job), I am skeptical that they measure anything or real predictive value. Test wise people can do surprisingly well in picking the right answer without even seeing the question.

    Reply
  9. Michael B

    One point that jumped out at me, not least because it is in bold type:

    "The only reason to use graphs or rather than quantitative correlational or regression data is to obscure reality."

    No one with any statistical credibility has thought this way for at least 50 years.

    https://en.wikipedia.org/wiki/Anscombe's_quartet

    The author may be correct in his conclusion but that sentence is not.

    Reply
  10. David44

    I think the key phrase here is "rather than": deBoer doesn't say one shouldn't use charts or graphs or that they can't illuminate in ways that the numerical data alone doesn't – after all, he uses them himself at various points in the essay! His claim is that if one ONLY uses charts or graphs and draws conclusions from them alone then this can seriously misrepresent the underlying data – but he argues that is what the "anti-SAT" side often does in order to overstate the income effect. (I have no independent knowledge of the question, and can't say if he's right about this!)

    Reply
  11. Chandra Sripada

    The SAT does have a sizable correlation (r~0.4) with SES, i.e. socioeconomic status—the author should not have downplayed that. But what drives the correlation? Some people assume that high SES lets you game the system and “test prep” your way to a high SAT score, in which case the SAT-SES correlation tells strongly against the SAT.

    The weight of evidence, however, is that a main driver is this: SES has a very strong influence on the development of academic abilities during childhood and adolescence, with concentrated poverty playing a particularly potent role. The SAT, in turn, is a good measure of these abilities, and thus the SAT and SES are correlated. Moreover, these abilities are important for college performance, so that’s why the SAT along with GPA are the two best predictors of college performance (with the SAT eclipsing GPA in recent years, likely due to grade inflation).

    So if you learn about the sizable SAT-SES correlation and your next move is to attack the SAT, you may be missing the point. Attack concentrated poverty, school inequality, lack of preschool, unsafe schools/neighborhoods, etc. And go out and argue for strong preferences in admissions for low SES students. Affirmative action based on SES is vastly underused even though it has astonishingly broad public support, even in states that are blood red. The corporate class that runs universities hate it b/c it is costly. Harvard gave massive “tips” to alumni, athletes, donors, etc. but virtually no advantage for low SES while sitting on a $40 billion endowment. In press coverage, race got all the attention, but the way that low SES students got the shaft got ignored–even by, actually especially by, “progressives”!

    Reply
  12. David Wallace

    It's not 'downplayed', as far as I can see – it's explicitly discussed. Your r~0.4 translates to an r-squared of ~0.16. The author links to a bunch of large studies with r-squared around 0.13 or 0.14 – i.e., consistent with the r~0.4 you cite, if maybe just a shade lower. He then interprets that r-squared value – correctly, so far as I can see, but in any rate in conformity with pretty standard statistical approaches – as telling us that only 13%-14% of variance in SAT score is attributable to SES.

    Reply
  13. Chandra Sripada

    David, he is downplaying because the correlation b/w SES and SAT is very large in terms of typical (reliably observed) effects seen in psychology and the social sciences. And it is already in interpretable units: the correlation says a one standard deviation decrease in SES yields a 0.4 standard deviation decrease in SAT – a meaningfully big drop. A familiar trick when you want to downplay a correlation is to turn it into an r squared and then talk about how much variance remains unexplained. R squared’s have desirable under the hood statistical properties and there are certain contexts when they more directly address the scientific question. But for the most part, they are unintuitive measures of effect size and can obscure what is otherwise recognized as a strong relationship, which is better understood via the correlation.

    Reply
  14. doris

    +1 Chandra

    As others have noted, a correlation of around .4 (which I'll here take on faith as accurately representing the SES-SAT correlation) represents a robust effect size in the context of much social science.

    More telling, perhaps, is that it likely represents a relationship detectable by "casual observation" unaided by statistical artifice. As Meyer et al. (2001) note in their classic treatment of effect sizes, the correlation between sex and arm strength is probably around .5; if so, and the SES-SAT correlation is around .4, I'm guessing it's a relationship large enough to be pretty noticeable if one was flipping through a pile of applications where both items are known. I'm further guessing that the relationship is large enough to matter practically; it's certainly larger than many relationships people act on, like the .11 relationship (per Meyer et al.) between antihistamine use and reduced cold symptoms.

    I'm not well-positioned to take sides on the contested issues concerning the utility and interpretation of r-squared, but if it contributes to the perception that the SES-SAT relationship is "minimal," it does seem misleading in this context, assuming a relationship in the vicinity of r = .4.

    Reply
  15. David Mathers

    Surely though what matters is not whether effect is large relative to other effects found in social science, but how much of the variance it explains. What we want to know is whether the reasonable attitude to the SAT is a) basically, your score is determined by your parent socio-economic status, so if we use it for admissions we'll favor the already privileged, or b) your score is mostly determined by stuff other than your parent socio-economic status and so we have to worry about this a lot less. I don't think in that context it is misleading to say it is small if the vast majority of the variance is explained otherwise. (Though it might be misleading if John Doris is right that the real relationship is almost certainly stronger.)

    Suppose that as it happens, social science never discovered any effects that explained more than 0.00000000000000000000000000000000000000000000001 of the variance between two groups. Then suppose we find the parental socio-economic status has a correlation with the SAT of around 0.000000000000000000000003. No one is going to say 'it's misleading to call this small, it is massive relative to other social scientific effects, even though it genuinely is massive compared to other effects in this scenario.

    Reply
  16. Joseph Rachiele

    Here is the comment on bias in testing from the 2007 paper by Kuncel and Hezlett that summarizes their meta-anlysis:

    "Overall and across tests, research has found that regression lines frequently do not differ by race or ethnic group. When they do, tests systematically favor minority groups (18–23). Tests do tend to underpredict the performance of women in college settings (24–26) (SOM text) but not in graduate school (18–20, 23)."

    Reply
  17. Joseph Rachiele

    Thanks for that comment. I wondered if you might have any references off hand for the thesis that SAT is now more predictive than GPA. A summary from the College Board in 2019 still gives GPA a slight edge and that Chicago Consortium study that got a lot of press found *much* greater prediction for grades among Chicago public school students. I suppose these results might vary a lot depending on student population? Anyway, just curious to learn more

    Reply
  18. Chandra Sripada

    @Joseph: https://senate.universityofcalifornia.edu/_files/underreview/sttf-report.pdf

    @David, Correlation conveys effect size well b/c it is in intuitive units: if sd income is 30k (use income as a proxy for SES) and sd SAT is 210 points (I think these are pretty close), going up 100k in income means going up around 275 points on the SAT. All of that is very back of the envelope but its clearly a big effect. Variance explained in contrast is a more technical notion and lacks this natural interpretation.

    Reply
  19. David Mathers

    @Chandra, oh, I don't understand any of the stats here at all. My point was just that how big the average social science effect is is irrelevant to what is relevantly "large" in this context, what matters is roughly how much one's success on the SAT actually depends on parental SES, whatever technical statistical notion best captures this notion of depending on.

    Reply
  20. Michael B

    Yes you are right, David44, thanks.

    I still wonder though: he wrote "rather than" which really does read like, "instead of".

    I do think (e.g.) "the only reason to use stocks rather than bonds or cash" is quite different to "the only reason to use stocks rather than stocks and bonds and cash" have quite different meanings.

    But, certainly, the author does not say "Never use graphs" and he does use them himself in the piece. So, perhaps I have not given him a charitable reading here.

    In any case, this is not helping this thread move forward on the substance so I'll shut up now. Cheers.

    Reply
  21. Joseph Rachiele

    Ah thanks! I had seen that but thought those findings might be explained by the selectivity of the sample?

    In case anyone is interested, here's the large College Board study I referenced. GPA and SAT are basically equally predictive of first year college GPA. (GPA has the tiniest edge.) Both help predict retention as well, the study notes

    https://www.researchgate.net/publication/334262776_Validity_of_the_SATR_for_Predicting_First-Year_Grades_and_Retention_to_the_Second_Year

    Reply
  22. David Wallace

    I’m not very moved either by the idea that r is more intuitive than r-squared (they are both rather technical measures of correlation, motivated more by their useful technical properties than by direct intuitive significance) or by the idea that someone misleads by giving one rather than the other (it is not difficult to calculate a square root) But I won’t push the point further as I don’t want to derail the main discussion.

    Reply
  23. Chandra Sripada

    To avoid getting derailed into discussions of r versus r-squared, I think it’s helpful to stick to real world units. So in the graph that de Boer shows, when you move from the second-to-bottom income bin (20-40k) to the top bin (>200k), that is a 300 point SAT difference. That is clearly an important difference.

    Looking more broadly in developmental and educational psychology, there is a huge amount of work spanning decades on how SES influences the development of academic abilities in childhood. Here is a summary of SES effects in the NAEP, a standardized test given nationally to US 8th graders.

    https://www.air.org/sites/default/files/US-National-State-Trends-in-Educational-Inequality-Due-to-SES-2003-2017-March-2021.pdf

    They find a ~ 1.2 sd difference b/w 75% SES and 25% SES, which is huge and bigger than the SES-SAT effect size (which is around 0.9 when converted). Generally, SES-based achievement gaps are larger in the US than elsewhere, they are likely getting larger over time, they vary across place within the US, and they are importantly related to concentrated poverty (I recommend Reardon et al "The geography of racial/ethnic test score gaps." American Journal of Sociology 2019).

    Given all that we know about the size and persistence of SES achievement gaps in childhood in the US, why would anyone be surprised that there is a large SES-SAT relationship? It would be truly miraculous if there weren’t. And why would anyone want to dismiss the SAT as measuring little more than “access to test prep”? The SES-SAT relationship reflects deep structural inequality that manifests in reduced development of academic abilities in lower SES American youth. That sociological reality, not the SAT, is what we need to attack.

    Reply
  24. L.F. Cooper

    Chandra Sripada, upthread, notes that affirmative action based on SES (class-based affirmative action, for short) is underused. I agree it should be used more.

    Of some interest in this connection is a 2019 op-ed by Richard Kahlenberg (link below). One of his premises seems to be that if race-based affirmative action were to be restricted by the courts or ruled out altogether, universities by default would turn to class-based affirmative action to diversify their student bodies. That may or may not be correct. Some universities might just decide they don't care that much about diversity. In any case, and without meaning to endorse every statement in it, here is the link to the piece:

    https://www.bostonglobe.com/opinion/2019/10/04/there-better-way-diversify-harvard/zU8GinH9WRhOEjTu7hrflK/story.html

    Reply
  25. Andrew Doyle

    A lot of people are talking about r = 0.4 (R2 = 0.14) and whether this is large.

    In a social science paper determining whether SES has _any_ effect on this would be a large enough effect to say "yes it does."

    But I think the poster's argument is not that SES has no effect on SAT (a stupid claim). He is arguing that the SAT is a lot more than an "income test" because while 0.4/0.1 is enough to show SES has an effect, it is not enough to show that SES and SAT scores are proxies for one.

    There is no contradiction between people here talking about how $X in income produces an increase of Y points on the SAT and what has been claimed in the original post.

    Reply
  26. Chandra Sripada

    The claims on the table are getting mushed up, and so let me lay out what I agree with in de Boer and what I am arguing against. de Boer takes himself to be arguing with a progressive interlocutor who repeats the well-known platitude that:

    (1) The SAT is only an income test.

    de Boer thinks he can counter that by arguing that:

    (2) SES explains only a “small” proportion of the variance in SAT scores.

    But (2) is just plain wrong because the effect of SES on SAT scores is big and no one should dispute that by focusing on hard to interpret r-squared values, as de Boer’s own WashPo graph shows and as John Doris helpfully elaborates on above.

    The actual reason why the SAT isn’t an income test is this:

    (3) SES affects SAT scores because SES strongly influences the development of academic abilities, and the SAT–as it is supposed to–measures those abilities. Moreover, the SAT’s relationship with college performance is essentially unchanged when you control for SES (as shown in the Sackett et al 2009 article).

    Unlike (2), (3) shows decisively that (1) is false or at least deeply misleading. Now, de Boer strongly agrees with (3) as well and adds additional evidence for it. So if you are keeping score, all agree that (1) is false and (3) is true, and since (2) is a murky claim in any case, the remaining disagreement here is likely not that important.

    Reply
  27. Matt McGue

    Non-philosopher here, who hopes it is okay to comment on this interesting discussion. My apologies in advance for the length.

    First, summary measures of rearing SES are usually a weighted composite of three variables: parent education, parent occupational status, and family income. Composite SES measures will differ depending on how these individual components are weighted although the important point here is that rearing SES is not equivalent to family income.
    P.R. Sackett et al. (2009, Psychological Bulletin, 135 (1): 1-22) reported a meta-analysis of associations between indicators of rearing SES and SAT scores. They report (in Table 7) from 30 studies with a combined N=30,980 a mean correlation between family income and SAT of .19 (in which case the r-squared is ~ 4%). As expected, correlations are higher for the other components of SES; namely parent education and occupation. While the mean correlation is likely an underestimate (because it was probably based on a single report – family incomes fluctuate – and was usually reported by the test taker – maybe not the best reporter), the available data does not support the conclusion that the SAT is a wealth test.

    Second, while there is overwhelming data supporting the existence of a correlation between family SES and educational attainment, I would say the causal basis for that correlation continues to be a matter for debate in the psychological literature. This is because SES is confounded with many other variables, including genetics, a subject of particular interest to me. A recent example from our own lab involved the use of adoption data to show that indeed high-SES parents do environmentally influence the educational chances of their children, although not through skill building (i.e., E.L. Anderson et al., 2020, Journal of Personality, early access). Of course, I would not want to make too much of a single study; rather, the point is that this is still an open question. My own hypothesis is that the non-genetic influence of parents is mediated primarily by what sociologists call social capital.

    Reply
  28. Gabriel

    Hey Professor Sripada, just wanted to comment a word of thanks for your analysis and communication of these issues. I've seen your comments on similar issues regarding the GRE on other blogs, and I've consistently appreciated your clear, well rounded, and fact based takes on these matters.

    Fwiw, as someone who tutored the SAT and ACT for several years in a wide range of settings (from, coincidentally, "inner city" areas in and around Detroit to Oakland county to the Northern suburbs of Chicago) and as someone who recently took the GRE for graduate admissions purposes and who also came from a "lower" economic bracket (pell grant receiving), I think your analysis is spot on. Tests like the SAT and GRE do measure skills and abilities that are important for academic success in the contemporary context, although obviously they are not perfect or all encompassing. These tests are no more of an educational barrier than any of the other metrics used for these such purposes, and in fact are in some ways more equitable than other available metrics. Unequal score outcomes reflect structural oppression and inequality more than they perpetuate it, and eliminating these tests or slandering them is usually sort of missing the point.

    Reply
  29. Chandra Sripada

    @Matt: Thanks for these thoughts. In my point (3) above, I put things in a causal way (SES “influences” the formation of academic skills). That reflects my personal model of the situation, but you are right that SES may not be causal due to confounding with say genes, etc. But causation is not needed for my underlying point. The more general point is that:

    (4) SES is associated with SAT scores because SES is associated with academic abilities, and the SAT–as it is supposed to–measures those abilities, and the SAT’s relationship with college performance is essentially unchanged when you control for SES

    I am not sure your position on (4), but I am guessing you think it is basically fine? Now I suppose your study does show that higher income in part predicts college attainment through a non-academic skills pathway. But that is not necessarily in tension with (4), since (4) is about college performance rather than attainment, and “in part” leaves open that the part in question is relatively small. Also, your odds ratios for college attainment for abilities (cognitive and non-cognitive) in Table 2 are very large, reinforcing the idea that it is abilities that (predominantly) matter for academic outcomes.

    @Gabriel: Thanks. I too tutored the SAT, in my case in the southside of San Antonio, which colors my impressions of how these tests work.

    Reply
  30. Matt McGue

    Hi Chandra,

    Sorry for the slow reply, I got distracted by the holiday (!). In any case, I agree with your point #4. Indeed this is precisely what the P.R. Sackett et al. (2012, Psychological Science) paper, which I think you or someone else cited earlier in this thread, shows. I also agree that cognitive and non-cognitive abilities matter most for academic success, although I suspect that social capital and income are ways parents can help get their kids into (but not necessarily through) college.

    I am no great fan of standardized ability tests for college admission, but I do think there is evidence (hard to track down though) that supports at least in part the College Board's claim (and the original motivation in developing the SAT) that standardized tests can help to identify academically talented individuals from disadvantaged backgrounds who might otherwise be missed. The paper by Card and Giuliano (PNAS, 2016) is an interesting example, although not formally college admissions. Regardless, I believe the critics of standardized tests have cleverly gotten them labelled as "wealth tests". My informal observation is that discrimination based on family income is intolerable for most Americans. IMO, the SAT is a dead test walking, hopefully not to be replaced by criteria even easier for wealthy families to game.

    Hope you are enjoying the holiday,

    Matt

    Reply
  31. Gordon Barnes

    The argument for the SAT is that it reliably predicts successful performance in college. That argument assumes that successful performance in college, in the system of Higher Education as it is, is an unquestionable standard of success. This is equivalent to assuming that the failure of some students in college is due to their deficiency, rather than some deficiency in current higher education, or some combination of the two. That assumption should be questioned, especially in this context. Most college faculty and administrators come from middle-class to upper-class backgrounds. Consequently, the culture of Higher Education is saturated with middle-class to upper-class values, norms and conventions. That is bound to affect students from low-income backgrounds who try to break into that world. By their own account, most students from lower income backgrounds feel alienated on college campuses from day one. My point is that, for lower-income students, “successful performance in college” requires that they successfully assimilate to a relatively alien world. If they fail, as many of them do, I am not willing to assume that this is entirely due to their academic deficiency alone.

    Reply
  32. harry b

    In response to Gordon Barnes (hi!). That's all true. But worth noting the following. I spend a quite a bit of time with people who work in leadership of enrollment management. You might ask yourself — 'why did so many schools go test-optional so fast?". A lot of EM leaders know, from their internal data, that SAT/ACT scores do little if any work, independently of their correlation with other variables, predicting college persistence or completion (I think this is consistent with the excellent comments here by Chandra Sripada). They also believe (and in some cases they have good data to support this) think that SAT/ACT scores make it harder for them to enroll lower-income students because lower-income students who would be successful but see that they are on the low end of the SAT/ACT range for that school simply don't apply, and go to less selective schools instead. So, say that your school states its average ACT score as 28. An upper middle class student with a 26 thinks "I'll give it a shot'. A working class kid with a 26 thinks "I'll try somewhere that I'm above average". (One factor here that I don't think has been mentioned is that the lower income student has much less access to high quality college counseling. High poverty high schools might have a 1:500 ratio or worse, whereas low poverty high schools might have a 1:100 ratio or better, and, obviously, the knowledge base of the latter concerning the behavior of the more selective college admissions practices will be very different from that of the former). Frustrating for many reasons, including that schools which are less selective are also less well-resourced, and there's ample evidence that lower income students benefit more than upper income students from attending better resourced institutions. An informal group of enrollment management leaders have been pushing for test-optional admission for years, and COVID gave them the argument they needed. I didn't watch the exact order in which institutions went test-optional, but did note that many of the early movers were places with VPs of enrollment management whom I know from their association with the informal test-optional movement.

    The VP-EM at Oregon State (Jon Boeckenstedt) has an influential, and remarkably frank, blog, here: https://jonboeckenstedt.net/

    Reply
  33. Chandra Sripada

    @Harry B: I’m skeptical that colleges “know, from their internal data, that SAT/ACT scores do little if any work.” The problem is that colleges already use test scores (or close proxies like subject tests or AP exam scores) in selecting their admitted class. So you observe a range restricted subset and do not get to observe the full pool that includes counterfactual students admitted with low test scores. To get an actual sense of how much work test scores do, you can statistically correct for range restriction (as in Kuncel et al, 2001) or do prospective studies that do not suffer from this problem. The NAEP (highly correlated with the SAT) is administered to a population representative sample of high schoolers and you can prospectively track college completion rates. "Advanced" level on the NAEP as opposed only "Basic" level is associated with 40 percentage point higher 6-year college completion—that is 40 percentage points, not percent. The difference holds among Pell grant recipients (i.e., low income), showing NAEP scores do not predict completion only in virtue of their association with SES. In short, test scores matter a lot. NAEP research reports are available here: https://www.nagb.gov/focus-areas/reports/preparedness-research.html

    Reply
  34. Chandra Sripada

    One more thing. It is a fair question: “Why did so many schools go test-optional so fast?”. The cynical answer is that it is much easier to throw a bone to “progressive” activists than to make serious often costly structural reforms, or battle legislators and elite interests to get the resources to make serious reforms.

    I hate to keep bringing up Harvard, but the litigation makes their process particularly clear. Harvard gives very strong preferences to several groups: rich and privileged prep school athletes (20% of their class), rich and privileged legacy students (another 15-20%), and wealthy Blacks and Hispanics, many from multi-generational elite immigrant families (another 20%). They give very weak preference to low SES applicants. Peter Arcidiacono’s (plaintiff’s expert) found the African-American race preference is very large if you are rich and essentially disappears if you are poor. The reason they favor the well-to-do is clear: Poor people cost a lot to educate, and they tend not to join the donor base. Whether Harvard admits it or not, cultivating the donor base is a core mission for their school, and nearly everything about their admissions process serves this end. This is a completely rotten system, so they have every incentive to point to the SAT and say “Look over there, that is the problem!”

    My own school, U. Michigan, released a major anti-racism task force report a couple of weeks ago, and it decries in a dozen different places that Proposition 6 outlawed affirmative action in Michigan, tying their hands on increasing diversity. But Prop 6 only banned race-based affirmative action. Michigan *right now* could give a much more substantial preference to low SES students—absolutely nothing in state law stops them. If they added family wealth and neighborhood disadvantage to their SES metric, the pool would skew heavily minority advancing racial diversity. They don’t do this because it would be very costly, and they feel they can’t afford it. So Michigan remains an outlier as the state school with the wealthiest student pool in the country. Given this, it makes perfect sense for Michigan administrators to switch to test optional in the coming years, and I expect it’s just a matter of time now. This has nothing to do with whether or not test scores convey useful information. They will do this to hand the vocal activist set a cheap victory, and then the school can go on with business as usual.

    Now multiply this same dynamic across hundreds of colleges. THAT is how we arrived at our current test-optional moment.

    Reply
  35. James Crouse

    Check out the following for more on the SAT and what should be done.

    The Case Against the SAT. Chicago: University of Chicago Press, 1988 (with Dale Trusheim).

    How Colleges Can Correctly Determine Selection benefits from the SAT. Harvard Educational Review, 1991, 61, 125-147. (with Dale Trusheim)

    This Time the College Board is Wrong. Harvard Educational Review, 1985, 55, 478486.

    Does the SAT Help Colleges Make Better Selection Decisions?
    Harvard Educational Review, 1985, 55, 195219.

    Reply
  36. Gordon Barnes

    This makes sense, and it is all consistent with my own experience with low income students over the years. Thanks very much for this information.

    Reply
  37. Gordon Barnes

    Those facts about Harvard are simply atrocious, but I’m not too surprised. My younger brother matriculated at Yale with a scholarship in the the early 90s. He expected his peers to be smart, curious kids like him. So he was disappointed to discover that most of them were thoughtless rich kids. He formed an organization called PAY (Poor at Yale), to protest Yale’s financial aid policies, but by the time he was a junior he could not afford the rising costs that were not covered by his scholarship. I think Yale now promises to cover everything for kids in his income bracket, but I bet that very few low income kids find their way to Yale.

    Reply
  38. Gregory C. Mayer

    (1) As a biologist, I am unfamiliar with the statistical conventions of the social sciences, but want to point out that Freddie de Boer's preference for r-squared over r is one that would be shared by many biologists. One advantage of r-squared is that it is readily generalizable to other statistical models in which the variation can be partitioned into "explained" and "unexplained" components (e.g., the analysis of variance). As to which is more "natural" or "intuitive", that might be a matter of taste and experience. r-squared has the immediately satisfying (to me) interpretation as ranging from 0 to 100% percent. The difference from 100% is what is left over, and still needs to be accounted for. Explaining r as the ratio of the changes in the standard deviations would seem, to me, to be less obvious to the uninitiated.

    As to what is "small", there is of course no hard and fast dividing line. A modern biostatistics text that was close to hand, in its example calculation of r-squared, gets a value of 22%, which it refers to as "moderate". If r-squared is 16%, that means 84% of the variation is unexplained; if you are looking to make a policy intervention, something among the 84% may be more promising to try to identify as an avenue of intervention.

    For an example of real data with an r of .346 and r-squared of 12% (not too far off from real r-squareds mentioned by de Boer), look at this graph: https://troyca.files.wordpress.com/2010/08/image008.jpg , to see how much else is going on in the Y variable which is not attributable to variation in X, despite the significance of the regression.

    To the initiated, both r and r-squared are perfectly well understood, and either will do. But is is unfair to dump on Freddie de Boer because he used r-squared.

    (2) A completely separate issue is that the predictive ability of SATs (and GPAs, for that matter) for academic success may decline for those colleges, such as regional branches of public universities, that are under tremendous financial pressure to maintain and increase "student success", the latter being a term of art among college administrators which refers to the awarding of credentials (degrees, certificates, etc.). At such schools, academic requirements are viewed as "impediments to student success", so the academic skills and background measured by the SAT may be less necessary for "success" as such requirements are revised.

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress