Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Daniel Fidel Ferrer's avatar
  2. Ryan Mitchell Wittingslow's avatar
  3. Panu Raatikainen's avatar
  4. Jason Stanley's avatar
  5. Daniel Greco's avatar
  6. Nobody's avatar
  7. Roger of Invisible America's avatar

The California antibody serology studies and the case of New York City

Brian Leiter

This offers a useful summary of the doubts about the recent serology studies in Santa Clara and Los Angeles, and then raises a concern based on New York City data:

[T]he current serology tests have relatively high percentages of false positives, ones which may be higher than the prevalence of COVID19 in the population itself. Here’s how to understand this. If you have a test that produces 5% false positives and your study finds an infection rate of 3% in the population it’s possible all your positives are false positives. You really don’t know what your results are telling you. There are also questions about how representative the samples were (a difficult task in these initial studies.) But the accuracy of the tests is the key issue, especially in populations where on a tiny fraction has been exposed….

[F]or something in the range of [the serology study's author's] IFR to be accurate [they estimate .1 to .2%], literally the entire population of New York City would have to have been infected already.

The numbers are straightforward: as of two days ago, there were 9,101 lab confirmed cases and 4,582 presumptive diagnosed cases for a total of 13,683 fatalities In New York City. The population of New York City is 8,398,748. That comes to either .11% or .16% depending on which death toll number is used.

I do not think anyone thinks 100% exposure is at all possible. Even if we assume what I think most experts would consider the highly unlikely possibility that 50% of New Yorkers have been infected with COVID19 that would mean a .33% IFR. To be generous, let’s say a third of the population of New York City had already been infected with COVID19 – very high but not inconceivable. That would mean a IFR of .49%.

Thoughts from readers on this analysis?  (Thanks to Ned Block for the pointer.)

ADDENDUM:  And for additional doubts, see this piece.  (Thanks to Dr. David Ozonoff for the pointer.)

Leave a Reply

Your email address will not be published. Required fields are marked *

15 responses to “The California antibody serology studies and the case of New York City”

  1. Eric Winsberg

    The tests do not, that we know of, have a high rate of false positives. They tested the test on 400 samples of blood from before the epidemic and found 2 false positives. Now, its true that this is a small sample, and so its *consistent* with that result that the test gives 5% false positives, but its pretty damn unlikely. (in fact it has a p value of .00000027) Moreover, there's good theoretical reason to think that these tests can trade sensitivity for specificity, and these tests are very insensitive. What you think about this really ought to depend on your prior regarding that theoretical reasoning. I think its quite unlikely that the test is less than 99% specific and extremely unlikely that its less than 97% specific. There's no way its only 95% specific. Now its also true that they did not propagate the uncertainty regarding the specificity through to their confidence interval on the degree of infection. That would have been better. But you really would need a prior pdf on the specificity to do this. Given the quality of *other* data that we have (which is mostly noise) I think people are being too critical of the California studies.

    Reply
  2. Eric Winsberg

    Relatedly, and just breaking, it now appears that there was community transmission of Covid-19 in Santa Clara County in late, if not mid, January. This is before the lockdown in Wuhan even started.

    hhttps://www.mercurynews.com/2020/04/21/coronavirus-earliest-covid-19-deaths-in-bay-area-occurred-in-february-not-march/

    Reply
  3. Robert A Gressis

    What's the significance of that? Does it mean that c19 has spread wider than we thought (on the grounds that we know how infectious it is, but it's been around longer)? Or that it's less infectious than we thought (on the grounds that we know how many people it's infected, and it's not as many as we would have expected)? Or something else?

    Reply
  4. Robert Lee

    I did a little digging…

    The false positive rate for the test in the study is reported to be 0.5% (2 of 391) rather than the 5% quoted. That would make the high side 95% confidence value be 4 of 391 or 1% false positive.

    The raw study results were 1.5% infection rate in 3330 respondents. These were then weight adjusted by demographics to 2.8%. That is a huge, huge adjustment. They don't have the raw data in the preprint, but they do have the demographics data. Notably, one of the groups (hispanic) was adjusted from 8% to 24.9%, while another group (white) was adjusted from 64 to 34%. To get a final number of 2.8 percent then the infection rate in the hispanic group was likely greater than 4% but less than 8%. They also adjusted by zip code and looking at the color maps they had to amplify some of them greatly as well. This adjustment markedly increases their error margins and I see no indication that they factored it into their final range.

    I see no mention of a questionaire to determine the likelihood of exposure of the participants or if they had exhibited any symptoms. Adding this would have markedly increased the value of the data.

    Finally, their high end estimate is based on a bit of data cherry picking.

    In conclusion, they made a lot of choices that markedly increased their ratios.

    What can be made from the wreckage?

    First, the false positive rate is apparently no higher than about 0.7% (A minimum falls out of reverse engineering the adjustments.) That is not terrible, but it is problematic if the true positive rate is small.

    Second, if you simply use their raw data (1.5% test positive) you get an actual infection rate of 1.1% with bounds more like 0.8 – 1.4 without the cherry picking. This results in ratios of 16 to 28. These are still quite high, but don't immediately conflict with the New York data. The conservative conclusion is that testing for active infection fell far short of the mark.

    Reply
  5. Robert Lee

    Wait, They did ask about "underlying comorbidities, and prior clinical symptoms" in digital form. However, they chose not to include that data or its analysis in the paper despite it being directly relevant to their conclusions. My opinion of Ioannidis just decreased markedly.

    Reply
  6. Lexington

    Not sure how you get that the high side of the 95% confidence range for the test is 1%, given the numbers. First, the numbers are, combining the 371 from the manufacturer and the 30 from the study, 399/401 (there were 2 false positives in the manufacturers samples). But what is the 95% confidence range? I go to the handy-dandy binomial calculator:

    https://stattrek.com/online-calculator/binomial.aspx

    Enter 401 as the number of trials, and 2 as the number of successes.

    Now enter .0179 as the probability of success on a single trial. Calculate, and P(X=x) is .975.

    So the 95% confidence range is [.06%,1.79%].

    Obviously, the unadjusted rate of 1.5% for positives in the study is within the upper bound.

    But it is worth noting that the more recent paper about LA apparently claims an unadjusted rate of 4% for positives, which would be well outside this range.

    I still don't trust the tests to be as reliable as the manufacturer claims. It's very early in the game in the development of these tests, and antibody tests in particular seem to be subject to a lot of problems with false positives. Until these tests are carefully vetted by independent parties, I don't see how firm inferences can be drawn.

    Of course, the best way to mitigate these problems is to conduct tests in a hot spot where the numbers will be big enough that the unreliability of the tests will not much interfere.

    Reply
  7. Lexington

    My comment above got a bit mangled, I'm guessing because the symbols I used were interpreted as some markup tag.

    It should read:

    Now enter .0179 as the probability of success on a single trial. Calculate, and P(X is less than or equal to x) is .975.

    Enter .0006 as the probability of success on a single trial, and P(X is greater than or equal to x) is .025.

    Reply
  8. Eric Winsberg

    Robert A Gressis:

    If it's been community spreading in that county since the middle of January, that certainly ups my credence that the 5% measurement is accurate. That was my only point.

    Reply
  9. Robert Lee

    Not sure where you are getting that 0.0179 value from. The best guess for the probability of success is 0.005, that is 2/400. Secondarily, the binomial test isn't quite the right way to go because, we can only have whole numbers. Thus, the confidence interval on 2 of 400 is 0 and 4 of 400 respectively.

    I tend not to use formulas to calculate probabilities but rather simulate the scenario hundreds/thousands of times. It is much easier to include any funky bits and you can crank out the scenario in Excel in a minute.

    Reply
  10. Lexington

    Apparently, New York state has been conducting a random survey of over 3000 supermarket customers across the state, and has found that 21% of subjects in NYC test positive for Covid antibodies. What may be particularly valuable about this study is that it seems to show only about 3.6% positive in upstate New York. Assuming these numbers haven't been adjusted or extrapolated or otherwise manipulated in some way, this would seem a pretty reliable result. The importance of the low 3.6% number is that it would seem to put a cap on the effect of false positives in the test, which would almost certainly have to be less than 3.6%. Removing 3.6% from 21% still leaves a sizable number, and should give us a relatively good ballpark estimate of the overall prevalence in NYC. Moreover, the 21% number seems consistent with the number of deaths and cases in NYC.

    On the other hand, there may be some significant selection effect if only supermarket customers are tested. Obviously, some supermarket customers may come quite frequently (such as younger people who feel less vulnerable), and others infrequently if at all. If this is not properly taken into account, the numbers might still be pretty far off.

    BL COMMENT: Can you provide a link to this study?

    Reply
  11. Eric Winsberg

    Robert Lee: I don't think hes "getting it" from anywhere. he's back calculating it by assume 2.5% of the probability is on either side of the confidence interval. .0179 is the number you have to plug in to get 2.5% chance of getting as few as 2 positive results out of 401.

    Reply
  12. George Stamets

    From the NY Times in the last hour or so:

    "1 in 5 New Yorkers May Have Had Covid-19, Antibody Tests Suggest"
    https://www.nytimes.com/2020/04/23/nyregion/coronavirus-antibodies-test-ny.html
    [note that it takes time to develop the relevant antibodies, so if these numbers are right as of when the tests were conducted, then they're some amount higher by now]

    "Hidden Outbreaks Spread Through U.S. Cities Far Earlier Than Americans Knew, Estimates Say"
    https://www.nytimes.com/2020/04/23/us/coronavirus-early-outbreaks-cities.html?action=click&module=Spotlight&pgtype=Homepage

    Reply
  13. Lexington

    Here's a link describing the NY study:

    https://thehill.com/policy/healthcare/494324-27m-new-yorkers-have-had-coronavirus-preliminary-data-shows

    Not sure if more details are yet available elsewhere.

    Reply
  14. Robert Lee

    Yup, I did the CI wrong. My best guess of the false positive rate being less than 0.7 still holds. From back calculating, it appears that the white ethnic group (n~2100) had a test positive rate of something like 1.0% or less. That would put the upper bound at about 1.5% if you assume those were all false positives. The study itself notes it but doesn't propagate the potential error.

    On a related note, The USC LA county study is nowhere to be found. The closest I got was a comment in a blog that the link to the "document" was removed because it was "unofficial." Likewise, the just released NY result is apparently without documentation. I sure hope they are solid because with three of these studies out there it is going to be very hard to unring this bell.

    Reply
  15. Dean C. Rowan

    The NYS testing described in comments above is being conducted by the state-run Wadsworth Center and by the medical school at University of Buffalo. I find no publicly available preliminary reports or data at either site.

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress