News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.
To be worth using, a detector needs not only (A) not get very many false positives, but also (B) get…
Everything you say is true, but what is the alternative? I don’t think people are advocating a return to in-class…
The discussion here assumes an institutional context where returning to supervised in-person assessment is at least theoretically feasible, a reasonable…
Cyber security professional here -reliably determining when a computational artifact (file, etc.) was created is *hard*. This is sorta why…
Agreed with the other commentator. It is extremely unlikely that Pangram’s success is due to its cheating by reading metadata.
I see this question as a bit naïve. There is metadata on every document created by a modern word processor…
There’s a simple way to test. Open a pre-2022 essay and copy-and-paste it into a new file.
I'm especially interested in discussions in the context of political science, but more general critical discussions (by philosophers or social scientists) are welcome. Links to discussions that are available on-line would also be useful. Thanks!
Raymond Nickerson, "Null Hypothesis Significance Testing: A Review of an Old and Continuing Controversy", Psychological Methods, 2000, Vol. 5, No. 2, pp. 241-301 is a very good overview of the debate about that issue.
https://xkcd.com/882/ is scarcely sophisticated philosophical commentary, but gets the basic point across pretty effectively.
There's a very accessible discussion of the issue in chapter nine of Jordan Ellenberg's book _How Not to Be Wrong_.
There's a nice piece on Aeon: 'It's time for science to abandon the term 'statistically significant' (https://aeon.co/essays/it-s-time-for-science-to-abandon-the-term-statistically-significant).
Colquhoun, the author of the Aeon piece, has a discussion of p values that I find more helpful than the Aeon piece: http://rsos.royalsocietypublishing.org/content/1/3/140216
Pretty much all of Andrew Gelman's blog. A good recent synopsis of problems and solutions here: http://onlinelibrary.wiley.com/doi/10.1111/brv.12315/full
Not sure exactly what you have in mind, but perhaps some of this is useful
http://www.polmeth.wustl.edu/files/polmeth/gill99.pdf
http://jonathanrenshon.com/Teaching/NPS/ResearchDesign/Using%20Graphs%20Instead%20of%20Tables.pdf
http://www.gwern.net/docs/statistics/2008-gerber.pdf
http://psychology.okstate.edu/faculty/jgrice/psyc5314/SignificanceSorceryLambdin2012.pdf
Obvious goto source here is Andrew Gelman's blog, andrewgelman.com, which is a continuing extremely sophisticated discussion of precisely these issues.
Gelman is professor of statistics and political science at Columbia.
Several accessibly written papers by Andrew Gelman address these questions
http://www.stat.columbia.edu/~gelman/research/published/
eg
http://www.stat.columbia.edu/~gelman/research/published/ForkingPaths.pdf
http://www.stat.columbia.edu/~gelman/research/published/pvalues3.pdf
http://www.stat.columbia.edu/~gelman/research/published/retropower_final.pdf
and see his blog.
There is an article on Stanford about the philosophy of statistics. Under the section dealing with problems of classical statistics there are issues related to statistical significance, centered on p-values, but it's not specific to social sciences.
Minor point for anyone who goes there: the author talks several times about p-values (and thus statistical significance) as bearing on whether or not one accepts or rejects the null hypothesis. E.g. "After all, the test leads to the advice to either reject the hypothesis or accept it, and this seems conceptually very close to giving a verdict of truth or falsity."
In statistics we NEVER accept a null hypothesis. We either reject it, or fail to reject it. And the decision is never given over to just a p-value, if at all possible – as if something magical happens at (say) p = 0.05 that doesn't happen at 0.050 000 001.
In fact, to the best of my recollection, in none of my statistics classes were we ever confronted with phrases like, 'So the results are statistically significant.' There are only ever pieces of evidence, of which p-values are but one, and people disagree about how strong they are, which is entirely context-dependent. (Would you get on a plane that, assuming everything was ship shape, still had a 5% chance of crashing?)
Many thanks for these excellent pointers.
The American Statistical Association released a statement earlier this year:
http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503
http://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108
Other useful sources:
"A Dirty Dozen: Twelve P-Value Misconceptions" http://www.perfendo.org/docs/BayesProbability/twelvePvaluemisconceptions.pdf
"Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing" http://tap.sagepub.com/content/18/1/69.abstract
Another thing I noticed is that, when people criticize the obsession of researchers with p-value (usually for good reasons), they often say that researchers should use confidence intervals. But, as this excellent paper shows, this can also lead to bad consequences: http://link.springer.com/article/10.3758/s13423-015-0947-8.
Deirdre McCloskey is worth readinghttp://www.deirdremccloskey.com/docs/jsm.pdf
Statistician William Briggs has got a long-running discussion of this issue, including a recent response to the ASA statement.
http://wmbriggs.com/?s=statistical+significance
Mostly on the bio/med sciences, but John Ioannidis's (Stanford Med) "Why Most Published Research Findings Are False" is a bracing commentary on these issues. (And open source.) Though, insofar as social sciences such as economics and political science are based on computer simulations of theoretical idealizations, p-values may not apply at all.
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
From Ioannidis's abstract: Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
—–
KEYWORDS:
Primary Blog
Leave a Reply to Barry Lam Cancel reply