P-curve is a statistical tool we developed about 15 years ago to help rule out selective reporting, be it p-hacking or file-drawering, as the sole explanation for a set of significant results. This post is about a forthcoming critique of p-curve in the statistics journal JASA (pdf). The authors identify four p-curve properties they object...| Data Colada
There is a recent QJE paper reporting a LinkedIn audit study comparing responses to requests by Black vs White young males. I loved the paper. At every turn you come across a clever, effortful, and effective solution to a challenge posed by studying discrimination in a field experiment. But, no paper is perfect, and this... The post [128] LinkedOut: The Best Published Audit Study, And Its Interesting Shortcoming appeared first on Data Colada.| Data Colada
Before we got distracted by things like being sued, we had been working on a series called Meaningless Means, which exposed the fact that meta-analytic averaging is (really) bad. When a meta-analysis says something like, “The average effect of mindsets on academic performance is d = .32”, you should not take it at face value.... The post [127] Meaningless Means #4: Correcting Scientific Misinformation appeared first on Data Colada.| Data Colada
When we design experiments, we have to decide how to generate and select the stimuli that we use to test our hypotheses. In a forthcoming JPSP article, “Stimulus Sampling Reimagined” (htm), we propose that for at least 60 years we have been thinking about stimulus selection in experiments in the wrong way [1]. Specifically, with... The post [126] Stimulus Plots appeared first on Data Colada.| Data Colada
In Colada[124] I summarized a co-authored critique (with Banki, Walatka and Wu) of a recent AER paper that proposed risk preferences reflect 'complexity' rather than preferences a-la Prospect Theory. Ryan Oprea, the AER author, has written a rejoinder (.pdf). Its first main point (pages 5-12), is that our results with medians are 'knife edge' (p.8),... The post [125] "Complexity" 2: Don't be mean to the median appeared first on Data Colada.| Data Colada
Kahneman and Tversky’s (1979) “Prospect Theory” article is the most cited paper in the history of economics, and it won Kahneman the Nobel Prize in 2002. Among other things, it predicts that people are risk seeking for unlikely gains (e.g., they pay more than $1 for a 1% chance of $100) but risk averse for... The post [124] "Complexity": 75% of participants missed comprehension questions in AER paper critiquing Prospect Theory appeared first on Data Colada.| Data Colada
This post delves into a disagreement I have with three prominent political scientists, Jens Hainmueller, Jonathan Mummolo, and Yiqing Xu (HMX), on a fundamental methodological question: how to analyze interactions in observational data? In 2019, HMX proposed the "binning estimator" for studying interactions, a technique that is now commonly used by political scientists. I argued... The post [123] Dear Political Scientists: The binning estimator violates ceteris paribus appeared first on D...| Data Colada
A forthcoming paper in the Quarterly Journal of Economics (QJE), "A Cognitive View of Policing" (htm), reports results from a field experiment showing that teaching police officers to "consider different ways of interpreting situations they encounter" led to "reductions in use of force, [and] discretionary arrests" (abstract). In this post I explain why, having spent... The post [122] Arresting Flexibility: A QJE field experiment on police behavior with about 40 outcome variables appeared fir...| Data Colada
There is a 2019 paper, in the journal Political Analysis (htm), with over 1000 Google cites, titled "How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice". The paper is not just widely cited, but is also actually influential. Most political science papers estimating interactions now-a-days, seem to... The post [121] Dear Political Scientists: Don't Bin, GAM Instead appeared first on Data Colada.| Data Colada
There is a classic statistical test known as the Kolmogorov-Smirnov (KS) test (Wikipedia). This post is about an off-label use of the KS-test that I don’t think people know about (not even Kolmogorov or Smirnov), and which seems useful for experimentalists in behavioral science and beyond (most useful, I think, for clinical trials and field... The post [120] Off-Label Smirnov: How Many Subjects Show an Effect in Between-Subjects Experiments? appeared first on Data Colada.| Data Colada
A forthcoming paper in Psych Methods (.pdf) had a set of coders evaluate 300 pre-registrations in terms of how informative they were about several study attributes (e.g., hypotheses, analysis, DVs). The authors analyzed the subjective codings and concluded that many pre-registrations in psychology, especially those relying on the AsPredicted template, provide insufficient information., Central to... The post [119] A Hidden Confound in a <i>Psych Methods</i> Pre‑registrations Critique appear...| Data Colada
As you may know, Harvard professor Francesca Gino is suing us for defamation after (1) we alerted Harvard to evidence of fraud in four studies that she co-authored, (2) Harvard investigated and placed her on administrative leave, and (3) we summarized the evidence in four blog posts. As part of their investigation, Harvard wrote a... The post [118] Harvard’s Gino Report Reveals How A Dataset Was Altered appeared first on Data Colada.| Data Colada
A previous version of this post was supposed to go live in January 2019. But the day before it was scheduled, the Data Colada team (Uri, Leif, and Joe) received an email that we took to be a potential death threat. After discussions with the local police, the FBI, and our families, we decided to... The post [117] The Impersonator: The Fake Data Were Coming From Inside the Lab appeared first on Data Colada.| Data Colada
“Any update on the lawsuit?” That is the most common question any of us is asked. It is usually preceded by an apologetic preamble, like, “sorry if this is a sensitive question,” or “I don’t know if you’re tired of talking about this, but…” The reality is that, for the most part, our actual sensitivity... The post [116] Our (First?) Day In Court appeared first on Data Colada.| Data Colada
Pre-registration is the best and possibly only solution to p-hacking. Ten years ago, pre-registrations were virtually unheard of in psychology, but they have become increasingly common since then. I was curious just how common they have become, and so I collected some data. This post shares the results. The data From the Web of Science... The post [115] Preregistration Prevalence appeared first on Data Colada.| Data Colada
We recently presented evidence of data tampering in four retracted papers co-authored by Harvard Business School professor Francesca Gino. She is now suing the three of us (and Harvard University). Gino’s lawsuit (.htm), like many lawsuits, contains a number of Exhibits that present information relevant to the case. For example, the lawsuit contains some Exhibits... The post [114] Exhibits 3, 4, and 5 appeared first on Data Colada.| Data Colada
Thank You A few months ago we reported evidence of data tampering in four papers that have since been retracted (or re-retracted) (Colada[109] .htm). A few weeks ago we were sued for doing so (Vox .htm). Lawsuits can be very expensive, and a thoughtful and generous group of colleagues and supporters started a fundraising campaign... The post [113] Data Litigada: Thank You (And An Update) appeared first on Data Colada.| Data Colada
This is the last post in a four-part series detailing evidence of fraud in four academic papers co-authored by Harvard Business School Professor Francesca Gino. It is worth reiterating two things. First, to the best of our knowledge, none of Gino’s co-authors carried out or assisted with the data collection for the studies in this... The post [112] Data Falsificada (Part 4): "Forgetting The Words" appeared first on Data Colada.| Data Colada
This is the third in a four-part series of posts detailing evidence of fraud in four academic papers co-authored by Harvard Business School Professor Francesca Gino. It is worth reiterating that to the best of our knowledge, none of Gino’s co-authors carried out or assisted with the data collection for the studies in this series.... The post [111] Data Falsificada (Part 3): "The Cheaters Are Out of Order" appeared first on Data Colada.| Data Colada
This is the second in a four-part series of posts detailing evidence of fraud in four academic papers co-authored by Harvard Business School Professor Francesca Gino. It is worth reiterating that to the best of our knowledge, none of Gino’s co-authors carried out or assisted with the data collection for the studies in this series.... The post [110] Data Falsificada (Part 2): "My Class Year Is Harvard" appeared first on Data Colada.| Data Colada
Microsoft has been making daily copies of the entire CRAN website of R packages since 2014. This archive, named MRAN, allows installing older versions of packages, which is valuable for reproducibility purposes. The 15,000+ R packages on CRAN are incessantly updated. For example, the package tidyverse depends on 109 packages; these packages accumulate 63 updates, just... The post [108] MRAN is Dead, long live GRAN appeared first on Data Colada.| Data Colada
This is the third post in a series (.htm) in which we argue/show that meta-analytic means are often meaningless, because they often (1) include invalid tests of the hypothesis of interest to the meta-analyst and (2) combine incommensurate results. The meta-analysis we discuss here explores how dishonesty differs across four different experimental paradigms (e.g., coin... The post [107] Meaningless Means #3: The Truth About Lies appeared first on Data Colada.| Data Colada
This post is the second in a series (.htm) in which we argue that meta-analytic means are often meaningless, because these averages (1) include invalid tests of the meta-analytic research question, and (2) aggregate incommensurable results. In each post we showcase examples of (1) and (2) in a different published meta-analysis. We seek out meta-analyses... The post [106] Meaningless Means #2: The Average Effect of Nudging in Academic Publications is 8.7% appeared first on Data Colada.| Data Colada
This post is the second in a series (see its introduction: htm) arguing that meta-analytic means are often meaningless, because (1) they include results from invalid tests of the research question of interest to the meta-analyst, and (2) they average across fundamentally incommensurable results. In this post we focus primarily on problem (2), though problem... The post [105] Meaningless Means #1: The Average Effect<BR> of Nudging Is d = .43 appeared first on Data Colada.| Data Colada
This post is an introduction to a series of posts about meta-analysis [1]. We think that many, perhaps most, meta-analyses in the behavioral sciences are invalid. In this introductory post, we make that case with arguments. In subsequent posts, we will make that case by presenting examples taken from published meta-analyses. We have recently written... The post [104] Meaningless Means: Some Fundamental Problems With Meta-Analytic Averages appeared first on Data Colada.| Data Colada
Mediation analysis is very common in behavioral science despite suffering from many invalidating shortcomings. While most of the shortcomings are intuitive [1], this post focuses on a counterintuitive one. It is one of those quirky statistical things that can be fun to think about, so it would merit a blog post even if it were... The post [103] Mediation Analysis is Counterintuitively Invalid appeared first on Data Colada.| Data Colada
This post shows how to run simulations (loops) in R that can go 50 times faster than the default approach of running code like: for (k in 1:100) on your laptop. Obviously, a bit of a niche post. There are two steps. Step 1 involves running parallel rather than sequential loops [1]. This step can... The post [102] R on Steroids: Running WAY faster simulations in R appeared first on Data Colada.| Data Colada
A recently published Nature paper (.htm) examined an interesting psychological hypothesis and applied it to a policy relevant question. The authors ran an ambitious field experiment and posted all their data, code, and materials. They also were transparent in showing the results of many different analyses, including some that yielded non-significant results. This is in... The post [101] Transparency Makes Research Evaluable: Evaluating a Field Experiment on Crime Published in <i>Nature</i> ap...| Data Colada
About a year ago I wrote Colada[95], a post on the threat R poses to reproducible research. The core issue is the 'packages'. When using R, you can run library(some_package) and R can all of a sudden scrape a website, cluster standard errors, maybe even help you levitate. The problem is that packages get updated... The post [100] Groundhog 2.0: Further addressing the threat R poses to reproducible research appeared first on Data Colada.| Data Colada
The paper titled "Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results" (.htm) is currently the most cited 2019 article in the Quarterly Journal of Economics (372 Google cites). It delivers bad news to economists running experiments: their p-values are wrong. To get correct p-values, the article explains, they need to... The post [99] Hyping Fisher: The Most Cited 2019 QJE Paper Relied on an Outdated Stata Default to Conclude ...| Data Colada
In the tenth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JMR) article entitled, “Goal Conflict Encourages Work and Discourages Leisure” (.htm). The article’s two key hypotheses are right there in the title: People who are faced with a goal conflict are (1) more likely to... The post [97] Data Replicada #10: Does Goal Conflict Affect Time Spent on Work and Leisure? appeared first on Data Colada.| Data Colada
A recent NBER paper titled "Gender and the Dynamics of Economics Seminars" (.htm) reports analyses of audience questions asked during 462 economics seminars, concluding that “women are asked more questions . . . and the questions asked of women are more likely to be patronizing or hostile . . . suggest[ing] yet another potential explanation... The post [96] Madam Speaker: Are Female Presenters Treated Worse in Econ Seminars? appeared first on Data Colada.| Data Colada
R, the free and open source program for statistical computing, poses a substantial threat to the reproducibility of published research. This post explains the problem and introduces a solution. The Problem: Packages R itself has some reproducibility problems (see example in this footnote [1]), but the big problem is its packages: the addon scripts that... The post [95] Groundhog: Addressing The Threat That R Poses To Reproducible Research appeared first on Data Colada.| Data Colada
In the ninth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Marketing Research (JMR) article entitled, “Advertising a Desired Change: When Process Simulation Fosters (vs. Hinders) Credibility and Persuasion” (.htm). Some products, such as weight loss programs, exist to help consumers attain a desired change. In this paper,... The post [94] Data Replicada #9: Are Progression Ads More Credible? appeared first on Data Colada.| Data Colada
This post introduces ResearchBox, a new platform for easily sharing data, code, materials, and pre-registrations. With a design and approach similar to AsPredicted, ResearchBox simplifies, standardizes, and organizes supporting materials for publishable research. Compared to the current leading platform, the OSF, ResearchBox is narrowly designed to make it easy for authors to share data, code,... The post [93] ResearchBox: Open Research Made Easy appeared first on Data Colada.| Data Colada
In the eighth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Marketing Research (JMR) article entitled, “The Left-Digit Bias: When and Why Are Consumers Penny Wise and Pound Foolish?” (.htm). In this paper, the authors offer insight into a previously documented observation known as the left-digit bias, whereby... The post [92] Data Replicada #8: Is The Left-Digit Bias Stronger When Prices Are Presented Side-By-Side? appeared first on Data...| Data Colada
The authors of a forthcoming AER article (.pdf), "Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics", painstakingly harvested thousands of test results from 25 economics journals to answer an interesting question: Are studies that use some research designs more trustworthy than others? In this post I will explain why I think their... The post [91] p-hacking fast and slow: Evaluating a forthcoming AER paper deeming some econ literatures less trustworthy appeared fi...| Data Colada
In the seventh installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JCR) article entitled, “Product Entitativity: How the Presence of Product Replicates Increases Perceived and Actual Product Efficacy” (.html). In this paper, the authors propose that “presenting multiple product replicates as a group (vs. presenting a... The post [90] Data Replicada #7: Does Displaying Multiple Copies of a Product Increase Its Perceived Eff...| Data Colada
In this sixth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JCR) article entitled, “The Impact of Resource Scarcity on Price-Quality Judgments” (.html). This one was full of surprises. The primary thesis of this article is straightforward: “Scarcity decreases consumers’ tendency to use price to judge... The post [89] Data Replicada #6: The Problem of (Weird) Differential Attrition appeared first on Data Colada.| Data Colada
A friend recently asked for my take on the Miller and Sanjurjo's (2018; pdf) debunking of the hot hand fallacy. In that paper, the authors provide a brilliant and surprising observation missed by hundreds of people who had thought about the issue before, including the classic Gilovich, Vallone, & Tverksy (1985 .htm). In this post:... The post [88] The Hot-Hand Artifact for Dummies & Behavioral Scientists appeared first on Data Colada.| Data Colada
In the fifth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JCR) article entitled, “The Influence of Product Anthropomorphism on Comparative Choice” (.html). A product becomes “anthropomorphized” when it is imbued with human-like features, such as a face or a name. For example, this camera, which... The post [87] Data Replicada #5: Do Human-Like Products Inspire More Holistic Judgments? appeared first on Data Colada.| Data Colada
We miss the old seminars and conferences. While we wait for those to happen again, we’ve decided to organize a seminar series ourselves. Most talks will probably be about behavioral science, but we are figuring things out as we go. The one thing that all talks will have in common is that all three of... The post [86] The Data Colada Seminar Series appeared first on Data Colada.| Data Colada
In this installment of Data Replicada, we report on Study 3 of a recently published Journal of Consumer Research article entitled, “Does Curiosity Tempt Indulgence?” (.htm). In that study, participants were induced to feel curious or not and then were asked to (hypothetically) choose between two gym memberships, one for a “normal” gym and one... The post [85] Data Replicada #4: The Problem of Hidden Confounds appeared first on Data Colada.| Data Colada
In the third installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JCR) article entitled, “The Uncertain Self: How Self-Concept Structure Affects Subscription Choice” (.htm). The central theory in the paper can be expressed in the following way: If you are uncertain about your own self-concept, then... The post [84] Data Replicada #3: Does Self-Concept Uncertainty Influence Magazine Subscription Choice? appeared first on Data ...| Data Colada
In this second installment of Data Replicada, we report two attempts to replicate a study in a recently published Journal of Consumer Research (JCR) article entitled, “Wine for the Table: Self-Construal, Group Size, and Choice for Self and Others” (.htm). Imagine that you are in a monthly book club and it is your job to... The post [83] Data Replicada #2: Do Self-Construal and Group Size Influence How People Make Choices on Behalf of a Group? appeared first on Data Colada.| Data Colada
In this post, we report our attempt to replicate a study in a recently published Journal of Marketing Research (JMR) article entitled, “Having Control Over and Above Situations: The Influence of Elevated Viewpoints on Risk Taking” (.htm). The article’s abstract summarizes the key result: “consumers’ views of scenery from a high physical elevation induce an... The post [82] Data Replicada #1: Do Elevated Viewpoints Increase Risk Taking? appeared first on Data Colada.| Data Colada
With more than mild trepidation, we are introducing a new column called Data Replicada. In this column, we will report the results of exact (or close) preregistered replications of recently published findings. Anyone who has been paying attention will have noticed that the publication of exact (or close) replications has become increasingly common. So why... The post [81] Data Replicada appeared first on Data Colada.| Data Colada
In a recent referee report I argued something I have argued in several reports before: if the effect of interest in a regression is an interaction, the control variables addressing possible confounds should be interactions as well. In this post I explain that argument using as a working example a 2011 QJE paper (.htm) that... The post [80] Interaction Effects Need Interaction Controls appeared first on Data Colada.| Data Colada
A PNAS paper (.htm) proposed that people object “to experiments that compare two unobjectionable policies” (their title). In our own work (.htm), we arrive at the opposite conclusion: people “don’t dislike a corporate experiment more than they dislike its worst condition” (our title). In a forthcoming PNAS letter, we identified a problem with the statistical...| Data Colada
Thinking about evidence and vice versa| Data Colada
This is the introduction to a four-part series of posts detailing evidence of fraud in four academic papers co-authored by Harvard Business School Professor Francesca Gino. In 2021, we and a team of anonymous researchers examined a number of studies co-authored by Gino, because we had concerns that they contained fraudulent data. We discovered evidence...| Data Colada
This post is co-authored with a team of researchers who have chosen to remain anonymous. They uncovered most of the evidence reported in this post. These researchers are not connected in any way to the papers described herein. *** In 2012, Shu, Mazar, Gino, Ariely, and Bazerman published a three-study paper in PNAS (.htm) reporting...| Data Colada
