Our trainees review webinars in their given fields and share abstracts to help colleagues outside their discipline make an informed choice about watching them. As our program bridges diverse disciplines, these abstracts are beneficial for our own group in helping one another gain key knowledge in each other’s fields. We are happy to share these here for anyone else who may find them helpful.
Reproducibility in fMRI: What is the problem?
Russel Poldrack, Professor of Psychology at Stanford University
July 27, 2020
University of Washington eScience Institute
Analysis by Olawale Salaudeen:
Functional Magnetic Resonance Imaging is a significant tool in psychology, neuroscience, etc., to understand the human brain and cognition. However, due to the many varying modalities that affect the inferences that are drawn from fMRI scans, reproducibility is a challenge. In this talk, Dr. Poldrack presents challenges in reproducibility and efforts in his lab and the broader community to mitigate those challenges. A few of the modalities that affect reproducibility are experimental design choices, small samples, group effects, and more.
For fMRI specifically, machine settings can affect the scan, and small sample sizes can affect effect-reliability in region activations (effect size relative to variance). Additionally, group effects may dominate more nuanced information that may be subgroup-dependent.
One strategy to make findings more reproducible is preregistration. In clinical trials, researchers were required to report the effect that aimed to measure, prior to conducting the trial. This requirement has been shown to correlate with more null effects, i.e., researchers cannot move the goal post, thereby biasing the findings to what is found in the trial by chance (prevents p-hacking).
One modality for fMRI specifically is workflow. The process of analysis for fMRI and prediction tasks, for example, are errors in the workflow that are hard to catch. This could simply be from small errors in code that lead to drastic errors in the final inference. So, one idea for addressing this is a code review and containerized codebases that can be reused with the same states and double-checked by others.
Professor Poldrack talks in more detail about these problems, potential solutions, and more. The topic of this talk is also more general than just fMRI; these are important considerations in all of science. The solutions that are in practice right now for fMRI may be useful and relevant for many fields.