“Reproducibility” in General
(Patil, Peng, and Leek 2016)
Seldomly Reported Inclusion/Exclusion
(Patil, Peng, and Leek 2016)
Solutions: RCT/CONSORT diagrams
Neuroimaging Reproducibility
(Patil, Peng, and Leek 2016)
Neuroimaging Reproducibility Starts w/Data
(Patil, Peng, and Leek 2016)
Controversy in Neuroimaging: Won’t Reproduce!
(Patil, Peng, and Leek 2016)
Gronenschild et al. (2012): Freesurfer Thickness
Gronenschild et al. (2012): Freesurfer Thickness
Gronenschild et al. (2012): Freesurfer Thickness
Gronenschild et al. (2012): Freesurfer Thickness
“The observed differences are similar in magnitude as effect sizes reported in accuracy evaluations and neurodegenerative studies.”
“Solutions”/Debugging: Containers
Docker 
Singularity 
Not Same as “Choosing a Pipeline”
(Patil, Peng, and Leek 2016)
No General Solutions for Pipeline Choices
Many combinations (Yan et al. 2013):
See which predicts outcome of interest. Cross validated, of course
Different Pipelines give Different Results
(Patil, Peng, and Leek 2016)
Original Authors May Claim “Incorrect”
(Patil, Peng, and Leek 2016)
It’s typical to have lots of software choices
Carp (2012) “The secret lives of experiments: methods reporting in the fMRI literature.” Neuroimage 63.1 (2012): 289-300.
(Carp 2012)
One Solution: Replication
(Patil, Peng, and Leek 2016)
Want External Replication
(Patil, Peng, and Leek 2016)
Minimum Reproducibility Goal
(Patil, Peng, and Leek 2016)
An R Platform for
Medical Imaging Analysis
What is Neuroconductor?
- A community of developers and users of R packages for imaging
- A website https://neuroconductor.org/.
- with tutorials and help
- A team helping developers and users (John, Adi Gherman, Ciprian Crainiceanu, Brian Caffo)
- A centralized repository of maintained packages
Goal: Centralize the packages (currently 73)
Need Workflows
- all R code
- interface/pipeline tool
- “native” R code
Complete pipeline
- preprocessing and analysis
Many Cases in Neuroimaging: Why?
(Patil, Peng, and Leek 2016)
Data: Submitting Not Required
|
|
|
|
|
|
|
|
R packages to access these repositories
- so if there, need ability to access
- neurovault - access neurovault
- neurohcp - Human Connectome Project
- Rxnat - XNAT interface (NITRC)
Conclusions
- Reproducible code a minimum
- Need data submitted (journals need to help)
- but need easy tools to access the data
- Analysis tools exist but need more
- Develop more standardization like BioConductor
- standard data structures
- publishable pipelines
Thanks
Example Packages
ANTsR
Based on ANTs: Advanced Normalization Tools
- State-of-the-art image processing pipelines
- Group has won challenges for imaging analysis
- Still actively maintained and developed
- Depends on the Insight ToolKit (ITK) medical image processing library
Using R as a Pipeline Tool: fslr
fslrpackage - call FSL from RRequires FSL to be installed (only *nix systems)
spm12r: Wrapper Functions for SPM
- Wraps some
MATLABcode to call SPM scripts - Using
Rsyntax (butMATLABruns the code) - Built from SPM batch commands
- Shown in worked example: http://johnmuschelli.com/talks/fmri_task_processing/index.html#1
neurohcp: Human Connectome Project
- Allows you to download data from Human Connectome Project
- The 1200 Subjects release: behavioral and 3T MR imaging data from 1206 healthy young adult participants. Standardized protocol.
- Tutorial: http://johnmuschelli.com/neuroc/neurohcp
malf.templates: Segmented T1-weighted Images
- Data from the MICCAI 2012 Challenge on Multi-atlas Labelling Data
- From OASIS project and the labeled data as provided by Neuromorphometrics, Inc. (http://Neuromorphometrics.com/)
Current limitations
- R is cross platform, but some packages that depend on *nix system
- Still in beta testing, but more likely to incorporate requests
- Rcpp requires compiled code, (see below)
- Licenses with data can be tricky
Image from: https://imgflip.com/i/22gplr
Training we are providing
Coursera Course: Introduction to Neurohacking In R
https://www.coursera.org/learn/neurohacking/
http://johnmuschelli.com/imaging_in_r/ ENAR 2018
Bibliography
Carp, Joshua. 2012. “The Secret Lives of Experiments: Methods Reporting in the fMRI Literature.” Neuroimage 63 (1). Elsevier:289–300.
Gronenschild, Ed HBM, Petra Habets, Heidi IL Jacobs, Ron Mengelers, Nico Rozendaal, Jim Van Os, and Machteld Marcelis. 2012. “The Effects of Freesurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements.” PloS One 7 (6). Public Library of Science:e38234.
Patil, Prasad, Roger D. Peng, and Jeffrey Leek. 2016. “A Statistical Definition for Reproducibility and Replicability.” bioRxiv. Cold Spring Harbor Laboratory. https://doi.org/10.1101/066803.
Yan, Chao-Gan, Brian Cheung, Clare Kelly, Stan Colcombe, R Cameron Craddock, Adriana Di Martino, Qingyang Li, Xi-Nian Zuo, F Xavier Castellanos, and Michael P Milham. 2013. “A Comprehensive Assessment of Regional Variation in the Impact of Head Micromovements on Functional Connectomics.” Neuroimage 76. Elsevier:183–201.