From 26eecfd17e0f4a708ed41ba19f4114da0cb27193 Mon Sep 17 00:00:00 2001 From: Hao-Ting Wang Date: Wed, 10 May 2023 11:49:32 -0400 Subject: [PATCH] typos and update content for neurolibre --- content/_config.yml | 4 +- content/_toc.yml | 3 +- content/docs/fmriprep.md | 4 +- content/docs/setup.md | 2 +- content/docs/software_implemetation.md | 44 ++++ content/docs/timeseires_metrics.md | 4 +- content/index.md | 33 ++- content/references.bib | 16 +- content/research_report/benchmark_methods.md | 245 ------------------ content/research_report/contributions.md | 13 +- content/research_report/results-atlas.md | 10 +- content/research_report/results-group.md | 20 +- .../research_report/software_implemetation.md | 109 -------- paper.bib | 12 - paper.md | 18 +- 15 files changed, 112 insertions(+), 425 deletions(-) create mode 100644 content/docs/software_implemetation.md delete mode 100644 content/research_report/benchmark_methods.md delete mode 100644 content/research_report/software_implemetation.md diff --git a/content/_config.yml b/content/_config.yml index 90828a1..b92bf07 100644 --- a/content/_config.yml +++ b/content/_config.yml @@ -1,7 +1,7 @@ # Book settings # Learn more at https://jupyterbook.org/customize/config.html -title: "A reproducible benchmark of resting-state fMRI denoising strategies using fMRIPrep and Nilearn" +title: "A reproducible benchmark of denoising strategies in resting-state fMRI connectivity using fMRIPrep and Nilearn" author: Hao-Ting Wang, Steven L Meisler, Hanad Sharmarke, Natasha Clarke, Nicolas Gensollen, Christopher J Markiewicz, François Paugam, Bertrand Thirion, Pierre Bellec copyright: "2023" logo: logo.png @@ -36,7 +36,7 @@ repository: html: use_issues_button: true use_repository_button: true - announcement: "⚠️This is a working draft for discussion and feedback!⚠️" + # announcement: "⚠️This is a working draft for discussion and feedback!⚠️" launch_buttons: notebook_interface: "classic" # The interface interactive links will activate ["classic", "jupyterlab"] diff --git a/content/_toc.yml b/content/_toc.yml index 0589649..f20e7e0 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -6,8 +6,6 @@ root: index.md parts: - caption: Research report chapters: - # - file: research_report/software_implemetation.md - # - file: research_report/benchmark_methods.md - file: research_report/results-group.md - file: research_report/results-atlas.md # - file: research_report/contributions.md @@ -18,6 +16,7 @@ parts: - file: supplementary_materials/ohbm2022abstract.md - caption: Workflow documentation chapters: + - file: docs/software_implemetation.md - file: docs/tldr.md - file: docs/setup.md - file: docs/fmriprep.md diff --git a/content/docs/fmriprep.md b/content/docs/fmriprep.md index 97cac86..5231624 100644 --- a/content/docs/fmriprep.md +++ b/content/docs/fmriprep.md @@ -11,13 +11,13 @@ If you wish to use the produced metrics to generate the book, feel free to skip We need to build the fMRIPrep singularity containers version `20.2.1` (fMRIPrep-slurm default) and `20.2.5`. For details, please find the documentation on [niprep](https://www.nipreps.org/apps/singularity/). Alternatively, you can execute the script prepared `get_fmriprep.sh`. - We recommand building the container in a tmux session. + We recommend building the container in a tmux session. This will take a few hours! Run it before going home. 2. `preprocessing/get_datasets.sh`: - While runing Step 1 you can get the OpenNeuro datasets in parallel. + While running Step 1 you can get the OpenNeuro datasets in parallel. Run the script in a tmux session. This will take an hour or so. diff --git a/content/docs/setup.md b/content/docs/setup.md index f7b5c4a..ad7052d 100644 --- a/content/docs/setup.md +++ b/content/docs/setup.md @@ -6,7 +6,7 @@ If not otherwise specified, the scripts were executive from the root of the proj If you are a member of the SIMEXP team, you can run the script as is; otherwise, please install other software and modify the path and variables accordingly. -We recommand to understand how to create and detach a tmux session before further reading. +We recommend to understand how to create and detach a tmux session before further reading. - [tmux cheat sheet](https://tmuxcheatsheet.com/) - [How to leave your script running in a tmux session](https://stackoverflow.com/questions/25331471/tmux-detach-while-running-script) diff --git a/content/docs/software_implemetation.md b/content/docs/software_implemetation.md new file mode 100644 index 0000000..9e885f3 --- /dev/null +++ b/content/docs/software_implemetation.md @@ -0,0 +1,44 @@ +# Software implementation + +For the full report, please see our preprint. + +The information here aims to provide context to understand the documentations. + +## Denoising workflow + +The denoising workflow is implemented through `nilearn`. +{numref}`fig-fmriprep-nilearn-denoise` presents the graphic summary of the workflow. + +The Python-based workflow describes the basic procedure to generate functional connectomes from fMRIPrep outputs with a Nilearn data loading routine +(e.g., `NiftiMapsMasker` or `NiftiLabelsMasker`), +fMRIPrep confounds output retrieval function (e.g., `load_confounds_strategy`), +and connectome generation routine (`ConnectivityMeasure`). +Path to the preprocessed image data is passed to load_confounds_strategy and the function fetches the associated confounds from the `.tsv` file. +The path of an atlas and the path of the preprocessed image file is then passed to the masker, along with the confounds, for time series extraction. +The time series are then passed to `ConnectivityMeasure` for generating connectomes. + + +```{figure} ../images/fig-1-masker.png +--- +height: 500px +name: fig-fmriprep-nilearn-denoise +--- +Example workflow of extracting denoised timeseries and functional connectomes from fMRIPrep outputs using nilearn. +``` + +## Benchmark workflow + +{numref}`fig-denoise-benchmark-workflow` presents the graphic summary of the benchmark workflow. + +The denoising benchmark workflow expands on the workflow in {numref}`fig-fmriprep-nilearn-denoise` (represented by the Nilearn logo in this figure). +We retrieved the datasets from OpenNeuro through DataLad and all steps indicated with the arrows are implemented with bash scripts written for the SLURM scheduler. +Atlases were either retrieved from the TemplateFlow archive or reformatted to fit the TemplateFlow format. +The extracted time series, denoising metrics, and all metadata for generating the report are available on [Zenodo](https://doi.org/10.5281/zenodo.6941757). + +```{figure} ../images/fig2-benchmark.png +--- +height: 500px +name: fig-denoise-benchmark-workflow +--- +Workflow of generating the full denoising benchmark. +``` \ No newline at end of file diff --git a/content/docs/timeseires_metrics.md b/content/docs/timeseires_metrics.md index 7bdaf65..6a9221a 100644 --- a/content/docs/timeseires_metrics.md +++ b/content/docs/timeseires_metrics.md @@ -33,7 +33,7 @@ SLURM account for job submission (default: rrg-pbellec) ``` - We created two separate scripts for descrete and probability atlas due to different memory requrement. + We created two separate scripts for discrete and probability atlas due to different memory requirement. You will find the output under: ``` /scratch/${USER}/fmriprep-denoise-benchmark/giga_timeseries/{DATASET_NAME}/{FMRIPREP_VERSION}/{UNIXTIME}/.slurm @@ -55,7 +55,7 @@ 4. `generate_metrics/*/slurm/metrics*.sh`: - Caculate metrics on denoising quality per atlas. + Calculate metrics on denoising quality per atlas. Use this line to submit all jobs at once. ```bash diff --git a/content/index.md b/content/index.md index 7865df3..723b46c 100644 --- a/content/index.md +++ b/content/index.md @@ -13,26 +13,23 @@ kernelspec: # Summary -Reducing contributions from non-neuronal sources is a crucial step in functional magnetic resonance imaging (fMRI) analyses. +Reducing contributions from non-neuronal sources is a crucial step in functional magnetic resonance imaging (fMRI) connectivity analyses. Many viable strategies for denoising fMRI are used in the literature, -and practitioners rely on denoising benchmarks for guidance in the selection of an appropriate choice for their study. -However, fMRI denoising software is an ever-evolving field, and the benchmarks can quickly become obsolete as the techniques or implementations change. -In this work, we present a fully reproducible denoising benchmark featuring a range of denoising strategies and evaluation metrics, +and practitioners rely on denoising benchmarks for guidance in the selection of an appropriate choice for their study. +However, fMRI denoising software is an ever-evolving field, and the benchmarks can quickly become obsolete as the techniques or implementations change. +In this work, we present a fully reproducible denoising benchmark featuring a range of denoising strategies and evaluation metrics for connectivity analyses, built primarily on the fMRIPrep {cite:p}`fmriprep1` and Nilearn {cite:p}`nilearn` software packages. -We apply this reproducible benchmark to investigate the robustness of the conclusions across two different datasets and two versions of fMRIPrep. -The majority of benchmark results were consistent with prior literature. -Scrubbing, a technique which excludes time points with excessive motion, -combined with global signal regression, is generally effective at noise removal. -Scrubbing however disrupts the continuous sampling of brain images and is incompatible with some statistical analyses, -e.g. auto-regressive modeling. In this case, a simple strategy using motion parameters, -average activity in select brain compartments, and global signal regression should be preferred. -Importantly, we found that certain denoising strategies behave inconsistently across datasets and/or versions of fMRIPrep, -or had a different behavior than in previously published benchmarks, especially ICA-AROMA. -These results demonstrate that a reproducible denoising benchmark can effectively assess the robustness of conclusions across multiple datasets and software versions. -Technologies such as BIDS-App {cite:p}`bidsapp`, the Jupyter Book {cite:p}`jupyter` and Neurolibre {cite:p}`neurolibre` provided the infrastructure to publish the metadata and report figures. -Readers can reproduce the report figures beyond the ones reported in the published manuscript. -With the denoising benchmark, we hope to provide useful guidelines for the community, -and that our software infrastructure will facilitate continued development as the state-of-the-art advances. +We apply this reproducible benchmark to investigate the robustness of the conclusions across two different datasets and two versions of fMRIPrep. +The majority of benchmark results were consistent with prior literature. +Scrubbing, a technique which excludes time points with excessive motion, combined with global signal regression, is generally effective at noise removal. +Scrubbing however disrupts the continuous sampling of brain images and is incompatible with some statistical analyses, e.g. auto-regressive modeling. +In this case, a simple strategy using motion parameters, average activity in select brain compartments, +and global signal regression should be preferred. +Importantly, we found that certain denoising strategies behave inconsistently across datasets and/or versions of fMRIPrep, +or had a different behavior than in previously published benchmarks. +These results demonstrate that a reproducible denoising benchmark can effectively assess the robustness of conclusions across multiple datasets and software versions. +In addition to reproducing core computations, interested readers can also reproduce or modify the figures of the article using the Jupyter Book project and the Neurolibre reproducible preprint server {cite:p}`neurolibre`. +With the denoising benchmark, we hope to provide useful guidelines for the community, and that our software infrastructure will facilitate continued development as the state-of-the-art advances. ![Overview of API.\label{top_level_fig}](./images/api_summary.png) diff --git a/content/references.bib b/content/references.bib index 315668f..f1c003c 100644 --- a/content/references.bib +++ b/content/references.bib @@ -1,12 +1,12 @@ @article{biswal_2010, -author = {Bharat B. Biswal and Maarten Mennes and Xi-Nian Zuo and Suril Gohel and Clare Kelly and Steve M. Smith and Christian F. Beckmann and Jonathan S. Adelstein and Randy L. Buckner and Stan Colcombe and Anne-Marie Dogonowski and Monique Ernst and Damien Fair and Michelle Hampson and Matthew J. Hoptman and James S. Hyde and Vesa J. Kiviniemi and Rolf Kötter and Shi-Jiang Li and Ching-Po Lin and Mark J. Lowe and Clare Mackay and David J. Madden and Kristoffer H. Madsen and Daniel S. Margulies and Helen S. Mayberg and Katie McMahon and Christopher S. Monk and Stewart H. Mostofsky and Bonnie J. Nagel and James J. Pekar and Scott J. Peltier and Steven E. Petersen and Valentin Riedl and Serge A. R. B. Rombouts and Bart Rypma and Bradley L. Schlaggar and Sein Schmidt and Rachael D. Seidler and Greg J. Siegle and Christian Sorg and Gao-Jun Teng and Juha Veijola and Arno Villringer and Martin Walter and Lihong Wang and Xu-Chu Weng and Susan Whitfield-Gabrieli and Peter Williamson and Christian Windischberger and Yu-Feng Zang and Hong-Ying Zhang and F. Xavier Castellanos and Michael P. Milham }, -title = {Toward discovery science of human brain function}, -journal = {Proceedings of the National Academy of Sciences}, -volume = {107}, -number = {10}, -pages = {4734-4739}, -year = {2010}, -doi = {10.1073/pnas.0911855107}, + author = {Bharat B. Biswal and Maarten Mennes and Xi-Nian Zuo and Suril Gohel and Clare Kelly and Steve M. Smith and Christian F. Beckmann and Jonathan S. Adelstein and Randy L. Buckner and Stan Colcombe and Anne-Marie Dogonowski and Monique Ernst and Damien Fair and Michelle Hampson and Matthew J. Hoptman and James S. Hyde and Vesa J. Kiviniemi and Rolf Kötter and Shi-Jiang Li and Ching-Po Lin and Mark J. Lowe and Clare Mackay and David J. Madden and Kristoffer H. Madsen and Daniel S. Margulies and Helen S. Mayberg and Katie McMahon and Christopher S. Monk and Stewart H. Mostofsky and Bonnie J. Nagel and James J. Pekar and Scott J. Peltier and Steven E. Petersen and Valentin Riedl and Serge A. R. B. Rombouts and Bart Rypma and Bradley L. Schlaggar and Sein Schmidt and Rachael D. Seidler and Greg J. Siegle and Christian Sorg and Gao-Jun Teng and Juha Veijola and Arno Villringer and Martin Walter and Lihong Wang and Xu-Chu Weng and Susan Whitfield-Gabrieli and Peter Williamson and Christian Windischberger and Yu-Feng Zang and Hong-Ying Zhang and F. Xavier Castellanos and Michael P. Milham }, + title = {Toward discovery science of human brain function}, + journal = {Proceedings of the National Academy of Sciences}, + volume = {107}, + number = {10}, + pages = {4734-4739}, + year = {2010}, + doi = {10.1073/pnas.0911855107}, } @article{sporns_organization_2004, diff --git a/content/research_report/benchmark_methods.md b/content/research_report/benchmark_methods.md deleted file mode 100644 index 8666dd4..0000000 --- a/content/research_report/benchmark_methods.md +++ /dev/null @@ -1,245 +0,0 @@ ---- -jupytext: - text_representation: - extension: .md - format_name: myst - format_version: 0.13 - jupytext_version: 1.14.1 -kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Benchmark methods - -## Datasets - -Dataset `ds000228` (N = 155) contains fMRI scans of participants watching a silent version of a Pixar animated movie "Partly Cloudy". -The dataset includes 33 adult subjects -(Age Mean(s.d.) = 24.8(5.3), range = 18--39; 20 female) -and 122 children subjects -(Age Mean(s.d.) = 6.7(2.3), range = 3.5--12.3; 64 female). -For more information on the dataset please refer to {cite:t}`richardson_development_2018`. - -```{code-cell} -:tags: [hide-input] - -import warnings - -warnings.filterwarnings('ignore') -import pandas as pd -from myst_nb import glue -from fmriprep_denoise.visualization import tables, utils - -path_root = utils.get_data_root() / "denoise-metrics" -fmriprep_version = 'fmriprep-20.2.1lts' - -desc = tables.lazy_demographic('ds000228', fmriprep_version, path_root) -desc = desc.style.set_table_attributes('style="font-size: 12px"') - -glue('ds000228_desc', desc) -``` - -Dataset `ds000030` includes multiple tasks collected from subjects with a variety of neuropsychiatric diagnosis , including ADHD, bipolar disorder, schizophrenia, and healthy controls. -The current analysis focused on the resting-state scans only. -Scans with an instrumental artifact (flagged under column `ghost_NoGhost` in `participants.tsv`) were excluded from the analysis pipeline. -Of 272 subjects, 212 entered the preprocessing stage. -Demographic information per condition can be found in {numref}`table-ds000030`. - -```{table} Demographic information of ds000030 -:name: table-ds000030 -| | Full sample | Healthy control | Schizophrenia | Bipolar disorder | ADHD | -|----------------:|------------:|----------------:|--------------:|-----------------:|------------:| -| N(female) | 212(98) | 106(54) | 30(8) | 41(19) | 35(17) | -| Age Mean(s.d.) | 33.2(9.3) | 31.8(8.9) | 37.2 (9.2) | 34.7 (8.9) | 32.5 (10.2) | -| Age Range | 21--50 | 21--50 | 22--49 | 21--50 | 21--50 | -``` - -```{code-cell} -:tags: [hide-input] - -desc = tables.lazy_demographic('ds000030', fmriprep_version, path_root) -desc = desc.style.set_table_attributes('style="font-size: 12px"') - -glue('ds000030_desc', desc) -``` - -## fMRI data preprocessing - -We preprocessed fMRI data using fMRIPrep LTS20.2.1 through [`fMRIPrep-slurm`](https://github.com/SIMEXP/fmriprep-slurm) with the following options: -``` ---use-aroma \ ---omp-nthreads 1 \ ---nprocs 1 \ ---output-spaces MNI152NLin2009cAsym MNI152NLin6Asym \ ---output-layout bids \ ---notrack \ ---skip_bids_validation \ ---write-graph \ ---resource-monitor -``` - -For the full description generated by fMRIPrep, please see [supplemental material](../supplementary_materials/CITATION.md). - -## Choice of atlases - -We extracted time series with regions of interest (ROI) defined by the following atlases: -Gordon atlas {cite:p}`gordon_atlas_2014`, -Schaefer 7 network atlas {cite:p}`schaefer_local-global_2017`, -Multiresolution Intrinsic Segmentation Template (MIST) {cite:p}`urchs_mist_2019`, -and Dictionary of Functional Modes (DiFuMo){cite:p}`difumo_2020`. -All atlases were resampled to the resolution of the preprocessed functional data. - -````{margin} -```{admonition} Warning message related to masker resampling -:class: note -[See source code](https://github.com/nilearn/nilearn/blob/d53169c6af1cbb3db3485c9480a3e7cb31c2537d/nilearn/maskers/nifti_labels_masker.py#L568-L573) -``` -```` - -Since DiFuMo and MIST atlases can include networks with disjointed regions under the same label, -we carried out further ROI extraction. -Labels are presented with the original number of parcels. -and we denote the number of extracted ROI in brackets. -Gordon and Schaefer atlas parcels use isolated ROI, -hence no further extraction was done. -The Schaefer 1000 parcels atlas was excluded since some regions would disappear after resampling the atlas to the shape of the processed fMRI data. - -- Gordon atlas: 333 -- Schaefer atlas: 100, 200, 300, 400, 500, 600, 800 -- Multiresolution Intrinsic Segmentation Template (MIST) {cite:p}`urchs_mist_2019`: 7, 12, 20, 36, 64, 122, 197, 325, 444, "ROI" (210 parcels, 122 split by the midline) -- DiFuMo atlas {cite:p}`difumo_2020`: 64 (114), 128 (200), 256 (372), 512 (637), 1024 (1158) - -Processes involved here are implemented through nilearn {cite:p}`nilearn`. -Time series were extracted using `nilearn.maskers.NiftiLabelsMasker` and `nilearn.maskers.NiftiMapsMasker`. -Connectomes were calculated using Pearson's Correlation, implemented through `nilearn.connectome.ConnectivityMeasure`. - -(framewise-displacement)= -## Participant exclusion based on motion - -We performed data quality control to exclude subjects with excessive motion leading to unusable data. -In the current report, we use framewise displacement as the metric to quantify motion. -Framewise displacement indexes the movement of the head from one volume to the next. -The movement includes the transitions on the three axes ($x$, $y$, $z$) and the respective rotation ($\alpha$, $\beta$, $\gamma$). -Rotational displacements are calculated as the displacement on the surface of a sphere of radius 50 mm {cite}`power_scrubbing_2012`. -fMRIPrep generates the dramewise displacement based on the formula proposed in {cite}`power_scrubbing_2012`. -The framewise displacement, denoted as FD, at each time point $t$ is expressed as: - -$$ -\text{FD}_t = |\Delta d_{x,t}| + |\Delta d_{y,t}| + -|\Delta d_{z,t}| + |\Delta \alpha_t| + |\Delta \beta_t| + |\Delta \gamma_t| -$$ - -```{code-cell} -:tags: [hide-input, remove-output] -from fmriprep_denoise.features.derivatives import get_qc_criteria - -stringent = get_qc_criteria('stringent') -glue('gross_fd', stringent['gross_fd']) -glue('fd_thresh', stringent['fd_thresh']) -glue('proportion_thresh', stringent['proportion_thresh'] * 100) -``` - -To ensure the analysis is performed in a realistic scenario we exclude subjects with high motion, -defined by the following criteria adopted from {cite:p}`parkes_evaluation_2018`: -mean framewise displacement > {glue:}`gross_fd` mm, -above {glue:}`proportion_thresh`% of volumes removed while scrubbing -with a {glue:}`fd_thresh` mm threshold. - -## Confound regression strategies - -Confound variables were retrieved using -(i) a basic API that retrieves different classes of confound regressors, -`nilearn.interfaces.fmriprep.load_confounds` (simplified as `load_confounds`); -and (ii) a higher level wrapper to implement common strategies from the denoising literature, -`nilearn.interfaces.fmriprep.load_confounds_strategy`(simplified as `load_confounds_strategy`). -The following section describes the logic behind the design of the API. -For documentation of the actual function, please see the latest version of [`nilearn` documentation](https://nilearn.github.io/stable/). - -We evaluated common confound regression strategies that are possible through fMRIPrep-generated confound regressors. -The connectome generated from high-pass filtered time series served as a baseline comparison. -Confound variables were accessed using the API `load_confounds_strategy`. -The detailed 11 strategies and a full breakdown of parameters used under the hood is presented in {numref}`table-examined_strategies`. - -:::{dropdown} Click to see {numref}`table-examined_strategies` - -```{table} Denoising strategies -:name: table-examined_strategies -| strategy | image | `high_pass` | `motion` | `wm_csf` | `global_signal` | `scrub` | `fd_thresh` | `compcor` | `n_compcor` | `ica_aroma` | `demean` | -|-----------------|--------------------------------|-------------|----------|----------|-----------------|---------|-------------|-----------------|-------------|-------------|----------| -| baseline | `desc-preproc_bold` | `True` | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | `True` | -| simple | `desc-preproc_bold` | `True` | `full` | `basic` | N/A | N/A | N/A | N/A | N/A | N/A | `True` | -| simple+gsr | `desc-preproc_bold` | `True` | `full` | `basic` | `basic` | N/A | N/A | N/A | N/A | N/A | `True` | -| scrubbing.5 | `desc-preproc_bold` | `True` | `full` | `full` | N/A | `5` | `0.5` | N/A | N/A | N/A | `True` | -| scrubbing.5+gsr | `desc-preproc_bold` | `True` | `full` | `full` | `basic` | `5` | `0.5` | N/A | N/A | N/A | `True` | -| scrubbing.2 | `desc-preproc_bold` | `True` | `full` | `full` | N/A | `5` | `0.2` | N/A | N/A | N/A | `True` | -| scrubbing.2+gsr | `desc-preproc_bold` | `True` | `full` | `full` | `basic` | `5` | `0.2` | N/A | N/A | N/A | `True` | -| compcor | `desc-preproc_bold` | `True` | `full` | N/A | N/A | N/A | N/A | `anat_combined` | `all` | N/A | `True` | -| compcor6 | `desc-preproc_bold` | `True` | `full` | N/A | N/A | N/A | N/A | `anat_combined` | `6 ` | N/A | `True` | -| aroma | `desc-smoothAROMAnonaggr_bold` | `True` | N/A | `basic` | N/A | N/A | N/A | N/A | N/A | `full` | `True` | -| aroma+gsr | `desc-smoothAROMAnonaggr_bold` | `True` | N/A | `basic` | `basic` | N/A | N/A | N/A | N/A | `full` | `True` | -``` -::: - -## Evaluation of the outcome of denoising strategies - -We used selected metrics described in the previous literature to evaluate the denoising results -{cite:p}`ciric_benchmarking_2017,parkes_evaluation_2018`. - -### Loss in temporal degrees of freedom - -The common analysis and denoising methods are based on linear regression. -Using more nuisance regressors can capture additional sources of noise-related variance in the data and thus improve denoising. -However, this comes at the expense of a loss of temporal degrees of freedom for statistical inference in further analysis. -This is an important point to consider alongside the denoising performance. -There are two factors constraining the degrees of freedom in the signal: -the number of volumes in a scan, -and the number of nuisance regressors used during denoising. -We calculate the number of regressors used for each strategy. -For the scrubbing-based strategy we further calculate the proportion of volume loss to number of volumes in a scan. - -### Quality control / functional connectivity (QC-FC) - -QC-FC {cite:p}`power_recent_2015` quantifies the correlation between mean framewise displacement and functional connectivity. -This is calculated by a partial correlation between mean framewise displacement and connectivity, -with age and sex as covariates. -The denoising methods should aim to reduce the QC-FC value. -Significance values were corrected for multiple comparisons using the false positive rate. - -### Distance-dependent effects of motion on connectivity - -To determine the residual distance-dependence of subject movement, -we first calculated the Euclidean distance between the centers of mass of each pair of parcels {cite:p}`power_scrubbing_2012`. -We then correlated the distance separating each pair of parcels and the associated QC-FC correlation of the edge connecting those parcels. -Closer parcels generally exhibit greater impact of motion on connectivity. -We expect to see a general trend of negative to zero correlation after confound regression. - -### Network modularity - -Confound regressors have the potential to remove real signals in addition to motion-related noise. -In order to evaluate this possibility, -we computed modularity quality, -an explicit quantification of the degree to which there are structured sub-networks in a given network - -in this case the de-noised connectome {cite:p}`satterthwaite_impact_2012`. -Modularity quality is quantified by graph community detection based on the Louvain method {cite:p}`rubinov2010`, -implemented in the Brain Connectome Toolbox. -If confound regression and censoring were removing real signals in addition to motion-related noise, -we would expect modularity to decline. -To understand the extent of correlation between modularity and motion, -we computed the partial correlation between subjects' modularity values and mean framewise displacement, -with age and sex as covariates. - -## Benchmark workflow - -{numref}`fig-denoise-benchmark-workflow` presents the graphic summary of the benchmark workflow. - - - -```{figure} ../images/fig2-benchmark.png ---- -height: 500px -name: fig-denoise-benchmark-workflow ---- -Workflow of generating the full denoising benchmark. -``` \ No newline at end of file diff --git a/content/research_report/contributions.md b/content/research_report/contributions.md index 50b4205..f897d32 100644 --- a/content/research_report/contributions.md +++ b/content/research_report/contributions.md @@ -21,7 +21,10 @@ We thank Chris Markiewicz[^6] for feedbacks related to fMRIPrep. Hao-Ting Wang and Pierre Bellec drafted the paper. Natasha Clarke[^1] revised and provided critical feed backs. -Please see the [original repository](https://github.com/SIMEXP/load_confounds#contributors-) for a full history of development and contributors. +All co-authors contributed to and approved the final version of the manuscript. + +Please see the [original repository](https://github.com/SIMEXP/load_confounds#contributors-) for a full history of development and contributors, +and this [issue](https://github.com/nilearn/nilearn/issues/2777) for a history of the integration in Nilearn and all the linked Pull Requests. [^1]: Centre de recherche de l'institut Universitaire de gériatrie de Montréal (CRIUGM), Montréal, Québec, Canada @@ -38,4 +41,10 @@ Please see the [original repository](https://github.com/SIMEXP/load_confounds#co [^7]: Psychology Department, Université de Montréal, Montréal, Québec, Canada ## Acknowledgments - + +The project is funded by IVADO PRF3, CCNA and J Courtois, the neuromind collaboration. +HTW and NC funded by Institut de valorisation des données (IVADO) postdoctoral research funding. +SLM was funded by the National Institute on Deafness and Other Communication Disorders (NIDCD; Grant 5T32DC000038). +CJM funded by NIMH 5R24MH117179. +FP funded by Courtois Neuromod. +PB funded by Fonds de Recherche du Québec - Santé (FRQ-S). diff --git a/content/research_report/results-atlas.md b/content/research_report/results-atlas.md index 27e7f79..41e0269 100644 --- a/content/research_report/results-atlas.md +++ b/content/research_report/results-atlas.md @@ -33,10 +33,10 @@ Please click on the launch button to lunch the Binder instance for interactive d In the report we used four atlases, three of them came with multiple parcellation schemes. -- Gordon atlas: 333 -- Schaefer 7 network atlas: 100, 200, 300, 400, 500, 600, 800 -- Multiresolution Intrinsic Segmentation Template (MIST): 7, 12, 20, 36, 64, 122, 197, 325, 444, “ROI” (210 parcels, 122 split by the midline) -- DiFuMo atlas: 64, 128, 256, 512, 1024 +- Gordon atlas {cite:p}`gordon_atlas_2014`: 333 +- Schaefer 7 network atlas {cite:p}`schaefer_local-global_2017`: 100, 200, 300, 400, 500, 600, 800 +- Multiresolution Intrinsic Segmentation Template (MIST) {cite:p}`urchs_mist_2019`: 7, 12, 20, 36, 64, 122, 197, 325, 444, “ROI” (210 parcels, 122 split by the midline) +- DiFuMo {cite:p}`difumo_2020`: 64, 128, 256, 512, 1024 ## Before we start: Loss of temporal degrees of freedom @@ -83,7 +83,7 @@ interactive( We can also plot them by each parcellation schemes. -This is the original way Ciric and colleagues (2017) presented their results! +This is the original way Ciric and colleagues {cite:p}`ciric_benchmarking_2017` presented their results! ### Gordon atlas diff --git a/content/research_report/results-group.md b/content/research_report/results-group.md index b4421c0..1ab14d2 100644 --- a/content/research_report/results-group.md +++ b/content/research_report/results-group.md @@ -37,21 +37,21 @@ datasets_baseline = {"ds000228": "adult", "ds000030": "control"} Here we provides alternative visualisation of the benchmark results from the manuscript. Please click on the launch button to lunch the Binder instance for interactive data viewing. -The benchmark was performed on two Long-Term Support (LTS) versions of fMRIPrep (`20.2.1` and `20.2.5`) and two OpenNeuro datasets (`ds000228` and `ds000030`). +The benchmark was performed on two Long-Term Support (LTS) versions of fMRIPrep (`20.2.1` and `20.2.5`) and two OpenNeuro datasets (`ds000228` {cite:p}`ds000228:1.1.0` and `ds000030` {cite:p}`ds000030:1.0.0`). For the demographic information and gross mean framewise displacement, it is possible to generate the report based on three levels of quality control filters (no filter, minimal, stringent). ## Sample and subgroup size change based on quality control criteria We would like to perform the benchmark on subjects with reasonable qulaity of data to reflect the decisions researchers make in data analysis. -We modified the criteria for filtering data from Parkes 2018 to suit our dataset better and ensure enough time points for functional connectivity analysis. +We modified the criteria for filtering data from {cite:p}`parkes_evaluation_2018` to suit our dataset better and ensure enough time points for functional connectivity analysis. The stringent threshold removes subjects based on two criteria: 1. removes subjects with mean framewise displacement above 0.25 mm 2. removes subjects with more than 80% of the volumes missing when filtering the time series with a 0.2 mm framewise displacement. -Parkes 2018 used a stricter criteria for remaining volumes (20%). However this will removed close to or more than 50% of the subjects from the datasets. +Parkes and colleagues {cite:p}`parkes_evaluation_2018` used a stricter criteria for remaining volumes (20%). However this will removed close to or more than 50% of the subjects from the datasets. -In addition, we included the minimal threshold from Parkes 2018 +In addition, we included the minimal threshold from {cite:p}`parkes_evaluation_2018` (removes subjects with mean framewise displacement above 0.55 mm) for readers to expore. @@ -143,7 +143,7 @@ interactive( We plotted the correlations among connectomes denoised with different denoisng strategies to get a general sense of the data. We see connectome denoised with or without global signal regressor formed two separate clusters. -The baseline and ICA-AROMA denoised connectome do not belong to any clusters. +The baseline and ICA-AROMA {cite:p}`aroma` denoised connectome do not belong to any clusters. ICA-AROMA potentially captures much more different source of noise than the others. ```{code-cell} @@ -208,7 +208,7 @@ interactive( ## Quality control / functional connectivity (QC-FC) -QC-FC (Power et al., 2015) quantifies the correlation between mean framewise displacement and functional connectivity. +QC-FC {cite:p}`power_recent_2015` quantifies the correlation between mean framewise displacement and functional connectivity. This is calculated by a partial correlation between mean framewise displacement and connectivity, with age and sex as covariates. The denoising methods should aim to reduce the QC-FC value. Significance tests associated with the partial correlations were performed, @@ -293,7 +293,7 @@ interactive( ## Residual distance-dependent effects of subject motion on functional connectivity (DM-FC) -To determine the residual distance-dependence of subject movement, we first calculated the Euclidean distance between the centers of mass of each pair of parcels (Power et al., 2012). +To determine the residual distance-dependence of subject movement, we first calculated the Euclidean distance between the centers of mass of each pair of parcels {cite:p}`power_scrubbing_2012`. Closer parcels generally exhibit greater impact of motion on connectivity. We then correlated the distance separating each pair of parcels and the associated QC-FC correlation of the edge connecting those parcels. We report the absolute correlation values and expect to see a general trend toward zero correlation after confound regression. @@ -325,9 +325,9 @@ interactive( Confound regressors have the potential to remove real signals in addition to motion-related noise. In order to evaluate this possibility, we computed modularity quality, -an explicit quantification of the degree to which there are structured subnetworks in a given network - in this case the denoised connectome (Satterthwaite et al., 2012). -Modularity quality is quantified by graph community detection based on the Louvain method (Rubinov & Sporns, 2010), -implemented in the Brain Connectivity Toolbox (Rubinov & Sporns, 2010). +an explicit quantification of the degree to which there are structured subnetworks in a given network - in this case the denoised connectome {cite:p}`satterthwaite_impact_2012`. +Modularity quality is quantified by graph community detection based on the Louvain method {cite:p}`rubinov2010`, +implemented in the Brain Connectivity Toolbox {cite:p}`rubinov2010`. ```{code-cell} from fmriprep_denoise.visualization import motion_metrics diff --git a/content/research_report/software_implemetation.md b/content/research_report/software_implemetation.md deleted file mode 100644 index b05cdc8..0000000 --- a/content/research_report/software_implemetation.md +++ /dev/null @@ -1,109 +0,0 @@ -# Software implementation - -fMRIprep {cite:p}`fmriprep1` is a popular minimal preprocessing software for functional MRI data. -'Minimal preprocessing' refers to motion correction, field unwarping, normalization, bias field correction, and brain extraction. -Denoising and smoothing are excluded from the workflow. -Instead, fMRIprep provides users with a large set of potential confound regressors that covers many denoising strategies. -The users have to select the confound regressors for denoising in subsequent analyses. -Our aim was to firstly, provide a easy and foolproof API for users to perform subsequent denoising of fMRIprep outputs. -The loaded format is compatible with `nilearn` analysis functions such as `NiftiMasker` and the GLM modules. -Secondly we will show how to generate denoised functional connectivity matrix from fMRIPrep outputs with nilearn. - -## Retrieve fMRIPrep confounds output - -To enable easy confound variable loading from fMRIPrep outputs, -confound variables were retrieved using -(i) a basic API that retrieves different classes of confound regressors, -`nilearn.interfaces.fmriprep.load_confounds` (simplified as `load_confounds`); -and (ii) a higher level wrapper to implement common strategies from the denoising literature, -`nilearn.interfaces.fmriprep.load_confounds_strategy` (simplified as `load_confounds_strategy`). - -The following section describes the logic behind the design of the API. -For documentation of the actual function, please see the latest version of [`nilearn` documentation](https://nilearn.github.io/stable/). - -### Basic noise components - -`load_confounds` provides an interface that groups subsets of confound variables into noise components and their parameters. -It is possible to fine-tune these subsets through the parameters provided. -The implementation will only support fMRIPrep 1.4.x series or above. -User has to keep the outputted functional derivatives directory unchanged. -User gains the maximum amount of freedoms from using this interface, -and some minimal constraints were implemented based on fMRIPrep documentations. -Warnings and errors will inform the user if the selected confounds were missing or not suited based on their requests. -The function returns the confound variables and a sample mask per input image. -The sample mask filters out si -The details of each parameter are described in the following section. - -Two types of regressors are always loaded with no additional parameters for user customisation: - -- `high_pass`: discrete cosine transformation basis regressors to handle low-frequency signal drifts. -- `non_steady_state` denotes volumes collected before the fMRI scanner had reached a stable state. - -`motion`, `wm_csf`, and `global_signal` share similar expansion options: - -- `motion`: head motion estimates translation/rotation (6 parameters). -- `wm_csf`: average signal extracted from masks of white matter and cerebrospinal fluid (2 parameters). -- `global_signal`: average signal extracted from brain mask (1 parameters). - -For the three parameters above, the user can select from the following four options: -- `basic`: original signal only (n parameter) -- `power2`: original signal + quadratic term (2 * n parameters) -- `derivatives`: original signal + temporal derivative (2 * n parameters) -- `full`: original signal + temporal derivatives + quadratic terms + quadratic terms temporal derivatives (4 * n parameters) - -`scrub` generates a mask to exclude volumes with excessive motion {cite:p}`power_scrubbing_2012`. -Three types of parameters can be used to determine volumes to be excluded: -- `fd_threshold`: set the head motion cut-off value determined by framewise displacement approach {cite:p}`power_scrubbing_2012`. -- `std_dvars_threshold`: set the head motion cut-off value determined by the standard deviation of root mean square approach {cite:p}`power_scrubbing_2012,jenkinson_2002`. -- `scrub`: after accounting for time frames with excessive motion, further remove segments shorter than the given number assigned. - -The CompCor {cite:p}`behzadi_compcor_2007` approach has two associated parameters: -- `compcor` allows users to select components generated by the temporal approach, - or the anatomical approach with specific details for the mask used in noise signal extraction. -- `n_compcor` states the number of principal components to retrieve. - -For the ICA-based approach, fMRIPrep implemented ICA-AROMA {cite:p}`aroma`. -Users must manually enable ICA-AROMA with flag `--use-aroma` when using fMRIPrep. -The parameter `ica_aroma` allows two approaches: -- `full`: Use fMRIPrep output with suffix `desc-smoothAROMAnonaggr_bold.nii.gz`. This is the standard and recommanded approach. -- `basic`: Use noise independent components only. Must be used with output with suffix `desc-preproc_bold.nii.gz`. - -### Pre-defined strategies - -`load_confounds_strategy` provides an interface to select currated confounds based on past literature, -with limited parameters for user customisation. -There are four possible strategies implementable from fMRIPrep confounds: -`simple` {cite:p}`fox_pnas_2005` (motion parameters and tissue signal), -`scrubbing` {cite:p}`power_scrubbing_2012`(volume censoring, motion parameters, and tissue signal), -`compcor` {cite:p}`behzadi_compcor_2007`(anatomical compcor and motion parameters), -and `aroma` {cite:p}`aroma`(ICA-AROMA based denoising and tissue signal). -All strategies, except `compcor`, provide an option to add global signal to the confound regressors. -The predefined strategies with their associated noise component is listed in {numref}`table-predefined_strategies`. -Parameters with `*` denote customisable parameters. - -:::{dropdown} Click to see {numref}`table-predefined_strategies` - -```{table} Denoising strategies -:name: table-predefined_strategies -| strategy | image | `high_pass` | `motion` | `wm_csf` | `global_signal` | `scrub` | `fd_thresh` | `std_dvars_threshold` | `compcor` | `n_compcor` | `ica_aroma` | `demean` | -|-------------|--------------------------------|-------------|----------|----------|-----------------|---------|-------------| --------------------- |------------------|-------------|-------------|----------| -| `simple` | `desc-preproc_bold` | `True` | `full`* | `basic`* | `None`* | N/A | N/A | N/A | N/A | N/A | N/A | `True`* | -| `scrubbing` | `desc-preproc_bold` | `True` | `full`* | `full` | N/A | `5` | `0.5` | `3`* | N/A | N/A | N/A | `True`* | -| `compcor` | `desc-preproc_bold` | `True` | `full`* | N/A | `None`* | N/A | N/A | N/A | `anat_combined`* | `all`* | N/A | `True`* | -| `aroma` | `desc-smoothAROMAnonaggr_bold` | `True` | N/A | `basic`* | `None`* | N/A | N/A | N/A | N/A | N/A | `full` | `True`* | -``` -::: - -## Denoising workflow - -The denoising workflow is implemented through `nilearn`. -{numref}`fig-fmriprep-nilearn-denoise` presents the graphic summary of the workflow. - - -```{figure} ../images/fig-1-masker.png ---- -height: 500px -name: fig-fmriprep-nilearn-denoise ---- -Example workflow of extracting denoised timeseries and functional connectomes from fMRIPrep outputs using nilearn. -``` \ No newline at end of file diff --git a/paper.bib b/paper.bib index 12021a9..ea960ac 100644 --- a/paper.bib +++ b/paper.bib @@ -17,18 +17,6 @@ @article{nilearn year = {2014} } -@article{bidsapp, - title = {{BIDS} apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods}, - volume = {13}, - issn = {1553-7358}, - doi = {10.1371/journal.pcbi.1005209}, - pages = {e1005209}, - number = {3}, - journal = {{PLOS} Computational Biology}, - author = {Gorgolewski, Krzysztof J. and Alfaro-Almagro, Fidel and Auer, Tibor and Bellec, Pierre and Capotă, Mihai and Chakravarty, M. Mallar and Churchill, Nathan W. and Cohen, Alexander Li and Craddock, R. Cameron and Devenyi, Gabriel A. and Eklund, Anders and Esteban, Oscar and Flandin, Guillaume and Ghosh, Satrajit S. and Guntupalli, J. Swaroop and Jenkinson, Mark and Keshavan, Anisha and Kiar, Gregory and Liem, Franziskus and Raamana, Pradeep Reddy and Raffelt, David and Steele, Christopher J. and Quirion, Pierre-Olivier and Smith, Robert E. and Strother, Stephen C. and Varoquaux, Gaël and Wang, Yida and Yarkoni, Tal and Poldrack, Russell A.}, - year = {2017} -} - @article{jupyter, title = {Jupyter: Thinking and Storytelling With Code and Data}, volume = {23}, diff --git a/paper.md b/paper.md index fa5c5a7..0edd095 100644 --- a/paper.md +++ b/paper.md @@ -50,17 +50,17 @@ affiliations: index: 6 - name: Psychology Department, Université de Montréal, Montréal, Québec, Canada index: 7 -date: 17 April 2023 +date: 11 May 2023 bibliography: paper.bib --- # Summary -Reducing contributions from non-neuronal sources is a crucial step in functional magnetic resonance imaging (fMRI) analyses. +Reducing contributions from non-neuronal sources is a crucial step in functional magnetic resonance imaging (fMRI) connectivity analyses. Many viable strategies for denoising fMRI are used in the literature, and practitioners rely on denoising benchmarks for guidance in the selection of an appropriate choice for their study. However, fMRI denoising software is an ever-evolving field, and the benchmarks can quickly become obsolete as the techniques or implementations change. -In this work, we present a fully reproducible denoising benchmark featuring a range of denoising strategies and evaluation metrics, +In this work, we present a fully reproducible denoising benchmark featuring a range of denoising strategies and evaluation metrics for connectivity analyses, built primarily on the fMRIPrep [@fmriprep1] and Nilearn [@nilearn] software packages. We apply this reproducible benchmark to investigate the robustness of the conclusions across two different datasets and two versions of fMRIPrep. The majority of benchmark results were consistent with prior literature. @@ -70,10 +70,9 @@ Scrubbing however disrupts the continuous sampling of brain images and is incomp e.g. auto-regressive modeling. In this case, a simple strategy using motion parameters, average activity in select brain compartments, and global signal regression should be preferred. Importantly, we found that certain denoising strategies behave inconsistently across datasets and/or versions of fMRIPrep, -or had a different behavior than in previously published benchmarks, especially ICA-AROMA. +or had a different behavior than in previously published benchmarks. These results demonstrate that a reproducible denoising benchmark can effectively assess the robustness of conclusions across multiple datasets and software versions. -Technologies such as BIDS-App [@bidsapp], the Jupyter Book [@jupyter] and Neurolibre [@neurolibre] provided the infrastructure to publish the metadata and report figures. -Readers can reproduce the report figures beyond the ones reported in the published manuscript. +In addition to reproducing core computations, interested readers can also reproduce or modify the figures of the article using the Jupyter Book project [@jupyter] and the Neurolibre [@neurolibre] reproducible preprint server. With the denoising benchmark, we hope to provide useful guidelines for the community, and that our software infrastructure will facilitate continued development as the state-of-the-art advances. @@ -92,6 +91,11 @@ Hao-Ting Wang and Pierre Bellec drafted the initial version of the paper, with c Please see the original repository for a history of initial development and [contributors](https://github.com/SIMEXP/load_confounds#contributors-), and this [issue](https://github.com/nilearn/nilearn/issues/2777 ) for a history of the integration in Nilearn and all the linked Pull Requests. - +The project is funded by IVADO PRF3, CCNA and J Courtois, the neuromind collaboration. +HTW and NC funded by Institut de valorisation des données (IVADO) postdoctoral research funding. +SLM was funded by the National Institute on Deafness and Other Communication Disorders (NIDCD; Grant 5T32DC000038). +CJM funded by NIMH 5R24MH117179. +FP funded by Courtois Neuromod. +PB funded by Fonds de Recherche du Québec - Santé (FRQ-S). # References \ No newline at end of file