MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction

Paper | Code | Dataset

Dataset

Brain Regions

Abstract

MTNeuro introduces a new dataset, annotations, and multiple downstream tasks that provide diverse ways to readout both macroscopic and microscopic brain architectural information from the same image. This multi-task neuroimaging benchmark is built on volumetric, micrometer-resolution X-ray microtomography imaging of a large thalamocortical section of mouse brain, which encompasses multiple cortical and subcortical regions and reveals dense reconstructions of the underlying microstructure (i.e., cell bodies, vasculature, and axons). We generated a number of different prediction challenges and evaluated several supervised and self-supervised models for brain-region prediction and pixel-level semantic microstructural segmentation. Our experiments not only highlight the rich heterogeneity of this dataset, but also provide insights into how self-supervised approaches can be used to learn representations that capture multiple attributes of a single image and perform well on a variety of downstream tasks.

Key Features

Three Dimensional Multi-Scale Annotated Dataset: The 3D x-ray microtomography dataset spans multiple brain areas and includes region of interest (ROI) annotations, densely annotated 3D cutouts, and semantic interpretable features.
Multi-Level Benchmark Tasks: Benchmark tasks feature both microscopic and macroscopic classification objectives.
Evaluation of Model Baselines: Both 2D and 3D training regimes are considered when training supervised and unsupervised models.

Dataset

We built our benchmark tasks on a large open access high-resolution (1.17μm isotropic) 3D microscopy database, which contains macroscopic-level ROI annotations, as well as 4 three-dimensional 256 × 256 × 360 cutouts from the somatosensory cortex (CTX), striatum (STR), ventral posterior region of thalamus (VP), and the zona incerta (ZI). These volumetric cutouts contain pixel-level microstuctural labels, identifying each point as either part of an axon, cell, blood vessel, or background.

The dataset and all corresponding labels are stored publicly in BossDB and accessed on-demand with Intern, a Python API library.

Provided are a PyTorch DataLoader for convenient algorithm development and testing, and example Jupyter Notebooks to assist with downloading the task cutouts in other frameworks.

Intern and DataLoader

Benchmark Tasks

Tasks

Task	Description
1. Image-Level Classification of Brain Area	Prediction of the brain region (somatosensory cortex, striatum, thalamus, zona incerta) to which a given image or volume belongs.
2. Pixel-level Segmentation of Microstructures	Prediction of neural microstructures (blood vessels, axons, cell bodies, background) from pixel-level annotations within the four core brain regions contained in the dataset.
3. Multi-task Decoding from Frozen Representations	Estimation of human-interpretable semantic features (such as the average cell size or axon density) from the representation of a given image or volume.

An overview of the three benchmark tasks. Click each row for more information.

Task 1: Image-Level Classification of Brain Area

Goal: Classification of 4 brain regions:

ROI-C1: prediction of the origin brain region of a sample within a same subvolume
ROI-C2: evaluation of generalization of the prediction with different test subvolumes
ROI-C3: evaluation of performance of prediction when trained with more subvolumes

Classes: Somatosensory Cortex (CTX), Striatum (STR), Thalamus (VP) and Zona Incerta (ZI)
Training sets:

ROI-C1: First 300 slices from sub-volume A (marked '1' in multi-task illustration) from each brain region.
ROI-C2: First 300 slices from sub-volume A from each brain region.
ROI-C3: First 300 slices from sub-volume A and all 360 slices from sub-volume B (marked '2') from each brain region.

Testing sets:

ROI-C1: Last 50 slices from sub-volume A from each brain region.
ROI-C2: All 360 slices from sub-volumes C and D (marked '3' and '4' in multi-task illustration) from each brain region.
ROI-C3: All 360 slices from sub-volumes C and D from each brain region.

Modalities:

3D classification: train with volumetric samples
2D classification: train with image samples (the third dimension corresponds to the sample index)

Get started with executing Task 1 here: [Jupyter Notebook]

Task 2: Pixel-level Segmentation of Microstructures

Goal: Segmentation of 3 or 4 anatomical microstructures at the level of pixels
Classes:

3 class segmentation: cell bodies, blood vessels, and axons + background
4 class segmentation: cell bodies, blood vessels, axons, and background

Training set: 256x256x300 sub-volumes of the 4 brain regions (3 regions (CTX, STR, VP) in the case of the 4 class segmentation)
Testing set: 256x256x50 sub-volumes of the 4 brain regions (3 regions (CTX, STR, VP) in the case of the 4 class segmentation)
Modality: 2D classification: train with image samples (the third dimension corresponds to the sample index)

Get started with executing Task 2 here: [Jupyter Notebook]

Task 3: Multi-task Decoding from Frozen Representations

Goal: Learn representations to observe how well they encode semantic properties of volumes or how relevant these features are for a specific downstream task
Semantic properties examples: blood vessels pixels, cell counts, axon density and average distance between cells
Modality: Exploration of readout of contextual features from the latent space learned by self-supervised models in Task 1

Get started with executing Task 3 here: [Jupyter Notebook]

Visualization of the representations learned by models trained in Task 1. Global semantic attributes for each image are visualized as different colors.

Evaluation of Model Baselines

Task 1: Image-Level Classification of Brain Area

	ROI - C1	ROI - C2	ROI - C3
Supervised Sup w/ Mixup	0.88 ± 0.03 0.90 ± 0.04	0.77 ± 0.03 0.78 ± 0.03	0.88 ± 0.02 0.90 ± 0.02
BYOL MYOW MYOW-m	0.88 ± 0.02 0.90 ± 0.02 0.94 ± 0.02	0.76 ± 0.02 0.78 ± 0.05 0.78 ± 0.03	0.97 ± 0.01 0.98 ± 0.01 0.98 ± 0.01
PCA NMF	0.59 0.62	0.25 0.27	0.07 0.50

Get started with executing Task 1 here: [Jupyter Notebook]

Task 2: Pixel-level Segmentation of Microstructures

2D Pixel-level Segmentation
	3-Class				4-Class without ZI
Method	Bg + Axons	Vessels	Cells	Avg.	Bg	Vessels	Cells	Axons	Avg.
2D U-Net (F1) 2D U-Net (IoU)	0.99 0.98	0.76 0.64	0.85 0.75	0.87 ± 0.012 0.79 ± 0.014	0.97 0.89	0.82 0.70	0.87 0.77	0.94 0.60	0.90 ± 0.003 0.74 ± 0.008
MAnet (F1) MAnet (IoU)	0.99 0.98	0.79 0.68	0.87 0.78	0.88 ± 0.003 0.81 ± 0.003	0.97 0.89	0.83 0.71	0.87 0.78	0.94 0.76	0.90 ± 0.002 0.78 ± 0.011
FPN (F1) FPN (IoU)	0.99 0.97	0.72 0.59	0.84 0.73	0.85 ± 0.01 0.76 ± 0.015	0.96 0.87	0.73 0.59	0.84 0.72	0.93 0.72	0.86 ± 0.004 0.72 ± 0.021
Unet++ (F1) Unet++ (IoU)	0.99 0.98	0.79 0.68	0.87 0.78	0.89 ± 0.002 0.81 ± 0.002	0.97 0.88	0.81 0.68	0.85 0.75	0.93 0.73	0.89 ± 0.015 0.76 ± 0.036
PAN (F1) PAN (IoU)	0.98 0.96	0.60 0.46	0.80 0.66	0.79 ± 0.035 0.69 ± 0.039	0.95 0.85	0.69 0.53	0.80 0.67	0.93 0.76	0.84 ± 0.007 0.70 ± 0.014
PSPNet (F1) PSPNet (IoU)	0.97 0.94	0.48 0.39	0.74 0.61	0.73 ± 0.013 0.65 ± 0.043	0.94 0.82	0.54 0.38	0.71 0.55	0.91 0.74	0.78 ± 0.012 0.62 ± 0.015

3D Pixel-level Segmentation
	3-Class				4-Class without ZI
Method	Bg + Axons	Vessels	Cells	Avg.	Bg	Vessels	Cells	Axons	Avg.
3D U-Net (F1) 3D U-Net (IoU)	0.99 0.98	0.77 0.65	0.87 0.76	0.88 ± 0.006 0.80 ± 0.007	0.93 0.81	0.76 0.62	0.80 0.67	0.87 0.50	0.84 ± 0.032 0.65 ± 0.045
VNetLight (F1) VNetLight (IoU)	0.99 0.97	0.75 0.61	0.83 0.70	0.85 ± 0.012 0.76 ± 0.013	0.90 0.78	0.65 0.46	0.73 0.58	0.76 0.43	0.76 ± 0.063 0.56 ± 0.061
HighResNet (F1) HighResNet (IoU)	0.99 0.97	0.74 0.61	0.84 0.72	0.85 ± 0.019 0.77 ± 0.026	0.89 0.73	0.51 0.35	0.73 0.58	0.77 0.42	0.72 ± 0.083 0.52 ± 0.075

Get started with executing Task 2 here: [Jupyter Notebook]

Task 3: Multi-task Decoding from Frozen Representations

Linear Readouts from Models Trained on a Single Subvolume (ROI-C1)
Method	Vessels	Axons	Cell Count	Cell Size	Dist (k=1)
Supervised Sup w/ Mixup	0.77 ± 0.06 0.82 ± 0.02	0.94 ± 0.01 0.95 ± 0.00	0.67 ± 0.06 0.71 ± 0.02	0.61 ± 0.05 0.67 ± 0.03	0.48 ± 0.05 0.47 ± 0.02
BYOL MYOW MYOW-m	0.85 ± 0.01 0.85 ± 0.01 0.87 ± 0.01	0.94 ± 0.01 0.94 ± 0.01 0.95 ± 0.01	0.75 ± 0.01 0.74 ± 0.01 0.77 ± 0.01	0.69 ± 0.01 0.69 ± 0.01 0.69 ± 0.01	0.49 ± 0.01 0.50 ± 0.02 0.51 ± 0.01
PCA NMF	0.75 0.81	0.82 0.85	0.55 0.59	0.47 0.55	0.31 0.34

Linear Readouts from Models Trained on Two Subvolumes (ROI-C3)
Method	Vessels	Axons	Cell Count	Cell Size	Dist (k=1)
Supervised Sup w/ Mixup	0.79 ± 0.02 0.75 ± 0.04	0.94 ± 0.02 0.88 ± 0.04	0.73 ± 0.02 0.64 ± 0.04	0.63 ± 0.04 0.54 ± 0.07	0.49 ± 0.02 0.37 ± 0.05
BYOL MYOW MYOW-m	0.88 ± 0.00 0.88 ± 0.01 0.87 ± 0.01	0.96 ± 0.00 0.96 ± 0.00 0.96 ± 0.01	0.79 ± 0.00 0.79 ± 0.01 0.78 ± 0.01	0.73 ± 0.01 0.72 ± 0.01 0.72 ± 0.01	0.53 ± 0.02 0.52 ± 0.01 0.53 ± 0.01
PCA NMF	0.75 0.75	0.82 0.83	0.53 0.56	0.46 0.49	0.29 0.31

Get started with executing Task 3 here: [Jupyter Notebook]

Citation

If you find this dataset or benchmark useful in your research, please cite the following papers:


            Quesada, J., Sathidevi, L., Liu, R., Ahad, N., Jackson, J.M., Azabou, M., ... & Dyer, E. L. (2022). 
            MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction. 
            Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.


            Prasad, J. A., Balwani, A. H., Johnson, E. C., Miano, J. D., Sampathkumar, V., De Andrade, V., ... & Dyer, E. L. (2020). 
            A three-dimensional thalamocortical dataset for characterizing brain heterogeneity. Scientific Data, 7(1), 1-7.

License

This software is available under the MIT License.
The X-ray Microtomography image dataset is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).

Neural Data Science Lab
Georgia Institute of Technology

Contributors:
Jorge Quesada, Lakshmi Sathidevi, Ran Liu, Nauman Ahad, Joy M. Jackson, Mehdi Azabou, Jingyun Xiao, Christopher Liding, Carolina Urzay, Eva L. Dyer

Brain Observatory Storage Service & Database
Johns Hopkins University Applied Physics Laboratory

Contributors:
William Gray-Roncal
Erik C. Johnson

For inquiries, contact Eva L. Dyer (evadyer@gatech.edu), Erik C. Johnson (erik.c.johnson@jhuapl.edu), or Jorge Quesada (jpacora3@gatech.edu).

Website hosted on Github. Updated August 2022.

MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction

Paper | Code | Dataset

Abstract

Key Features

Dataset

Benchmark Tasks

Task 1: Image-Level Classification of Brain Area

Task 2: Pixel-level Segmentation of Microstructures

Task 3: Multi-task Decoding from Frozen Representations

Evaluation of Model Baselines

Task 1: Image-Level Classification of Brain Area

Task 2: Pixel-level Segmentation of Microstructures

2D Pixel-level Segmentation

3D Pixel-level Segmentation

Task 3: Multi-task Decoding from Frozen Representations

Linear Readouts from Models Trained on a Single Subvolume (ROI-C1)

Linear Readouts from Models Trained on Two Subvolumes (ROI-C3)

Citation

License

Neural Data Science LabGeorgia Institute of Technology

Brain Observatory Storage Service & DatabaseJohns Hopkins University Applied Physics Laboratory

Neural Data Science Lab
Georgia Institute of Technology

Brain Observatory Storage Service & Database
Johns Hopkins University Applied Physics Laboratory