Finding datasets that naturally feature missing sequences (where patients were simply not scanned with the full protocol) is rare because public repositories usually curate data to ensure completeness. However, several high-profile datasets are specifically designed or widely used to address the problem of missing modality imputation and robust segmentation.
These are categorized into datasets with explicit challenges for synthesis (simulated missingness) and real-world clinical datasets (inherent missingness).
1. The Gold Standard: BraTS (Brain Tumor Segmentation)1
This is the most widely used resource for missing sequence research. While the core dataset is complete (containing T1, T1ce, T2, and FLAIR), it is the standard benchmark for simulating missing data.
- BraTS Synthesis (BraSyn) Challenge: Since 2023, the BraTS challenge has included a specific track called BraSyn.2
- The Data: They provide subjects with all 4 modalities (T1, T1ce, T2, FLAIR) for training.
- The Task: In the validation/test sets, one or more modalities are intentionally “missing” (withheld), and you must synthesize the missing volume or perform segmentation without it.
- Access: Available via the Synapse.org BraTS portal.
- Standard BraTS Usage: Even outside the specific challenge, most “missing modality” papers (e.g., HeMIS, U-HVED) use the standard BraTS dataset. They artificially “drop” channels during training/testing to prove their models can handle incomplete data.
2. Real-World Clinical Missingness: ADNI
The Alzheimer’s Disease Neuroimaging Initiative (ADNI) is a massive longitudinal study. Unlike BraTS, the missing data here is often real and unintentional (due to patient dropout, protocol changes over decades, or different scanner capabilities).
- Nature of Missingness:
- Longitudinal Gaps: A patient might have a T1 and FLAIR at Month 0, but only a T1 at Month 6.
- Protocol Variations: Some patients have advanced sequences (like DTI or ASL) while others do not.
- Use Case: This is the primary dataset used for longitudinal imputation (predicting a future missing scan) or multi-modal imputation (filling in missing biomarkers).
- Access: Requires an application to ADNI LONI.
3. Healthy Control Synthesis: IXI Dataset
The IXI (Information eXtraction from Images) dataset is the standard playground for proving “Image-to-Image Translation” concepts (e.g., GANs, Diffusion Models).
- The Data: Contains T1, T2, and PD (Proton Density) images, plus some MRA (Angiography) and DTI from ~600 healthy subjects.
- Why it’s used: It is cleaner than tumor datasets. Researchers assume one modality is “missing” (e.g., “Input: T1, Target: T2”) to test synthesis algorithms.
- Access: Openly available without registration at Brain Development.org.
4. Other Notable Datasets
- ISLES (Ischemic Stroke Lesion Segmentation): Stroke imaging protocols are notoriously inconsistent in clinical practice. ISLES datasets often contain Perfusion (PWI) and Diffusion (DWI) maps. Older iterations (like ISLES 2015/2018) are frequently used to test models when perfusion maps are “missing.”
- MSSEG (Multiple Sclerosis Segmentation): Many MS datasets (like the MICCAI 2016 challenge) aggregate data from different centers. Due to multi-site variance, they are excellent for testing models that must be robust to missing or domain-shifted FLAIR sequences.
- OASIS (Open Access Series of Imaging Studies): Similar to ADNI but open access. OASIS-3 contains longitudinal MRI (T1, T2, FLAIR) and PET data where not every subject has every modality at every time point.
Summary Recommendation
| If you need… | Use this Dataset |
| A standardized benchmark to compare your method against others | BraTS (BraSyn Challenge) |
| Real clinical messiness (inconsistent protocols over time) | ADNI or OASIS-3 |
| Healthy anatomy to test pure synthesis (T1 $\rightarrow$ T2) | IXI |
| Stroke-specific missing perfusion maps | ISLES |
To simulate missing MRI sequences (modalities) on the BraTS dataset, you generally have two options: using specific challenge scripts (from the BraSyn challenge) or implementing on-the-fly augmentation (Modality Dropout) using libraries like MONAI or TorchIO.
Here are the best ways to do this in Python.
1. The “Official” Method: BraSyn Toolkit
The BraTS BraSyn (Brain Tumor Synthesis) challenge released specific scripts to generate missing modality data. This is best if you want to create a static dataset where files are physically “missing” or replaced with zeros to benchmark against others.
- Repository: BraSyn 2024 Tutorial / Challenge Code
- Key Script: They provide a script called
drop_modality.py(or similar variants in their repo) that takes the 4-channel BraTS data and randomly drops one.
Simplified Logic (what their script does):
Python
import numpy as np
import nibabel as nib
import random
def drop_random_modality(image_path, output_path):
# Load 4D BraTS image (240, 240, 155, 4) -> (T1, T1ce, T2, FLAIR)
img_obj = nib.load(image_path)
data = img_obj.get_fdata()
# Pick a random channel to drop (0=T1, 1=T1ce, 2=T2, 3=FLAIR)
# BraSyn often ensures at least 1 modality remains
missing_idx = random.randint(0, 3)
# Zero out the channel (common strategy)
data[..., missing_idx] = 0
# OR: Delete the channel entirely if your model handles variable input sizes
# data = np.delete(data, missing_idx, axis=3)
new_img = nib.Nifti1Image(data, img_obj.affine, img_obj.header)
nib.save(new_img, output_path)
print(f"Dropped channel {missing_idx} for {image_path}")
2. The “On-the-Fly” Method: MONAI (Recommended)
If you are training a model, you shouldn’t save new files to disk. Instead, simulate missingness dynamically during training using MONAI. This technique is often called “Modality Dropout.”
Library: [suspicious link removed]
MONAI has a powerful transform system. You can use RandLambda to zero out channels.
Python
from monai.transforms import (
Compose,
LoadImaged,
EnsureChannelFirstd,
RandLambdad
)
import numpy as np
# Define a function to randomly zero out one channel
def zero_out_channel(x):
# x shape is (C, H, W, D) -> (4, 240, 240, 155)
# Create a copy to avoid modifying original if cached
x = x.clone()
# Randomly pick a channel to drop
idx = np.random.randint(0, 4)
x[idx, ...] = 0
return x
# Create the transform pipeline
train_transforms = Compose([
LoadImaged(keys=["image", "label"]),
EnsureChannelFirstd(keys=["image"]),
# ... other transforms (Resample, Normalize) ...
# Apply Modality Dropout with 50% probability
RandLambdad(keys=["image"], func=zero_out_channel, prob=0.5)
])
# Now use this in your Dataset/DataLoader
3. The TorchIO Method
TorchIO is excellent for 3D MRI augmentation. While it focuses on physics-based augmentation (motion, bias field), you can inject a custom “Lambda” transform for missing data.
Python
import torchio as tio
import torch
import random
def modality_dropout(tensor):
# tensor shape is (C, H, W, D)
if random.random() < 0.5: # 50% chance to drop a channel
idx = random.randint(0, 3)
tensor[idx] = 0
return tensor
# Define the transform
transforms = [
tio.Lambda(modality_dropout), # Custom missing modality
tio.RandomAffine(), # Standard augmentation
]
transform = tio.Compose(transforms)
# Apply to a subject
subject = tio.Subject(
mri=tio.ScalarImage('brats_t1_t1ce_t2_flair.nii.gz'),
)
transformed_subject = transform(subject)
Summary of Approaches
| Approach | Best For… |
| BraSyn Scripts | Creating a static benchmark dataset to compare exactly with other papers. |
| MONAI | Training Deep Learning models (U-Net/Transformers). It integrates seamlessly into the training loop and GPU pipeline. |
| TorchIO | If you are already using it for complex augmentations (like simulation of motion artifacts) and want to add missing modalities to the stack. |