MEMOIR-VLM: a multimodal vision-language model for Alzheimer's disease classification and question answering
Sohail Gidwani
, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Paul M. Thompson, and the Alzheimer's Disease Neuroimaging Initiative
Published 1 Oct 2026Frontiers in Computational Neuroscience
doi: 10.3389/fncom.2026.1902258Cite
2,363
ADNI subjects
68.2%
3-class Bal. Acc.
91.3%
CN vs Dem Bal. Acc.
78.7%
OASIS-3 zero-shot Bal. Acc.
Real Alzheimer's patients rarely have every scan, so MEMOIR-VLM reads whatever a patient has: MRI, diffusion MRI and clinical scores. One model predicts diagnosis, severity, age and sex, even with a scan missing. Clinical scores carry most of the diagnosis: diffusion MRI adds a small, measurable gain on top of them, and imaging clearly wins only on age. A retrieval layer explains each prediction from similar patients. It doesn't make it more accurate.
The clinical ask
Why Alzheimer's assessment needs several scan types at once, and why most systems break when one of them is missing.
Alzheimer's disease assessment is inherently multimodal. Structural MRI captures neurodegeneration, DTI reflects white-matter microstructural integrity, and cognitive and biomarker variables summarize clinical status. In practice, however, these modalities are not always available together. DTI is often missing, cognitive batteries may be incomplete, and many deep learning systems require complete inputs or only produce a categorical label.
This project asks whether a single missing-modality-aware model can integrate whatever data is available, classify disease stage, estimate clinical severity, retrieve similar historical cases, and support natural-language questions over brain-scan-derived representations.
Primary target
One multimodal model for Alzheimer's diagnosis and clinical prediction that keeps working when inputs are missing. The encoder predicts CN / MCI / Dementia, CN vs Dementia, CDR-SB severity, age, and sex. The frozen encoder then feeds a retrieval-augmented language layer that returns comparable cases and plain-language summaries, so a prediction can be inspected rather than just read.
Dataset
Development and internal testing used 2,363 ADNI subjects, with zero-shot external validation on 1,048 OASIS-3 subjects.
The Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort was filtered to subjects with valid diagnostic labels and T1 MRI, then deduplicated to one scan per subject before any split, leaving 2,363 subjects. DTI-FA was available for 930 of them (39.4%). The FAISS reference index is built only from training subjects, so a test subject can never retrieve itself.
External validation used 1,048 OASIS-3 subjects (one session each: 751 CN, 297 impaired), which have T1 MRI and clinical scores but no processed diffusion data. None of it was used for training, model selection, or index construction.
ADNI · 80 / 20 split, stratified by diagnosis
| Class | Train | Test | Total |
|---|---|---|---|
| CN - Cognitively Normal | 669 | 168 | 837 |
| MCI - Mild Cognitive Impairment | 650 | 163 | 813 |
| Dementia | 570 | 143 | 713 |
| Total | 1,889 | 474 | 2,363 |
100%
T1 MRI (9DOF 2mm)
39.4%
DTI FA coverage · 930 subjects
Clinical score availability · training set
91.5%
APOE
91.2%
MMSE
90.3%
ADAS-11
89.3%
ADAS-13
63.9%
MoCA
MEMOIR-VLM avoids imputing missing imaging modalities. Individual missing clinical scores, such as an absent MoCA, are mean-imputed.
Model architecture
Three modality encoders, per-modality masking, and the cross-attention fusion that feeds five prediction heads.
Three modality-specific encoders produce ℓ2-normalized 512-d embeddings. Each passes through a masking gate that randomly drops modalities during training (T1: 10%, DTI: 30%, Clinical: 5%). Masked embeddings are fused via 8-head cross-attention with a learnable pool query, producing z_f ∈ ℝ⁵¹² that feeds five MLP task heads: DX 3-class, DX Binary, Sex, Age, and CDR-SB.
The model is a multimodal vision-language model with missing-modality support. Three modality-specific encoders produce ℓ2-normalized 512-d embeddings, each gated by a per-modality masking probability during training. The masked embeddings are fused via 8-head cross-attention with a learnable pool query, producing a fused representation z_f ∈ ℝ⁵¹² that feeds five MLP task heads.
Imaging encoders
Two independent 3D ResNet-18 networks for T1 MRI and DTI FA maps. Input volumes are 91×109×91, output is 512-d followed by a linear projection + LayerNorm.
Clinical encoder
MLP over 4 continuous clinical scores (ADAS-11, ADAS-13, MMSE, MoCA) plus an APOE genotype embedding. Output: 512-d, ℓ2-normalized. CDR-SB is a prediction target, never an input.
Design decision: targets stay out of the inputs
Diagnosis, sex, age, and CDR-SB are prediction targets, not inputs. An earlier version also fed CDR-SB into the clinical encoder. CDR-SB is one of the targets and tracks diagnosis closely, so that was a cognitive-score leak; the final model removes it, and every number on this page comes from the corrected model unless it is marked original. The remaining scores still correlate with diagnosis, which is itself partly derived from cognitive testing, so the paper makes no claim that the model finds signal those scores do not already carry.
Training procedure
Two stages: contrastive pre-training across modality pairs, then multi-task fine-tuning at differential learning rates.
Training runs in two stages. Stage 1 is contrastive pre-training: a pairwise CLIP/InfoNCE loss between all three modality pairs (T1–DTI, T1–Clinical, DTI–Clinical), computed only on pairs where both modalities are present. Stage 2B is multi-task fine-tuning across five heads with differential learning rates (backbone 10⁻⁵, heads 5×10⁻⁴). The best composite checkpoint came at epoch 5, after which the model began to overfit, and that checkpoint is the one evaluated.
During training, modalities are randomly dropped so the model learns to work with any subset at inference time. T1 is dropped 10% of the time, DTI 30%, Clinical 5%. DTI gets the highest drop rate because only 39.4% of subjects have it, so the model needs to handle "missing DTI" as the normal case, not the exception.
Stage 1 · Contrastive
30 epochs · pairwise InfoNCE
lr 1×10⁻⁴ · AdamW · cosine anneal
Stage 2B · Multi-task
5 joint heads · best at epoch 5
focal (γ=2) + smooth L1 · AMP FP16
Results
The corrected encoder on the held-out 474-subject ADNI test set, beside the original numbers it replaced.
Evaluated on the held-out 474-subject test set with all available modalities. The final column is the corrected model. The original column is the earlier version that still had CDR-SB among its inputs, kept so the correction is visible rather than silently overwritten.
| Task | Final | Original |
|---|---|---|
| DX 3-class balanced accuracyprimary | 0.682 | 0.707 |
| DX 3-class macro-F1 | 0.681 | 0.703 |
| CN vs Dementia balanced accuracy | 0.913 | 0.933 |
| Sex accuracy | 0.555 | 0.575 |
| Age MAE (years) ↓ | 5.96 | 6.31 |
| CDR-SB MAE ↓ | 1.11 | 0.97 |
Original: the earlier model with CDR-SB among its inputs. CDR-SB error rises to 1.11 because the original figure had the target partly present in its own inputs; that is the leak being removed, not the model getting worse. No AUC is listed because the paper reports AUC only for the original model.
CN vs Dementia
0.913 Bal. Acc.
The extremes of the disease spectrum separate cleanly. MCI remains the hardest boundary: it sits between CN and dementia and overlaps both.
Severity & age
1.11 CDR-SB MAE · 5.96 yr age MAE
Learned jointly with diagnosis from the same fused embedding, which is also what retrieval searches in the VQA extension below.
Modality ablation
Masking inputs on the same trained model: where the diagnostic signal comes from, and which differences survive a significance test.
Mask a modality
Same trained weights, masked at inference
- DX 3-class · n = 474
- 0.682
- CN vs Dementia · n = 311
- 0.913
The full model. +0.016 over clinical only on 3-class, p = 0.498: not distinguishable from the baseline.
Every modality subset runs through the same trained weights, with the missing modalities masked at inference. The chart shows the four configurations that include clinical scores. With the clinical encoder masked entirely, imaging alone retains DX3 0.58 and DX2 0.84 (T1 + DTI), and DTI-FA by itself sits near the three-class chance line at 0.392.
DX 3-class (Bal. Acc.)
CN / MCI / Dementia · n = 474
DX Binary (Bal. Acc.)
CN vs Dementia · n = 311
Highlighted: the only configuration whose gain over clinical-only is statistically supported. On CN vs Dementia no configuration separates from clinical-only.
Paired differences · DX 3-class
Positive favors the second configuration · n = 474
AClinical only → DTI + Clinical
0.666 → 0.701
p 0.012
+0.034[+0.009, +0.060]
BClinical only → Full
0.666 → 0.682
p 0.498
+0.016[−0.031, +0.062]
CClinical only → T1 + Clinical
0.666 → 0.683
p 0.488
+0.016[−0.028, +0.062]
DFull → Clinical MLP
0.682 → 0.705
p 0.379
+0.023[−0.023, +0.068]
EFull → AutoGluon
0.682 → 0.699
p 0.549
+0.016[−0.031, +0.064]
Stratified paired-bootstrap 95% intervals (10,000 resamples of the test set) with McNemar's exact p. Only A clears zero; B to E are secondary, and Holm adjustment takes each to p = 1.00. On the 179 test subjects who actually have DTI, A roughly doubles to +0.071 [+0.002, +0.141]. The model was trained once, so the intervals do not capture training-seed variability.
What the ablation supports
Clinical scores carry most of the diagnostic signal. DTI contributes a statistically measurable gain when combined with clinical information for 3-class diagnosis, but imaging alone is not sufficient for staging, so imaging and cognitive assessment are complementary rather than redundant. The full model is not significantly better than clinical-only on diagnosis, and simple clinical baselines are statistically indistinguishable from it, not better. On CN vs Dementia none of the comparisons clears zero. Where imaging does win clearly is age: imaging-only beats clinical-only by 0.63 years MAE [+0.14, +1.11], Wilcoxon p = 0.005.
Retrieval-augmented VQA extension
Retrieval, reranking, and generation over similar cases, and the audit that found where its headline accuracy came from.
The frozen five-head encoder produces the fused embedding z_f ∈ ℝ⁵¹². FAISS retrieves the top-50 similar training subjects; a cross-encoder re-scores the top 20 and keeps 5. Only the retrieved textual captions reach the LLM, which never sees raw brain images. Three LLM backbones are compared: Mistral 7B, Gemma 4 26B MoE, and MedGemma 1.5 4B.
The encoder gives you a prediction. What it doesn't give you is an explanation, or any way to ask follow-up questions in natural language. That's what the VQA extension adds. The frozen encoder turns a query case into a 512-d fused embedding. FAISS retrieves the 50 most similar training subjects, a cross-encoder re-scores the top 20 of those, and the 5 most relevant go to Mistral 7B as context for its answer.
The LLM never receives raw brain images. T1, DTI, and clinical inputs are encoded into a 512-dimensional fused representation. FAISS retrieves similar training subjects in embedding space, and only retrieved textual captions are passed to the LLM as context.
Text encoder
all-MiniLM-L6-v2, MLM-pretrained on 26,889 clinical sentences (25K synthetic + 1,889 real captions), then contrastively aligned to the imaging embedding space. 384-d output projected up to 512-d.
Retrieval + rerank
FAISS IndexFlatIP over 1,889 ℓ2-normalized training vectors, exact inner-product search, top-50. Cross-encoder ms-marco-MiniLM-L-6-v2 re-scores the top 20 and keeps 5.
The 94.7% was label exposure, not diagnosis. With labeled captions the pipeline reached 94.7% diagnosis accuracy. But the captions of retrieved training subjects state their diagnosis, which opens two routes to the answer without any inference: the cross-encoder can rerank neighbors by matching the diagnosis text, and the language model can copy the majority label from its context. The final paper treats 94.7% as a label-exposure artifact.
The deployment-realistic test withholds the diagnosis field, since a new patient's diagnosis is unknown. Under masked captions no evaluated LLM exceeds a retrieval-only k-NN majority vote of 67.3%, which itself only matches the encoder's 68.2% 3-class accuracy.
Diagnosis, labels masked
3-class · LLMs and k-NN on 150 stratified test subjects
The encoder figure is its balanced accuracy on the full 474-subject test set. Retrieval does not beat the encoder, and no language model beats retrieval.
Where the diagnosis comes from
The multimodal encoder performs diagnosis. The retrieval-augmented language layer provides case-based reasoning, comparable historical cases, and natural-language summaries grounded in retrieved evidence. Building the reference index from label-free captions is a requirement of that setup, not an implementation detail.
LLM backbone comparison
Three backbones on identical retrieved context, and a ranking that flips once the labels are hidden.
Three models were given the same retrieved context: Mistral 7B Instruct v0.3 (general-purpose, dense), Gemma 4 26B MoE (larger, mixture-of-experts), and MedGemma 1.5 4B IT (smaller, fine-tuned on medical data), all quantized to 4-bit NF4. Retrieval is identical across them, so any difference comes from the generation step alone.
Diagnosis · labeled
label-exposure artifact
Diagnosis · masked
deployment-realistic · tick = k-NN 0.673
Format adherence
parseable outputs
Mistral 7B
Instruct v0.3
Gemma 4 26B
MoE
MedGemma 1.5 4B
IT · medical FT
Diagnosis · labeled
label-exposure artifact
Diagnosis · masked
deployment-realistic · tick = k-NN 0.673
Format adherence
parseable outputs
Same retrieved context across all three models; only the generation model changes. Under labeled captions Mistral 7B looked strongest because it copied the stated labels most faithfully. With labels masked the ranking inverts: Gemma 4 26B leads, and still falls short of the retrieval-only vote.
Why Mistral 7B stays in the pipeline
Not for diagnosis. In the label-masked setting Gemma 4 26B performs best among the tested LLMs, but none surpasses the retrieval-only baseline. Mistral 7B remains the natural-language interface backbone because it holds the output format (about 100% parseable, against 66.7% for MedGemma), produces high-fidelity text, and runs in about 4.5 GB of GPU memory with 4 to 6 s end-to-end inference.
Text quality · Mistral 7B, top-10 context, 150 subjects
| Metric | VQA | Captioning |
|---|---|---|
| BLEU | 0.066 ± 0.038 | 0.044 ± 0.025 |
| ROUGE-L | 0.345 ± 0.084 | 0.211 ± 0.057 |
| BERTScore | 0.894 ± 0.022 | 0.866 ± 0.017 |
| SBERT | 0.811 ± 0.076 | 0.767 ± 0.077 |
Lexical overlap is modest because the references are structured templates and the model writes free prose; semantic similarity is strong. The layer reads as a readable, evidence-grounded interface rather than a diagnostic classifier.
External validation
The unchanged ADNI-trained weights, run zero-shot on 1,048 OASIS-3 subjects with no diffusion imaging at all.
OASIS-3 has T1 MRI and clinical scores but no processed diffusion maps, so the same ADNI-trained weights run on it with the DTI branch masked: exactly the missing-modality path the model was trained for. No OASIS-3 data touched training, model selection, or the retrieval index.
CN vs impaired · n = 1,048
0.787 Bal. Acc. · 0.889 AUC
Binary diagnosis transfers across sites, scanners, and protocols with a whole modality absent.
What does not transfer
0.524 3-class · 8.43 yr age MAE
Three-class accuracy degrades, driven by an MCI class that is small in OASIS-3. Age fails because OASIS-3 skews younger than ADNI. CDR-SB MAE is nominally lower (0.90 vs 1.11), but only because the cohort is mostly cognitively normal.
A model that needed every modality, or imputed the missing one, could not run here without extra machinery. This is the clearest evidence that the missing-modality mechanism is load-bearing. OASIS-3 is still a North American research cohort, so it shows cross-site transfer, not generalization to diverse clinical populations.
Key findings
What the corrected results support, and what they do not.
Corrected encoder performance
After removing CDR-SB from the input space, MEMOIR-VLM reaches 91.3% balanced accuracy for CN vs Dementia and 68.2% for 3-class diagnosis. MCI remains the hardest class boundary.
One set of weights for every modality subset
One shared set of weights operates across all seven non-empty combinations of T1w MRI, DTI-FA, and clinical inputs, with no separate model per subset and no imaging imputation. Only 39.4% of subjects had usable DTI.
DTI adds measurable 3-class signal alongside clinical context
Adding DTI-FA to clinical inputs raises 3-class balanced accuracy from 0.666 to 0.701, a +0.034 gain whose 95% interval excludes zero (McNemar p = 0.012). Imaging alone is not enough for staging.
RAG VQA is interpretability infrastructure, not a better classifier
The 94.7% labeled-caption accuracy was a label-exposure artifact. Once diagnosis labels are removed from retrieved captions, no evaluated LLM exceeds the retrieval-only baseline. The encoder remains the diagnostic component.
Zero-shot external validation
On 1,048 OASIS-3 subjects with diffusion imaging entirely absent, the unchanged ADNI-trained model retains 78.7% balanced accuracy for CN vs impaired (AUC 0.889).
Faithful natural-language summaries
VQA summaries reach BERTScore 0.894 and SBERT cosine similarity 0.811, which supports the language layer as a readable clinical interface.
Publication
Published · Frontiers in Computational Neuroscience
Open access under CC BY, volume 20. The work was done at the Imaging Genetics Center, Keck School of Medicine of USC.
- Received
- 7 Jun 2026
- Revised
- 30 Jul 2026
- Accepted
- 27 Aug 2026
- Published
- 1 Oct 2026
Gidwani SH, Chattopadhyay T, Thomopoulos SI and Thompson PM (2026) MEMOIR-VLM: a multimodal vision-language model for Alzheimer's disease classification and question answering. Front. Comput. Neurosci. 20:1902258. doi: 10.3389/fncom.2026.1902258
Future work extends MEMOIR-VLM to amyloid prediction supervised on PET-derived labels, with cognitive variables kept out of the inputs, the most direct route to testing what the imaging pathway carries on its own.