Learning From Less: Weak Labels and Contrastive Learning in Medical Imaging

The bottleneck is labels, not images

Medical imaging has no shortage of images. What it lacks is expert time to label them. As a doctor building AI for cervical cancer screening, I run into this every week.

A single cervical cytology slide can hold tens of thousands of cells. Asking a cytopathologist to outline every abnormal one is slow and expensive. At the scale deep learning needs, it is simply unrealistic, especially in health systems where pathologists are already scarce.

What we usually have instead is cheaper information: a diagnosis for the whole slide, a report attached to a scan, or thousands of images nobody has labelled at all. Two families of methods turn that cheaper information into useful models. Weakly supervised learning learns from coarse or noisy labels, and contrastive learning learns from no labels at all. This post explains both, shows how they fit together, and flags where they can mislead you.

Weakly supervised learning: making coarse labels work

Weak supervision means training with labels that are coarser, sparser or noisier than the task you actually care about. In medical imaging, three kinds come up again and again.

Inexact labels describe the whole image but not where the finding is, such as a slide reported as high-grade lesion with no cell marked. Incomplete labels cover only part of the data, for example a few hundred annotated images among tens of thousands. Inaccurate labels contain errors, often because they were mined automatically from free-text reports, as with large public chest X-ray collections such as CheXpert [1].

The workhorse for inexact labels is multiple instance learning (MIL). An image is treated as a bag of patches, and only the bag carries a label. The classic assumption is that a bag is positive if at least one patch is positive, and negative only if every patch is negative.

That assumption mirrors clinical reasoning. One cluster of abnormal cells is enough to make a slide abnormal, wherever it sits. Campanella and colleagues showed in 2019 that this scales: they trained pathology classifiers on 44,732 slides from 15,187 patients using only the diagnoses already written in pathology reports [2].

Attention-based MIL made the idea more flexible [3]. Instead of trusting the single most suspicious patch, the model learns a weight for every patch and pools them. Later methods such as CLAM [4] and TransMIL [5] built on this foundation.

The attention weights have a useful side effect. Projected back onto the slide, they form a heatmap of where the model looked, giving rough localisation from labels that never contained any location. Class activation maps play a similar role in radiology, turning an image-level classifier into a coarse lesion finder [6].

Contrastive learning: learning before any label arrives

Contrastive learning takes a different route: it learns useful features with no labels at all. The model sees two randomly altered versions of the same image, such as two crops with different brightness, and learns to place them close together. Versions of different images are pushed apart.

To succeed, the network must learn what survives those alterations, which tends to be shape, texture and structure. Those features can then be reused for a diagnostic task that needs far fewer labels. SimCLR [7] and MoCo [8], both published in 2020, made this practical, and DINO-style self-distillation later reached similar results without explicit negative pairs [9].

Medical researchers soon found smarter ways to choose positive pairs. MICLe treats different images of the same patient as positives, so the model learns what a condition looks like across views rather than across crops [10]. ConVIRT paired chest X-rays with their radiology reports, learning images and text together [11].

That image-text idea, popularised in general AI by CLIP [12], now underpins medical models such as BiomedCLIP [13] and, in pathology, CONCH [14]. Large self-supervised pathology encoders such as UNI [15] and Virchow [16], trained with DINOv2 self-distillation on more than 100,000 and about 1.5 million slides respectively, have become a common starting point for new projects.

Putting them together

In computational pathology the two ideas now work as a pair, and the combination fits cervical cytology closely.

Figure 1. The two-stage pipeline. Expert labels enter only in the second stage, and only at slide level.

Stage one uses the slides you already have but cannot label. A patch encoder is pretrained with contrastive or related self-supervised learning on your own archive. It learns what cells, nuclei and background look like under your stains and scanners.

Stage two freezes that encoder and trains a small attention-MIL head on slide-level diagnoses, such as Bethesda categories from routine cytology reports [4, 15]. Because the encoder already understands cell morphology, the head needs far fewer labelled slides. Its attention weights then point reviewers to the cells that drove each decision.

Where these methods can mislead you

Augmentations borrowed from natural photographs can erase the signal you need. The SimCLR authors found that the choice of augmentations largely defines what the model learns [7]. Colour jitter can distort the staining differences that separate normal from abnormal cells, and an aggressive crop can remove a small lesion entirely. Choose augmentations with a pathologist or radiologist rather than copying an ImageNet recipe.

Contrastive learning also has a false negative problem. Two different patients with the same disease are treated as a pair to push apart, which works against learning what the disease has in common. Patient-aware positives [10], diverse batches and non-contrastive methods [9] all reduce the damage.

Weak labels invite shortcut learning. A slide-level model may learn the scanner, the staining batch or the hospital instead of the disease, because those correlate with the label in training. Zech and colleagues showed that chest X-ray models could identify which hospital an image came from, and performed worse on data from outside hospitals [17]. Testing on data from a different site is the only reliable check.

Finally, an attention heatmap is not an explanation. High attention means a patch influenced the prediction, not that it holds the lesion a clinician would point to. Treat heatmaps as a prompt for review, and validate localisation against expert annotation on at least a subset of cases.

Why this matters for low-resource health systems

Label efficiency is more than an engineering convenience. Sub-Saharan Africa has been estimated to have at best a tenth of the pathology coverage of high-income countries, roughly one pathologist per million people, so expert annotation is the scarcest resource in the pipeline [18, 19]. Methods that need slide-level diagnoses instead of cell-level outlines change what is feasible.

They also make local data usable. Self-supervised pretraining can run on thousands of unlabelled slides from our own hospitals, adapting the model to local stains, scanners and populations before a single label is spent. The labels we do collect can then go where they matter most: validation.

None of this removes the need for rigorous clinical evaluation. It shifts the question from whether we can afford to label enough data to whether we have tested the model well enough to trust it. That is a far better question to be stuck on.

References

1. Irvin J, Rajpurkar P, Ko M, et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence. 2019;33(1). doi:10.1609/aaai.v33i01.3301590 https://doi.org/10.1609/aaai.v33i01.3301590
2. Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med. 2019;25(8):1301-1309. doi:10.1038/s41591-019-0508-1 https://doi.org/10.1038/s41591-019-0508-1
3. Ilse M, Tomczak JM, Welling M. Attention-based deep multiple instance learning. Proceedings of the 35th International Conference on Machine Learning. PMLR 80; 2018:2127-2136. https://proceedings.mlr.press/v80/ilse18a.html
4. Lu MY, Williamson DFK, Chen TY, Chen RJ, Barbieri M, Mahmood F. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat Biomed Eng. 2021;5(6):555-570. https://pmc.ncbi.nlm.nih.gov/articles/PMC8711640/
5. Shao Z, Bian H, Chen Y, Wang Y, Zhang J, Ji X, Zhang Y. TransMIL: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems 34 (NeurIPS); 2021. https://arxiv.org/abs/2106.00908
6. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminative localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016:2921-2929. doi:10.1109/CVPR.2016.319 https://doi.org/10.1109/CVPR.2016.319
7. Chen T, Kornblith S, Norouzi M, Hinton G. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning. PMLR 119; 2020:1597-1607. https://proceedings.mlr.press/v119/chen20j.html
8. He K, Fan H, Wu Y, Xie S, Girshick R. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2020:9729-9738. doi:10.1109/CVPR42600.2020.00975 https://doi.org/10.1109/CVPR42600.2020.00975
9. Caron M, Touvron H, Misra I, Jégou H, Mairal J, Bojanowski P, Joulin A. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); 2021:9650-9660. https://arxiv.org/abs/2104.14294
10. Azizi S, Mustafa B, Ryan F, et al. Big self-supervised models advance medical image classification. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); 2021:3478-3488. doi:10.1109/ICCV48922.2021.00346 https://doi.org/10.1109/ICCV48922.2021.00346
11. Zhang Y, Jiang H, Miura Y, Manning CD, Langlotz CP. Contrastive learning of medical visual representations from paired images and text. Proceedings of Machine Learning for Healthcare. PMLR 182; 2022. (Preprint first released 2020, arXiv:2010.00747.) https://proceedings.mlr.press/v182/zhang22a.html
12. Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning. PMLR 139; 2021:8748-8763. https://proceedings.mlr.press/v139/radford21a.html
13. Zhang S, Xu Y, Usuyama N, et al. A multimodal biomedical foundation model trained from fifteen million image-text pairs (BiomedCLIP). NEJM AI. 2025;2(1). doi:10.1056/AIoa2400640 https://doi.org/10.1056/AIoa2400640
14. Lu MY, et al. A visual-language foundation model for computational pathology (CONCH). Nat Med. 2024;30(3). doi:10.1038/s41591-024-02856-4 https://doi.org/10.1038/s41591-024-02856-4
15. Chen RJ, Ding T, Lu MY, et al. Towards a general-purpose foundation model for computational pathology (UNI). Nat Med. 2024;30(3):850-862. doi:10.1038/s41591-024-02857-3 https://doi.org/10.1038/s41591-024-02857-3
16. Vorontsov E, Bozkurt A, Casson A, et al. A foundation model for clinical-grade computational pathology and rare cancers detection (Virchow). Nat Med. 2024;30(10):2924-2935. doi:10.1038/s41591-024-03141-0 https://doi.org/10.1038/s41591-024-03141-0
17. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 2018;15(11):e1002683. doi:10.1371/journal.pmed.1002683 https://doi.org/10.1371/journal.pmed.1002683
18. Adesina A, Chumba D, Nelson AM, et al. Improvement of pathology in sub-Saharan Africa. Lancet Oncol. 2013;14:e152-e157. https://pubmed.ncbi.nlm.nih.gov/23561746/
19. Wilson ML, Fleming KA, Kuti MA, Looi LM, Lago N, Ru K. Access to pathology and laboratory medicine services: a crucial gap. Lancet. 2018;391(10133):1927-1938.