BioMistral-Clinical: A Scalable Approach to Clinical LLMs via Incremental Learning and RAG
ISBN
979-8-89176-303-6
Type
conference paper
Date Issued
2025-12
Author(s)
Abstract
The integration of large language models (LLMs) into clinical medicine represents a major advancement in natural language processing (NLP). We introduce BioMistral-Clinical 7B, a clinical LLM built on BioMistral-7B (Labrak et al., 2024), designed to support continual learning from unstructured clinical notes for real-world tasks such as clinical decision support. Using the augmented-clinical notes dataset provided by Hugging Face (2024), we apply prompt engineering to transform unstructured text into structured JSON capturing key clinical information (symptoms, diagnoses, treatments, outcomes). We employ selfsupervised continual learning (SPeCiaL) (Caccia and Pineau, 2021) to achieve efficient incremental training. Evaluation on MedQA (Jin et al., 2021) and MedMCQA (Pal et al., 2022) shows that BioMistral-Clinical 7B improves accuracy on MedMCQA by nearly 10 points (37.4% vs. 28.0%) over the base model, while maintaining comparable performance on MedQA (34.8% vs. 36.5%). Building on this, we propose the BioMistral-Clinical System, which integrates Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) to enrich responses with relevant clinical cases retrieved from a structured vector database. The full system enhances clinical reasoning by combining domain-specific adaptation with contextual retrieval.
Language
English (United States)
HSG Classification
contribution to scientific community
Refereed
Yes
Book title
Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics
Publisher
The Asian Federation of Natural Language Processing and The Association for Computational Linguistics
Start page
1171
End page
1184
Event Location
Mumbai, India
Event Date
December 2025
Contact Email Address
christina.niklaus@unisg.ch
File(s)![Thumbnail Image]()
Name
2025.findings-ijcnlp.71.pdf
Type
Main Article
Size
1.59 MB
Format
Adobe PDF
Checksum (MD5)
50e6ed18ec3246e3f927f4254cbd7d64