Federated Self-Paced Learning Signals Enhanced LLM Training For Long-Context Adverse Drug Events Detection In Clinical Text
DOI:
https://doi.org/10.29284/538mtv24Keywords:
Biomedical Informatics, Federated Learning, LLM, Self-Paced Learning, Bioinformatics, Adverse Drug Events, EHR, PEFT, Qwen 2.5Abstract
Detecting adverse drug events (ADEs) in free-text clinical notes is important for patient safety yet challenging because mentions of ADEs are rare and many paragraphs contain language that can be confusing, for example negated or historical statements. We study paragraph-level ADE detection with a compact large language model, Qwen2.5-1.5B, trained using self-paced learning that starts with clearer, easier paragraphs and gradually introduces harder, more ambiguous ones. We build a paragraph-level dataset from n2c2 notes and simulate a multi-hospital setting by partitioning the corpus into non-identical silos for federated learning. The model fine-tunes only LoRA adapters while keeping the backbone frozen, which keeps memory and communication low. In centralised training the system reaches 88.92 macro-F1 on n2c2, about 17 points higher than the strongest prior transformer baseline in our compilation. In federated training across five silos the main schedule achieves 88.59 macro-F1 with client participation 0.60 over 50 rounds, and a compute-capped schedule with 20 rounds and two local epochs attains 88.41 macro-F1. A no-differential-privacy ablation reaches 88.68 macro-F1. Communication is modest, with approximately 12.1 MB uplink and 10.9 MB downlink per participating round, which totals about 0.69 GB per client for the main schedule and 0.27 GB for the compute-capped schedule. Client-level Gaussian noise with σ≈0.41 yields ε≈2.4 at δ=10⁻⁶ in our accounting. These results show that a small LLM with self-paced learning delivers strong ADE screening and that the gains carry over to a federated setting with practical bandwidth and privacy budgets.
References
1. Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA. 1998;279(15):1200–1205. doi:10.1001/jama.279.15.1200.
2. Hazell L, Shakir SAW. Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29(5):385–396. doi:10.2165/00002018-200629050-00003.
3. Henry S, Buchan K, Filannino M, Stubbs A, Uzuner O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc. 2020;27(1):3–12. doi:10.1093/jamia/ocz166.
4. Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234–1240. doi:10.1093/bioinformatics/btz682.
5. Beltagy I, Lo K, Cohan A. SciBERT: A pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China: Association for Computational Linguistics; 2019. p. 3615–3620. doi:10.18653/v1/D19-1371.
6. Peng Y, Yan S, Lu Z. Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets. arXiv [Preprint]. 2019. doi:10.48550/arXiv.1906.05474.
7. McMahan HB, Moore E, Ramage D, Hampson S, Agüera y Arcas B. Communication-efficient learning of deep networks from decentralized data. In: Singh A, Zhu J, editors. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017); 2017 Apr 20–22; Fort Lauderdale, FL, USA. PMLR; 2017. p. 1273–1282.
8. Kumar MP, Packer B, Koller D. Self-paced learning for latent variable models. In: Lafferty JD, Williams CKI, Shawe-Taylor J, Zemel RS, Culotta A, editors. Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems (NeurIPS 2010); 2010 Dec 6–9; Vancouver (BC), Canada. Red Hook (NY): Curran Associates, Inc.; 2010. p. 1189–1197.
9. Yang A, Yang B, Zhang B, Hui B, Zheng B, Yu B, et al. Qwen2.5 technical report. arXiv [Preprint]. 2024. doi:10.48550/arXiv.2412.15115.
10. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. LoRA: Low-rank adaptation of large language models. arXiv [Preprint]. 2021. doi:10.48550/arXiv.2106.09685.
11. Beltagy I, Peters ME, Cohan A. Longformer: The long-document transformer. arXiv [Preprint]. 2020. doi:10.48550/arXiv.2004.05150.
12. Wei Q, Ji Z, Li Z, Du J, Wang J, Xu J, et al. A study of deep learning approaches for medication and adverse drug event extraction from clinical text. J Am Med Inform Assoc. 2020;27(1):13–21. doi:10.1093/jamia/ocz063
13. Dai HJ, Su CH, Wu CS. Adverse drug event extraction in EHR via cascading architectures with different sequence labeling models. J Am Med Inform Assoc. 2020;27(1):47–55. doi:10.1093/jamia/ocz120
14. Chen L, Gu Y, Ji X, Sun Z, Li H, Gao Y, et al. Extracting medications and associated adverse drug events using NLP system combining knowledge base and deep learning. J Am Med Inform Assoc. 2020;27(1):56–64. doi:10.1093/jamia/ocz141
15. Ju M, Nguyen NTH, Miwa M, Ananiadou S. An ensemble of neural models for nested adverse drug events and medication extraction with subwords. J Am Med Inform Assoc. 2020;27(1):22–30. doi:10.1093/jamia/ocz075
16. El-Allaly E, Sarrouti M, En-Nahnahi N, Ouatik El Alaoui S. MTTLADE: Multi-task transfer learning for adverse drug events extraction. Inf Process Manag. 2021;58(3):102473. doi:10.1016/j.ipm.2020.102473
17. Kumar MP, Packer B, Koller D. Self-paced learning for latent variable models. In: Lafferty JD, Williams CKI, Shawe-Taylor J, Zemel RS, Culotta A, editors. Advances in Neural Information Processing Systems 23 (NIPS 2010). Red Hook (NY): Curran Associates, Inc.; 2010. p. 1189–1197.
18. McMahan HB, Moore E, Ramage D, Hampson S, Agüera y Arcas B. Communication-efficient learning of deep networks from decentralized data. In: Singh A, Zhu J, editors. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017); 2017 Apr 20–22; Fort Lauderdale (FL), USA. PMLR; 2017. p. 1273–1282.
19. Dayan I, Roth HR, Zhong A, Harouni A, Gentili A, Abidin AZ, et al. Federated learning for predicting clinical outcomes in patients with COVID-19. Nat Med. 2021;27(10):1735–1743. doi:10.1038/s41591-021-01506-3.
20. Li S, Liu P, Nascimento GG, Wang X, Leite FRM, Chakraborty B, et al. Federated and distributed learning applications for electronic health records and structured medical data: a scoping review. J Am Med Inform Assoc. 2023;30(12):2041–2049. doi:10.1093/jamia/ocad170.
21. Teo ZL, Jin L, Li S, Miao D, Zhang X, et al. Federated machine learning in healthcare: a systematic review on clinical applications and technical architecture. Cell Rep Med. 2024;5(2):101419. doi:10.1016/j.xcrm.2024.101419.
22. Mathew J, Stripelis D, Ambite JL. Federated named entity recognition. arXiv [Preprint]. 2022. doi:10.48550/arXiv.2203.15101.
23. Ge S, Wu F, Wu C, Qi T, Huang Y, Xie X. FedNER: Privacy-preserving medical named entity recognition with federated learning. arXiv [Preprint]. 2020. doi:10.48550/arXiv.2003.09288.
24. Bonawitz K, Ivanov V, Kreuter B, Marcedone A, McMahan HB, Patel S, et al. Practical secure aggregation for privacy-preserving machine learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS ’17); 2017 Oct 30–Nov 3; Dallas (TX), USA. New York (NY): Association for Computing Machinery; 2017. p. 1175–1191. doi:10.1145/3133956.3133982.
25. Beutel DJ, Topal T, Mathur A, Qiu X, Fernandez-Marques J, Gao Y, et al. Flower: a friendly federated learning research framework. arXiv [Preprint]. 2020. doi:10.48550/arXiv.2007.14390.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
