Enhancing Personally Identifiable Information Recognition Using Compact Transformer With Hybrid Sequential Architecture
DOI:
https://doi.org/10.29284/m14da163Keywords:
Compact Transformer, Named Entity Recognition, Personal Identifiable Information, Hybrid Sequential Architecture.Abstract
The rapid expansion of unstructured data across organizational repositories has significantly increased the risk of sensitive information leakage and regulatory non-compliance. Named Entity Recognition for Personally Identifiable Information plays a crucial role in automated data protection systems by identifying and classifying sensitive entities such as personal names, identification numbers, addresses, and financial information. While traditional rule-based approaches offer interpretability, they often suffer from limited generalization and low recall. In contrast, deep learning-based approaches have demonstrated superior performance but are frequently constrained by high computational complexity and memory requirements. This research investigates the effectiveness of compact Transformer architectures for PII-NER, focusing on the DeBERTa v3 xsmall model as the primary backbone. With approximately 22 million backbone parameters, DeBERTa v3 xsmall provides a favorable trade-off between model size and representational power. To further enhance sequence modeling capabilities, this study integrates Bidirectional Long Short-Term Memory, Bidirectional Gated Recurrent Unit, and Conditional Random Fields layers on top of the Transformer encoder. Experiments were conducted on the English portion of the AI4Privacy PII dataset containing approximately 43,000 annotated samples. The experimental results demonstrate that the hybrid architecture combining DeBERTa v3 xsmall + BiGRU + CRF achieves the best performance, with an F1-score of 0.9139 after 15 training epochs. The findings indicate that integrating compact Transformer backbones with sequential modeling and structured prediction layers significantly improves label consistency and entity boundary detection while maintaining computational efficiency. This study highlights the practical feasibility of deploying lightweight yet high-performance PII-NER systems in real-world environments with limited hardware resources.
References
1. T. Areskoug, and V. Mörk, “Detection of personal data in unstructured text using deep learning models,” 2025.
2. M. D. Drovo, M. Chowdhury, S. I. Uday, and A. K. Das, “Named entity recognition in Bengali text using merged hidden Markov model and rule base approach,” In 2019 7th International Conference on Smart Computing & Communications (ICSCC), pp. 1-5, June 2019.
3. W. C. Chang, C. H. Tsou, and A. Huang, “Fast rule-based NER in SpaCy for chest radiography reports with CheXpert’s 14 Categories,” In 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1-4, July 2024.
4. N. A. D. Dharviyanti and N. Wilantika, “Rule-Based NER for crime information extraction through online news site,” In 2024 International Conference on Information Technology Research and Innovation (ICITRI), pp. 99-104, September 2024.
5. Y. S. Chuang, C. T. Lee, R. Brandon, T. D. Tran, O. Tokede, M. F. Walji, et al., “Extracting periodontitis diagnosis in clinical notes with RoBERTa and regular expression,” In 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI), pp. 200-205, June 2023.
6. Z. Toh, W. Wang, M. Lan, and X. Li, “An NER-based product identification and lucene-based product linking approach to CPROD1 challenge: Description of submission system to CPROD1 Challenge,” In 2012 IEEE 12th International Conference on Data Mining Workshops, pp. 869-871, December 2012.
7. I. Augenstein, L. Derczynski, and K. Bontcheva, “Generalisation in named entity recognition: A quantitative analysis,” Computer Speech & Language, vol. 44, pp. 61-83, 2017.
8. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”, ArXiv, 2019.
9. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” arXiv, 2019.
10. M. N. M. M. Noor and M. F. M. Badauraudine, “Evaluating small BERT based models on automated essay scoring task,” Information Research Communications, vol. 2, pp. 175-181, 2025.
11. Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou, “Mobilebert: a compact task-agnostic bert for resource-limited devices,” In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 2158-2170, July 2020.
12. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, et al., “Tinybert: Distilling bert for natural language understanding,” In Findings of the association for computational linguistics: EMNLP 2020, pp. 4163-4174, November 2020.
13. I. Turc, M. Chang, K. Lee, and K. Toutanova, “Well-read students learn better: On the importance of pre-training compact models,” arXiv, 2019.
14. P. He, J. Gao, and W. Chen, “Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing,” arXiv, 2021.
15. A. Graves, S. Fernández, and J. Schmidhuber, “Bidirectional LSTM networks for improved phoneme classification and recognition,” In International conference on artificial neural networks, pp. 799-804, September, 2005.
16. R. Panchendrarajan, and A. Amaresan, “Bidirectional LSTM-CRF for named entity recognition,” In Proceedings of the 32nd Pacific Asia conference on language, information and computation, 2018.
17. K. L. Thant, K. Nongpong, Y. K. Thu, T. Aung, K. H. Wai, and T. M. Oo, “myNER: Contextualized Burmese named entity recognition with bidirectional lstm and fastText embeddings via joint training with pos tagging,” In 2025 IEEE International Conference on Cybernetics and Innovations (ICCI), pp. 1-6, April, 2025.
18. X. Liu, Y. Wang, X. Wang, H. Xu, C. Li, and X. Xin, “Bi-directional gated recurrent unit neural network based nonlinear equalizer for coherent optical communication system,” Opt. Express, vol. 29, no. 4, pp. 5923–5933, February 2021.
19. L. Anand, “LSTM-and GRU-Based architectures for text sequence modeling,” 2022.
20. J. Deng, L. Cheng, and Z. Wang, “Self-attention-based BiGRU and capsule network for named entity recognition,” arXiv, 2020.
21. J. Lafferty, A. McCallum, and F. C. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” In Proceedings of the Eighteenth International Conference on Machine Learning (ICML '01), pp. 282–289, 2001.
22. J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler, “Efficient object localization using convolutional networks,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 648-656, 2015.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
