Optimization of Feminacare Chatbot Application Using SeaLLM Model
DOI:
https://doi.org/10.57152/ijirse.v5i1.2043Keywords:
Chatbot, SeaLLM, Retrieval-Augmented Generation (RAG), Women's Health, FeminacareAbstract
Feminacare, a women's health consultation application with a chatbot, previously used the LSTM model, which had an accuracy of 61% but often gave less relevant responses. This research proposes the use of the SeaLLM model with the Retrieval-Augmented Generation (RAG) approach to improve the accuracy of the chatbot. Based on three trials in the evaluation of 120 questions, the chatbot obtained a mean accuracy of 87%, with a precision of 93%, recall of 89%, and F1-score of 91%. Compared to previous models, this approach produced more relevant and accurate responses. Overall, this research proves that the application of SeaLLM with RAG can improve the effectiveness of chatbots in providing women's health information. However, further improvements are still needed, especially in handling more complex questions by expanding the dataset.
References
M. Muliyono and S. Sumijan, “Identifikasi Chatbot dalam Meningkatkan Pelayanan Online Menggunakan Metode Natural Language Processing,” J. Inform. Ekon. Bisnis, vol. 3, pp. 142–147, 2021, doi: 10.37034/infeb.v3i4.102.
L. Xu, L. Lu, M. Liu, C. Song, and L. Wu, “Nanjing Yunjin intelligent question-answering system based on knowledge graphs and retrieval augmented generation technology,” Herit. Sci., vol. 12, no. 1, pp. 1–23, 2024, doi: 10.1186/s40494-024-01231-3.
L. Athota, V. K. Shukla, N. Pandey, and A. Rana, “Chatbot for Healthcare System Using Artificial Intelligence,” ICRITO 2020 - IEEE 8th Int. Conf. Reliab. Infocom Technol. Optim. (Trends Futur. Dir., pp. 619–622, 2020, doi: 10.1109/ICRITO48877.2020.9197833.
J. N. K. Wah, “Revolutionizing e-health: the transformative role of AI-powered hybrid chatbots in healthcare solutions,” Front. Public Heal., vol. 13, no. February, pp. 1–14, 2025, doi: 10.3389/fpubh.2025.1530799.
M. Laymouna, Y. Ma, D. Lessard, T. Schuster, K. Engler, and B. Lebouché, “Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review,” J. Med. Internet Res., vol. 26, pp. 1–28, 2024, doi: 10.2196/56930.
L. Li, “Studies advanced in chatbots based on deep learning,” Appl. Comput. Eng., vol. 6, no. 1, pp. 678–683, 2023, doi: 10.54254/2755-2721/6/20230921.
S. Pandey and S. Sharma, “A comparative study of retrieval-based and generative-based chatbots using Deep Learning and Machine Learning,” Healthc. Anal., vol. 3, no. May, p. 100198, 2023, doi: 10.1016/j.health.2023.100198.
Y. Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Trans. Intell. Syst. Technol., vol. 15, no. 3, 2024, doi: 10.1145/3641289.
Y. Ke et al., “Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report,” 2024, [Online]. Available: http://arxiv.org/abs/2402.01733
F. Koto, R. Mahendra, N. Aisyah, and T. Baldwin, “IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces,” 2024, [Online]. Available: http://arxiv.org/abs/2404.01854
F. Koto, “Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia,” 2024, [Online]. Available: http://arxiv.org/abs/2409.08564
A. Balaguer et al., “RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture,” 2024, [Online]. Available: http://arxiv.org/abs/2401.08406
R. Lakatos, P. Pollner, A. Hajdu, and T. Joo, “Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems,” pp. 1–17, 2024, [Online]. Available: http://arxiv.org/abs/2403.09727
M. Fatehkia, J. K. Lucas, and S. Chawla, “T-RAG: Lessons from the LLM Trenches,” pp. 1–22, 2024, [Online]. Available: http://arxiv.org/abs/2402.07483
K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval Augmentation Reduces Hallucination in Conversation,” Find. Assoc. Comput. Linguist. Find. ACL EMNLP 2021, pp. 3784–3803, 2021, doi: 10.18653/v1/2021.findings-emnlp.320.
M. ?orec?u and E. ?orec?u, “An Alternative Application to CHATGPT that Uses Reliable Sources to Enhance the Learning Process,” Int. Conf. KNOWLEDGE-BASED Organ., vol. 29, no. 3, pp. 113–119, 2023, doi: 10.2478/kbo-2023-0084.
G. F. Febrian and G. Figueredo, “KemenkeuGPT: Leveraging a Large Language Model on Indonesia’s Government Financial Data and Regulations to Enhance Decision Making,” 2024, [Online]. Available: http://arxiv.org/abs/2407.21459
P. Zhao et al., “Retrieval-Augmented Generation for AI-Generated Content: A Survey,” no. March, 2024, [Online]. Available: http://arxiv.org/abs/2402.19473
D. A. Ramadhan, D. Ramadhani, and U. Riau, “Classification of Riau Batik Motifs Using the Convolutional Neural Network ( CNN ) Algorithm,” vol. 07, no. 03, pp. 201–211, 2024.
