Comparison of Convolutional Neural Network and Recurrent Neural Network Algorithms for Indonesian Sign Language Recognition

Dani Harmade; Afif Fathin; Nur Jannah Nai'mah Zainal

doi:10.57152/predatecs.v3i2.2090

Authors

Dani Harmade Universitas Islam Negeri Sultan Syarif Kasim Riau, Indonesia https://orcid.org/0009-0009-1044-4653
Afif Fathin Universitas Islam Negeri Sultan Syarif Kasim Riau, Indonesia
Nur Jannah Nai'mah Zainal International Islamic University Malaysia, Malaysia

DOI:

https://doi.org/10.57152/predatecs.v3i2.2090

Keywords:

Convolutional Neural Network, Recurrent Neural Network, SIBI, Sign Language

Abstract

Effective communication is a fundamental human need; however, for people with hearing impairments in Indonesia, interaction relies heavily on the Indonesian Sign Language System (Sistem Isyarat Bahasa Indonesia – SIBI). Although deep learning has been widely applied in sign language recognition, comprehensive comparative studies focusing specifically on SIBI remain limited, particularly in evaluating the performance gap between different neural network architectures. This study addresses this gap by comparing the effectiveness of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) in classifying SIBI hand gesture images. An augmented SIBI dataset was trained using the Adam optimizer to improve generalization and recognition performance. The experimental results reveal a significant performance difference between the two models, where CNN achieved a precision, recall, and F1-score of 94%, while RNN obtained a precision of 76% recall of 74%, and F1-score of 73%. These findings demonstrate that CNN is substantially more effective for image-based SIBI recognition because it extracts spatial features more effectively than the sequential processing mechanism of RNN. This research contributes empirical evidence for selecting appropriate deep learning architectures in SIBI recognition systems and offers practical implications for developing more accurate and reliable assistive communication technologies in educational and accessibility contexts.

References

B. Sumaiya, S. Srivastava, V. Jain, and V. Prakash, “The Role of Effective Communication Skills in Professional Life,” World J. English Lang., vol. 12, no. 3, pp. 134–140, 2022, doi: 10.5430/wjel.v12n3p134.

Suharjito, N. Thiracitta, and H. Gunawan, “SIBI Sign Language Recognition Using Convolutional Neural Network Combined with Transfer Learning and non-trainable Parameters,” Procedia Comput. Sci., vol. 179, no. 2019, pp. 72–80, 2021, doi: 10.1016/j.procs.2020.12.011.

F. Wijaya, L. Dahendra, E. S. Purwanto, and M. K. Ario, “Quantitative analysis of sign language translation using artificial neural network model,” Procedia Comput. Sci., vol. 245, no. C, pp. 998–1009, 2024, doi: 10.1016/j.procs.2024.10.328.

S. Dwijayanti, Hermawati, S. I. Taqiyyah, H. Hikmarika, and B. Y. Suprapto, “Indonesia Sign Language Recognition using Convolutional Neural Network,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 10, pp. 415–422, 2021, doi: 10.14569/IJACSA.2021.0121046.

E. Moriarty, “‘Sign to me, not the children’: Ideologies of language contamination at a deaf tourist site in Bali,” Lang. Commun., vol. 74, pp. 195–203, 2020, doi: 10.1016/j.langcom.2020.06.002.

F. E. Dianastiti, S. Suwandi, and B. Setiawan, “Contributing Factors and Challenges in Mastering Academic Writing Skills: Multiple Case Studies of Deaf Students in Inclusive Universities in Indonesia,” Int. J. Lang. Educ., vol. 8, no. 1, pp. 20–35, 2024, doi: 10.26858/ijole.v8i1.60905.

D. Indra, Purnawansyah, S. Madenda, and E. P. Wibowo, “Indonesian sign language recognition based on shape of hand gesture,” Procedia Comput. Sci., vol. 161, pp. 74–81, 2019, doi: 10.1016/j.procs.2019.11.101.

Y. Obi, K. S. Claudio, V. M. Budiman, S. Achmad, and A. Kurniawan, “Sign language recognition system for communicating to people with disabilities,” Procedia Comput. Sci., vol. 216, no. 2022, pp. 13–20, 2022, doi: 10.1016/j.procs.2022.12.106.

A. Young, R. Oram, and J. Napier, “Hearing people perceiving deaf people through sign language interpreters at work: on the loss of self through interpreted communication,” J. Appl. Commun. Res., vol. 47, no. 1, pp. 90–110, 2019, doi: 10.1080/00909882.2019.1574018.

F. Fitriyani, L. Q. Ainii, R. Jannah, and S. Maryam, “Analysis of Sign Language Skills in Improving Communication and Learning for Deaf Children,” Contin. Educ. J. Sci. Res., vol. 5, no. 1, pp. 30–39, 2024, doi: 10.51178/ce.v5i1.1757.

L. Ismail, N. Shahin, H. Tesfaye, and A. Hennebelle, “VisioSLR?: A Vision Data-Driven Framework for Sign Language Video Recognition and Performance Evaluation on Fine-Tuned YOLO Models,” Procedia Comput. Sci., vol. 257, pp. 85–92, 2025, doi: 10.1016/j.procs.2025.03.014.

T. Haamann and D. Basten, “The role of information technology in bridging the knowing-doing gap: an exploratory case study on knowledge application,” J. Knowl. Manag., vol. 23, no. 4, pp. 705–741, Jan. 2019, doi: 10.1108/JKM-01-2018-0030.

J. Homepage, I. Gusti, A. Oka Aryananda, and F. Samopa, “MALCOM: Indonesian Journal of Machine Learning and Computer Science Comparison of the Accuracy of The Bahasa Isyarat Indonesia (BISINDO) Detection System Using CNN and RNN Algorithm for Implementation on Android,” Malcom Indones. J. Mach. Learn. Comput. Sci., vol. 4, no. 3, pp. 1111–1119, 2024, doi: https://doi.org/10.57152/malcom.v4i3.1465.

B. Paneru, B. Paneru, and K. N. Poudyal, “Advancing human-computer interaction: AI-driven translation of American Sign Language to Nepali using convolutional neural networks and text-to-speech conversion application,” Syst. Soft Comput., vol. 6, no. October, p. 200165, 2024, doi: 10.1016/j.sasc.2024.200165.

D. Kumari and R. S. Anand, “Isolated Video-Based Sign Language Recognition Using a Hybrid CNN-LSTM Framework Based on Attention Mechanism,” Electronics, vol. 13, no. 7, 2024, doi: https://www.mdpi.com/2079-9292/13/7/1229.

J. Huang and V. Chouvatut, “Video-Based Sign Language Recognition via ResNet and LSTM Network,” J. Imaging, vol. 10, no. 6, 2024, doi: doi.org/10.3390/jimaging10060149.

A. Baihan, A. I. Alutaibi, M. Alshehri, and S. K. Sharma, “Sign language recognition using modified deep learning network and hybrid optimization?: a hybrid optimizer ( HO ) based optimized CNNSa-LSTM approach,” Sci. Rep., vol. 14, no. 1, 2024, doi: https://www.nature.com/articles/s41598-024-76174-7.

S. Kumar, R. Rani, and U. Chaudhari, “MethodsX Real-time sign language detection?: Empowering the disabled community,” MethodsX, vol. 13, no. May, p. 102901, 2024, doi: 10.1016/j.mex.2024.102901.

S. Thamear, A. Al, L. Salman, Y. Azhana, A. Saif, and M. Khadim, “Instant Sign Language Recognition by WAR Strategy Algorithm Based Tuned Machine Learning,” Int. J. Networked Distrib. Comput., vol. 12, no. 2, pp. 344–361, 2024, doi: 10.1007/s44227-024-00039-8.

I. H. Sarker, “Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions,” SN Comput. Sci., vol. 2, no. 6, pp. 1–20, 2021, doi: 10.1007/s42979-021-00815-1.

L. Alzubaidi et al., Review of deep learning: concepts, CNN architectures, challenges, applications, future directions, vol. 8, no. 1. Springer International Publishing, 2021. doi: 10.1186/s40537-021-00444-8.

T. Talaei Khoei, H. Ould Slimane, and N. Kaabouch, “Deep learning: systematic review, models, challenges, and research directions,” Neural Comput. Appl., vol. 35, no. 31, pp. 23103–23124, 2023, doi: 10.1007/s00521-023-08957-4.

A. Abulwafa, “A Survey of Deep Learning Algorithms and its Applications,” Nile J. Commun. Comput. Sci., vol. 0, no. 0, pp. 0–0, 2022, doi: 10.21608/njccs.2022.139054.1000.

M. Kaur and A. Mohta, “A Review of Deep Learning with Recurrent Neural Network,” Proc. 2nd Int. Conf. Smart Syst. Inven. Technol. ICSSIT 2019, no. Icssit, pp. 460–465, 2019, doi: 10.1109/ICSSIT46314.2019.8987837.

W. Fang, Y. Chen, and Q. Xue, “Survey on Research of RNN-Based Spatio-Temporal Sequence Prediction Algorithms,” J. Big Data, vol. 3, no. 3, pp. 97–110, 2021, doi: 10.32604/jbd.2021.016993.

K. Choudhary et al., “Recent advances and applications of deep learning methods in materials science,” npj Comput. Mater., vol. 8, no. 1, 2022, doi: 10.1038/s41524-022-00734-6.

K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9908 LNCS, pp. 630–645, 2016, doi: 10.1007/978-3-319-46493-0_38.

D. Hindarto, “Use ResNet50V2 Deep Learning Model to Classify Five Animal Species,” J. JTIK (Jurnal Teknol. Inf. dan Komunikasi), vol. 7, no. 4, pp. 758–768, 2023, doi: 10.35870/jtik.v7i4.1845.

E. Firasari, N. Khasanah, F. L. D. Cahyanti, D. N. Kholifah, U. Khultsum, and F. Sarasati, “Performance Evaluation of ResNet50 and MobileNetV2 in Skin Cancer Image Classification with Various Optimizers,” in 2024 International Conference on Information Technology Research and Innovation (ICITRI), International Conference on Information Technology Research and Innovation (ICITRI), 2024, pp. 376–380. doi: 10.1109/ICITRI62858.2024.10698943.

M. Reyad and A. M. Sarhan, “A modified Adam algorithm for deep neural network optimization,” Neural Comput. Appl., vol. 35, no. 23, pp. 17095–17112, 2023, doi: 10.1007/s00521-023-08568-z.

M. A. Mahjoubi et al., “Optimizing ResNet50 performance using stochastic gradient descent on MRI images for Alzheimer’s disease classification,” Intell. Med., vol. 11, no. October 2024, p. 100219, 2025, doi: 10.1016/j.ibmed.2025.100219.

A. Bintang, “Sistem Isyarat Bahasa Indonesia (SIBI),” Kaggle. Accessed: May 14, 2025. [Online]. Available: https://www.kaggle.com/datasets/alvinbintang/sibi-dataset

A. I. Putri and Mustakim, "Implementation of Convolutional Neural Network for Hijab Classification Using Deep Learning Approach," 2025 IEEE International Conference on Artificial Intelligence and Mechatronics Systems (AIMS), Sumedang, Indonesia, 2025, pp. 1-6, doi: 10.1109/AIMS66189.2025.11229507.