Real-Time Indonesian Sign Language Recognition Using Long Short-Term Memory and IndoBERT Contextual Modeling

Amanda Betania Maritza; Rizka Ardiansyah

doi:10.57152/malcom.v6i3.2839

Authors

Amanda Betania Maritza Universitas Tadulako
Rizka Ardiansyah Universitas Tadulako

DOI:

https://doi.org/10.57152/malcom.v6i3.2839

Keywords:

Contextual Language Modeling, IndoBERT, LSTM, Real-Time Recognition, SIBI

Abstract

Communication barriers between deaf and hearing individuals remain a major challenge in inclusive education and social interaction. Most Indonesian Sign Language (Sistem Isyarat Bahasa Indonesia = SIBI) recognition systems focus solely on isolated gesture classification, without contextual understanding. This study proposes a real-time SIBI recognition system integrating LSTM-based temporal modeling and IndoBERT contextual language modeling. Hand keypoints were extracted from gesture sequences and processed using Long Short-Term Memory (LSTM) to recognize dynamic alphabet gestures. The predicted letter sequences were refined using a Trie-based lexical filter and IndoBERT to generate contextually appropriate word predictions. The dataset consisted of 780 gesture sequences representing 26 alphabet classes. The results showed that the system achieved 89.42% accuracy, 91.57% precision, 89.42% recall, and 88.74% F1-score while maintaining real-time performance at an average processing speed of 249.18 FPS and a latency of 129.46 ms per sample. Statistical evaluation confirmed that camera distance significantly affected recognition performance, while integrating IndoBERT improved contextual word-prediction accuracy from 18.2% to 58.64% (3.2× improvement). McNemar analysis further verified that the improvement was statistically significant, highlighting the effectiveness of contextual language modeling for enhancing semantic interpretation SIBI recognition. The proposed framework improves semantic interpretation and real-time usability for SIBI-based assistive communication systems.

Downloads

Download data is not yet available.

References

A. Podury, N. T. Jiam, M. Kim, J. I. Donnenfield, and A. Dhand, “Hearing and sociality: the implications of hearing loss on social life,” Front. Neurosci., vol. 17, no. 1245434, Oct. 2023, doi: 10.3389/fnins.2023.1245434.

P. A. Rodríguez-Correa, A. Valencia-Arias, O. N. Patiño-Toro, Y. Oblitas Díaz, and R. Teodori De la Puente, “Benefits and development of assistive technologies for Deaf people’s communication: A systematic review,” Front. Educ. (Lausanne)., vol. 8, no. 1121597, Apr. 2023, doi: 10.3389/feduc.2023.1121597.

S. Syamsuddin, T. Pristiwaluyo, W. A Saleh, and Z. Zulfitrah, “Development of Indonesian Sign Language System (SIBI) Dictionary Application for Students with Special Needs,” AL-ISHLAH: Jurnal Pendidikan, vol. 15, no. 3, pp. 3132–3143, Sep. 2023, doi: 10.35445/alishlah.v15i3.3529.

K. M. Concha, F. P. Zenteno, and J. T. Alfaro, “Use of artificial intelligence in libraries a systematic review, 2019-2023,” South African Journal of Library and Information Science, vol. 90, no. 2, Aug. 2024, doi: 10.7553/90-2-2387.

D. Van??ek, Y. I. Yorulmaz, and D. Dobrovská, “A holistic exploration of student attitudes toward AI use in higher education: an international comparison,” Cogent Education, vol. 12, no. 1, 2025, doi: 10.1080/2331186X.2025.2571691.

Asma Syafira and R. A. Yuana, “Interactive Mobile Applications for Social Communication in Deaf and Hearing-Impaired Populations: A Systematic Review,” International Journal of Interactive Mobile Technologies (iJIM), vol. 20, no. 02, pp. 78–95, Jan. 2026, doi: 10.3991/ijim.v20i02.57207.

M. Sari and E. R. Jamzuri, “Hand Sign Recognition of Indonesian Sign Language System (SIBI) Using Inception V3 Image Embedding and Random Forest,” Jurnal Rekayasa Sistem dan Teknologi Informasi, vol. 9, no. 2, pp. 258–265, Apr. 2025, doi: 10.29207/resti.v9i2.6156.

S. Shivani and S. Bal Gupta, “A Comprehensive Analysis of Recognition of Hand Gestures using Machine Learning,” Makara Journal of Technology, vol. 29, no. 1, pp. 48–60, Apr. 2025, doi: 10.7454/mst.v29i1.1679.

J. Huang and V. Chouvatut, “Video-Based Sign Language Recognition via ResNet and LSTM Network,” J. Imaging, vol. 10, no. 6, p. 149, Jun. 2024, doi: 10.3390/jimaging10060149.

K. Xia, W. Lu, H. Fan, and Q. Zhao, “A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation,” Sensors, vol. 22, no. 23, p. 9107, Dec. 2022, doi: 10.3390/s22239107.

K. Navendu and V. Sahula, “Word Level Sign Language Recognition using MediaPipe and LSTM-GRU Network,” in 2024 IEEE International Symposium on Smart Electronic Systems (iSES), IEEE, Jul. 2024, pp. 13–18. doi: 10.1109/iSES63344.2024.00014.

T. Purbojo and A. P. Wijaya, “Enhancing Pose-Based Sign Language Recognition: A Comparative Study of Preprocessing Strategies with GRU and LSTM,” Advance Sustainable Science, Engineering and Technology, vol. 7, no. 2, Feb. 2025, doi: 10.26877/asset.v7i2.1658.

I. V. Lemmuela, M. Ayub, and O. Karnalim, “Dynamic Sign Language Recognition in Bahasa using MediaPipe, Long Short-Term Memory, and Convolutional Neural Network,” Journal of Information Systems Engineering and Business Intelligence, vol. 11, no. 1, pp. 17–29, Feb. 2025, doi: 10.20473/jisebi.11.1.17-29.

E. Altiarika and W. P. Sari, “Pengembangan Deteksi Realtime untuk Bahasa Isyarat Indonesia dengan Menggunakan Metode Deep Learning Long Short Term Memory dan Convolutional Neural Network,” Jurnal Teknologi Informatika dan Komputer, vol. 9, no. 1, pp. 1–13, Mar. 2023, doi: 10.37012/jtik.v9i1.1272.

P. Ho and H. Santoso, “LSTM-Based Hand Gesture Recognition for Indonesian Sign Language System (SIBI) on Affix, Alphabet, Number, and Word,” Journal of Applied Informatics and Computing (JAIC), vol. 9, no. 3, p. 928, Jun. 2025, doi: 10.30871/jaic.v9i3.9607.

C. A. Sari, E. H. Rachmawanto, Z. Saifullah, C. Jatmoko, and D. Sinaga, “Real-time detection of indonesian sign language (ISL) gestures based on long short-term memory,” Journal of Soft Computing Exploration, vol. 5, no. 3, pp. 251–262, Sep. 2024, doi: 10.52465/joscex.v5i3.452.

D. Bala, B. Sarkar, M. I. Abdullah, M. A. Hossain, and I. Abdullah, “American Sign Language Alphabets Recognition using Convolutional Neural Network,” Article in International Journal of Knowledge Based Computer Systems, vol. 6, no. 1, pp. 33–40, Jun. 2021, [Online]. Available: https://www.researchgate.net/publication/352878275

G. H. Samaan et al., “MediaPipe’s Landmarks with RNN for Dynamic Sign Language Recognition,” Electronics (Switzerland), vol. 11, no. 19, p. 3228, Oct. 2022, doi: 10.3390/electronics11193228.

W. Jintanachaiwat et al., “Using LSTM to translate Thai sign language to text in real time,” Discover Artificial Intelligence, vol. 4, no. 1, Dec. 2024, doi: 10.1007/s44163-024-00113-8.

R. R. Agustin, H. Maulana, and E. P. Mandyartha, “Detection of Actions BISINDO (Indonesian Sign Language) Into Text-to-Speech using Long Short-Term Memory with Mediapipe Holistics,” Jurnal Teknik Informatika (Jutif), vol. 5, no. 4, pp. 1051–1061, Nov. 2023, doi: 10.52436/1.jutif.2024.5.4.1492.

L. Katriel and R. Wiguna, “Sentence-Level Indonesian Sign Language (BISINDO) Recognition Using 3D CNN-LSTM and 3D CNN-BiLSTM Models (Indonesian Sign Language); 3D Convolutional Neural Network (3D CNN); Long Short-Term Memory (LSTM) Network; Bidirectional Long Short-Term Memory (BiLSTM); Connectionist Temporal Classification (CTC),” IJACSA) International Journal of Advanced Computer Science and Applications, vol. 16, no. 9, pp. 433–443, 2025, doi: 10.14569/IJACSA.2025.0160941.

Y. Setiawan and L. A. Wulandhari, “Comparative Analysis of IndoBERT and LSTM for Multi-Label Text Classification of Indonesian Motivation Letter,” JOIN (Jurnal Online Informatika), vol. 10, no. 27, pp. 260–269, 2025, doi: 10.15575/join.v10i2.1499.

D. Setiawan, D. U. Iswavigra, and E. Anggiratih, “Implementation of IndoBERT for Sentiment Analysis of the Constitutional Court’s Decision Regarding the Minimum Age of Vice Presidential Candidates,” Scientific Journal of Informatics, vol. 12, no. 3, pp. 397–406, 2025, doi: 10.15294/sji.v12i3.26360.