Development of a Question Answering System Using the Best Matching 25 (BM25) Method to Support Final Project Completion for Informatics Engineering Students at UNISSULA
Keywords:
Academic Information, BM25, Information Retrieval, Question Answering System, Text ProcessingAbstract
This study developed and evaluated a BM25-based question-answering system to support final-project information retrieval for Informatics Engineering students at Universitas Islam Sultan Agung. The system used the official Final Project Guidebook and Supervision Logbook as its knowledge corpus and integrated document reconstruction, heading-aware chunking, paragraph overlap, lexical preprocessing, synonym-based query expansion, heading-weighted indexing, BM25 ranking, and source-grounded answer presentation. Evaluation was conducted using 50 manually mapped ground-truth queries and measured through Recall@1, Recall@3, Recall@5, and Mean Reciprocal Rank. The results showed that 41 queries retrieved the relevant passage at rank 1, eight at rank 2, and one at rank 3. Recall@1 reached 82.00%, while Recall@3 and Recall@5 both achieved 100.00%, with an MRR of 0.9067. These findings indicate that the proposed lexical retrieval architecture can reliably recover authoritative academic information while maintaining transparency, interpretability, and source traceability. The remaining limitation concerns first-rank discrimination among procedurally similar passages, suggesting that future improvements should prioritize reranking or hybrid retrieval without sacrificing complete Top-3 coverage. The approach also provides a reproducible baseline for evaluating more complex semantic retrieval methods in academic information environments.
Downloads
References
Agustiani, A. D., Putri, S. M., Hutahaean, J., Rizqi, M., Alifi, M. R., & Hodijah, A. (2026). Evaluating RAG performance on small language models for low-resource devices through chunking and retrieval methods. Jurnal Online Informatika, 11(1), 243–255. https://doi.org/10.15575/join.v11i1.1733
Alsaedi, S., Gad-Elrab, A. A., Noaman, A., & Eassa, F. (2024). Two-level information-retrieval-based model for bug localization based on bug reports. Electronics, 13(2), 321. https://doi.org/10.3390/electronics13020321
Amin, A., Bustomi, T., & Harianto, K. (2024). Implementasi chatbot sebagai asisten virtual untuk membantu mahasiswa dalam mendapatkan informasi akademik. Sebatik, 28(2), 1–12. https://doi.org/10.46984/sebatik.v28i2.0000
Atmojo, T. T., & Kunang, Y. N. (2023). Machine learning-based e-archive for archives management of South Sumatra Province. Journal of Information Systems and Informatics, 5(4), 1491–1507. https://doi.org/10.51519/journalisi.v5i4.566
Buckland, M. K., & de Fremery, W. (2024). Relevance, works and context engineering. Proceedings from the Document Academy, 11(2), Article 5. https://doi.org/10.35492/docam/11/2/5
Bunda, Y. P., Siregar, M. F., Sains, F., Studi, P., Informasi, S., Tjut, U., & Dhien, N. (2025). Pengembangan Question Answering System Menggunakan Large Language Model UU TNI 2025. Jurnal Pendidikan Dan Teknologi Indonesia, 5(10), 3041-3050. https://doi.org/10.52436/1.jpti.1396
Darmawan, I., Arifin, M. N., Ubaidi, U., Ramadhani, N., & Dewi, N. P. (2025). Algoritma Okapi Bm25 Dalam Sistem Pencarian Informasi Berbasis Teks. Insand Comtech: Information Science and Computer Technology Journal, 10(1), 30-37. https://doi.org/10.53712/jic.v10i1.2649
Eboseremen, B., Adebayo, A., Essien, I., Afuwape, A., Soneye, O., & Ofori, S. (2021). The role of natural language processing in data-driven research analysis. International Journal of Multidisciplinary Research and Growth Evaluation, 2(1), 935-942. https://doi.org/10.54660/.IJMRGE.2021.2.1.935-942
Fahreza, R. A., & Anwar, M. (2026). Routing struktural adaptif: Arsitektur hybrid chunking untuk RAG pada dokumen pembelajaran teknik informatika. Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), 6(Nomor Jurnal), 1356–1365. https://doi.org/10.58578/masaliq.v6i3.10134
Gienapp, L., Scells, H., Deckers, N., Bevendorff, J., Wang, S., Kiesel, J., Syed, S., Fröbe, M., Zuccon, G., Stein, B., Hagen, M., & Potthast, M. (2024). Evaluating generative ad hoc information retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1916–1929). Association for Computing Machinery. https://doi.org/10.1145/3626772.3657849
Ikhsan, M. F., & ’Uyun, S. . . (2026). Analisis Perbandingan Metode Information Retrieval (IR) Pada Implementasi Pencarian Ayat Al-Qur’an. Journal Software, Hardware and Information Technology, 6(1), 83–92. https://doi.org/10.24252/shift.v6i1.272
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 6769–6781). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.550
Lasminingrum, D. P., Puspaningrum, E. Y., & Mulyo, B. M. (2026). Perbandingan metode lexical dan semantic retrieval: BM25, IndoSBERT baseline, dan IndoSBERT fine-tuned pada pencarian dokumen berbahasa Indonesia. Prosiding Seminar Nasional Penelitian dan Pengabdian kepada Masyarakat LPPM Universitas 'Aisyiyah Yogyakarta, 4. https://proceeding.unisayogya.ac.id/index.php/prosemnaslppm/article/view/2088
Lawami, R. H., Harahap, N. S., & Iskandar, I. (2024). Pembuatan Chatbot Telegram untuk Layanan Pencarian Al-Quran. Jurnal Indonesia: Manajemen Informatika dan Komunikasi, 5(2), 1846-1857. https://doi.org/10.35870/jimik.v5i2.761
Lawrie, D., Mayfield, J., Yang, E., Yates, A., MacAvaney, S., Pradeep, R., ... & Soldani, L. (2025). Neuclirbench: A modern evaluation collection for monolingual, cross-language, and multilingual information retrieval. arXiv preprint arXiv:2511.14758. https://doi.org/10.48550/arXiv.2511.14758
Mallia, A., Mackenzie, J., Suel, T., & Tonellotto, N. (2022). Faster learned sparse retrieval with guided traversal. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery. https://doi.org/10.1145/3477495.3531774
Moffat, A. (2022). Batch evaluation metrics in information retrieval: Measures, scales, and meaning. IEEE Access, 10, 105564-105577. https://doi.org/10.1109/ACCESS.2022.3211668
Molavi, M., & Zeinali, H. (2023, July). SLT at SemEval-2023 Task 1: Enhancing visual word sense disambiguation through image text retrieval using BLIP. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023) (pp. 1921-1925). https://doi.org/10.18653/v1/2023.semeval-1.264
Mursidah, E., Ambarwati, L., & Karima, F. A. (2022). Implementasi Chatbot layanan informasi pendaftaran mahasiswa baru program pascasarjana Departemen Teknik Informatika ITS. J. Ilm. Nero, 7(1), 43-52. https://doi.org/10.21107/nero.v7i1.276
Raj, M., & Mishra, N. (2025). Exploring new approaches for information retrieval through natural language processing. arXiv preprint arXiv:2505.02199. https://doi.org/10.48550/arXiv.2505.02199
Sager, P. J., Kamaraj, A., Grewe, B. F., & Stadelmann, T. (2025). Deep Retrieval at CheckThat! 2025: identifying scientific papers from implicit social media mentions via hybrid retrieval and re-ranking. arXiv preprint arXiv:2505.23250. https://doi.org/10.48550/arXiv.2505.23250
Schofield, J., Tian, S., Truong, H. T. T., & Heil, M. (2025). DS@ GT at CheckThat! 2025: exploring retrieval and reranking pipelines for scientific claim source retrieval on social media discourse. arXiv preprint arXiv:2507.06563. https://doi.org/10.48550/arXiv.2507.06563
Setiawan, A., Abidin, Z., & Imammudin, M. (2025). Impact of preprocessing on Indonesian extractive summarization using LexRank, TextRank, DivRank, and cosine similarity. G-Tech: Jurnal Teknologi Terapan, 9(4), 2311–2321. https://doi.org/10.70609/g-tech.v9i4.8306
Siregar, M. E., Benedicta, L., Adrielvino, N. A., Park, S. B., Gading, J., Boulevard, S., Tower, N., Sangereng, C., & Dua, K. K. (2025). Pengembangan sistem pencarian resep makanan dengan implementasi text preprocessing dan BM25. Jurnal Nasional Komputasi dan Teknologi Informasi, 8(3), 1458–1465. https://doi.org/10.32672/jnkti.v8i3.9137
Solihin, A. A., Utomo, F. S., & Barkah, A. S. (2026). Impact of Stopword Variation on Qur'anic Text Classification using Support Vector Machine and Backpropagation. Journal of Innovation Information Technology and Application (JINITA), 8(1), 173-183. https://doi.org/10.35970/jinita.v8i1.3069
Suparto, A., Clement, M. J., Rahman, A., & Irsyad, H. (2025). Implementasi Preprocessing Dan Synonym Expansion Untuk Sistem Temu Kembali Berita Bahasa Indonesia. Jurnal Software Engineering and Computational Intelligence, 3(01), 10-17. https://doi.org/10.36982/jseci.v3i01.5405
Wang, L., Yang, N., & Wei, F. (2023, December). Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 9414-9423). https://doi.org/10.18653/v1/2023.emnlp-main.585
Wang,S. (2024). Exploring the fusion of natural language processing and information retrieval. Applied and Computational Engineering, 76, 27-32. https://doi.org/10.54254/2755-2721/76/20240559
Wei, Y. C., Li, F. S., & Glass, J. (2023). Expand, rerank, and retrieve: Query reranking for open-domain question answering. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 12131–12147). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.768
Zhang, W., & Lin, J. (n.d.). Agentic Information Retrieval. Association for Computing Machinery.
