MultiModal Explainable AI and Blockchain Integration for Automated Halal Verification of Cosmetic Products

Endang Supriyati; Mohammad Iqbal; Tri Listyorini

doi:10.31436/iiumej.v27i2.4059

Authors

Endang Supriyati Muria Kudus University https://orcid.org/0000-0003-0386-582X
Mohammad Iqbal Muria Kudus University https://orcid.org/0000-0002-7102-0629
Tri Listyorini Muria Kudus University https://orcid.org/0000-0002-1334-7519

DOI:

https://doi.org/10.31436/iiumej.v27i2.4059

Keywords:

Explainable Artificial Intelligence (XAI), Multi-Modal Learning, Optical Character Recognition (OCR), Halal Verification System, Blockchain

Abstract

This research develops a framework that integrates blockchain technology and MXAI (Multimodal Explainable Artificial Intelligence) to automate the authentication and verification of halal cosmetic products. Two data sources were used: text extracted by OCR (Optical Character Recognition) and text manually input from cosmetic labels. After the pre-processing stage, the text data are analyzed using zero-shot classification to determine the inspection results, namely halal, haram, or syubhat. The inspection is conducted using MXAI, with decisions based on confidence scores and SHAP values. The inspection results are converted into digital reports as SHA-256 hashes and stored as Merkle roots on the blockchain, allowing users to download certificates as QR codes. The halal status experiment on the cosmetics dataset achieved an accuracy of 94.5% for classification, with a 3.3% improvement over baseline models, evaluated using stratified 5-fold cross-validation. This system enhances transparency, accountability, and public trust in automated halal certification. The contribution of this research is the integration of MXAI and blockchain technology into a single intelligent halal verification system, which can be extended to other supply chain sectors.

ABSTRAK: Kajian ini menggabungkan teknologi rantaian blok dan MXAI (Kecerdasan Buatan Penjelasan Multimodal) bagi automasi dan jaminan pengesahan produk kosmetik halal. Dua sumber data digunakan: teks yang diekstrak oleh OCR (Pengesahan Optik Karakter) dan teks yang dimasukkan secara manual daripada label kosmetik. Selepas peringkat pra-pemprosesan, data teks dianalisis menggunakan klasifikasi sifar-tembakan bagi menentukan keputusan pemeriksaan, iaitu halal, haram, atau syubhat. Pemeriksaan dijalankan menggunakan MXAI, dengan keputusan berdasarkan skor keyakinan dan nilai SHAP. Keputusan pemeriksaan ditukar menjadi laporan digital dalam bentuk hash SHA-256 dan disimpan sebagai punca Merkle pada rantaian blok, membolehkan pengguna memuat turun sijil dalam bentuk kod QR. Eksperimen status halal pada set data kosmetik menunjukkan ketepatan 94.5% untuk prestasi pengelasan, dengan peningkatan 3.3% berbanding model asas, dinilai menggunakan pengesahan silang 5-lipatan berstrata. Sistem ini meningkatkan ketelusan, akauntabiliti, dan kepercayaan awam dalam pensijilan halal automatik. Sumbangan penyelidikan ini ialah melalui integrasi MXAI dan teknologi rantaian blok dalam satu sistem pengesahan halal pintar tunggal, yang boleh diperluas kepada sektor rantaian bekalan lain.

Downloads

Download data is not yet available.

References

A. Azwar and Abur Hamdi Usman, “Artificial Intelligence In The Halal Industry: Trends And Global Research Opportunities,” Int. J. Halal Ind., vol. 1, no. 1, pp. 46–67, 2025, doi: 10.20885/ijhi.vol1.iss1.art4.

A. Masood, S. R. H. Hati, and A. A. Rahim, “Halal Cosmetics Industry for Sustainable Development: a Systematic Literature Review,” Int. J. Bus. Soc., vol. 24, no. 1, pp. 141–163, 2023, doi: 10.33736/ijbs.5609.2023.

F. Omar, A. Nabot, and R. Alqirem, “Transforming Decision Support Systems Through Artificial Intelligence: Enhancing Analytics, Automation, and Interaction,” 2025, pp. 479–491. doi: 10.1007/978-3-031-90271-0_34.

A. K. B. Arnob, R. R. Chowdhury, N. A. Chaiti, S. Saha, and A. Roy, “A Comprehensive Systematic Review Of Intrusion Detection Systems: Emerging Techniques, Challenges, and Future Research Directions,” J. Edge Comput., vol. 4, no. 1, pp. 73–104, 2025, doi: 10.55056/jec.885.

N. Nurhayati, A. Chelbi, A. Ben Mabrouk, and F. C. Young, “Unleashing Halal Potential: Exploring The Dynamics of Halal Logistics In Tangerang City, Indonesia,” Agrociencia, pp. 1–13, Mar. 2025, doi: 10.47163/agrociencia.v59i3.3272.

F. Sunmola, G. Baryannis, A. Tan, K. Co, and E. Papadakis, “Holistic Framework for Blockchain-Based Halal Compliance in Supply Chains Enabled by Artificial Intelligence,” Systems, vol. 13, no. 1, pp. 1–26, 2025, doi: 10.3390/systems13010021.

K. Kumar, V. Kumar, Seema, M. K. Sharma, A. A. Khan, and M. J. Idrisi, “A Systematic Review of Blockchain Technology Assisted with Artificial Intelligence Technology for Networks and Communication Systems,” J. Comput. Networks Commun., vol. 2024, pp. 1–15, Feb. 2024, doi: 10.1155/2024/9979371.

V. T. Hoang et al., “Halal or Not: Knowledge Graph Completion for Predicting Cultural Appropriateness of Daily Products,” IEEE Access, vol. 13, no. December 2024, pp. 15158–15167, 2025, doi: 10.1109/ACCESS.2025.3528488.

A. Ghosh et al., “Blockchain-Assisted Serverless Framework for AI-Driven Healthcare Applications,” Int. J. Adv. Comput. Sci. Appl., vol. 16, no. 5, pp. 473–482, 2025, doi: 10.14569/IJACSA.2025.0160546.

S. Bhatia and A. S. Albarrak, “A Blockchain-Driven Food Supply Chain Management Using QR Code and XAI-Faster RCNN Architecture,” Sustain., vol. 15, no. 3, 2023, doi: 10.3390/su15032579.

R. Salama, D. Cacciagrano, and F. Al-Turjman, “Connecting AI and Blockchain to Improve Security of Financial Services,” Barolli, L. Adv. P2P, Parallel, Grid, Cloud Internet Comput., vol. 232., doi: https://doi.org/10.1007/978-3-031-76462-2_7.

M. Vasileiou et al., “Digital Transformation of Food Supply Chain Management Using Blockchain: A Systematic Literature Review Towards Food Safety and Traceability,” Bus. Inf. Syst. Eng., 2025, doi: 10.1007/s12599-025-00948-0.

Nor Aina Mardhiah Nazim and Syarifah Md Yusof, “Halal Legislation Enforcement of Food Premises in Malaysia,” Glob. J. al thaqafah, vol. SI, no. 1, pp. 151–164, Jul. 2023, doi: 10.7187/GJATSI072023-12.

S. Wicaksono, M. K. H. Al-Asy’ari, E. D. N. El-Adibah, and Iffahasanah, “MAQASHID SHARIA PROGRESSIVE: Anatomical and Transformational of Halal Institutions in UIN KHAS Jember,” El-Mashlahah, vol. 13, no. 2, pp. 107–132, 2023, doi: 10.23971/el-mashlahah.v13i2.7370.

D. K. Elwaked, N. M. Kassim, and R. Thurasamy, “Enhancing Internal Integration Through AI: A Resource-Based View With Data-Driven Culture As Moderator,” J. Logist. Informatics Serv. Sci., vol. 12, no. 2, pp. 209–228, 2025, doi: 10.33168/JLISS.2025.0213.

N. Zulkeefli, Siti Nur Izzati, Hashim, “Comparison of CNN-based Algorithms for Halal Logo Recognition,” J. Syst. Manag. Sci., vol. 12, no. 5, pp. 155–168, 2022, doi: 10.33168/JSMS.2022.0510.

S. Al-Thani, Shaikha Nasser;Anjum, “Comparative performance of ChatGPT, Gemini, and final-year emergency medicine clerkship students in answering multiple-choice questions: implications for the use of AI in medical education,” Int. J. Emerg. Med., vol. 18, no. 1, p. 146, 2025, doi: 10.1186/s12245-025-00949-6.

M. T. Zamir, S. U. Khan, and S. U. Khan, “Explainable AI-Driven Analysis of Radiology Reports Using Text and Image Data: Experimental Study,” JMIR Form. Res., vol. 9, 2025, doi: 10.2196/77482.