Evaluating A Custom-Developed Ai Chatbot for Enhancing Gamified Phishing Awareness Training in A Small Non-Profit Organization: A Comparative Case Study at ThesciencefFor Africa Foundation

Abstract

This thesis investigates the effectiveness of gamification, enhanced with a Retrieval- Augmented Generation (RAG) GPT-4-powered AI chatbot, in strengthening phishing awareness training among staff at the Science for Africa Foundation. In light of the escalating cybersecurity threats targeting non-governmental organizations (NGOs), the study responds to the growing concern that conventional training approaches—often passive and one-size-fits-all—fail to engage users meaningfully or build lasting behavioural change. By integrating gamified training modules with an interactive, real-time chatbot assistant, this research explores a novel, blended approach designed to improve user engagement, information retention, and practical response to phishing attempts. The study is grounded in literature tracing back to the 1990s, which identified user awareness and education as key pillars in phishing defence. More recent studies have emphasized the value of gamification in cybersecurity training, citing increased motivation and better learning outcomes. However, limited research has examined the role of generative AI, especially domain- specific chatbots, in complementing gamified learning environments. This study fills that gap by combining gamification and generative AI into a unified training platform and assessing its impact in a real-world organizational setting. A comparative case study design was employed involving two groups of employees. The control group underwent traditional, classroom-style phishing awareness training, while the intervention group engaged with Click Armor gamified modules, supported by a custom-developed RAG-enhanced GPT-4 chatbot capable of answering phishing- related queries and reinforcing learning in real time. A total of 58 participants—drawn from a diverse employee base across departments—took part in the study. Data collection tools included InfoSight phishing simulation tests, structured observation checklists, engagement surveys, and focus group discussions. Quantitative analysis using descriptive statistics revealed that the intervention group outperformed the control group in both engagement levels and phishing recognition scores. Qualitative analysis, conducted through thematic coding of focus group transcripts, further demonstrated that the AI chatbot contributed to a more personalized and responsive learning experience. Participants reported greater confidence, clarity, and enthusiasm during training sessions, citing the chatbot’s immediate feedback and contextual explanations as particularly helpful. The study concludes that gamified phishing awareness training, when augmented with AI chatbot support, offers a significantly more effective and scalable solution for NGOs. It recommends broader adoption of such technologies in cybersecurity education to address the evolving threat landscape in resource-constrained sectors.