10.29327/1884517.29-12
Context: Ethical risks in Large Language Models (LLMs) are difficult to validate reproducibly because principles such as Fairness, Accountability, and Transparency (FAT) are often stated as abstract guidelines rather than measurable requirements. From a Requirements Engineering perspective, these principles can be treated as non-functional requirements, but current evaluation practices frequently rely on non-deterministic evaluators or LLM-as-a-judge approaches, limiting reproducibility. Goal: This paper proposes a systematic framework for the automated validation of ethical properties derived from FAT principles in LLMs. Method: We introduce an ethical fuzzing framework that operationalizes six of eleven FAT-derived ethical risks (R-F1, R-F2, R-F4, R-A2, R-T1, R-T2) as testing modules combining mutation-based, generation-based, differential, metamorphic, and adversarial techniques. The framework relies on deterministic oracles defined a priori, combining statistical metrics, semantic similarity via fixed SBERT embeddings, and structural verification, without depending on human or LLM-based judges. Results: The framework was evaluated on three commercial LLMs, GPT (gpt-5.2), Gemini (gemini-3-flash), and DeepSeek (deepseek-chat), comprising 8{,}880 cases and 14,160 API calls. Results show a global failure rate of 32.69% [31.72; 33.67], with the highest rates in counterfactual fairness (R-F1: 79.17%) and sensitivity to irrelevant attributes (R-T2: 70.00%). They also reveal a strong asymmetry in R-T1 between direct explanation (22.50-27.00%) and meta-explanation (0.00-1.50%), suggesting that LLM explanations may be more reliable in abstract descriptions than in concrete decision cases. Conclusion: The framework offers Requirements Engineering a mechanism to translate abstract ethical principles into measurable and auditable acceptance criteria, supporting evidence-based validation practices for AI-based systems.
Keywords: AI Ethics; Ethical Fuzzing; Large Language Models; Fairness Testing; Accountability; Transparency
@inproceedings{wer202611,
author = {Vasconcelos, J. L. and Canedo, E. D.},
title = {A Fuzzing-Based Framework for Automated Ethical Risk Detection in Large Language Models},
booktitle = {Anais do Workshop em Engenharia de Requisitos - Proceedings of the 29th Workshop on Requirements Engineering (WER2026)},
year = {2026},
issn = {2675-0066},
isbn = {978-65-02-19591-8},
doi = {10.29327/1884517.29-12}
}