WCSE

Enhanced Knowledge Distillation for YOLO via Attention Mechanisms in Smart City Applications

Abstract— This paper explores the application of knowledge distillation (KD), particularly in the context of YOLO models used for real-time object detection in smart city scenarios. KD is a technique that transfers knowledge from a large teacher model to a smaller student model, enabling the latter to achieve higher accuracy while maintaining efficiency for deployment on resource-limited devices. As a popular method for distillation, Channel-wise Distillation (CWD) has been widely used for classification tasks. However, it faces limitations when applied to YOLO models for object detection, such as susceptibility to background noise. To address these issues, we introduce attention modules into the CWD framework. These modules help the model focus on relevant features, reducing background noise and improving detection accuracy, especially in complex scenes. We evaluate the performance of applying attention modules on three popular detection tasks in smart city applications, and the results demonstrate the effectiveness of our proposed method in enhancing the precision and generalization ability of YOLO models.

Index Terms— Knowledge distillation, Object detection, YOLO, Smart city

Yanbo Wang, Shiyong Wang, Weichao Lan, Zhikun Hong, Hang Yao, Yang Wang
Inspur Smart City Technology Co., Ltd, CHINA
Lili Zhang
Inspur Group Co., Ltd, CHINA

[Download]

Cite: Yanbo Wang, Shiyong Wang, Weichao Lan, Lili Zhang, Zhikun Hong, Hang Yao, Yang Wang, "Enhanced Knowledge Distillation for YOLO via Attention Mechanisms in Smart City Applications", 2025 the 15th International Workshop on Computer Science and Engineering (WCSE 2025), pp. 10-14, Jeju Island, South Korea, June 28-30, 2025.