ISBN: 978-981-18-5852-9 DOI: 10.18178/wcse.2022.04.069
Multi-dimensional Metamorphic Testing Method in Vehicle Intelligent Identification System
Abstract— The image recognition systems, like other Deep Learning-driven systems, face the problem of Oracle Problem. In order to solve the problem of a single metamorphic relation in the image recognition system testing, a multi-dimensional metamorphic testing method is proposed. First, the multi-dimensional metamorphic relations of general image scene and transformation image background region is constructed; Then Then the derived test images are generated by image transformation technology, and the authenticity and naturalness of the derived test images are detected by introducing structural similarity. Finally, the test results are analyzed by Mean Absolute Error. Experiments conducted on Baidu AI service platform, Jingdong AI open platform and Aliyun visual intelligence open platform show that the multi-dimensional metamorphic testing method detects 3724, 2432 and 2747 inconsistencies in the identification results of the three platforms, and each metamorphic relation in the multi-dimensional metamorphic testing method can detect hundreds of inconsistent behaviors in recognition results. The image recognition system is tested more comprehensively in different image scenes, and the defects of the system are exposed more easily.
Index Terms— metamorphic testing; metamorphic relation; image recognition systems; oracle problem.
Jiaze Sun
School of Computer Science and Technology, Xi'an University of Posts and Telecommunications
Yanyan Zhang
School of Computer Science and Technology, Xi'an University of Posts and Telecommunications
Shuyan Wang
School of Computer Science and Technology, Xi'an University of Posts and Telecommunications
Wei Wu
School of Computer Science and Technology, Xi'an University of Posts and Telecommunications
Cite: Jiaze Sun, Yanyan Zhang, Shuyan Wang, Wei Wu, " Multi-dimensional Metamorphic Testing Method in Vehicle Intelligent Identification System, " WCSE 2022 Spring Event: 2022 9th International Conference on Industrial Engineering and Applications, pp. 581-588, Sanya, China, April 15-18, 2022.