WCSE

A CNN-BiGRU Text Classification Model Merging Positional Encoding and Attention Mechanism

Abstract— Aiming at the problems that Convolution Neural Network (CNN) and Bi-directional Gated Recurrent Unit (BiGRU) have insufficient ability to extract text features and cannot obtain the word weight that has a great impact on classification in text, a text classification model based on positional encoding and attention mechanism is proposed. Firstly, positional encoding is introduced before CNN input to obtain the word vector containing text position information, which improves the limited ability of convolution sliding window to extract word association information. At the same time, multichannel CNN and BiGRU are used to obtain local important features and global context information respectively. Finally, the attention mechanism is introduced to fuse the important output features of each channel, and the classification result is obtained by Softmax. The experimental results of two public data sets show that the model can effectively extract text semantic features and improve the accuracy of text classification.

Index Terms— positional encoding, attention mechanism, Convolution Neural Network, Bi-directional Gated Recurrent Unit, feature fusion

Mengna Zhang
School of Computer, Xi’an University of Posts and Telecommunications
Weiwei Kong
School of Computer, Xi’an University of Posts and Telecommunications; Guangxi Key Laboratory of Trusted Software, Guilin University of Electronic Technol
Jinbao Teng
School of Computer, Xi’an University of Posts and Telecommunications
Ze Wang
School of Computer, Xi’an University of Posts and Telecommunications

[Download]

Cite: Mengna Zhang, Weiwei Kong, Jinbao Teng, Ze Wang, "A CNN-BiGRU Text Classification Model Merging Positional Encoding and Attention Mechanism," WCSE 2022 Spring Event: 2022 9th International Conference on Industrial Engineering and Applications, pp. 1-10, Sanya, China, April 15-28, 2022.