To address the issue of inefficient inference computation of modern artificial neural networks caused by abundant gated neural units within, where the weight parameters are not adequately compressed, a method termed as hierarchical symmetric singular value decomposition (HSSVD) of matrices was proposed. Further, parallel compression and acceleration method for gated neural units was proposed for recurrent neural network (RNN) and Transformer. Firstly, based on the two factor matrices achieved by the normal singular value decomposition (SVD) for large-scale weight matrix, their linearly independent column vectors and row vectors were separately folded and split again by SVD to construct the HSSVD format of the weight matrix, then the optimal rank selection algorithm of HSSVD under the constraint of absolute error was proposed. Secondly, the parallel compression method for general gated neural units was proposed, and the corresponding acceleration algorithms orienting to the specialized gated neural units in RNN and Transformer were also proposed. Lastly, to deal with the requirement of translating natural sentences to structured query language (SQL) language, which was familiar in financial software engineering, the effect of the proposed method in practical engineering application was given. The experimental results demonstrate that, compared with existing typical tensor decomposition methods, HSSVD can achieve satisfactory effects of weight parameters compression and inference computation acceleration for both RNN and Transformer at the same time. On the typical video recognition datasets UCF11 and YCF, under the precondition of the total number of parameters of RNN is about 8 400, HSSVD can gain 88.1% and 88.9% recognition accuracy, and the time consumptions of inference are just 13.9 ms and 14.1 ms, respectively. On the benchmark image recognition dataset ImageNet, under the precondition of the total number of parameters of Transformer is about 11.9×10⁶, HSSVD can obtain the highest 82.9% recognition accuracy, and the inference efficiency can reach 3 195.9 images per second. For the engineering application of translating natural sentences to SQL language, HSSVD can bring 89.3% memory economy and 41.6% generating efficiency optimization under the condition of lossless lightweight.