Orthogonal Transform Domain Based Neural Network Layers

In this thesis, we present a family of orthogonal-transform-based neural network (OTNN) layers to improve the performance of neural networks. We explore the concept of using orthogonal transforms in DNN layers to create OTNN layers that can better capture the important features of the input data. The thesis describes the design and implementation of OTNN layers using various orthogonal transforms such as Discrete Cosine Transform (DCT), Hadamard Transform (HT), and Discrete Wavelet Transform (DWT). Convolutional filtering operations are performed in the transform domain using element-wise multiplications, taking advantage of well-known convolution theorems. Trainable soft-thresholding layers that remove noise in the transform domain bring nonlinearity to the transform domain layers. The proposed layers reduce the number of parameters and multiplications significantly while improving the accuracy results of regular ResNets on the ImageNet-1K classification task. For example, compared to the vanilla ResNet-50 model, the DCT-based revised ResNet-50 reaches 0.82\% higher center-crop top-1 accuracy on the ImageNet-1K dataset with 11.5\% fewer parameters and 11.5\% fewer MACs. Furthermore, the proposed layers can be inserted with a batch normalization layer before the global average pooling layer in the conventional ResNets as an additional layer to improve classification accuracy with a negligible increase in the number of parameters and computational cost. For instance, without changing the convolutional base, an extra DCT-based layer improves the center-crop top-1 accuracy of ResNet-18 0.74\% on the ImageNet-1K dataset with only 2.3\% extra parameters and 0.3\% extra MACs. Moreover, we propose a hybrid quantum-classical HT layer for quantum computers. Since computing the $2\times2$ quantum HT is $O(1)$ in quantum computers and the quantum gates can be computed in parallel, the computational complexity of the hybrid quantum-classical HT on a $N$-length vector is $O(N)$. On the other hand, the complexity of the classical fast HT is $O(N\log N)$. Therefore, the HT-based layer gains benefit from quantum computers in terms of computational efficiency. In practice, we extended the OTNN layers into other tasks such as ECG-derived respiration rate estimation and ember fire detection, and the OTNN layers improved the performance of the classical models in these tasks. In the end, we present a Fourier-analysis-based pruning method for MobileNet-V2 and present its application to video-based wildfire detection. Overall, the thesis provides a comprehensive exploration of the concept of using orthogonal transforms in DNN layers to create OTNN layers that can improve accuracy and reduce the computational cost of DNNs. The proposed OTNN layers have potential applications in various fields, including image classification, detection, and data reconstruction.