Tensor Methods: Revolutionizing Data Science and Machine Learning via Higher-Order Arrays By Dimitris G. Chachlakis Modern engineering systems collect large volumes of data measurements across diverse sensing modalities. These measurements can naturally be arranged in higher-order arrays of scalars which are commonly referred to as multi-way arrays or tensors. A 1-way tensor is a standard vector, a 2-way tensor is a standard matrix, and a 3-way array is a data cube of scalars. For higher-order tensors, visualization on paper is a challenging task and is left to the imagination. Tensors find important applications across fields of science and engineering such as data analysis, pattern recognition, computer vison, machine learning, wireless communications, and biomedical signal processing, among others. Data that can be naturally arranged in tensors are, for instance, Amazon reviews in the form of a data cube with modes user-product-word, crime activity of a city in the form of a 4-way array with modes day-hour-community-crime type, Uber pickups across a city in the form of a 4-way tensor with modes date-hour-latitude-longitude, and ratings of Netflix movies in the form of a data cube with modes rating-customer/usermovie title, to name a few.
Fig. 1: Visual illustration of 1-way, 2-way, and 3-way tensors.
Fig. 2: An image of downtown Rochester stored as a data cube. Original image credit: Patrick Ashley. Storing, processing, and analyzing tensor data in their natural form enables the discovery of patterns and underlying data structures that would have otherwise stayed hidden. Tensor methods comprise an appealing set of mathematical tools for processing tensor data and are widely considered an extension of the standard matrix methods (e.g., Principal-Component Analysis, Singular Value Decomposition, Independent Component Analysis, etc.) that have been widely used over the past decades. At their core, tensor methods rely on theory of tensor algebra which is also known as multi-linear algebra. Tensor algebra is an extension of the standard matrix algebra (linear algebra). For context, linear algebra operates on matrices and vectors –e.g., operations in the form of matrix-to-matrix product or vector-to-matrix product. Tensor algebra generalizes standard linear algebra and defines operations on tensors –e.g., operations in the form of tensor-to-matrix product and tensor-to-tensor product, to name a few. On one hand, tensor algebra has many similarities with matrix algebra, but on the other hand, it exhibits some notable differences. Despite this fact, tensors and tensor algebra exhibit many meritorious properties and have found applications in a diverse array of applications. For instance, tensor operations are highly parallelizable enabling faster training and inference, they are flexible in the sense that they allow training and inference to be decoupled, and tensors naturally model higher-order relationships in the processed data, to name a few.
34 | The ROCHESTER ENGINEER APRIL 2021
campus feature article
