19 Autoencoders
A central goal of deep learning is to discover representations of data that are useful for one or more subsequent applications. One well-established approach to learning internal representations is called the auto-associative neural network or autoencoder. This consists of a neural network having the same number of output units as inputs and which is trained to generate an output y that is close to the input x. Once trained, an internal layer within the neural network gives a representation z(x) for each new input. Such a network can be viewed as having two parts. The first is an encoder, which maps the input x into a hidden representation z(x), and the second is a decoder, which maps the hidden representation onto the output y(z). If an autoencoder is to find non-trivial solutions, it is necessary to introduce some form of constraint, otherwise the network can simply copy the input values to the outputs. This constraint might be achieved, for example, by restricting the dimensionality of z relative to that of x or by requiring z to have a sparse represen-
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 C. M. Bishop, H. Bishop, Deep Learning, https://doi.org/10.1007/978-3-031-45468-4_19
563