Geodesics, Splines, and Embeddings in Spaces of Images

Abstract

This thesis focuses on two main mathematical topics: geodesic and spline interpolation of images using the meta­morphosis model, and learning low bending and low distortion image manifold embeddings. <br /> The metamorphosis model, originally introduced by Miller, Trouve, and Younes, is a well-known approach for image morphing, which involves computing a visually appealing transition between two images such that semantically corresponding regions are mapped onto each other. The model defines Riemannian manifold where the geodesic paths between images are obtained by minimization of the path energy measuring the dissipation caused by the motion field and the image intensity change along the motion paths. The existing color-based approaches, proposed by Berkels, Effland, and Rumpf, suffer from the appearance of blending and the loss of sharp interfaces along the geodesic paths. To overcome these issues, this thesis proposes a metamorphosis model in a deep feature space, which combines image channels with semantic features obtained by a deep convolutional neural network-based extraction operator. In addition, an anisotropic regularization operator depending on the magnitude of image gradients, is applied to allow for discontinuities in deformation fields. As the main contribution, we study the variational time discretization, prove the existence of time discrete geodesic paths for this model, and discuss their convergence to the time continuous counterpart in the context of Mosco-convergence. The fully discrete model is based on finite difference approximation and cubic Hermite spline approximation. Numerical results show that this approach outperforms color-based morphing in terms of visual quality. <br /> The above approach allows for smooth interpolation between two fixed images. The next step is to find a smooth interpolation between multiple fixed key frame images. Both the color-based and the deep feature-based piecewise geodesic interpolation lack smoothness at the interpolation data points, resulting in jerking when ani­mating the resulting family of images. In several different shape spaces, the solution to this problem is provided by cubic splines, defined as minimizers of functional penalizing acceleration terms. We propose a spline interpolation approach within the metamorphosis model based on the minimization of a functional which combines quadratic functionals of the Eulerian motion acceleration and the second material derivative of the image intensity. Although physically intuitive, this separation of flow and image intensity acceleration does not result in a fully Riemannian model. Building upon the above-discussed model, we introduce a time discrete model and study its convergence to the time continuous model. This allows for establishing the existence of metamorphosis splines as minimizers of the time continuous spline functional. We complement the time discretization with a finite difference-based spatial discretization and a stable B-spline approximation of warping operation. Numerical results demonstrate the robustness and versatility of our method, showing smoothness, stability, and other qualitative properties compared to piecewise geodesic interpolation. <br /> Processing of high-dimensional data in their original form, with underlying very high- or even infinite-dimensio­nal manifold, an example of which is the metamorphosis model, is a challenging task, both in theoretical and numerical computation senses. To this end, it represents a substantial simplification if the data lies on a low­dimensional manifold. This leads to the task of the identification of given data sets consisting of objects embedded in a very high-dimensional space with points in a low-dimensional manifold, as one of the central machine learn­ing tasks. Autoencoders, consisting of an encoder and a decoder, represent a widely used tool for this task. The encoder embeds the input data manifold into a lower-dimensional latent space, while the decoder represents the in­verse map, providing a parametrization of the data manifold by the latent manifold. Further data processing tasks, an example of which is interpolation, may be substantially simplified by the suitable regularity and structure of the embedded manifold. To this end, we propose and analyze a regularization for learning the encoder component of an autoencoder, based on a loss functional that prefers isometric and extrinsically flat embeddings. The training data consists of pairs of nearby points on the input manifold, together with their Riemannian distance and average. These values can be efficiently evaluated due to the hand-crafted construction of the datasets. The discrete sam­pling loss functional is computed via Monte Carlo integration. Its limit, as the number of samples tends to infinity, is given by the sampling strategy dependent nonlocal continuous loss functional. The Mosco-limit of these func­tionals, as the distances of points in sampled pairs tend to zero, is identified as a local geometric loss functional that promotes isometry and flatness of the embedding. Numerical tests on images encoding different explicitly given data manifolds, validate the effectiveness of this approach in obtaining smooth manifold embeddings and enabling reasonable interpolation between nearby points on the manifold by the linear interpolation in the latent space.