posted on 2020-12-01, 00:00authored byVignesh Ganapathiraman
Finding representations of data that are useful for the underlying prediction task has been an important and active pursuit in machine learning. Modern-day deep learning algorithms have been hugely successful in making accurate data-driven predictions by automatically learning highly discriminative
representations of data via multiple nonlinear transformations. In addition to
learning discriminative representations, it is also useful to encode problem /
task specific information (priors) directly in the representations. For instance, in
image classification, it is useful to enforce the constraint that a machine
learning model's prediction should not change when the image is perturbed
spatially a.k.a translation invariance. One way to realize non-trivial structured priors in a model is via carefully designed losses and regularizers. However regularization approaches result in a hard non-convex optimization problem. Non-convex optimization problems often pose serious challenges in training and seldom result in guaranteed optimal solutions (global optima).
In this thesis, we address the above challenges in representation learning by:
(1) exploring priors that encode non-trivial problem / data specific structures
(2) designing efficient convex models and training algorithms to automatically learn these structures n a data-dependent fashion (3) providing learning and approximation guarantees wherever possible. Our first approach towards convex representation learning is to explicitly model structured priors in the latent layer of a two-layer neural network. The resulting non-convex optimization problem is then relaxed to obtain a convex model that is able to obtain all the structural regularities of the original non-convex problem. Second, we develop a new convex representation learning framework, based on semi-inner-product spaces, to model the so-called generalized invariances in an efficient and scalable manner.
History
Advisor
Zhang, Xinhua
Chair
Zhang, Xinhua
Department
Computer Science
Degree Grantor
University of Illinois at Chicago
Degree Level
Doctoral
Degree name
PhD, Doctor of Philosophy
Committee Member
Reyzin, Lev
Ziebart, Brian
Liu, Bing
Schuurmans, Dale