Local Elasticity: A Phenomenological Approach Toward Understanding Deep LearningSpeaker: Hangfeng He, University of PennsylvaniaTime: Wednesday, Nov 17, 2021, 10:00AM - 11:00AM, Eastern Time Zoom Link: contact tml.online.seminars@gmail.com Abstract:Motivated by the iterative nature of training neural networks, we ask: If the weights of a neural network are updated using the induced gradient on an image of a tiger, how does this update impact the prediction of the neural network at another image (say, an image of another tiger, a cat, or a plane)? To address this question, we introduce a phenomenon termed local elasticity. Roughly speaking, our experiments show that modern deep neural networks are locally elastic in the sense that the change in prediction is likely to be most significant at another tiger and least significant at a plane, at late stages of the training process. We further illustrate some implications of local elasticity by relating it to the neural tangent kernel and improving on the generalization bound for uniform stability. Finally, we offer a local-elasticity-focused agenda for future research toward a theoretical foundation for deep learning.Speaker's BioHangfeng He is a fifth-year Ph.D. student, working with Dan Roth and Weijie Su, in the Department of Computer and Information Science at the University of Pennsylvania. His research interests include machine learning and natural language processing, with a focus on moving beyond scale-driven learning. Specifically, he works on incidental supervision for natural language understanding, the interpretability of deep neural networks, question answering, reasoning in natural language, and structured data modeling. |