Research | Lee Optimization Group

Model Compression As AI models become larger and more computationally demanding, compression is essential for making them practical and sustainable. We develop optimization methods that reduce training and inference costs while preserving model performance. Our work focuses on extreme compression across the AI model lifecycle.

Extreme Sparsity (ICLR '26, ICML '25, EMNLP '24) We push the limits of LLM sparsity through advanced optimization frameworks such as ADMM, aiming for robust generalization even at extreme compression levels.

Joint Compression We develop unified frameworks that integrate pruning, quantization, and distillation to reduce error accumulation and preserve core model capabilities, including reasoning, in highly compressed states.

Distributed Training The shift toward massive-scale models has made distributed training essential for workloads that no single machine can support. From an optimization perspective, the key challenge is maintaining efficiency while managing the overhead of communication between devices. Our research develops robust algorithms for decentralized and high-latency training environments.

Communication-Efficient Training (arXiv '26) We design methods that reduce the heavy data-sharing requirements of distributed training through information compression, reduced synchronization, and zeroth-order optimization.

Asynchronous Training (arXiv '26) We create algorithms that allow devices to work independently and reduce waiting time, with a focus on correcting errors caused by slightly outdated information.

Device-Heterogeneous Training We develop adaptive strategies for imbalanced computational resources across devices, aiming for consistent global convergence despite variations in hardware performance.

Deep Learning Applications Optimization is not only a subject of study in itself, but also a versatile lens through which we tackle challenges in interpretability, uncertainty quantification, continual learning, and other real-world deep learning systems. Our research explores how optimization principles can be extended and applied to address such challenges.

Trustworthy Machine Learning (NeurIPS '25, ICML '23, ACL '26) We bridge complex model architectures and human understanding, from uncovering model logic in realistic scenarios to evaluating the reliability of model outputs.

Continual Adaptation (NeurIPS '25) We study constrained optimization problems where models adapt to new knowledge without disrupting what has already been learned, opening principled approaches to continual learning.

Advanced Optimization Optimization has long been a source of crucial ideas that drastically enhance all corners of deep neural network training, and many of its most impactful questions remain open. Our research investigates these questions and develops optimization principles and algorithms for modern deep learning systems.

Optimization for Generalization (UAI '25, ICML '25) We study the role of loss landscape curvature in generalization and develop flatness-oriented optimization strategies with both theoretical advantages and scalability for modern deep neural networks.

Zeroth-Order Optimization for Black-Box Model Training (ICLR '25) We develop methods that operate without direct gradient access, motivated by settings such as proprietary model APIs and decentralized learning systems where gradients are unavailable.

Research Focus

Collaborators