2026年7月1日（周三）下午两点半，深圳河套学院荣幸地邀请到明尼苏达大学双城分校电气与计算机工程系的洪明毅教授，在B401教室带来主题为“大语言模型预训练优化：全新算法、理论与研究洞见”的学术报告分享。诚挚欢迎学院师生及对该领域感兴趣的产学研界同仁参与讲座。

Title

大语言模型预训练优化：

全新算法、理论与研究洞见

Optimization for Large Language Model Pre-training: New Algorithms, Theory, and Insights

About the Speaker

洪明毅 教授

Prof. Mingyi Hong

洪明毅现任明尼苏达大学双城分校电气与计算机工程系教授。他的研究方向为面向信号处理、机器学习及基础大模型的优化理论与算法开发。其研究成果曾斩获两项IEEE信号处理学会最佳论文奖（2021年、2022年）、国际华人数学家联盟最佳论文奖（2020年）等多项学术荣誉。他是亚马逊学者，2022年IEEE皮埃尔- 西蒙・拉普拉斯青年技术成就奖、2025年运筹学与管理科学学会（INFORMS）优化分会埃贡・巴拉斯奖得主，同时为IEEE会士。

Swipe up for EN version

Mingyi Hong is a Professor in the Department of Electrical and Computer Engineering at the University of Minnesota, Minneapolis. His research has been focused on developing optimization theory and algorithms for applications in signal processing, machine learning and foundation models. His work has received two IEEE Signal Processing Society Best Paper Awards (2021, 2022), an International Consortium of Chinese Mathematicians Best Paper Award (2020), among others. He is an Amazon Scholar, and he is the recipient of the 2022 Pierre-Simon Laplace Early Career Technical Achievement Award from IEEE, and the 2025 Egon Balas Prize from INFORMS Optimization Society. He is a Fellow of IEEE.

Abstract

大语言模型（LLM）预训练已成为当今资源消耗最庞大的计算任务之一，需要耗费海量算力、内存与能源。其核心是一类看似简单、实际求解难度极高的非凸随机优化问题，实际训练效果极大程度取决于优化器的选择。虽然Adam等自适应优化器已经成为事实上的行业标准，但仍存在若干基础性问题尚未得到解答：优化器真正必需的内存开销究竟是多少？基础随机梯度下降（SGD）为何在该场景下性能表现极差？涅斯捷罗夫动量这类经典加速思想，能否稳定适配现代神经网络架构？

本次报告将结合实证分析、理论研究与算法设计，介绍一系列针对上述问题的最新研究工作。首先，洪教授将介绍SCALE（随机列归一化末层动量优化器），这是一款基于SGD构建的极简优化器。该算法结合按列梯度归一化，且仅在输出层施加动量机制，内存占用仅为Adam 的35%–45%，性能却能与Adam持平甚至更优。其次，洪教授将分析SGD在大语言模型预训练中表现乏力的内在原因，并指出随机梯度存在两大关键特性：（1）权重范数与梯度范数的比值严重失衡，且该比值会随批次规模增大而不断升高；（2）输出层各类词元对应的梯度差异巨大。二者共同压缩了SGD可用的学习率区间。随后证明基于上述规律设计简单的梯度裁剪策略，就能大幅缩小SGD与Adam之间的性能差距。第三，洪教授将介绍EMA-Nesterov算法，它是涅斯捷罗夫前瞻动量的指数移动平均变体，能够稳定加速效果；在凸优化场景下保留经典加速收敛速率，并且在各类大语言模型预训练基准测试中，持续提升Adam、SOAP、Muon等主流优化器的训练效果。综合来看，上述研究为大规模神经网络训练的优化器设计提供了全新视角，同时引出优化理论与现代机器学习交叉领域中诸多有待探索的研究方向。

Swipe up for EN version

Pre-training large language models (LLMs) has become one of the most resource-intensive computational tasks of our time, demanding enormous amounts of compute, memory, and energy. At its core lies a deceptively simple, yet notoriously difficult, nonconvex stochastic optimization problem, whose practical performance depends heavily on the choice of optimizer. Although adaptive optimizers such as Adam have been the de facto standard, a number of fundamental questions remain unresolved: How much memory overhead is truly necessary? Why does plain stochastic gradient descent (SGD) perform so poorly in this regime? And can classical acceleration ideas, such as Nesterov’s momentum, be made to work reliably for modern neural architectures?

In this talk, Prof. Hong will present a series of recent works that address these questions through a combination of empirical analysis, theoretical study, and algorithm design. First, Prof. Hong will discuss SCALE (Stochastic Column-normAlized Last-layer momEntum), a minimalist optimizer that is built upon SGD that matches or exceeds the performance of Adam while using only 35–45% of its memory footprint, by combining column-wise gradient normalization with momentum applied only at the output layer. Second, Prof. Hong will examine why SGD struggles in LLM pre-training and identify two key properties of the stochastic gradients: (1) a disproportionately large weight-to-gradient norm ratio that grows with the batch size, and (2) a severe per-token-class disparity at the output layer, that together restrict SGD’s admissible learning rate. They then show that simple clipping mechanisms exploit this understanding to close most of the gap to Adam. Third, Prof. Hong will introduce EMA-Nesterov, an exponential moving-average variant of Nesterov’s lookahead that stabilizes acceleration, retains the classical accelerated convergence rate in the convex regime, and consistently improves modern optimizers (Adam, SOAP, Muon) across language-model pre-training benchmarks. Taken together, these works offer a fresh perspective on optimizer design for large-scale neural network training, and point toward a number of open research questions at the interface of optimization theory and modern machine learning.

Host

王祥丰 教授

Prof. Xiangfeng Wang

Date & Time

2026年7月1日（星期三）

下午14:30-16:00

July 1, 2026, Wednesday,

2:30pm-4:00pm

Venue

深圳河套学院B401教室

（深圳市福田区福保街道红棉路6号，地图导航“深圳河套学院-南门”）

Room B401, Shenzhen Loop Area Institute

(6 Hongmian Rd, Fubao Sub-Street, Futian District, Shenzhen, navigate to "Shenzhen Loop Area Institute (South Gate)" on the map.)