余弦退火

AI学习的“变速箱”：深入浅火腿余弦退火

在人工智能，特别是深度学习领域，我们常常会听到各种高深莫测的技术名词。其中，“余弦退火”（Cosine Annealing）就是一个听起来有些抽象，但实际上非常巧妙和实用的优化策略。今天，我们就用大白话和生活中的例子，一起揭开它的神秘面纱。

余弦退火交互式演示

AI如何“学习”？从“下山寻宝”说起

想象一下，你是一位寻宝高手，听说在大山深处有一个藏宝地。这个藏宝地就隐藏在山势最低的“山谷”里。你的任务就是从山顶出发，找到这个最低的山谷。

在AI训练中，“寻找山谷”这个过程，就是让模型学习数据的规律，找到最优的参数组合，从而达到最好的预测或识别效果。这里的“山谷”，指的是损失函数（Loss Function）的最小值点，而我们每走一步调整参数的过程，就是“优化”。

那么，你是怎么下山的呢？你不可能闭着眼睛乱跑，而是需要根据当前所处位置的坡度，来决定下一步怎么走，走多远。这个“走多远”，就是我们AI学习中的一个核心概念——学习率（Learning Rate）。

  • 学习率高（步子大）： 如果你刚开始在山顶，地势很陡峭，你可以迈开大步往前冲，这样能快速下到山谷的大致区域。AI模型在训练初期通常会设置一个较高的学习率，以快速探索参数空间，避免训练过慢。
  • 学习率低（步子小）： 当你逐渐靠近山谷底部时，地势变得平缓，如果你还迈着大步，很可能会一不小心就跨过了最低点，又跳到另一边的山上，甚至在谷底附近来回震荡，永远找不到精确的最低点。这时候，你就需要把步子放小，小心翼翼地慢慢挪动，才能精准地找到谷底。AI模型在训练后期也需要一个较低的学习率，以便更精细地优化参数，收敛到最优解。

所以，学习率不是一成不变的，它是需要不断调整的。这种调整学习率的策略，我们称之为学习率调度器（Learning Rate Scheduler）。余弦退火，就是一种非常优雅和高效的学习率调度器。

余弦退火：一种“顺应自然”的步速调整法

你可能见过很多调整学习率的方法，比如每训练几轮（epoch）就把学习率设为原来的一半（步长衰减），或者线性地让学习率逐渐减小。这些方法固然有效，但余弦退火却提供了一种更为平滑和自然的方式。

“余弦”指的是数学中的余弦函数，它的曲线是像波浪一样起伏的。余弦退火的灵感就来源于此，它让学习率随着训练的进行，按照余弦函数曲线的形状来变化。

具体来说，在一个训练周期内（比如你计划走多长时间下山）：

  1. 初期： 学习率会从一个较高的值开始，但下降的速度相对较慢。这就像你刚下山时，虽然知道要往下走，但还没有完全进入状态，可以稳健地迈步。
  2. 中期： 学习率下降的速度会加快。这对应余弦曲线在中间部分下降最快的阶段。这个时候，你已经大致锁定了山谷的位置，可以加速冲刺，快速接近目标。
  3. 后期： 学习率下降的速度又会逐渐减慢，最终会降到一个非常小的值。这就像你到达山谷底部，需要非常细微的调整才能找到最准确的藏宝点一样。AI模型通过这种方式，可以在训练后期进行微调，避免错过最优解。

这种曲线变化的好处是，它给了模型在训练初期足够的“探索”能力，又在训练后期提供了足够的“精细优化”能力，而且整个过程非常平滑，避免了学习率突然变化带来的不稳定性。

余弦退火的好处与最新应用

余弦退火不仅能帮助模型找到更好的解，还有助于模型收敛得更快、更稳定。它能够让模型在优化过程中更好地“跳出”局部最优解（就像下山时，偶尔迈个大步可以越过一些小坑，避免困在小坑里）。

在最新的AI发展中，“余弦退火”这个概念也一直在演进和应用：

  • 与“热重启”结合 (Cosine Annealing with Warm Restarts)： 这是目前非常流行的一种变体。想象一下，你找到了一个山谷，但你怀疑附近还有没有更深的山谷。于是，你在这个山谷停留一阵子后（学习率降到最低），突然又“瞬移”回了高处（学习率瞬间恢复到最大值），然后再次按照余弦曲线下山。这种周期性的重启和学习率衰减，可以鼓励模型探索更广阔的参数空间，从而更有可能找到全局最优解，并提高模型的泛化能力。许多框架如PyTorch都内置了 CosineAnnealingWarmRestarts 类来实现这一功能。例如，最近的研究表明，在训练大型转化器增强残差神经网络时，余弦退火在降低损失方面是有效的。
  • 在大型模型训练中的应用： 余弦退火在诸如大语言模型（LLMs）等需要长时间训练的复杂模型中尤为重要。例如，在2025年10月24日的最新文章中提到，在训练一个17M参数的中文GPT模型时，就采用了线性预热（warm-up）与余弦退火机制相结合的动态调度策略，以确保模型平稳收敛。
  • 与“学习率预热”（Warmup）结合： 在训练初期，模型参数是随机初始化的，如果一开始学习率就很高，可能会导致模型不稳定。因此，通常会将余弦退火与学习率预热策略结合。预热阶段会先用一个很小的学习率让模型“热身”，慢慢提高学习率，然后再进入余弦退火阶段，这样能进一步提高训练的稳定性。
  • 新的变体和优化： 研究人员还在探索余弦退火的更多可能性，例如2024年3月的一项研究提出了“循环对数退火”（cyclical log annealing）方法，它采用了比余弦退火更激进的重启机制，有望在某些在线凸优化框架中发挥作用。

结语

“余弦退火”就像AI模型学习过程中的一个智能“变速箱”，它根据学习的阶段，自动调整学习率的大小，让模型既能快速探索，又能精细收敛。这种基于数学之美的优化策略，使得AI模型能够更有效、更稳定地找到“宝藏”，在各个领域发挥出更大的潜力。

The “Gearbox” of AI Learning: A Deep Dive into Cosine Annealing

In the field of artificial intelligence, especially deep learning, we often hear various esoteric technical terms. Among them, “Cosine Annealing” is one that sounds somewhat abstract, but is actually a very clever and practical optimization strategy. Today, let’s unveil its mystery using plain language and real-life examples.

Interactive Demo of Cosine Annealing

How Does AI “Learn”? Starting from “Treasure Hunting Down the Mountain”

Imagine you are a treasure hunter who hears that there is a treasure hidden deep in the mountains. This treasure is hidden in the “valley” with the lowest elevation. Your task is to start from the top of the mountain and find this lowest valley.

In AI training, the process of “finding the valley” involves the model learning the patterns of data and finding the optimal combination of parameters to achieve the best prediction or recognition results. Here, the “valley” refers to the minimum point of the Loss Function, and the process of adjusting parameters with each step we take is called “optimization”.

So, how do you go down the mountain? You can’t run blindly with your eyes closed; instead, you need to decide how to go and how far to go based on the slope of your current location. This “how far to go” corresponds to a core concept in AI learning—Learning Rate.

  • High Learning Rate (Big Steps): If you are just starting at the top of the mountain seeking a path, the terrain is steep, and you can take big strides forward to quickly get down to the general area of the valley. AI models usually set a higher learning rate in the early stages of training to quickly explore the parameter space and avoid slow training.
  • Low Learning Rate (Small Steps): As you gradually approach the bottom of the valley, the terrain becomes flatter. If you continue to take big strides, you might accidentally step over the lowest point and jump to the mountain on the other side, or even oscillate back and forth near the bottom of the valley, never finding the precise lowest point. At this time, you need to shorten your steps and move slowly and carefully to pinpoint the bottom of the valley. AI models also need a lower learning rate in the later stages of training to optimize parameters more finely and converge to the optimal solution.

Therefore, the learning rate is not static; it needs to be constantly adjusted. This strategy of adjusting the learning rate is called a Learning Rate Scheduler. Cosine Annealing is a very elegant and efficient learning rate scheduler.

Cosine Annealing: A “Nature-Conforming” Pace Adjustment Method

You may have seen many methods for adjusting learning rates, such as halving the learning rate every few training rounds (step decay) or linearly decreasing the learning rate. While these methods are effective, Cosine Annealing offers a smoother and more natural approach.

“Cosine” refers to the cosine function in mathematics, whose curve fluctuates like a wave. The inspiration for Cosine Annealing comes from this, allowing the learning rate to change according to the shape of the cosine function curve as training progresses.

Specifically, within a training cycle (e.g., how long you plan to walk down the mountain):

  1. Early Stage: The learning rate starts at a relatively high value, but the rate of decrease is relatively slow. This is like when you just started going down the mountain; although you know you need to go down, you haven’t fully gotten into the rhythm yet, so you can take steady steps.
  2. Middle Stage: The rate of decrease of the learning rate speeds up. This corresponds to the phase where the cosine curve drops the fastest in the middle. At this time, you have roughly locked onto the position of the valley and can accelerate your sprint to quickly approach the target.
  3. Late Stage: The rate of decrease of the learning rate slows down again, eventually dropping to a very small value. This is like arriving at the bottom of the valley, needing very fine adjustments to find the most accurate treasure spot. Through this method, the AI model can perform fine-tuning in the later stages of training to avoid missing the optimal solution.

The benefit of this curve change is that it gives the model sufficient “exploration” ability in the early stages of training, and sufficient “fine optimization” ability in the later stages, and the entire process is very smooth, avoiding instability caused by sudden changes in learning rate.

Benefits and Latest Applications of Cosine Annealing

Cosine annealing not only helps the model find better solutions but also aids the model in converging faster and more stably. It enables the model to better “jump out” of local optima during the optimization process (just like taking a big step occasionally when going down a mountain can cross some small pits, avoiding getting stuck in them).

In the latest AI developments, the concept of “Cosine Annealing” has also been evolving and applied:

  • Cosine Annealing with Warm Restarts: This is currently a very popular variant. Imagine you found a valley, but you suspect there might be a deeper valley nearby. So, after staying in this valley for a while (learning rate drops to the minimum), you suddenly “teleport” back to a high place (learning rate instantly restores to the maximum value), and then go down the mountain again according to the cosine curve. This periodic restart and learning rate decay can encourage the model to explore a broader parameter space, thereby making it more likely to find the global optimal solution and improve the model’s generalization ability. Many frameworks like PyTorch have built-in CosineAnnealingWarmRestarts classes to implement this functionality. For example, recent research shows that cosine annealing is effective in reducing loss when training large Transformer-enhanced residual neural networks.
  • Application in Large Model Training: Cosine annealing is particularly important in complex models that require long training times, such as Large Language Models (LLMs). For instance, an article from October 24, 2025, mentioned that when training a 17M parameter Chinese GPT model, a dynamic scheduling strategy combining linear warm-up with cosine annealing was used to ensure the model converged smoothly.
  • Combination with “Warmup”: In the early stages of training, model parameters are randomly initialized. If the learning rate is high from the start, it may cause model instability. Therefore, cosine annealing is usually combined with a learning rate warmup strategy. The warmup phase starts with a very small learning rate to let the model “warm up,” slowly increasing the learning rate, and then entering the cosine annealing phase, which can further improve training stability.
  • New Variants and Optimizations: Researchers are also exploring more possibilities for cosine annealing. For example, a study in March 2024 proposed a “cyclical log annealing” method, which adopts a more aggressive restart mechanism than cosine annealing and is expected to play a role in certain online convex optimization frameworks.

Conclusion

“Cosine Annealing” is like an intelligent “gearbox” in the AI model learning process. It automatically adjusts the size of the learning rate according to the stage of learning, allowing the model to both explore quickly and converge finely. This optimization strategy based on the beauty of mathematics allows AI models to find “treasures” more effectively and stably, unleashing greater potential in various fields.