SARSA

揭秘SARSA：智能体如何在“摸着石头过河”中学习（面向非专业人士）

在人工智能的浩瀚领域中，有一种方法让机器能够像人类一样通过“试错”来学习，这就是强化学习（Reinforcement Learning, RL）。强化学习的核心思想是：智能体（agent）在一个环境中行动，获得奖励或惩罚，然后根据这些反馈来调整自己的行为，以期在未来获得更多的奖励。而SARSA，就是强化学习家族中一个非常重要的成员。

想象一下你正在学习玩一个新游戏，比如走迷宫。你一开始可能不知道怎么走，会四处碰壁（惩罚），偶尔也会找到正确的路径（奖励）。久而久之，你会记住哪些路能通向宝藏，哪些路是死胡同。SARSA算法，就是让机器以更系统、更“脚踏实地”的方式，去学习这种“摸着石头过河”的策略。

SARSA：一个“行动派”的学习方法

SARSA这个名字本身就揭示了它的工作原理，它是“State-Action-Reward-State-Action”这五个英文单词首字母的缩写，翻译过来就是“状态-行动-奖励-新状态-新行动”。这五个元素构成了一个完整的学习回路，也是SARSA算法更新其知识（或者说“Q值”）的基础。

我们用一个日常生活中的例子来具体理解这五个概念：

假设你是一个机器人，你的任务是学习如何最快地从客厅（起始点）走到厨房并泡一杯咖啡（获得奖励）。

  1. 状态（State, S）：这代表你当前所处的情况。比如，你现在在“客厅”里，这就是一个状态。
  2. 行动（Action, A）：这是你在当前状态下可以选择执行的操作。在客厅里，你可能选择“向厨房方向走”、“打开电视”、“坐下”等。
  3. 奖励（Reward, R）：这是你执行一个行动后环境给你的即时反馈。如果你“向厨房方向走”了一步，也许会得到一个小小的正奖励（比如 +1分），因为它让你更接近目标；如果你撞到了墙，可能会得到一个负奖励（比如 -5分）。当你成功泡到咖啡时，会得到一个很大的正奖励（比如 +100分）。
  4. 新状态（Next State, S’）：这是你执行行动A之后所到达的下一个状态。你从“客厅”执行“向厨房方向走”后，现在可能处于“走廊”这个新状态。
  5. 新行动（Next Action, A’）：这是SARSA最关键的地方。在你到达“走廊”这个新状态(S’)后，你根据你当前的策略，会决定下一步要执行的行动A’。比如，你可能决定在“走廊”里“继续向厨房方向走”，这就是你的新行动A’。

SARSA正是将这连续的五元组——（当前状态S，当前行动A，获得的奖励R，新状态S’，基于当前策略选择的新行动A’）——作为一个整体来学习和更新自己的行为准则。

SARSA与“更贪婪”的Q-learning有何不同？

SARSA算法常常与另一个著名的强化学习算法Q-learning拿来比较。它们的核心目的都是学习一个“Q值”（Quality Value），这个Q值代表在某个状态下采取某个行动能获得的长期总奖励的预期。拥有一个准确的Q值表，智能体就能选择在每个状态下Q值最高的行动，从而实现最优策略。

主要区别在于它们如何利用“新行动（A’）”来更新Q值：

  • SARSA（“在线/在策略”学习）：它是一个“实干派”。它会真的根据当前正在使用的策略（包括探索性行动）在S’状态选择一个A’，然后用这个真实发生的(S, A, R, S’, A’)序列来更新Q值。就像一个学开车的学员，他会根据自己当前的驾驶习惯（即使偶尔不完美）来总结经验，调整下一回的操作。这种方式让SARSA的学习过程更加“保守”和“安全”，因为它考虑到自己当前的探索行为可能带来的后果。比如，在一个有悬崖的迷宫里，SARSA会倾向于学习一条远离悬崖但可能稍长的路径，因为它在探索时会“实际走一步”进入悬崖并感受到巨大的惩罚，从而避免这条危险路径。

  • Q-learning（“离线/离策略”学习）：它是一个“理想派”。它在S’状态下，不考虑自己当前策略下一步会选择哪个行动，而是假设自己下一步总是会选择能带来最大Q值的那个理想行动来更新Q值。这就像一个学开车的学员，他会想象一个最完美的司机下一步会怎么操作，然后用这个“最优”的想象来指导自己当前行为的改进。Q-learning在学习时更“贪婪”，因为它总是假设未来会采取最优行动，因此它更容易找到环境中的最优策略。然而，如果环境中有很大的负面奖励（比如悬崖），Q-learning在探索时可能会因为假设未来总是最优而“掉入悬崖”，导致学习不稳定。

简单来说，SARSA是“我实际怎么做，就怎么学”，它关注的是“按照我的当前策略走下去的Q值”；Q-learning是“如果我未来总是做最好的选择，我当前应该怎么做”，它关注的是“未来最优选择能带来多大的Q值”。

SARSA的应用与优缺点

因为SARSA是“在策略”学习，它根据智能体实际采取的行动序列进行学习，这使得它在某些场景下特别有用：

  • 在线学习：如果智能体必须在真实环境中边学习边行动（例如，一个自动驾驶汽车在真实的道路上学习），SARSA就非常合适，因为它考虑了智能体在学习过程中采取的实际行动，以及这些行动可能带来的风险。它能学习到一个更稳健、更安全的策略，即使这个策略不总是“理论上最优”的。
  • 避免危险：在一些环境中，犯错的成本很高（例如，机器人操作机械臂，一旦操作失误可能造成物理损坏），SARSA的“保守”特性使其能够学习到避免危险区的策略。

优点:

  • 稳定性好：由于其“在策略”的特性，SARSA在学习过程中通常具有较好的稳定性。
  • 对环境探索更安全：它会把探索性动作纳入到更新中，所以在有负面奖励的风险区域，它会学习避免这些区域，从而更安全地探索。
  • 收敛速度较快：在某些情况下，SARSA算法的收敛速度较快。
  • 适合在线决策：如果代理是在线学习，并且注重学习期间获得的奖励，那么SARSA算法更加适用。

缺点:

  • 可能收敛到次优策略：由于它受到当前探索策略的限制，有时可能会收敛到一个次优策略，而不是全局最优策略。
  • 学习效率可能受限：如果探索策略效率不高，学习速度可能会受到影响。

SARSA 的发展与未来

SARSA算法最早由G.A. Rummery和M. Niranjan在1994年的论文中提及，当时被称为“Modified Connectionist Q-Learning”，随后在1996年由Rich Sutton正式提出了SARSA的概念。作为强化学习的基础算法之一，许多针对Q-learning的优化方法也可以应用于SARSA上。

尽管SARSA是一个相对传统的强化学习算法，但其“在策略”的学习方式在需要考虑实时性和安全性的应用中仍有其独特的价值。例如，在机器人控制、工业自动化等领域，智能体需要根据当前实际的动作来评估并更新其策略，SARSA可以帮助它们在复杂且充满不确定性的环境中，学习出既高效又安全的行为模式。

总而言之，SARSA算法就像一位“脚踏实地”的学徒，它通过真实地体验每一次尝试，从自己的实际行为中吸取教训，一步一个脚印地提升自己的技能。这种学习方式虽然可能不像Q-learning那样追求最极致的“理想”表现，但在很多需要谨慎和即时反馈的现实应用中，SARSA却能提供一个更加稳健和安全的解决方案。

Revealing SARSA: How Agents Learn by “Feeling the Stones to Cross the River” (For Non-Experts)

In the vast field of Artificial Intelligence, there is a method that allows machines to learn through “trial and error” just like humans, which is Reinforcement Learning (RL). The core idea of RL is: an agent acts in an environment, receives rewards or punishments, and then adjusts its behavior based on these feedbacks to gain more rewards in the future. SARSA is a very important member of the reinforcement learning family.

Imagine you are learning to play a new game, like navigating a maze. At first, you might not know how to go, hitting walls everywhere (punishment), and occasionally finding the right path (reward). Over time, you will remember which roads lead to treasure and which are dead ends. The SARSA algorithm enables machines to learn this strategy of “feeling the stones to cross the river” in a more systematic and “down-to-earth” way.

SARSA: An “Action-Oriented” Learning Method

The name SARSA itself reveals its working principle. It is an acronym for the first letters of five English words: “State-Action-Reward-State-Action”. These five elements constitute a complete learning loop and are the basis for the SARSA algorithm to update its knowledge (or “Q-value”).

Let’s use a daily life example to specifically understand these five concepts:

Suppose you are a robot, and your task is to learn how to get from the living room (start point) to the kitchen and make a cup of coffee (get reward) as quickly as possible.

  1. State (S): This represents your current situation. For example, you are now in the “living room”, which is a state.
  2. Action (A): This is the operation you can choose to perform in the current state. In the living room, you might choose “walk towards the kitchen”, “turn on the TV”, “sit down”, etc.
  3. Reward (R): This is the immediate feedback given by the environment after you perform an action. If you take a step “towards the kitchen”, you might get a small positive reward (e.g., +1 point) because it brings you closer to the goal; if you hit a wall, you might get a negative reward (e.g., -5 points). When you successfully make coffee, you get a large positive reward (e.g., +100 points).
  4. Next State (S’): This is the next state you reach after performing action A. After you execute “walk towards the kitchen” from the “living room”, you might now be in the “hallway”, which is the new state.
  5. Next Action (A’): This is the most critical part of SARSA. After you arrive at the new state “hallway” (S’), you will decide the next action A’ to execute based on your current strategy. For example, you might decide to “continue walking towards the kitchen” in the “hallway”, which is your new action A’.

SARSA uses this continuous quintuple—(Current State S, Current Action A, Received Reward R, New State S’, New Action A’ selected based on current strategy) — as a whole to learn and update its behavioral guidelines.

How is SARSA Different from the “Greedier” Q-learning?

The SARSA algorithm is often compared with another famous reinforcement learning algorithm, Q-learning. Their core purpose is to learn a “Q-value” (Quality Value), which represents the expected long-term total reward of taking a certain action in a certain state. With an accurate Q-value table, the agent can choose the action with the highest Q-value in each state, thereby achieving the optimal strategy.

The main difference lies in how they use the “Next Action (A’)” to update the Q-value:

  • SARSA (“On-Policy” Learning): It is a “realist”. It will actually choose an A’ in the S’ state based on the currently used strategy (including exploratory actions), and then use this actually occurring sequence (S, A, R, S’, A’) to update the Q-value. Like a student learning to drive, he will summarize experience and adjust the next operation based on his current driving habits (even if occasionally imperfect). This method makes SARSA’s learning process more “conservative” and “safe” because it considers the consequences that its current exploratory behavior might bring. For example, in a maze with a cliff, SARSA will tend to learn a path that is far from the cliff but possibly slightly longer, because during exploration it will “actually walk a step” into the cliff and feel the huge punishment, thereby avoiding this dangerous path.

  • Q-learning (“Off-Policy” Learning): It is an “idealist”. In the S’ state, it does not consider which action its current strategy will choose next, but assumes that it will always choose the ideal action that brings the maximum Q-value next to update the Q-value. Like a student learning to drive, he will imagine how a perfect driver would operate next, and then use this “optimal” imagination to guide the improvement of his current behavior. Q-learning is “greedier” in learning because it always assumes that the optimal action will be taken in the future, so it is easier to find the optimal strategy in the environment. However, if there are large negative rewards in the environment (such as a cliff), Q-learning might “fall off the cliff” during exploration because it assumes the future is always optimal, leading to unstable learning.

Simply put, SARSA is “Learn as I actually do“, focusing on “the Q-value of proceeding according to my current strategy”; Q-learning is “What should I do now if I always make the best choice in the future“, focusing on “how much Q-value the future optimal choice can bring”.

Applications, Pros and Cons of SARSA

Because SARSA is “on-policy” learning, it learns based on the sequence of actions actually taken by the agent, which makes it particularly useful in certain scenarios:

  • Online Learning: If the agent must learn while acting in a real environment (e.g., an autonomous car learning on a real road), SARSA is very suitable because it considers the actual actions taken by the agent during the learning process and the risks these actions may entail. It can learn a more robust and safer strategy, even if this strategy is not always “theoretically optimal”.
  • Avoiding Danger: In some environments where the cost of making mistakes is high (e.g., a robot operating a mechanical arm, where a mistake could cause physical damage), SARSA’s “conservative” nature enables it to learn strategies to avoid danger zones.

Pros:

  • Good Stability: Due to its “on-policy” nature, SARSA usually has good stability during the learning process.
  • Safer Environment Exploration: It includes exploratory actions in updates, so in risky areas with negative rewards, it learns to avoid these areas, thus exploring more safely.
  • Faster Convergence: In some cases, the SARSA algorithm converges faster.
  • Suitable for Online Decision Making: If the agent is learning online and cares about the rewards obtained during learning, the SARSA algorithm is more applicable.

Cons:

  • May Converge to Sub-optimal Policy: Because it is limited by the current exploration strategy, it may sometimes converge to a sub-optimal strategy rather than the global optimal strategy.
  • Learning Efficiency May Be Limited: If the exploration strategy is inefficient, the learning speed may be affected.

Development and Future of SARSA

The SARSA algorithm was first mentioned by G.A. Rummery and M. Niranjan in a 1994 paper, called “Modified Connectionist Q-Learning” at the time, and then formally proposed as SARSA by Rich Sutton in 1996. As one of the fundamental algorithms of reinforcement learning, many optimization methods for Q-learning can also be applied to SARSA.

Although SARSA is a relatively traditional reinforcement learning algorithm, its “on-policy” learning method still holds unique value in applications requiring real-time performance and safety. For example, in fields like robot control and industrial automation, agents need to evaluate and update their strategies based on current actual actions. SARSA can help them learn efficient and safe behavior patterns in complex and uncertain environments.

In summary, the SARSA algorithm is like a “down-to-earth” apprentice. It learns lessons from its actual behavior by truly experiencing every attempt, improving its skills step by step. Although this learning method may not pursue the most extreme “ideal” performance like Q-learning, SARSA can provide a more robust and safe solution in many real-world applications that require caution and immediate feedback.