“SCM” in the AI Field: Revealing the Mystery of Causality, Moving Towards a Smarter Future

In the vast field of Artificial Intelligence (AI), when we talk about the abbreviation “SCM”, many non-experts might be confused. Even for insiders, this abbreviation might trigger different associations. Most commonly, it might refer to “Supply Chain Management”, a field where AI technology is widely applied. AI improves the efficiency and resilience of supply chains by optimizing logistics, inventory, and forecasting demand. For example, AI can predict commodity demand based on historical data and real-time market conditions to reduce the risk of stockouts or overstocking. AI is also used in supply chains to optimize routes, improve warehouse management, and even improve customer service through chatbots. In this sense, SCM is a manifestation of AI’s powerful application capabilities and a model of AI empowering traditional industries.

However, in the core theories and frontier research of AI, especially in the eyes of scientists and researchers pursuing deeper intelligence, “SCM” represents a completely different, yet more fundamental and profound concept—Structural Causal Model. It is not an application scenario of AI, but one of the key theoretical tools for AI itself to achieve the grand goal of “understanding the world”.

What this article will explore in depth is this “Structural Causal Model” (SCM) which has disruptive potential in the AI field. We will use examples from daily life to explain this abstract concept in simple terms.

I. What is a Structural Causal Model (SCM)?

Imagine you are a very smart child who knows nothing about the world. You see many things happening: it gets dark, the light turns on; you press a switch, the light also turns on. You might think that both “getting dark” and “pressing the switch” are related to “the light turning on”. But which one is the cause and which one is merely an association? If you want the light to turn on, should you wait for it to get dark or go press the switch?

This is the difference between “causality” and “association”. Structural Causal Model (SCM) is precisely a set of mathematical frameworks used by AI to understand this “causal relationship”. It not only tells us that A and B happen together (association), but more importantly, it reveals that “A causes B” (causality).

The core of SCM includes three main components:

1. Variables: Represent various events or states we want to study. For example, “getting dark”, “switch state”, “whether the light is on” in the example above.
2. Structural Equations: These equations describe the direct causal relationships between variables. Each equation represents how a variable is determined by its direct cause variables. For example, “Whether the light is on = f(switch state, whether the bulb works properly, whether there is electricity)”. Here, f is a function or rule. Importantly, this function points from “cause” to “effect”, not the other way around.
3. Exogenous Variables: Also known as error terms or disturbance terms. They represent external factors that are not explicitly modeled in the model but still affect the results. In our “light is on” example, “whether the bulb works properly” and “whether there is electricity” might be exogenous variables. They are not directly controlled by “switch state” immediately, but will affect the result of “light is on”.

To use a vivid metaphor, if our world is a complex machine, traditional machine learning is like simply predicting the next result by observing the results of pressing different buttons on the machine. Structural Causal Model (SCM), on the other hand, is like trying to draw the “design blueprints and user manual“ of this machine. It describes which parts (variables) are connected in what way (structural equations), how a change in one part directly or indirectly affects other parts, and what external factors (exogenous variables) might interfere with the operation of the machine. With this blueprint, we can not only predict the machine’s behavior but also better understand “why” the machine operates that way, and even be able to proactively “modify” the machine’s design (perform interventions) to achieve the effects we want.

II. Why Does AI Need Structural Causal Models (SCM)?

Our current AI technologies, especially deep learning, have made amazing achievements in “associative learning”. For example, AI can learn to identify cats and dogs in pictures, predict future housing prices, or generate realistic language text by analyzing massive amounts of data. But these powerful capabilities are mostly based on discovering statistical associations in data.

However, relying solely on association brings huge limitations:

1. The Paradox of “Ice Cream Sales Rise, Drowning Incidents Also Increase”: This is just a classic example of association rather than causation. The real cause is the hot summer, which leads to both an increase in ice cream sales and more people going swimming (thus increasing the risk of drowning). If AI only sees the association, it might offer a ridiculous suggestion: “To reduce drowning incidents, we should ban the sale of ice cream!” Clearly, AI lacking causal understanding might make wrong decisions.
2. Difficulty in Performing “Intervention” and “Counterfactual” Reasoning:
   • Intervention: If we know that “pressing the switch” causes “the light to turn on”, we can actively press the switch to control the light. This is the basis for AI to perform tasks and actively change the world. SCM allows AI to answer questions like “What happens if I intervene in this system?”.
   • Counterfactuals: This is a more advanced form of causal reasoning that allows us to think “What would present be like if things in the past had been different?”. For example, “If I hadn’t stayed up late yesterday, I wouldn’t be so sleepy today.” This ability is crucial for AI to perform error attribution, improve decision-making, and plan for the future.
3. Explainability and Trust: Many current AI models are considered “black boxes”. We only know they give a result, but we don’t know why. SCM makes the AI’s decision-making process more transparent and explainable by clarifying the causal paths between variables. For example, when a doctor uses AI to assist in diagnosing diseases, if AI can explain “Because the patient has symptoms X and Y, and these symptoms lead to disease Z, therefore the diagnosis is Z”, this will greatly enhance the doctor’s trust in AI.
4. Robustness and Generalization: Models based on association often perform poorly when the data distribution changes. For example, after learning traffic patterns on sunny days, AI might not be able to navigate effectively on rainy days. But a model based on causality, because it understands the underlying mechanism, can better adapt even if the environment changes. Knowing “wet roads lead to longer braking distances”, this causal relationship usually holds true regardless of the city or car model.

III. Recent Progress and Future Prospects of Structural Causal Models (SCM)

In recent years, with the development of the field of causal inference, the importance of SCM in AI has become increasingly prominent, becoming the core of Causal AI. Researchers are exploring how to combine SCM with currently powerful machine learning models (such as deep learning and large language models, LLMs) to make up for the deficiencies of traditional AI in causal understanding.

• Combination with Large Models: Current generative AI (such as Large Language Models, LLMs), although capable of human-like conversation and content creation, often generates text based on statistical associations, lacking real causal reasoning capabilities. “They don’t understand the ‘reasons’ and causal relationships behind customer behaviors.” Introducing SCM into LLMs is expected to enable these models not only to “say what” but also to “understand why it is said” and “what would happen if that were done”, thereby improving their decision interpretability and reducing bias and risk.
• Explainable AI (XAI): SCM naturally provides powerful tools for XAI. By constructing and analyzing causal graphs, AI systems can explain the reasons for their predictions or decisions more clearly, which is crucial for high-risk applications (such as healthcare and autonomous driving).
• Automated Causal Discovery: Researchers are dedicated to developing algorithms capable of automatically discovering causal relationships from data (i.e., building SCMs), rather than relying entirely on human experts to specify these relationships.

Back to our initial metaphor of “design blueprints and user manual”. AI is growing from an assistant who can only “mimic” machine operators into an engineer who can “interpret” and even “improve” machine design plans. The Structural Causal Model (SCM) is precisely this crucial design blueprint, guiding AI beyond superficial associations to touch the deep logic of how things work, enabling AI to truly understand, predict, and intervene in the world, thus moving towards the future of Artificial General Intelligence.

Revealing SARSA: How Agents Learn by “Feeling the Stones to Cross the River” (For Non-Experts)

In the vast field of Artificial Intelligence, there is a method that allows machines to learn through “trial and error” just like humans, which is Reinforcement Learning (RL). The core idea of RL is: an agent acts in an environment, receives rewards or punishments, and then adjusts its behavior based on these feedbacks to gain more rewards in the future. SARSA is a very important member of the reinforcement learning family.

Imagine you are learning to play a new game, like navigating a maze. At first, you might not know how to go, hitting walls everywhere (punishment), and occasionally finding the right path (reward). Over time, you will remember which roads lead to treasure and which are dead ends. The SARSA algorithm enables machines to learn this strategy of “feeling the stones to cross the river” in a more systematic and “down-to-earth” way.

SARSA: An “Action-Oriented” Learning Method

The name SARSA itself reveals its working principle. It is an acronym for the first letters of five English words: “State-Action-Reward-State-Action”. These five elements constitute a complete learning loop and are the basis for the SARSA algorithm to update its knowledge (or “Q-value”).

Let’s use a daily life example to specifically understand these five concepts:

Suppose you are a robot, and your task is to learn how to get from the living room (start point) to the kitchen and make a cup of coffee (get reward) as quickly as possible.

1. State (S): This represents your current situation. For example, you are now in the “living room”, which is a state.
2. Action (A): This is the operation you can choose to perform in the current state. In the living room, you might choose “walk towards the kitchen”, “turn on the TV”, “sit down”, etc.
3. Reward (R): This is the immediate feedback given by the environment after you perform an action. If you take a step “towards the kitchen”, you might get a small positive reward (e.g., +1 point) because it brings you closer to the goal; if you hit a wall, you might get a negative reward (e.g., -5 points). When you successfully make coffee, you get a large positive reward (e.g., +100 points).
4. Next State (S’): This is the next state you reach after performing action A. After you execute “walk towards the kitchen” from the “living room”, you might now be in the “hallway”, which is the new state.
5. Next Action (A’): This is the most critical part of SARSA. After you arrive at the new state “hallway” (S’), you will decide the next action A’ to execute based on your current strategy. For example, you might decide to “continue walking towards the kitchen” in the “hallway”, which is your new action A’.

SARSA uses this continuous quintuple—(Current State S, Current Action A, Received Reward R, New State S’, New Action A’ selected based on current strategy) — as a whole to learn and update its behavioral guidelines.

How is SARSA Different from the “Greedier” Q-learning?

The SARSA algorithm is often compared with another famous reinforcement learning algorithm, Q-learning. Their core purpose is to learn a “Q-value” (Quality Value), which represents the expected long-term total reward of taking a certain action in a certain state. With an accurate Q-value table, the agent can choose the action with the highest Q-value in each state, thereby achieving the optimal strategy.

The main difference lies in how they use the “Next Action (A’)” to update the Q-value:

• SARSA (“On-Policy” Learning): It is a “realist”. It will actually choose an A’ in the S’ state based on the currently used strategy (including exploratory actions), and then use this actually occurring sequence (S, A, R, S’, A’) to update the Q-value. Like a student learning to drive, he will summarize experience and adjust the next operation based on his current driving habits (even if occasionally imperfect). This method makes SARSA’s learning process more “conservative” and “safe” because it considers the consequences that its current exploratory behavior might bring. For example, in a maze with a cliff, SARSA will tend to learn a path that is far from the cliff but possibly slightly longer, because during exploration it will “actually walk a step” into the cliff and feel the huge punishment, thereby avoiding this dangerous path.

• Q-learning (“Off-Policy” Learning): It is an “idealist”. In the S’ state, it does not consider which action its current strategy will choose next, but assumes that it will always choose the ideal action that brings the maximum Q-value next to update the Q-value. Like a student learning to drive, he will imagine how a perfect driver would operate next, and then use this “optimal” imagination to guide the improvement of his current behavior. Q-learning is “greedier” in learning because it always assumes that the optimal action will be taken in the future, so it is easier to find the optimal strategy in the environment. However, if there are large negative rewards in the environment (such as a cliff), Q-learning might “fall off the cliff” during exploration because it assumes the future is always optimal, leading to unstable learning.

Simply put, SARSA is “Learn as I actually do“, focusing on “the Q-value of proceeding according to my current strategy”; Q-learning is “What should I do now if I always make the best choice in the future“, focusing on “how much Q-value the future optimal choice can bring”.

Applications, Pros and Cons of SARSA

Because SARSA is “on-policy” learning, it learns based on the sequence of actions actually taken by the agent, which makes it particularly useful in certain scenarios:

• Online Learning: If the agent must learn while acting in a real environment (e.g., an autonomous car learning on a real road), SARSA is very suitable because it considers the actual actions taken by the agent during the learning process and the risks these actions may entail. It can learn a more robust and safer strategy, even if this strategy is not always “theoretically optimal”.
• Avoiding Danger: In some environments where the cost of making mistakes is high (e.g., a robot operating a mechanical arm, where a mistake could cause physical damage), SARSA’s “conservative” nature enables it to learn strategies to avoid danger zones.

Pros:

• Good Stability: Due to its “on-policy” nature, SARSA usually has good stability during the learning process.
• Safer Environment Exploration: It includes exploratory actions in updates, so in risky areas with negative rewards, it learns to avoid these areas, thus exploring more safely.
• Faster Convergence: In some cases, the SARSA algorithm converges faster.
• Suitable for Online Decision Making: If the agent is learning online and cares about the rewards obtained during learning, the SARSA algorithm is more applicable.

Cons:

• May Converge to Sub-optimal Policy: Because it is limited by the current exploration strategy, it may sometimes converge to a sub-optimal strategy rather than the global optimal strategy.
• Learning Efficiency May Be Limited: If the exploration strategy is inefficient, the learning speed may be affected.

Development and Future of SARSA

The SARSA algorithm was first mentioned by G.A. Rummery and M. Niranjan in a 1994 paper, called “Modified Connectionist Q-Learning” at the time, and then formally proposed as SARSA by Rich Sutton in 1996. As one of the fundamental algorithms of reinforcement learning, many optimization methods for Q-learning can also be applied to SARSA.

Although SARSA is a relatively traditional reinforcement learning algorithm, its “on-policy” learning method still holds unique value in applications requiring real-time performance and safety. For example, in fields like robot control and industrial automation, agents need to evaluate and update their strategies based on current actual actions. SARSA can help them learn efficient and safe behavior patterns in complex and uncertain environments.

In summary, the SARSA algorithm is like a “down-to-earth” apprentice. It learns lessons from its actual behavior by truly experiencing every attempt, improving its skills step by step. Although this learning method may not pursue the most extreme “ideal” performance like Q-learning, SARSA can provide a more robust and safe solution in many real-world applications that require caution and immediate feedback.

Revealing the AI Superstar: Soft Actor-Critic (SAC) Algorithm—Helping You Learn Like a Gym Coach!

In the vast world of AI, there is a field called Reinforcement Learning (RL), which allows machines to learn through “trial and error”, just like humans learn to walk or ride a bicycle. In this field, the Soft Actor-Critic (SAC) algorithm is undoubtedly a highly acclaimed star. It is not only effective but also highly efficient in learning, making it a powerful tool for complex tasks such as robot control and autonomous driving.

Today, let’s unveil its mystery using concepts from daily life.

1. Reinforcement Learning: An Endless Game of “Exploration and Reward”

Imagine you are training a puppy to learn to shake hands. When the puppy successfully extends its paw, you give it a treat as a reward; if it just wags its tail, you don’t reward it, or even slightly correct it. Through constant attempts, the puppy finally learns that “shaking hands” leads to rewards.

This is the core idea of reinforcement learning: an “Agent” (like the puppy) takes “Actions” (extending a paw, wagging a tail) in an “Environment” (the training scenario you set up), and the environment gives “Rewards” or “Punishments” based on the actions. The agent’s goal is to find an optimal action strategy through repeated attempts to maximize the long-term cumulative reward.

2. Actor-Critic: The “Brain Duo” with Division of Labor

In early reinforcement learning, the agent’s brain might have only one part: either focusing on deciding how to act (“Actor”) or focusing on evaluating how good the action is (“Critic”). But people soon discovered that combining these two functions makes learning more efficient. This is the “Actor-Critic” architecture.

“Actor” Network: The Decision Maker

You can imagine the “Actor” as a professional “Action Coach”. Facing the current situation (e.g., the puppy sees you extending your hand), it decides what action to take next (e.g., extending the left paw or the right paw) based on its experience and judgment. Its task is to provide an action strategy.

“Critic” Network: The Evaluator

The “Critic” acts like a “Value Appraiser”. When the “Action Coach” proposes an action, the “Value Appraiser” gives a “score” based on the expected outcome of this action, telling the coach how good this action is, or roughly how much total reward can be obtained in the future after executing this action.

These two roles work together: the Action Coach proposes actions, the Value Appraiser evaluates them, and the Action Coach then adjusts its strategy based on the evaluation results to propose better actions next time. Through constant cycles, they make the agent smarter and smarter.

3. Where is the “Soft”? SAC’s Uniqueness—Encouraging the Spirit of Exploration

The most special thing about SAC lies in the word “Soft”. In traditional reinforcement learning, agents often only pursue the “highest reward”, that is, finding an optimal path and executing it unswervingly. But this sometimes brings problems:

• Premature convergence to local optima: Just like a novice driver who gets used to a familiar route, even if this route is always congested at certain times, he rarely tries to take a detour to find a new highway shortcut.
• Instability: If the environment changes slightly, the original optimal path may no longer apply, and the agent becomes confused at once.

The “Soft” in the SAC algorithm is precisely to solve these problems. While pursuing maximized rewards, it also adds a unique element: maximizing the strategy’s “Entropy”.

Entropy: A Measure of “Uncertainty” and “Diversity”

“Entropy” here can be simply understood as the diversity or randomness of actions.

For example:

• Low Entropy (Deterministic): An old driver who only knows one route to work every day and never tries other paths. His strategy is very deterministic.
• High Entropy (Randomness/Diversity): A curious explorer who takes this road today and that road tomorrow, even if it’s a bit further, just to see if there are new sceneries or faster hidden paths. His strategy has high entropy.

SAC’s strategy is not only to get high rewards but also to make its action strategy as “random” and “scattered” as possible, rather than concentrating on just one action. In layman’s terms, it encourages the agent to “also try different methods and accumulate experience while getting rewards!”

This is like a gym coach teaching you to work out: he will not only tell you how to do moves to achieve the best results but also encourage you to occasionally try some new postures or use different equipment to train the same body part. The benefits of doing this are:

1. Stronger Exploration Ability: By trying different actions, the agent can discover more potential, even better strategies, avoiding falling into “local optimal solutions” too early. Just like that explorer, who might actually find a hidden path with beautiful scenery and time-saving one day.
2. Higher Robustness: Diversified strategies mean it doesn’t rely on a specific successful path. When the environment changes, it has more alternatives to cope with and is less likely to “crash”. Just like when you work out, with more varied movements, your body coordination and adaptability to different sports will be stronger.
3. Better Sample Efficiency: SAC is an “Off-policy” algorithm. It stores all past experiences in an “Experience Replay Buffer” and then samples from it to learn. Because exploration is encouraged, the experience in this buffer will be very rich and diverse, allowing the agent to learn more from “old experience”, thereby greatly improving learning efficiency without needing to interact with the environment repeatedly in large quantities. It’s a bit like you not only learn from your own workout experience but also watch various training videos posted by fitness influencers in the past to draw experience.
4. More Stable Training: SAC usually uses techniques like “Double Q-networks” to reduce the bias of overestimating action values, which greatly improves the stability of the training process. Just like a gym coach evaluating your movements from multiple angles to ensure that what is corrected is not a wrong estimation.

4. SAC’s Success Secret and Applications

In summary, the reason why the SAC algorithm stands out in the field of reinforcement learning is that it cleverly balances “Exploration” and “Exploitation”:

• Exploitation: Execute known good actions as much as possible to get rewards.
• Exploration: Try some new actions even if they don’t look optimal, to discover better potential strategies.

By maximizing the objective of “Reward + Policy Entropy”, SAC performs excellently in many complex tasks, especially adept at handling scenarios with Continuous Action Spaces (example: robot joints can perform infinite fine movements, not just discrete actions like “forward, backward”).

It is widely used in:

• Robot Control: Allowing robots to complete various fine operations more flexibly and autonomously.
• Autonomous Driving: Helping unmanned vehicles make safer and smarter decisions in complex road conditions.
• Game AI: Training AI to play various highly complex strategy games.

As of 2024 and 2025, the SAC algorithm and its variants remain popular choices in deep reinforcement learning research and applications. Researchers are constantly optimizing its mathematical principles, network architecture, and improving deployment effects in actual scenarios, such as dynamically adjusting the importance of entropy through adaptive temperature parameters to further improve the stability and performance of the algorithm.

Summary

The SAC algorithm is like a professional and innovative gym coach: it not only knows how to get you high scores (high rewards) but also knows how to make you stronger, more robust, and more comprehensive by encouraging you to “try more and not be partial” (high entropy). It is this emphasis on “Soft” exploration that keeps SAC shining on the AI stage, pushing the boundaries of agent learning and evolution in a complex world.

ResNet: The “Highway” of Deep Learning—Letting AI See Deeper and More Accurately

In the wave of artificial intelligence, we often marvel at AI’s extraordinary abilities in fields like image recognition, autonomous driving, and medical diagnosis. Behind these capabilities lies a technology known as “Deep Learning”, and within deep learning, there is a crucial “neural network” architecture. Its emergence is like opening up “highways” on the path of AI learning, enabling AI to see deeper and learn more accurately. This revolutionary architecture is what we are going to explore in depth today—Residual Network (ResNet).

1. The “Dilemma” of Deep Learning: The Deeper the Better, But Also Harder to Learn?

Imagine you are training a “little detective” to identify objects in pictures. At first, you teach him some simple features, such as circles being apples and squares being boxes. Through a few layers of “learning” (shallow layers of neural networks), he performs quite well. So you think, if you let him learn more deeply and identify more subtle features, such as the texture of apples or the material of boxes, wouldn’t he become a “master detective”?

In the field of deep learning, people once believed: The more layers a neural network has, theoretically the richer the features it can learn, and the better the performance should be. This is like the more knowledge the little detective learns, the stronger his ability. Therefore, researchers frantically stacked the layers of neural networks, from a dozen to dozens of layers.

However, reality was not so wonderful. When the number of network layers reached a certain level, the performance not only did not improve but began to decline. It’s like the little detective learned too many complicated things, and his memory and understanding became worse, even “forgetting” the simple knowledge he learned before. Why does this happen?

There are two main problems here:

• Gradient Vanishing/Exploding Problem:
  • Vanishing: Imagine you assign 100 questions to the little detective, and the answer to each question affects the answer to the next. If you make a small mistake on the first question, this mistake might become negligible after being passed 100 times, causing you to be unable to effectively correct the initial mistake. In neural networks, each layer transmits a “learning signal” (gradient). If the network is too deep, these signals will gradually decay to near zero during backpropagation, causing the parameters of the earlier layers to not be effectively updated, and learning stagnates.
  • Exploding: Conversely, if the signal is constantly amplified during transmission, it will cause the parameters to update too quickly, making the network unstable.
• Degradation Problem:
  • Even if the gradient vanishing/exploding problem is solved by some technical means, people found that when simply increasing the number of network layers without changing its basic structure, the training error of deep networks is actually higher than that of shallow networks. This indicates that deep networks do not always learn better “feature representations”, and they even struggle to learn an “identity mapping” (i.e., learning nothing, just passing the input to the output, keeping it as is). If it can’t even “keep it as is”, then learning more complex patterns is even harder.

This is like assigning a complex task with 200 steps to the little detective; not only did he not become smarter, but his ability to complete simple tasks actually regressed.

2. ResNet’s “Brainstorm”: Opening a “Shortcut”

Faced with this dilemma, He Kaiming and others from Microsoft Research Asia proposed a revolutionary solution in 2015—Residual Network (ResNet).

The core idea of ResNet is very ingenious. It introduces a mechanism called “Residual Connection“ or “Skip Connection“.

Let’s use a more vivid metaphor to explain:

Suppose the little detective needs to learn the concept of “cat”. The traditional method is that you give him a picture, and he analyzes it layer by layer from beginning to end, such as:
Eyes -> Nose -> Mouth -> Fur -> Overall contour… then outputs “This is a cat”.

If this analysis process is too long, he might get “lost” at some link in the middle, or the information might become “distorted”.

ResNet’s approach is to add a “bypass“ or “shortcut“ to this analysis flow. What is this shortcut?

It allows input data to directly skip one or more layers in the network and then merge with the output processed by these layers.

Specifically, when the little detective analyzes a picture, besides the original layer-by-layer indepth analysis path, there is a “direct train”:
He will first take a look at the original picture (this is input $X$), and he has a “team” to analyze this picture in detail (this represents the original network layers, learning a complex mapping $F(X)$). At the same time, he himself also keeps a “copy” of the original picture (this is the input $X$ passed through the shortcut). When the team finishes analyzing, he will add the team’s analysis result $F(X)$ and the original copy $X$ he kept to get the final conclusion: $F(X) + X$.

Why is this useful? The key lies in that, in this way, the network no longer directly learns how to transform from $X$ to $F(X)+X$, but only needs to learn the “residual“ ($F(X)$), which is the difference, between the original input and the expected output.

It’s like:

• Formerly (Traditional Network): You want the little detective to learn the complete features of the cat $H(X)$ directly from input $X$. If $H(X)$ is hard to learn, he won’t learn it well.
• Now (ResNet): You tell the little detective that he doesn’t need to generate a cat feature map from scratch. He just needs to find the “difference“ $F(X)$ between the original picture $X$ and the target cat feature map $H(X)$. Then adding this difference $F(X)$ to the original picture $X$ yields $H(X)$.

Learning this “difference” $F(X)$ is often much easier than directly learning the complex $H(X)$. Even in extreme cases, if the original picture $X$ is already good enough and is almost a cat, then the network only needs to learn $F(X) = 0$ (i.e., do nothing) so that $H(X) = X$. Learning the identity mapping of “doing nothing” is a piece of cake for residual networks.

This mechanism effectively alleviates the gradient vanishing problem because gradients can be backpropagated directly through the “shortcut”, ensuring that earlier layers can also receive effective learning signals.

3. The Power of ResNet: Deeper, Stronger, More Stable

The emergence of ResNet completely broke the bottleneck of deep network training in the past and brought advantages in many aspects:

• Making training ultra-deep networks possible: ResNet makes it possible to build deep networks with hundreds or even thousands of layers, such as variants like ResNet-50, ResNet-101, ResNet-152, etc. The more layers, usually the stronger the feature extraction capability. In the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC), ResNet successfully trained a network with as many as 152 layers and won championships in multiple tasks such as image classification, object detection, object localization, and instance segmentation in one fell swoop.
• Solving Gradient Vanishing/Exploding: Through residual connections, gradients can flow easier, allowing parameters in deep layers of the network to be effectively updated.
• Significant improvement in model performance: On tasks like image classification, ResNet achieved state-of-the-art performance at the time, drastically reducing the error rate.
• Easier to optimize: Learning the residual function $F(x)$ is usually simpler than learning the original complex function $H(x)$, making the training process more stable and convergence faster.

4. ResNet’s Family and New Progress

ResNet is not static; its core ideas have inspired numerous subsequent variants and improvements:

• Wide ResNet (WRN): Instead of continuing to increase depth, it works on the width of the network (i.e., the number of channels per layer), which can improve model expression capability while reducing training time.
• DenseNet: Through denser connections, the output of each layer is passed to all subsequent layers, further promoting the flow of information and gradients, reducing the number of parameters.
• ResNeXt: Introduced grouped convolution and the concept of “cardinality”, improving model performance by increasing the number of parallel paths.
• SENet (Squeeze-and-Excitation Networks): Introduced attention mechanisms on the basis of ResNet, allowing the network to learn the importance of each feature channel, thereby improving feature expression capabilities.

Today, ResNet and its variants remain indispensable infrastructure in the field of computer vision. The latest research and applications are still emerging:

• Remote Sensing Image Analysis: Research in 2025 demonstrates the enhanced application of ResNet in land use classification of satellite images (such as Sentinel-2), significantly improving classification accuracy by identifying complex patterns and features.
• Climate Prediction: In the prediction study of the Indian Ocean Dipole (IOD), ResNet is used to fuse sea surface temperature and sea surface height data, capturing ocean dynamic processes, extending the prediction lead time to 8 months, outperforming traditional methods.
• Multi-domain Applications: ResNet shows strong capabilities in various computer vision tasks such as image classification, object detection, face recognition, medical image analysis (such as pneumonia prediction), and image segmentation, and often serves as the “backbone network” for various more complex tasks to extract features.
• Combining with Frontier Technologies: ResNet is also combined with technologies like data pruning. Researchers found that by selecting training samples, ResNet may achieve exponential scaling during training, breaking the limits of traditional power-law scaling. Even in 2025, there is a view that although “Transformer giants” are prevalent, basic architectures like ResNet and the underlying gradient descent principles are still the “essential methods” of AI progress and will evolve in a smarter and more collaborative way.

5. Conclusion

The birth of ResNet is a milestone in the history of deep learning. It is like building “highways” for AI learning, allowing information to flow unimpeded in deeper networks, effectively solving the “getting lost” and “amnesia” problems in deep neural network training. It is not only a theoretical breakthrough but also brought significant performance improvements in practical applications, greatly promoting the development of artificial intelligence, especially in the field of computer vision. Understanding ResNet is understanding how AI moves from imitation to deeper cognition, and it is also an excellent perspective to appreciate the charm of deep learning.

Exploring RoBERTa: A More “Robust” AI Language Understander

Imagine if AI were a student learning human language, then RoBERTa (Robustly Optimized BERT approach) would undoubtedly be a “super student” who has undergone rigorous training and uses extremely efficient learning methods. It does not start learning from scratch but builds upon another excellent student, BERT (Bidirectional Encoder Representations from Transformers), becoming stronger and better at understanding language through a series of “devilish training”.

The emergence of BERT was a major leap in the field of Natural Language Processing (NLP). It showed us the potential for AI to understand text content, not just recognize keywords. BERT learns language through two methods: “Cloze test” (Masked Language Model, MLM) and “judging sentence relevance” (Next Sentence Prediction, NSP). Simply put, it is like a student being asked to fill in missing words in a sentence and judge whether two adjacent sentences are truly coherent. Through training on massive texts, BERT learned word collocations, sentence structures, and even some common-sense language rules.

However, like all top students, there are always people exploring how to take them to the next level. The Facebook AI Research team launched RoBERTa in 2019. Its core idea is to perform “Robust Optimization” on BERT’s training strategy, enabling the model to demonstrate more powerful capabilities in language understanding tasks. So, how does RoBERTa achieve this?

RoBERTa’s “Secret Training Manual”

We can understand RoBERTa’s optimization strategy as equipping the “language student” BERT with more advanced learning tools and a more scientific study plan, making its learning process more focused.

1. Dynamic Masking: More Flexible “Cloze Test”

   • BERT’s “Reviewing Old Questions”: In BERT’s training, if a word in a sentence is masked (e.g., “The weather is [MASK] good today”), the pattern of this “cloze test” for this sentence is usually fixed throughout the training process. The AI student might gradually memorize that “very” should be filled in here after seeing “The weather is [MASK] good today” multiple times, rather than truly understanding the context.
   • RoBERTa’s “Daily New Questions”: RoBERTa adopts a “Dynamic Masking” mechanism. This means that every time the model sees the same sentence, the masked words may change randomly. This is like a teacher giving you different cloze test questions every time, forcing you not to memorize by rote but to truly understand the meaning of the sentence and the contextual relationship, thereby learning more solidly and comprehensively.

2. Larger Training Batches and More Data: Massive Reading and Concentrated Training

   • BERT’s “Small Class Learning”: When BERT trains, the number of texts processed each time (called “batch size”) is relatively small, and the amount of data is relatively limited.
   • RoBERTa’s “Thousand-Person Classroom”: RoBERTa uses a massive dataset far exceeding BERT, such as a combination of BookCorpus and OpenWebText, with a data volume of 160GB. At the same time, it also adopts a larger batch size, increasing from BERT’s 256 to 8K. This is like letting the AI student read a huge library, and in each study session, they can process and understand massive text information simultaneously. Larger batches allow the model to see more examples of different contexts, thereby better summarizing and learning the universal laws of language.

3. Removing “Next Sentence Prediction” (NSP) Task: Focusing on Core Abilities

   • BERT’s “Multi-task Learning”: In addition to the cloze test, BERT needs to complete a “Next Sentence Prediction” (NSP) task during training, which is to judge whether two sentences are consecutive. Researchers at the time believed that this helped the model understand document-level context.
   • RoBERTa’s “Streamlining”: RoBERTa’s experiments found that the NSP task did not improve model performance as much as imagined, and could even be removed. It’s like this top student discovered that an additional “guessing game” task didn’t really help him understand language better, but instead distracted him. Therefore, RoBERTa simply abandoned the NSP task and devoted all its energy to the core language modeling task of “cloze test”, making it more profound in understanding single sentences and paragraphs.

4. Longer Training Time: Diligent Study leads to Success

   • This point is the most intuitive and easiest to understand. RoBERTa was trained for a longer time than BERT, using more computing resources. Just like a student who spends more time studying and practicing than others, they naturally achieve a higher level of proficiency and understanding.

RoBERTa’s Outstanding Achievements and Far-reaching Influence

Through the series of optimizations mentioned above, RoBERTa achieved significant performance improvements in multiple Natural Language Processing benchmarks (such as GLUE), surpassing the original version of BERT. It demonstrated stronger generalization ability and accuracy in tasks such as text classification, question answering systems, and sentiment analysis.

Although Large Language Models (LLMs) have emerged one after another in recent years, constantly refreshing various records, the training strategies and optimization ideas introduced by RoBERTa, such as dynamic masking, large-scale data, and batch training, have become cornerstones and standard practices for many subsequent excellent models. It proves that under the existing model architecture, model performance can be significantly improved through “more robust” training methods. This has important guiding significance for the development of the entire NLP field. Even with newer and more powerful models today, RoBERTa remains an indispensable part of the development history of AI language understanding, and many of its principles and optimization ideas are still widely studied and applied.

The “Learning Master” in the AI Field: Exploring the Reptile Algorithm

In the vast world of Artificial Intelligence (AI), the way a model learns new knowledge is its core capability. Imagine when we humans learn new skills, we don’t start from scratch every time. For example, once you learn to ride a bicycle, you will learn to ride an electric bike or a motorcycle much faster because you have mastered the general skill of “balance”. The AI field has a similar pursuit, which is to let the model learn to “infer other things from one fact” and master the “method of learning”. This is the core concept we are going to popularize today—Meta-Learning.

Among the many meta-learning algorithms, there is one proposed by OpenAI called Reptile, which has become a striking “learning master” with its “simple is beautiful” design philosophy. Reptile means “creeping animal” in English, but here it does not refer to a biological reptile, but an efficient AI algorithm. So, how exactly does Reptile make AI smarter? Let’s find out.

Core Concept: Meta-Learning—The Ability to “Learn to Learn”

Before diving into Reptile, let’s talk about meta-learning. Traditional machine learning models are like “specialized students” who are very good at solving a specific problem, such as distinguishing between cats and dogs. If you ask it to identify cars and airplanes, it has to start learning from scratch, just like it has never seen these new things before.

The goal of Meta-Learning is to make the AI model a “top student” who can not only learn specific knowledge but also learn how to learn new knowledge more efficiently. For example, a top student does not memorize the solution to every problem by rote, but masters the general methods and techniques for solving problems. When encountering a new type of problem, he can quickly find the key points, draw inferences, and master it quickly. Meta-learning empowers AI with this ability to “learn to learn”. It no longer just learns “Task A”, but learns “the method of learning Task A, B, C…”.

Enter Reptile: The “Learning Master” of Simplicity

The Reptile algorithm, proposed by OpenAI in 2018, is unique in the field of meta-learning because its design is extremely simple yet effective. Imagine you are an experienced chef (AI model). You have learned cooking techniques for many cuisines (initial parameters of the model). Now, you need to learn a brand new dish that you have never touched before.

• Traditional approach: Every time you learn a new dish, you may have to start from the most basic steps like washing and cutting vegetables, consuming a lot of time.
• Goal of Meta-Learning: You hope to master a set of general “recipe learning methods” so that next time, whether it is Sichuan cuisine or Cantonese cuisine, you can get started quickly.

Reptile is such an efficient “recipe learning method”. It does not pursue complicated theoretical derivations, but allows the model to quickly adapt to new tasks through a very intuitive and easy-to-operate way.

Reptile’s “Secret Learning Manual” (Working Principle)

The core idea of Reptile can be illustrated vividly with our chef example:

1. Initial “General Skillset”: Your starting point in cooking (initial parameters of the AI model) is the “general skillset” accumulated from your years of experience.
2. Quick Adaptation to New Dishes: Now, you receive a task to cook a new dish. You won’t start from scratch, but based on your “general skillset”, quickly try to make this new dish. During this process, you will make some quick adjustments and learning (perform Stochastic Gradient Descent, SGD, on a small amount of data).
3. “Reviewing the Old to Learn the New” Adjusting the General Skillset: After cooking a few new dishes, you find that in order to cook these dishes well, you have adjusted in a certain direction (such as paying more attention to heat control, or being more proficient in seasoning). What Reptile does is to fine-tune your “general skillset” towards the common direction reflected after learning these new dishes. It doesn’t care “how many steps you adjusted” or the “path of adjustment” when you cooked each dish, it only looks at the skill state of the dish you finally cooked successfully, and then moves your initial “general skillset” slightly closer to these successful states.

This process is repeated constantly: learn some new tasks, then perform quick fine-tuning on these tasks, and finally update the initial parameters of the model based on the results of the fine-tuning, making these initial parameters “smarter” and able to adapt to future new tasks faster.

In more technical language, the Reptile algorithm will:

• Randomly sample a task from the task distribution (e.g., a new dish).
• Perform a small amount of gradient descent on this task (quickly try cooking).
• Update the model’s initial parameters to bring them closer to the final parameters learned on this task (adjust your basic cooking skills based on the experience of successful cooking).
• Repeat the above steps, cycling over and over.

Why is Reptile Efficient?

Before Reptile appeared, MAML (Model-Agnostic Meta-Learning) was another important milestone in the field of meta-learning. Although MAML is powerful, it requires calculating complex second-order derivatives, which involves a large amount of calculation and is relatively complex to implement.

The ingenuity of Reptile lies in that its performance is comparable to MAML, but it is simpler, easier to implement, and more computationally efficient. It avoids the complexity of unrolling computational graphs and calculating high-order derivatives in MAML, and achieves the goal of meta-learning simply through standard Stochastic Gradient Descent (SGD) and a clever parameter update strategy. As some researchers have said, Reptile demonstrates the “Occam’s Razor principle” in the AI field: often the most elegant solutions are born from the rejection of complexity. When the entire field was struggling in second-order derivatives, Reptile opened a new era of meta-learning with a line of averaging operations.

Application Scenarios of Reptile: “Few-Shot Learning” by Inference

The Reptile algorithm is particularly useful in Few-Shot Learning scenarios. What is few-shot learning? It refers to the capability of a model to learn to recognize new categories with only a very small amount (e.g., 1 to 5) of samples.

For example: Traditional image recognition models may need thousands of pictures of cats to learn to recognize “cats”. A model trained by a meta-learning algorithm like Reptile may only need to see one picture of a new animal (such as an “okapi” that has never been seen before) to quickly recognize this animal, because it has learned the general ability of “how to distinguish animal features”. OpenAI once released an interactive demo where users can freely draw a few figures as category samples, then draw a new figure, and the Reptile model can quickly classify it.

Summary and Outlook

The Reptile algorithm provides a powerful and practical tool for the field of meta-learning with its simple and efficient characteristics. It allows AI models to learn “the method of learning”, thereby demonstrating the ability to quickly adapt and draw inferences when facing brand new tasks. This technology has huge potential in scenarios where data is scarce and new models need to be deployed quickly, such as medical diagnosis, personalized recommendation, and new product design.

The success of Reptile also reminds us that on the road of AI exploration, sometimes the most elegant and powerful solutions come precisely from the simplification of complexity and a profound understanding of basic principles.

The “Memory Master” of the AI Field: How the Reformer Model Handles Massive Information

In the vast universe of Artificial Intelligence (AI), the Transformer model is undoubtedly a shining star, empowering many powerful large language models like ChatGPT. However, even the Transformer faces huge challenges when processing extremely long text sequences, such as insufficient memory and excessive computational costs. Imagine if an AI had to read and understand a massive tome like “War and Peace” in one go; a traditional Transformer might frequently “crash” or “forget words”. To solve this problem, scientists at Google Research proposed an innovative model in 2020 called “Reformer”—the Efficient Transformer.

The Reformer model is like an “information processing master” with extraordinary memory and efficient working methods. Through ingenious design, it maintains the powerful capabilities of the Transformer while greatly improving the efficiency of processing long sequence data, enabling it to handle contexts of up to a million words requiring only 16GB of memory. This allows AI to handle massive data such as entire books, ultra-long legal documents, gene sequences, and even high-resolution images with ease.

So, how does Reformer, this “memory master”, achieve this? It mainly relies on two core technological innovations: Locality-Sensitive Hashing (LSH) Attention mechanism and Reversible Residual Networks.

1. Locality-Sensitive Hashing (LSH) Attention Mechanism: From “Needle in a Haystack” to “Categorized Search”

The Dilemma of Traditional Transformers:
We know that the core of the Transformer is the “Attention Mechanism”, which allows the model to “pay attention” to all other words in the sequence when processing each word, thereby capturing the complex relationships between words. This is like looking for an acquaintance in a large room; you need to look around at everyone in the room to judge which one is the person you are looking for. For short sequences, this is effective. But if the number of people in the room (sequence length) becomes very large, say thousands or even hundreds of thousands, identifying them one by one becomes very time-consuming and laborious. The calculation volume grows quadratically ($O(L^2)$), and memory consumption is also huge. This is like looking for a needle in a haystack, extremely inefficient.

Reformer’s Solution: LSH Attention
The LSH attention mechanism introduced by Reformer is like equipping this “seeker” with a smart event planner. Before the event starts, the planner groups all guests into many small groups based on their hobbies, dressing styles, and other characteristics, and puts similar people in the same group. When you want to find someone, you only need to know roughly which group they belong to, and then look directly in that group, without having to compare everyone in the venue.

In the AI model, LSH uses hash functions to assign similar “information blocks” (such as word vectors in text) to the same “bucket”. When calculating attention, Reformer no longer lets each information block pay attention to all other information blocks, but only to those in the same “bucket” or adjacent “buckets”. In this way, the computational complexity is greatly reduced from quadratic $O(L^2)$ to $O(L \log L)$, making it possible to process long sequences of tens of thousands or even millions of levels.

2. Reversible Residual Networks: The “Smart Bookkeeping Method” that Saves Worry and Effort

The Dilemma of Traditional Deep Learning Models:
Deep learning models are usually composed of many stacked layers. In order to perform backpropagation during training (i.e., adjusting the model’s internal parameters based on output errors), the model needs to remember the intermediate results calculated by each layer (called “activations”). This is like a company that, in order to check accounts, must record the details of income and expenditure for every department and every link completely, and save many copies. If the model has many layers and the sequence is long, these intermediate results will occupy huge memory space and quickly exhaust the memory of the computing device.

Reformer’s Solution: Reversible Residual Networks
Reformer’s Reversible Residual Networks are like introducing a “smart bookkeeping method”. It no longer needs to save every intermediate bill. Instead, it designs a clever way to reverse deduce the input value of the previous layer from the output value of the current layer when needed. This is like a brilliant accountant who only needs the current general ledger and a small amount of key information to reverse restore all itemized expenditures and incomes when needed, without piling up all original vouchers.

Specifically, the reversible residual layer divides the input data into two parts; only one part is processed, and the other part is combined with the processing result in some way. During backpropagation, it can accurately recover the activation value of the previous layer through mathematical inverse operations, thereby avoiding the huge memory overhead caused by storing all intermediate activation values. This method greatly reduces the amount of memory required during model training, making it independent of the number of network layers and only related to the sequence length, thus enabling the training of deeper models that process longer sequences.

3. Chunking for Feed-Forward Networks: “Task Segmentation Execution”

In addition to the two major innovations mentioned above, Reformer also adopts the technique of Chunking Feed-Forward Networks. In the Transformer architecture, besides the attention layer, the Feed-Forward Network layer is also an important component. For very long sequences, the Feed-Forward Network can still occupy a large amount of memory. Reformer divides the calculation task of the Feed-Forward Network into small chunks and processes them one by one. This is like reading a long novel; you don’t read the whole content in one breath, but read it in sections, processing one section after reading it, so you don’t need to remember all the details of the whole book in your mind at the same time, thus saving “brain” memory.

Reformer’s Significance and Applications

These innovations of Reformer enable it to process sequences much longer than traditional Transformers with lower computing resources and memory consumption. This means that AI models can better understand and generate long articles, summarize entire papers, analyze genomic data, process long audio or video, and even generate high-resolution images. For example, the Reformer model can summarize, generate text, or perform sentiment analysis on an entire novel on a single machine.

Although Reformer is a model proposed in 2020, the ideas it pioneered, such as LSH attention and reversible layers, remain important milestones in the development of efficient Transformer architectures. Today, as large language models continue to pursue larger scales and longer contexts, Reformer’s philosophy provides valuable ideas for how to build more efficient and scalable AI models. It can be said that Reformer is like an early pathfinder, pointing out the way forward for later AI “memory masters”.

RedPajama: The “Open Source Recipe” and “Data Treasure” in the AI Field

In the current wave of Artificial Intelligence (AI), Large Language Models (LLMs) are undoubtedly the deserving stars, capable of writing poetry, programming, and conversing—almost omnipotent. However, behind these powerful models often hides an unknown secret—the massive amounts of data they rely on for learning, and the technical details required to train these models, are often “privatized” by a few commercial companies. It’s like a top-tier restaurant that only displays delicious dishes but never publishes its exclusive “recipes”, making it difficult for many researchers and small teams to explore and innovate deeply.

It is against this backdrop that the “RedPajama” project came into being. It is like a “public interest organization” dedicated to breaking monopolies and sharing knowledge, aiming to make the powerful capabilities of AI more transparent, open, and accessible.

What is RedPajama? The “Open Source Key” to the AI World

Imagine building a magnificent skyscraper; you need detailed design blueprints and a large amount of building materials. In the world of AI, large language models are that skyscraper, and its “blueprints” and “building materials” are training data and model architectures. The construction details and training data of many leading AI models, such as some foundation models behind ChatGPT, are not disclosed to the public or only partially disclosed. This greatly limits other researchers from innovating and customizing based on them.

RedPajama is a collaborative project initiated by multiple institutions including Together, Ontocord.ai, ETH DS3Lab, Stanford CRFM, and Hazy Research, aimed at creating a leading, fully open-source large language model ecosystem. Its core philosophy is that if top AI models are built based on publicly available data and methods, then anyone can verify their working principles and improve upon them, thereby driving progress in the entire AI field. It’s as if a top chef’s secret dish is very popular, and the RedPajama project decided to do it themselves, reconstructing the “cooking recipe” and required “ingredients” of this dish based on public clues, and sharing them with everyone for free.

The Core of RedPajama: A Massive and High-Quality “Data Feast”

To train a smart and powerful language model, the most critical thing is to have enough and good enough text data, just like a child learning to speak needs to hear a lot of language input. One of the core contributions of the RedPajama project is the construction of two milestone massive datasets: RedPajama-V1 and RedPajama-V2.

1. RedPajama-V1: The Pioneer of Replicating “Secret Recipes”

Initially, the RedPajama project set its sights on a model called LLaMA. Although LLaMA is not fully open source, its published dataset composition attracted widespread attention. The goal of RedPajama-V1 was to “replicate” LLaMA’s training dataset. This is like a group of world-class bakers who, by analyzing an already public cake, learned its main ingredients (flour, sugar, eggs), and then tried their best to purchase ingredients themselves according to its formula and proportions, making a cake very close in taste and quality, and fully publishing this “flour formula” and “production steps”.

RedPajama-V1 contains over 1.2 trillion “tokens”. You can understand “tokens” as the smallest text units processed by the model, which can be words, punctuation marks, or even parts of words. This data comes from various open resources on the Internet, including English CommonCrawl data, C4 dataset, code on GitHub, Wikipedia, books (such as Project Gutenberg and Books3), academic papers on ArXiv, and Q&A content on Stack Exchange. The project team carefully pre-processed and filtered this raw data to ensure data quality.

2. RedPajama-V2: The Expanded and Optimized “Data Treasure”

If RedPajama-V1 successfully replicated an existing recipe, then RedPajama-V2 groundbreakingly built an unprecedented “ingredient warehouse” and attached detailed “quality inspection labels” to each ingredient.

In October 2023, the RedPajama project team released RedPajama-V2, which is a larger and more powerful dataset. This dataset contains an astonishing 30 trillion filtered and deduplicated tokens (the raw data volume exceeds 100 trillion tokens). This is equivalent to a huge library containing books with 30 trillion words, and these books are not only vast in number but also preliminarily organized and classified.

The uniqueness of RedPajama-V2 lies in that it not only provides massive text but also additionally provides more than 40 pre-calculated “data quality annotations” or “quality signals”. This is like an intelligent ingredient warehouse: you can get massive ingredients, but each ingredient bag is not only written with the product name but also attached with dozens of detailed quality indicators such as “freshness score”, “origin score”, “sweetness index”, etc. This allows developers to select only the data most suitable for their model training according to their own needs, just like picking ingredients, or to process data with different weights. For example, a model that pays more attention to generating rigorous articles might focus more on selecting texts with higher “academic paper” quality. This dataset covers English, French, Spanish, German, and Italian.

RedPajama-V2 is considered the largest publicly available dataset specifically for large language model training to date. It provides a cornerstone for the community, which can be used not only to train high-quality LLMs but also for in-depth research on data selection and management strategies.

The Goals and Profound Significance of RedPajama

The core goals of the RedPajama project and the impact it brings are multifaceted:

• Promoting the Democratization of AI: Many of the most powerful models remain commercial closed-source or partially open, which limits research, customization, and use with sensitive data. RedPajama aims to eliminate these restrictions by providing fully open models and data, allowing more people to access, understand, and improve AI technology. This is like building a public library so that knowledge is no longer the privilege of a few.
• Fostering Innovation and Research: By providing high-quality open-source datasets and models, RedPajama provides a common starting point for researchers and developers worldwide. They can experiment and innovate on this basis without having to invest huge resources from scratch to collect and process data. This is like providing unified, standard building blocks, where everyone can build their own unique creative works based on these blocks.
• Improving Transparency and Reproducibility: In the field of AI, the transparency of model training and the reproducibility of results are very important. RedPajama makes the entire training process more transparent by publishing its dataset construction methods and sources, allowing researchers to better understand how the model learns and reproduce its results. This helps build trust and reliability in AI technology.
• Developing Open Source Models: In addition to datasets, the RedPajama project is also committed to developing Base models and Instruction tuning models. They have released the RedPajama-INCITE series of models, including 3 billion and 7 billion parameter models, which even surpass other open-source models of the same scale in some aspects. They plan to release model weights under permissive open-source licenses like Apache 2.0, which will allow commercial applications, further lowering the threshold for AI innovation.

Looking to the Future: The “Shared Garden” of the AI Field

The RedPajama project is not just about data and models; it is more of a spirit—a spirit of openness, collaboration, and sharing. By providing huge open datasets and their quality signals, RedPajama is building a “Shared Garden” in the AI field. In this garden, everyone can pick high-quality “seeds” (data) according to their own needs and plant their own “flowers of intelligence” (AI models), thereby jointly promoting the prosperous development of artificial intelligence technology.

With the release of large-scale, high-quality, multi-language datasets like RedPajama-V2, we can expect to see more innovative AI models emerge. These models will not only be more powerful, but their development process will also be more transparent and fair, truly benefiting all of humanity with the power of AI.

ReLU Variants

The wave of Artificial Intelligence (AI) is changing our lives, and behind this wave, neural networks play a core role. In neural networks, there is a seemingly inconspicuous but crucial component that determines whether a neuron is “activated” and the intensity of the activation. This is what we will discuss in depth today—Activation Functions. In particular, we will focus on an activation function called ReLU (Rectified Linear Unit) and its various “improved versions” or “variants”.

Starting from the “Switch”: What is an Activation Function?

Imagine our brain, where billions of neurons transmit electrical signals through complex connection networks. After receiving signals from other neurons, each neuron decides whether to get “excited” based on the sum of these signals and passes the signal to the next neuron. If the signal strength is not enough, it may “remain silent”; if the signal is strong enough, it will “light up” and transmit information.

In AI neural networks, the activation function plays the role of this “neuron switch”. It receives an input value (usually the weighted sum of all previous input signals) and then outputs a processed value. This output value will determine whether the neuron is activated and the degree of its activation. If all neurons simply transmit numerical values, the entire network will only perform linear operations, and even the most complex network can only solve simple problems. The activation function introduces non-linearity, enabling neural networks to learn and simulate more complex and non-linear patterns in the real world, just like letting your computer recognize cat and dog pictures instead of just doing simple addition and subtraction.

Simple but Powerful: The Original ReLU

Long ago, neural networks mainly used activation functions like Sigmoid or Tanh. They are like traditional “faucet switches”—turn a little, a little water flows; turn all the way, water flows maximally. However, when the water flow is very small or very large, the pressure (gradient) change in the pipe becomes very gentle, making it difficult to precisely adjust the valve (parameters). This is the so-called “gradient vanishing” problem, making the training of deep neural networks extremely slow and difficult.

To solve this problem, researchers introduced a “simple and crude” but very effective activation function—ReLU (Rectified Linear Unit).

You can imagine it as a “one-way gate“ or a “positive signal light“:

• If the input is a positive number, it lets the signal pass through unchanged (for example, if you give it 5 volts, it outputs 5 volts).
• If the input is a negative number, it completely cuts off the signal and outputs 0 (for example, if you give it -3 volts, it outputs nothing, total darkness).

The advantages of ReLU are obvious:

• Very fast calculation: Because it only involves simple judgment and output, unlike the previous faucet switches that required complex mathematical operations (exponential functions).
• Solved the gradient vanishing problem for positive signals: For positive inputs, its “slope” (gradient) is fixed and will not become gentle at both ends like old-fashioned switches.

However, this “one-way gate” also has its troubles, which is the “Dying ReLU“ problem. Imagine if a neuron always receives negative inputs, then it will always output 0, its “switch” will be turned off forever, unable to be activated again, and unable to update its own learning parameters. This is like a water pipe that is completely blocked and can no longer flow water—this water pipe (neuron) is “dead”.

Striving for Perfection: Various “Variants” of ReLU

To overcome these limitations of ReLU, scientists designed a series of smarter and more flexible “upgraded” activation functions based on the “one-way gate”, which we call ReLU Variants. Their goals are to maintain the advantages of ReLU while avoiding or mitigating problems like “Dying ReLU” to improve the learning ability and stability of neural networks.

Let’s look at several major ReLU variants:

1. Leaky ReLU: Letting in a Little Light

To solve the “Dying ReLU” problem, the most direct way is to let the “completely closed gate” leak a little bit.

• Metaphor: Imagine a “leaky faucet“. When the input is positive, it still releases water normally; but when the input is negative, it no longer closes completely but leaks a little bit of water (a very small negative value, such as 0.01 times the input value).
• Principle: The characteristic of Leaky ReLU is: $f(x) = \max(0.01x, x)$. This means that when the input $x$ is less than 0, it outputs $0.01x$ instead of 0.
• Advantage: By allowing a tiny non-zero gradient in the negative region, even if the neuron’s input is always negative, it can transmit weak signals, thus avoiding the risk of “death” and continuing to participate in learning.

2. PReLU (Parametric ReLU): The Learnable Gate

The “leak” ratio (0.01) in Leaky ReLU is fixed. Can we let the neural network learn this optimal “leak” ratio by itself? This is PReLU.

• Metaphor: This is a “smart leaking faucet“. Its leak ratio in the negative region is not a fixed 0.01, but allows the neural network to learn a most suitable ratio parameter $a$ during the training process.
• Principle: The characteristic of PReLU is: $f(x) = \max(ax, x)$, where $a$ is a learnable parameter.
• Advantage: By introducing learnable parameters, PReLU can adaptively adjust the slope of the negative region according to the characteristics of the data, thereby achieving better performance.

3. ELU (Exponential Linear Unit): Smoother Drainage Pipe

Besides letting the negative region have a slope, we also care about whether the output values can be evenly distributed around 0, which is also important for the stability of network training. ELU improves on this.

• Metaphor: Imagine a “smoothly transitioning drainage bend“. When input is positive, it outputs normally; when input is negative, it is no longer a linear “leak”, but uses an exponential function form to smoothly output negative values, and these negative outputs can help the average output of the entire network be closer to zero, making training more stable.
• Principle: The characteristic of ELU is: when $x > 0$, $f(x) = x$; when $x \le 0$, $f(x) = \alpha(e^x - 1)$, where $\alpha$ is a hyperparameter (usually set to 1).
• Advantage: ELU not only solves the “Dying ReLU” problem but also helps the mean of the network output approach zero through its smooth negative value output, thereby speeding up learning and improving the model’s robustness to noise.

4. Swish / SiLU: The “Thinking” Smart Dimmer

In recent years, as the complexity of deep learning models has continued to increase, some more advanced activation functions have begun to emerge. Among them, Swish (or SiLU) and GELU are currently very popular choices in large models (such as Transformer).

• Metaphor: This is not a simple switch, but a “smart dimmer“. It doesn’t just look at whether the signal is positive or negative, but uses a “self-gating” mechanism to decide how much to output, and the output changes are very soft and smooth.
• Principle: The Swish function is usually defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$, where $\beta$ is a constant or learnable parameter. When $\beta = 1$, it is SiLU (Sigmoid Linear Unit): $f(x) = x \cdot \text{sigmoid}(x)$.
• Advantage: The curve of Swish/SiLU is very smooth and non-monotonic (in some regions, the output value will drop first and then rise, which makes them show “memory” and “forgetting” characteristics in some cases). Most importantly, it has the characteristics of no upper bound, lower bound, and smoothness, which can prevent gradient saturation during training, and performs better than ReLU on many tasks, especially in deep networks.

5. GELU (Gaussian Error Linear Unit): Probabilistic Fuzzy Gate

GELU is another very popular and excellent activation function, especially favored by large Transformer models in the field of Natural Language Processing.

• Metaphor: It is a “fuzzy gate with a bit of randomness“. It doesn’t simply truncate negative values like ReLU, nor does it fixedly “leak” a little like Leaky ReLU, but decides whether to let the signal pass with a bit of “probability” based on the input value. This “probability” is based on the Gaussian distribution (a common bell-shaped curve distribution), so it can regulate signals more finely and intelligently.
• Principle: The definition of GELU is $f(x) = x \cdot P(X \le x)$, where $P(X \le x)$ is the cumulative distribution function $\Phi(x)$ of the standard normal distribution. In other words, it is the input value $x$ multiplied by its cumulative probability in the Gaussian distribution.
• Advantage: GELU combines the advantages of ReLU and the idea of Dropout (a technique to prevent overfitting), improving the model’s generalization ability by introducing randomness. Its smoothness and non-linear characteristics make it perform excellently when processing complex data, especially language data, and it is commonly used in large pre-trained models like BERT and GPT.

Summary and Outlook

From the initial simple “switch” ReLU to SiLU and GELU that can “learn”, “think”, and even carry a bit of “probability”, the evolution of activation functions demonstrates the spirit of continuous exploration and innovation in the field of Artificial Intelligence.

The importance of these ReLU variants lies in their ability to:

• Solve ReLU’s drawbacks: Such as the “Dying ReLU” problem.
• Improve model performance: Smoother and more flexible functions can better fit complex data.
• ** enhance training stability**: Reduce the risk of gradient vanishing or explosion, making the model easier to train.

Of course, just as there is no panacea that cures all diseases, there is no “best” activation function suitable for all scenarios. Which ReLU variant to choose often depends on the specific task, data characteristics, and model architecture. But it is certain that these carefully designed activation functions are undoubtedly an important force driving the continuous development of AI technology. In the future, as AI models become larger and more complex, we may see more ingenious and efficient activation functions emerge, continuing to play the key role of letting machines “think” in neural networks.

RMSprop

The “Guiding Light” of AI Training: A Simple Explanation of RMSprop Optimization Algorithm

In the vast world of Artificial Intelligence (AI), we often hear the term “training models”. Imagine training an AI model is like teaching a student new knowledge. The student needs to constantly solve problems and correct mistakes to improve. In the AI field, this process of “correcting mistakes” and guiding the model to learn in the right direction relies on various “Optimizers”. Today, we are going to talk about RMSprop, which is one of the many excellent optimizers. It acts like an experienced mountain guide, helping AI models find the best path for learning more efficiently and stably.

What is RMSprop?

The full name of RMSprop is “Root Mean Square Propagation”. It sounds a bit technical, but its core idea is actually very intuitive—adaptively adjusting the “step size” of learning.

In the process of training an AI model, our goal is to constantly adjust the internal “parameters” of the model (which can be understood as various knowledge points in the student’s brain) so that the model makes the fewest mistakes when performing specific tasks (such as recognizing pictures, translating languages). This process of adjusting parameters is what we call “Gradient Descent”.

Vivid Metaphor: The Wisdom of a Mountaineer

To better understand RMSprop, let’s imagine the story of a mountaineer. The goal of this mountaineer is to find the lowest point of the valley (this lowest point is the “optimal solution” or “minimum of the loss function” in our AI model training).

1. Stochastic Gradient Descent (SGD): A Blindfolded Mountaineer
   The earliest “mountaineer”—Stochastic Gradient Descent (SGD), usually walks blindfolded. He takes fixed-size steps each time, in the direction of the steepest slope (gradient) he feels under his feet.

   • Problem: If the mountain road goes straight down, SGD can walk well. But if the terrain is steep at times and gentle at others, or like a narrow “valley” with steep slopes on both sides but a gentle slope in the valley bottom direction, this mountaineer might sway left and right in this valley, wasting a lot of energy on unnecessary oscillations and advancing very slowly.

2. RMSprop: A Wise Guide with “Historical Experience”
   RMSprop is a smarter mountaineer. He no longer walks completely blindly but possesses a special “memory” system capable of remembering “how steep the slopes he has walked” in a certain direction recently.

   • Adaptive Steps: When he finds that a certain direction (update of a certain parameter) has always been particularly steep in the past (when the gradient changes greatly), indicating that the “terrain” in this direction might be complicated or full of “noise”, he will be careful and take smaller steps to avoid “overshooting” or falling into unnecessary oscillations. Conversely, if he finds that a certain direction has always been relatively gentle in the past (when the gradient changes little), he will boldly take larger steps to speed up progress.
   • “Root Mean Square” Memory: RMSprop’s “memory” method is calculating the “exponential decay average” of the squared gradients. This is like a continuously updated record of “average steepness”. It does not simply remember all historical information but gives recent slope information greater weight, while information from long ago gradually fades. This “memory” allows it to better adapt to changing terrain conditions.

How Does RMSprop Do It? (Technical Reveal)

RMSprop achieves its “wisdom” through the following core mechanisms:

1. Accumulating Squared Gradients: For each parameter in the model (imagine each coordinate axis in the valley), it calculates the square of the gradient at each update.
2. Exponential Moving Average: It does not directly use the square of all historical gradients but calculates an “exponential decay average”. This means that the squared gradient values of the last few times have a greater impact on the average, while the squared gradient values from long ago have a diminishing impact. This average value can be seen as a “historical record” or an estimate of the “oscillation degree” of the gradient change amplitude of this parameter.
3. Adjusting Learning Rate: When updating parameters, RMSprop divides the original learning rate (our “maximum step size”) by the square root (i.e., root mean square) of this “exponential decay average”.
   • If the gradient change was large in the past, the root mean square is large, so after dividing by it, the actual learning step size will become smaller.
   • If the gradient change was small in the past, the root mean square is small, and the actual learning step size will become larger.

This mechanism effectively solves the problem of inconsistent strides in different dimensions of traditional gradient descent, especially for those directions where gradients change greatly, it can effectively suppress oscillation and make the training process more stable. Geoff Hinton once suggested that in practice, the decay coefficient (parameter measuring the weight of old gradient information) is usually set to 0.9, and the initial learning rate can be set to 0.001.

Pros and Limitations of RMSprop

Pros:

• Solving Adagrad’s Problem: Before RMSprop, the Adagrad optimizer also tried adaptive learning rates, but it would accumulate the square of gradients without limit, causing the learning rate to become smaller and smaller, and training might stop prematurely. RMSprop effectively solves this problem through exponential decay average.
• More Stable Training: By adaptively adjusting the learning rate for different parameters, RMSprop can effectively handle gradient oscillation and improve the stability of training.
• Wide Applicability: It is particularly suitable for dealing with complex, non-convex (i.e., having many “bumps and hollows”) error surfaces, as well as non-stationary (the objective function changes all the time) objectives.

Limitations:

• Although RMSprop can adaptively adjust the learning rate of each parameter, it still requires us to manually set a global learning rate (i.e., the “maximum step size” mentioned earlier), and the choice of this value will still affect the training effect.

RMSprop and Adam: The Story of Successors

After the emergence of RMSprop, the evolution of AI optimization algorithms did not stop. Another very popular optimizer—Adam (Adaptive Moment Estimation)—was further developed on the basis of RMSprop. Adam not only inherits the advantages of RMSprop’s adaptive learning rate but also introduces the concept of “Momentum”, which can be understood as adding “inertia” or “memory inertia”. This makes Adam perform better than RMSprop on many tasks and has become one of the most commonly used optimizers in deep learning today.

Nevertheless, RMSprop is still a very important and effective optimization algorithm, which is still the first choice in some specific scenarios, and it laid the foundation for subsequent more advanced optimization algorithms.

Summary

RMSprop is like an experienced mountain guide, providing intelligent step size suggestions for every step (every parameter update) in AI model training by “remembering” the “average steepness” of the historical terrain. It effectively improves the problems of traditional gradient descent and paves the way for the development of subsequent more advanced optimization algorithms (such as Adam). Understanding RMSprop not only helps us train AI models better but also gives us a deeper understanding of those seemingly complex technical concepts in the AI world.