ROUGE Score

Revealing the AI Text Evaluation “Reviewer”: ROUGE Score, It Decides If You Write Well!

In the wave of artificial intelligence, we see amazing text content generated by various AI models every day, from automatic summarization to machine translation, and intelligent question answering. But how do we know if these AI-generated texts are “good” or “bad”? Does it accurately convey the meaning of the original text? Or is it fluent and natural, capturing the key points? To answer these questions, the AI field has introduced a variety of evaluation metrics, one of the very important and widely used ones is the ROUGE score.

ROUGE, the full name being “Recall-Oriented Understudy for Gisting Evaluation”, literally translates to “Recall-Oriented Summarization Evaluation Stand-in”. As the name suggests, it was originally designed for automatic summarization tasks to measure the degree of similarity between machine-generated summaries and human-written “standard answers” (i.e., reference summaries). You can think of it as the “grading teacher” for AI text creation, using a relatively objective set of standards to score the AI.

ROUGE’s “Scoring Principle”: Like “Checking Answers”

The core idea of ROUGE is actually very simple: it scores by calculating the degree of overlap of common words or phrases between the computer-generated text and one or more human reference texts. Is it a bit like when we were kids finishing homework, and then checking against the standard answer to see how many words and sentences we got right? ROUGE uses this “checking answers” method to judge the quality of AI-generated text.

ROUGE is not a single indicator, but a collective term for a group of indicators, mainly including: ROUGE-N, ROUGE-L, and ROUGE-S. They each measure text similarity from different angles.

1. ROUGE-N: “Precise Match” of Words and Phrases

ROUGE-N measures the overlap of “N-grams” between machine-generated text and reference text.

• What is an N-gram? simply put, an N-gram is a sequence of N consecutive words in a text.
  • If N=1, it is a “unigram”, i.e., a single word.
  • If N=2, it is a “bigram”, i.e., a phrase consisting of two consecutive words.

For example:
Suppose your AI model generated a sentence: “The cat sits on the mat.“
And the standard answer is: “The little cat sits on the soft mat.“

• ROUGE-1 (Unigram): It compares the overlap of single words in the two sentences.

  • Words present in both sentences are: “cat”, “sits”, “on”, “the”, “mat”.
  • A high ROUGE-1 score usually means that the AI’s text captured most of the keywords.

• ROUGE-2 (Bigram): It compares the overlap of phrases consisting of two consecutive words.

  • AI Generated: “The cat”, “cat sits”, “sits on”, “on the”, “the mat”.
  • Standard Answer: “The little”, “little cat”, “cat sits”, “sits on”, “on the”, “the soft”, “soft mat”.
  • Overlapping phrases are: “cat sits”, “sits on”, “on the”.
  • A high ROUGE-2 score indicates that the AI not only grasped the keywords but also preserved the local order relationship between words, making the generated phrases more human-like.

You can imagine ROUGE-N as a comparison of a “shopping list”. If you listed “apples, milk, bread”, and the standard list is “apples, oranges, milk, bread”, then ROUGE-1 will find that “apples, milk, bread” all match. If the standard list is “fresh milk, whole wheat bread”, and you wrote “milk, bread”, ROUGE-2 will check if you even matched phrases like “milk” and “bread”.

2. ROUGE-L: “Skeleton Match” of Long Sentence Structure and Main Information

ROUGE-L measures the overlap of the Longest Common Subsequence (LCS) between machine-generated text and reference text. The “subsequence” here does not have to be continuous, but must maintain the original word order.

For example:
AI Generated: “The meeting discussed budget cuts and market expansion.“
Standard Answer: “Today’s meeting mainly discussed issues such as market expansion and budget cuts.“

LCS might be: “meeting discussed… budget cuts”, “market expansion”. ROUGE-L will find the longest part in the two sentences with consistent word order.

It’s like you are retelling the plot of a long movie. You might not remember every line of dialogue verbatim, but you will remember the key plot points and the order in which they happened. For example: “The protagonist met a mentor, obtained a magic item, and finally defeated the villain.” Even if you used your own words to describe it, ROUGE-L can identify the similarity of this sequence of key events. A higher ROUGE-L score indicates that the AI-generated text is more consistent with the reference text in terms of overall structure and main information flow.

3. ROUGE-S: “Jump Match” of Core Concepts

ROUGE-S (Skip-gram ROUGE) is a more flexible metric that considers the overlap of “skip-bigrams” (skip-n-gram). That is, even if there are other words between two words, as long as they maintain relative order in the original sentence, ROUGE-S can identify them as a match.

For example:
Standard Answer: “This fast and important policy will soon bring positive changes.“
AI Generated: “Policy brings positive changes.“

In this case, ROUGE-S can identify skipped bigrams like “policy… brings”, “brings… changes”, even if words like “fast and important”, “soon bring” are omitted in between.

ROUGE-S is like organizing notes after listing to a speech. You might not write down every word, but you will connect some related and important words, even if they are not closely connected in the speech. A high ROUGE-S score indicates that the AI-generated text can capture core conceptual associations, even if the expression is simplified or changed.

Additional Considerations: Precision, Recall, and F1 Score

ROUGE scores are usually presented together with Precision, Recall, and F1-measure.

• Recall: Imagine you have a “treasure chest” full of all important information (reference text). Recall tells you what proportion of important information the AI-generated text has fished out of this treasure chest. As the name implies, ROUGE scores are recall-oriented.
• Precision: Now look at the AI-generated text itself. Precision tells you what proportion of information in the text generated by the AI itself truly comes from the “treasure chest” (i.e., is accurate and relevant).
• F1 Score (F-measure): It is the harmonic mean of Precision and Recall, which can be seen as a comprehensive assessment of both, balancing the comprehensiveness and accuracy of the generated text.

In layman’s terms:

• High Recall, Low Precision: The AI is like a “chatterbox”, saying a lot for fear of missing something, but there is a lot of nonsense or irrelevant information.
• High Precision, Low Recall: The AI is very “concise”, every sentence it says is accurate, but it may miss a lot of important information.
• High F1 Score: The AI generates text “just right”, with neither nonsense nor missing key points.

Pros and Cons of ROUGE: Objective but Not “Smart” Enough

Advantages of ROUGE:

• Strong Objectivity: ROUGE provides a quantifiable standard for assessing text quality, reducing the subjectivity of manual evaluation, and facilitating comparison and benchmarking between models.
• Easy to Calculate: The calculation method based on word overlap is relatively intuitive and efficient.
• Widely Used: ROUGE is a mainstream evaluation tool in multiple NLP fields such as text summarization and machine translation.

Limitations of ROUGE:

However, ROUGE is not perfect, it also has its “not smart” side:

1. Stays on Semantic Surface: ROUGE mainly focuses on literal overlap of words or phrases, so it cannot capture semantic similarity well. For example, “very big” and “huge” have similar meanings, but in ROUGE’s view, they are different words, which may lower the score. It also doesn’t understand synonyms and paraphrasing.
2. Ignores Context and Coherence: ROUGE cannot understand the overall context of the text, logical relationships between sentences, and the fluency and readability of the text. A summary with a high ROUGE score may just be a pile of key phrases from the original text, but reads fragmented.
3. Insensitive to Factual Accuracy: It doesn’t care if the content generated by AI is true or involves “hallucinations”. For example, AI may generate a summary that is grammatically correct and has high word overlap with the original text, but the actual content is inconsistent with the facts.
4. May Be Biased Towards Long Summaries: Since it focuses more on recall, it sometimes favors summaries that contain more words and are longer, because long summaries are more likely to have more word overlap with the reference text.
5. Dependence on Reference Summaries: ROUGE requires high-quality human-written reference summaries as “standard answers”. Creating these reference summaries is often time-consuming and costly, and different reference summaries may lead to different scores.

Looking to the Future: Smarter Evaluation Methods

Given the limitations of ROUGE, researchers are constantly exploring smarter and more comprehensive evaluation methods. For example:

• BERTScore: It uses the word vectors of pre-trained language models (such as BERT) to measure semantic similarity. Even if the words are different but the meanings are similar, it can give a higher score. This is like no longer just looking at whether words are exactly the same, but understanding from a deeper level whether their meanings are similar.
• Human Evaluation: although time-consuming, humans are still the “gold standard” for evaluating text quality, able to judge semantic accuracy, logical coherence, fluency, and other aspects that AI finds difficult to capture.
• LLM-based Evaluation: Large Language Models (LLMs) themselves can also be used to evaluate summary quality, judging its relevance, coherence, and factual accuracy, sometimes even without reference summaries. But this also faces the arbitrariness and bias problems of LLMs themselves.

Summary

The ROUGE score is an important tool for measuring the quality of AI-generated text (especially summaries). By calculating the degree of overlap of words or phrases, it provides us with a quantitative and objective evaluation standard. ROUGE-N focuses on precise matching of words and phrases, ROUGE-L focuses on long sentence structure and main information flow, while ROUGE-S captures the association of core concepts more flexibly.

However, we must also clearly recognize the limitations of ROUGE—it is like a rigorous but unsympathetic “grading teacher”, capable of detecting many superficial errors, but unable to give effective judgments on the deep meaning, logical coherence, and factual accuracy of the text. Therefore, when evaluating AI-generated text, we often need to combine multiple automated metrics such as ROUGE and BERTScore, supplemented by human evaluation, to have a more comprehensive and in-depth understanding of AI’s text capabilities.

ReAct

Unveiling the “Left and Right Hands” of AI Thinking: An In-Depth but Accessible Guide to the ReAct Framework

Imagine you have an extremely intelligent assistant who is well-read, has a photographic memory, is eloquent, and can give you a plausible-sounding answer to almost any question you ask. This is the “Large Language Model” (LLM) we often hear about now. However, this assistant also has a small flaw: he only lives in his own world of knowledge, unable to go online to check the latest information, unable to use a calculator to help you with accounts, let alone call a restaurant to make a reservation. Even worse, sometimes he fabricates information that sounds very real but is actually wrong, which is called “hallucination” in the AI field.

So, how can we make this smart assistant more “grounded” and reliable? The answer is—the ReAct framework.

ReAct: Your AI Assistant Now Can “Think” and “Act”!

The name ReAct itself reveals its core secret: it combines “Reasoning“ and “Acting“. Simply put, ReAct empowers large language models with a human-like problem-solving capability: think first, then take action based on the thinking result, then think further based on the feedback from the action, and repeat until the problem is solved.

Let’s use a vivid analogy to understand it.

The “Thinking” of LLMs: Like a Detective’s Inner Monologue

When a detective receives a complex case, he won’t immediately identify the murderer. He will first analyze the clues in his mind, envision various possibilities, and formulate an investigation plan, such as “Who might this fingerprint belong to? I need to check the police database.” or “Who did the victim have a grudge against? I have to talk to his colleagues.” This internal brainstorming and logical reasoning process is the “Reasoning” part of the large language model. It breaks down the problem step by step, plans strategies, weighs pros and cons, and even corrects previous ideas.

The “Thinking” of LLMs: Like a Detective’s “Eighteen Martial Arts”

Just thinking without acting cannot solve the case. After the detective figures out what needs to be done, he will truly “act”: call the forensic doctor, visit witnesses, query information, use fingerprint recognition equipment, etc. These “actions” are the various “tools” or interfaces that the LLM can call within the ReAct framework. For example, it can be a search engine (to query the latest information), a calculator (to perform precise calculations), an external database (to get specific data), an API interface (to control external systems, such as booking tickets or sending emails), etc.

“Observation”: Feedback from Action

After the detective takes action, he will get a result: a fingerprint was found, a witness provided a new clue, no relevant records were found in the database, etc. These results are “Observations” in ReAct. Just like a detective thinks again after receiving new clues, the large language model will also feed the “observed” results back to its “thinking” module, thereby adjusting the next plan or action, forming a continuously iterative problem-solving process.

The Operational Flow of ReAct: Getting deeper like a detective solving a case

Imagine the process of an AI detective solving the case (task) “Do I need an umbrella in London today?”:

1. AI Detective Receives Task: User asks: “I’m in London, do I need an umbrella today?”
2. Thought: The AI detective analyzes in his mind: “The user is asking about the weather in London today, especially about the possibility of rain. I need to get real-time weather information for London today.”
3. Action: The AI detective decides to use the “Weather Query Tool” (such as a weather API). Call the tool and pass parameters: “Query London’s weather for today.”
4. Observation: The weather query tool returns the result: “London is sunny turning to cloudy today, with a 20% chance of precipitation.”
5. Thought: The AI detective analyzes the observation result: “The probability of precipitation is not high. Generally, a 20% chance of rain means no special need for an umbrella. I can give an answer.”
6. Final Answer: The AI detective replies: “The probability of rain in London is low today, you probably don’t need an umbrella.”

Through this “Thought-Action-Observation” loop, the AI model is no longer a passive “Question-Answering Machine”, but an active “Problem Solver”.

Superpowers Brought by ReAct

The ReAct framework gives large language models the following “superpowers”:

• More Accurate and Reliable: By obtaining factual information through external tools, the possibility of the model “making things up” (hallucination) is greatly reduced, and results are more truthful and credible.
• Handling Complex Tasks: Capable of breaking down complex tasks into a series of small thinking and action steps, approaching the goal step by step, solving difficult problems that are hard to complete by memory alone.
• Connecting to the Real World: Making up for the defect that LLMs cannot directly perceive and affect the external world, allowing AI to go online, calculate, and operate real-world tools.
• Enhanced Interpretability: Since the AI’s thinking and action process is explicitly shown step by step, we can clearly see its problem-solving logic, which helps us understand, debug, and trust AI.
• Real-time Information Access: The LLM’s own knowledge base may be static, but through tools like search engines, ReAct allows AI to access the latest real-time information.

ReAct Did Not Appear Out of Nowhere: Difference from “Chain-of-Thought”

Before ReAct, a technique called “Chain-of-Thought” (CoT) was popular in the AI field. CoT allows large language models to generate a series of intermediate reasoning steps before answering a question, just like a human writes down every calculation step when solving a math problem. This indeed improved the reasoning ability of LLMs.

However, the disadvantage of CoT is that it relies entirely on the model’s internal knowledge and reasoning, and cannot interact with the external world. This is like a detective who, although capable of thinking, cannot leave the office to conduct field investigations. Therefore, CoT is still prone to factual errors or “hallucinations”.

ReAct goes a step further by combining CoT’s “thinking” with actual “action”, forming a closed loop of “Thought-Action-Observation”. This allows AI not only to think about how to solve a problem but also to put it into practice and correct its thinking based on practice results, thereby achieving stronger problem-solving capabilities.

ReAct in Daily Life

The application of ReAct goes far beyond weather queries. For example:

• Intelligent Customer Service: AI customer service is no longer just answering common questions; it can understand user intent through “thinking”, and then “act” to query databases, initiate refund processes, or even connect to human customer service.
• Personalized Education: AI can “think” about students’ learning progress and weaknesses, and then “act” to recommend customized course materials and generate practice questions.
• Travel Planning: AI can “think” about your preferences and budget, and then “act” to search for flight and hotel information, and even compare prices.

Conclusion

The emergence of the ReAct framework is an important milestone in the history of large language model development. It arms AI from a language expert who “can only talk” into an intelligent agent who “can both think and act”. By empowering AI with the ability to interact with the external world, ReAct is leading us towards a smarter, more autonomous AI era, making AI truly a capable assistant in our lives and work.

RMSNorm

In the vast universe of Artificial Intelligence (AI), Large Language Models (LLMs) are evolving at an astonishing speed, capable of understanding and generating human language, and even engaging in creative writing. Hidden deep within the “brains” of these complex models are many key “unsung heroes” that ensure the model learns stably and efficiently. The “RMSNorm” we are going to popularize today is one of them. It is an ingenious Normalization technique, acting like an “intelligent volume regulator” in the AI world, making complex computational processes orderly.

Why Do AI Models Need an “Intelligent Volume Regulator”?

Imagine a huge factory assembly line where each workstation (each layer of a neural network) receives semi-finished products from the previous workstation, processes them, and then passes them on. If the parts passed from the previous workstation vary in size and shape, the next workstation will find it difficult to process efficiently, and might even shut down because it cannot adapt to this “chaos”. In AI models, this “chaos” is called “Internal Covariate Shift” or “Gradient Problems” (Vanishing/Exploding Gradients).

Specifically, when a layer of a neural network updates its parameters, it causes the distribution of input data for subsequent layers to change. This continuous change forces subsequent layers to constantly adapt to new input distributions, slowing down training speed and affecting model stability, much like workers on an assembly line constantly adjusting tools to adapt to ever-changing parts. Furthermore, data being too large or too small can lead to gradients disappearing during backpropagation (Vanishing Gradients, model cannot learn) or exploding (Exploding Gradients, model training crashes), just like volume being too low to hear or too high to be deafening.

To solve these problems, scientists introduced “Normalization Layers”. Their goal is like an intelligent quality inspection and adjustment station on the assembly line, ensuring that the semi-finished products output by each workstation meet unified standards, keeping data within a suitable “volume” range, thereby improving training stability and efficiency.

RMSNorm: A More “Concise” Intelligent Volume Regulator

Among various normalization techniques, the most famous is Layer Normalization (LayerNorm). RMSNorm (Root Mean Square Normalization) is a simpler and more efficient “intelligent volume regulator” that has emerged in the era of LLMs.

What is Root Mean Square (RMS)?

To understand RMSNorm, we first need to understand the concept of “Root Mean Square” (RMS). In daily life, we may have heard of the “effective voltage” or “effective current” of alternating current. This “effective value” is a kind of RMS. Instead of simply calculating the average of a set of numbers, it first squares all numbers, then calculates the average of these squared values, and finally takes the square root. It measures the “average intensity” or “energy” of a set of values, is more sensitive to extreme values, and better reflects the overall “vitality”.

A vivid analogy is: Suppose you have a band, and the volume of each instrument (the output of a “neuron” in a neural network) varies. RMSNorm is like an intelligent sound engineer who only focuses on the “energy” of the volume. It calculates the “average energy” (RMS) of each instrument’s sound, and then adjusts the overall volume of each instrument based on this energy value. It’s not about adjusting the sound of all instruments to exactly the same pitch or timbre, but ensuring their overall loudness is in a comfortable and clear range, avoiding one instrument being too loud to drown out others, or one instrument being too quiet to hear.

How RMSNorm Works

The way RMSNorm works is very straightforward:

1. Calculate Root Mean Square: For all input data (or a vector) of a certain layer of the neural network, it first calculates the RMS of these values.
2. Scaling: Then, it divides each input value by this calculated RMS.
3. Optional Gain Adjustment: Usually, it also multiplies by a learnable “gain” parameter (γ), allowing the model to fine-tune the overall magnitude of the data after normalization to achieve optimal performance.

Unlike the widely used LayerNorm, RMSNorm omits the step of subtracting the mean (re-centering) during the normalization process. LayerNorm adjusts both the “center” of the data (making the mean close to 0) and the “size” (making the variance close to 1), while RMSNorm focuses on adjusting the “size” (i.e., overall magnitude) of the data, ensuring its “average energy” is in a stable range.

Why is RMSNorm So Popular?

This “simplification” of RMSNorm is not cutting corners, but brings many advantages, making it an increasingly important choice in modern AI models, especially Large Language Models (LLMs):

1. Significantly Improved Computational Efficiency: Omitting the step of calculating the mean means fewer floating-point operations. For LLMs with tens or hundreds of billions of parameters, every saving in calculation means a huge reduction in resource and time costs. Original research papers show that RMSNorm can reduce model training runtime by 7% to 64%.
2. More Stable Model Training: Although simplified, RMSNorm retains the most important “rescaling invariance” property of normalization. This means that no matter how much the input data is scaled up or down, RMSNorm ensures that the overall magnitude of its output remains stable, effectively preventing gradient vanishing or exploding problems during training.
3. Simpler Code Implementation: Due to simpler mathematical formulas, RMSNorm is easier to implement and deploy in code, reducing the complexity of development and maintenance.
4. Shining in LLMs: Many leading large language models, such as Meta’s LLaMA family, Mistral AI’s models, and Google’s T5 and PaLM models, have chosen RMSNorm as their core normalization technique. It has been proven to provide stable and efficient training in large-scale Transformer architectures, becoming an important driving force for LLM technology development.
5. Continuous Optimization and Innovation: Researchers are constantly exploring the potential of RMSNorm. For example, latest techniques like “Flash Normalization” are trying to fuse RMSNorm operations with subsequent linear layer calculations to further optimize LLM inference speed and efficiency. In addition, when quantizing models to low precision to reduce memory and computational requirements, extra RMSNorm layers can also help maintain model stability and performance.

Summary

As an important concept in the field of artificial intelligence, RMSNorm plays an indispensable role in frontier applications such as large language models due to its simplicity, efficiency, and stability. It acts like an “intelligent volume regulator” in AI models, silently ensuring that the “energy” of data flow inside the neural network is always kept in the optimal state, allowing complex AI systems to run stably and continuously break through performance boundaries. Understanding RMSNorm not only helps us deeply understand the operating mechanism of contemporary AI models but also allows us to see that sometimes the most elegant and powerful solutions come from ingenious simplification of complex problems.

Q-Learning

The “Explorer” of Artificial Intelligence: Simple Explanation of Q-Learning

Imagine you are dropped into a completely strange city with no map and no guide. Your task is to find a legendary, delicious restaurant. You might start walking aimlessly, finding a place to eat when hungry, but you will also remember which intersections brought you closer to the destination and which choices led you to taste delicious food (or step on a landmine). Every attempt and feedback helps you accumulate experience, so that next time you encounter a similar situation, you can make a better choice.

This process of finding delicious food works surprisingly similarly to a very interesting algorithm in the field of artificial intelligence—Q-learning. Q-learning is a core and important algorithm in Reinforcement Learning. Reinforcement learning is a branch of machine learning. Its core idea is to let an “Agent” interact continuously with the “Environment” and learn how to take the best action to achieve a preset goal based on the “Reward” or “Punishment” obtained after each action, just like a child learns to ride a bicycle through trial and error.

What is Q-Learning? — The “Secret Manual” for Scoring Actions

The core of Q-learning lies in its attempt to learn something called “Q-value”. Here “Q” can be understood as an abbreviation for “Quality”. Q-value represents the long-term “goodness” or magnitude of future potential returns that can be obtained by taking a specific “Action” in a specific “State”.

We can imagine the Q-value as an “Action Secret Manual” or “Scoring Handbook” for the agent. When the agent faces a choice, it will consult this manual to see how many points it can get by choosing different actions in the current situation. The higher the score, the better the “quality” of this action, and the more worthy it is to be taken.

Five Elements of Q-Learning: Agent, Environment, State, Action, and Reward

To understand how Q-learning works, we first need to recognize several basic components of its world:

1. Agent: This is the learner itself, such as “you” looking for a restaurant in a strange city, or an AI program playing a game, a cleaning robot, etc.
2. Environment: The external world where the agent is located, which contains all the information the agent can perceive. For you looking for a restaurant, the environment is the entire city; for the AI playing a game, the environment is the game interface; for the cleaning robot, the environment is the room map and obstacles.
3. State: The specific situation of the environment at a certain moment. For example, your specific location in the city coordinate system, the health volume and location of the game character, or which corner of the room the robot is currently in.
4. Action: The choice the agent can make in a certain state. You can choose to go east or west; the game character can choose to attack or defend; the robot can choose to move forward or turn.
5. Reward: The feedback signal given by the environment after the agent executes an action. These feedbacks can be positive (such as finding a restaurant, defeating an enemy, cleaning up), or negative (such as getting lost, being attacked by an enemy, hitting an obstacle). The goal of the agent is to maximize the accumulated reward it receives.

The Mystery of the Q-Table: The “Treasure Map” of Experience

The core operating mechanism of Q-learning is that it builds and updates a data structure called “Q-table”. You can imagine the Q-table as a constantly updated “Experience Handbook” or “Star Rating Guide”. Each row of this handbook represents a possible state, each column represents an action that can be taken, and each cell in the table stores the Q-value of taking that action in that state.

For example, in a simple maze game:

Initially, all Q-values in the Q-table are usually initialized to 0 or random values. This means that the agent has no preference for any action in any state at the beginning; it is just clueless.

Learning Process: From “Groping” to “Mastery”

So, how does the agent learn through the Q-table? This process can be summarized as constant “trial and error” and “summarizing experience”:

1. Observe State: The agent first observes its current state, such as where it is in the maze.
2. Choose Action (Exploration and Exploitation): This is one of the most interesting points in Q-learning. The agent needs to balance “Exploration” and “Exploitation”.
   • Exploration: Just like a child in a toy store always wants to try playing with new toys to see if there are any surprises. In Q-learning, this means the agent will randomly choose an action, even if it is not sure if this action is the best. This “exploration” is to discover new possibilities and potential greater rewards.
   • Exploitation: Just like you go to your favorite restaurant when you are hungry because you know it tastes good and is unlikely to go wrong. In Q-learning, this means the agent will consult the Q-table and choose the action with the highest current Q-value. This is the “optimal” choice based on existing experience.
   • To balance the two, Q-learning usually adopts a strategy called ε-greedy (epsilon-greedy): most of the time (say 90% probability), I will “greedily” choose the action with the highest Q-value (exploitation); but occasionally (say 10% probability), I will randomly choose an action (exploration), just like occasionally trying a new restaurant.
3. Execute Action and Get Feedback: The agent executes the chosen action, and then the environment gives it a reward (or punishment) and takes it to a new state.
4. Update Q-value: This is the core step of Q-learning. The agent updates the corresponding Q-value in the Q-table based on the reward just obtained and the new state entered. This update process is based on a mathematical formula. Simply put, it considers:
   • Immediate reward obtained from the current action.
   • Maximum possible future reward. The agent looks one step ahead and estimates what the best reward it can get in the future if it takes the optimal action in the new state.
   • “Discount Factor” (γ): This is a value between 0 and 1, which determines whether the agent values immediate rewards more or future rewards more. If γ is close to 1, the agent is “far-sighted” and will sacrifice some immediate small benefits for long-term interests; if γ is close to 0, the agent is “short-sighted” and only pursues immediate benefits.
   • “Learning Rate” (α): This is also a value between 0 and 1, which determines how much each learning affects the Q-value. A large learning rate means the agent updates faster but may be unstable; a small learning rate means slow updates but may be more stable.

Through such constant repetition, the agent makes a large number of attempts in the environment and corrects its Q-table. Over time, the Q-values in the Q-table will gradually stabilize, accurately reflecting the true “quality” of taking various actions in various states, thereby allowing the agent to learn how to maximize its accumulated reward.

Advantages and Limitations of Q-Learning

As a cornerstone of the reinforcement learning field, Q-learning has significant advantages:

• Model-Free: This is one of the most attractive parts of Q-learning. It does not need to know the operating rules or models of the environment in advance (such as the complete map of the maze, or the precise consequences of each action in the game). The agent learns entirely through interaction with the environment, which makes it very suitable for tasks where the environment is complex, rules are unknown, or difficult to model.
• Off-policy: When learning the “optimal policy”, Q-learning does not need to rely on the policy the agent actually takes. This means the agent can learn the optimal action guidance while exploring unknown paths.

However, Q-learning also has some limitations:

• “Curse of Dimensionality”: If the number of states or actions in the environment is enormous (for example, pixels in a high-resolution image as states, or a robot having infinite joint angles as actions), the Q-table will become extremely huge and impossible to store and update. This is called the “curse of dimensionality”.
• Slow Convergence Speed: In complex environments, Q-learning may require massive attempts to make Q-values converge to the optimum, and the learning process will be very long.

From Q-Learning to Deep Q-Network (DQN): Breaking the “Curse of Dimensionality”

To overcome the limitations of Q-learning in dealing with complex, high-dimensional problems, researchers introduced Deep Learning technology, giving birth to Deep Q-Network (DQN). DQN no longer uses a traditional Q-table to store Q-values, but uses a Deep Neural Network to approximate Q-values. The input of this neural network is the current state, and the output is the Q-value of each possible action.

DeepMind successfully applied DQN to Atari games in 2014, allowing AI to reach human expert levels in multiple classic games, shocking the world. The emergence of DQN greatly expanded the application range of Q-learning, enabling reinforcement learning to solve more complex and realistic problems.

Real-World Applications of Q-Learning

Q-learning and its variants (such as DQN) have penetrated every aspect of our lives:

• Game AI: Making NPCs (Non-Player Characters) in games behave more intelligently and realistically, and even surpassing humans in complex games such as Go and Atari games.
• Robot Control: Helping robots learn to navigate, grasp objects, and complete tasks in complex environments without pre-programming.
• Recommendation Systems: Intelligently recommending products, movies, music, or news based on user historical behavior and feedback, providing personalized experiences.
• Traffic Signal Control: Relieving urban traffic congestion by optimizing traffic light timing.
• Healthcare: Showing potential in treatment plan optimization, personalized medication dosage, chronic disease management, and clinical decision support systems.
• Education Sector: Providing students with personalized learning paths, adaptive learning platforms, and intelligent tutoring systems to improve learning efficiency and effectiveness.
• Financial Sector: Optimizing trading strategies, conducting customer relationship management, and adapting to dynamic financial markets.
• Energy Management: Optimizing power system scheduling and improving energy utilization efficiency, such as building energy management systems.

Summary

As a cornerstone algorithm of reinforcement learning, Q-learning provides a powerful “trial-and-error learning” framework for artificial intelligence. By building and updating an “Action Secret Manual” (Q-table), it allows the agent to gradually learn how to make optimal decisions in various situations without knowing the environment model in advance, thereby maximizing long-term rewards. Although Q-learning faces challenges when facing large state spaces, through combination with deep learning, it has evolved into more powerful algorithms such as DQN, greatly expanding its application boundaries and playing an increasingly important role in many fields such as games, robotics, healthcare, and finance. With the continuous development of artificial intelligence technology, Q-learning and its derived family will continue to serve as the core “brain” of intelligent systems, helping us build a smarter and more efficient future.

REINFORCE

Understanding AI in Depth but Simply: REINFORCE, Teaching You to Optimize Decisions Like a Life Mentor

The wave of Artificial Intelligence is sweeping the globe, and “Reinforcement Learning” is particularly drawing attention. Unlike supervised learning which relies on a large amount of labeled data, or unsupervised learning which looks for internal structures in data, it learns through “trial and error”. Among the many algorithms in reinforcement learning, there is a classic and important cornerstone—REINFORCE. Although its name sounds professional, its core idea is as simple and powerful as our daily learning methods.

What is Reinforcement Learning? (A Simple Review)

Imagine you are teaching a puppy to fetch a ball. You don’t tell it how to walk, how to open its mouth, or how to hold the ball in every step. Instead, you wait for it to make a move—for example, if it touches the ball, you give it a treat (reward); if it runs away, you don’t (punishment). Through constant attempts, the puppy slowly learns what actions bring rewards and what don’t. This is the core of reinforcement learning: An Agent takes Action in an Environment, receives a Reward, and then adjusts its Policy to finally learn how to maximize the total reward.

Enter REINFORCE: A “Big Picture” Life Mentor

In the world of reinforcement learning, the agent needs a “brain” to decide what to do in a given situation, and this “brain” is its Policy. A policy can be understood as a set of codes of conduct, an action guide, or a “habit” of taking action when faced with different scenarios. It is usually represented by a parameterized probability distribution, for example, implemented through a neural network, where the input is the current state and the output is the probability of each possible action.

Traditional reinforcement learning methods, such as Value-Based Methods, try to evaluate the “goodness” or “badness” of each action—that is, their value, and then choose the action with the highest value. This is like ordering in a restaurant, looking at which dish has the highest rating, and then ordering that dish.

REINFORCE is different. It belongs to a type of Policy Gradient method. As the name suggests, it does not directly evaluate the value of each action, but directly optimizes the “policy” itself. It is like a life mentor, not entangling in the right or wrong of a specific decision you made, but reviewing the total result after you finish a whole thing (a “life segment” or “episode”), and then telling you: “Did your ‘habit’ (policy) make the result of this thing good or bad? If it was good, adjust a little more in this direction next time; if it was bad, adjust a little less in this direction next time.”

Analogy: Learning to Ride a Bicycle

Imagine you are learning to ride a bicycle, which is the task your agent needs to solve.

• Agent: Yourself.
• Environment: Road, bicycle, wind, obstacles, etc.
• Action: Pedaling, holding handlebars, leaning body, etc.
• Reward: Riding a distance without falling (positive reward), falling (negative reward).
• Policy: The “set of rules” in your brain that makes movements based on the current situation (e.g., the bike is tilting, need to turn). At first, it might be clumsy and random attempts.

When you first try to ride a bike, you might fall many times. The REINFORCE algorithm won’t immediately say “Wrong!” every time your bike tilts a little to the left. Instead, it lets you complete the entire “riding attempt” (an “Episode”)—for example, from the starting point to where you fall.

If the result of this episode is: you rode 10 meters and fell, then the performance under this “policy” is low. REINFORCE will review all the actions you took in this 10 meters (and the states when these actions occurred), and then say: “It seems that your ‘habit’ (policy) along the way didn’t work well overall, and you need to adjust it properly next time.” It will give “negative reinforcement” to all the actions taken in the process that “might have led to failure” based on your failure experience, reducing the probability of their reoccurrence.

Conversely, if you successfully rode 100 meters without falling, and even successfully turned a corner, then the performance under this “policy” is high. REINFORCE will review all the actions you took in the process, and then say: “Your ‘habit’ (policy) was great overall this time! Next time you encounter a similar situation, tend to do these things more.” It will give “positive reinforcement” to all actions that “might have led to success”, increasing the probability of their reoccurrence.

The core of REINFORCE is: It waits for a complete “episode” to end, and adjusts the “probability” of performing all previous actions based on the total reward of this episode. The probability of executing good action combinations will increase; the probability of executing bad action combinations will decrease. The REINFORCE algorithm calculates the gradient of policy parameters directly through the trajectory data obtained by sampling, and then updates the current policy to move closer to the goal of maximizing the expected return of the policy.

How REINFORCE Works (Simplified Version)

Technically, REINFORCE updates policy parameters by calculating the Policy Gradient.

1. Policy Construction: Usually a neural network, inputting the current state of the environment and outputting the probability of each possible action.
2. Run Episode: The agent performs a series of actions in the environment according to the current policy until a terminal state is reached (e.g., task completion or failure).
3. Calculate Total Return: Record the reward obtained at each step in this episode and calculate a total return (usually considering the decay of future rewards, i.e., discounted cumulative reward). This total return is the “score” that measures how well the current “policy” performed in the current episode.
4. Update Policy: REINFORCE utilizes the actions, states of each step recorded previously, and the total reward of the entire episode to calculate a “gradient”. This gradient indicates the direction in which the “policy parameters” should be adjusted to obtain higher total rewards in the future.
   • If the total reward is high, then all actions executed in this episode will be considered “good” attempts, and their probabilities will be increased in the policy.
   • If the total reward is low, then all actions executed in this episode will be considered “bad” attempts, and their probabilities will be decreased in the policy.

This process is like a teacher grading a complex exam paper. Instead of grading the right or wrong of each small question, they look at your final total score. If the total score is high, you are encouraged to maintain and strengthen your learning method; if the total score is low, you are asked to reflect and adjust your learning method.

Pros and Cons of REINFORCE

Pros:

• Simple, Intuitive, Easy to Implement: The concept is relatively easy to understand, it is the basis of policy gradient methods, and the structure is relatively simple.
• Direct Policy Optimization: REINFORCE directly optimizes the policy without estimating the value function, avoiding bias and variance issues in value function estimation.
• Suitable for Stochastic Policies: Naturally suitable for stochastic policies, able to introduce exploration mechanisms to help agents discover better action paths.
• Suitable for Continuous Action Spaces: Can directly output the probability distribution of actions, very suitable for scenarios where actions are not discrete choices but continuous values (such as the force and direction of a joystick).

Cons:

• High Variance: This is the biggest pain point of REINFORCE. Because it uses the total reward of the episode to update the policy of each step, if the overall reward of an episode is high, but a certain action in it is actually terrible, it will also be wrongly “encouraged”; and vice versa. This leads to an unstable learning process, like when riding a bicycle, sometimes although you fell, a certain auxiliary action of yours was actually correct, but because the overall outcome was bad, it might also be “punished”.
• Slow Convergence Speed: Due to high variance, REINFORCE often requires a large number of training episodes to converge to a good policy, and the efficiency is low.
• Low Sample Efficiency: It is a Monte Carlo method, which must wait until the entire episode ends to perform an update, resulting in low sample efficiency.

Improvements and Latest Progress of REINFORCE

Due to the high variance and low efficiency of REINFORCE, researchers have developed many more advanced and stable policy gradient algorithms on this basis, which can be seen as the evolution and optimization of the REINFORCE idea.

1. REINFORCE with Baseline:
   To solve the high variance problem, researchers introduced a “Baseline”. When calculating the gradient, a baseline value is subtracted from the total reward. This baseline value is usually an estimate of the state value function (i.e., the expected average reward that can be obtained in the current state).
   This is like when a teacher grades an exam paper, they no longer just look at your total score, but give you a “passing line” or “average score”. If your performance exceeds the baseline, you can get some positive adjustments even if the total score is not high; if it is lower than the baseline, negative adjustments are made. The introduction of baseline can significantly reduce the variance of gradient estimation, improve learning stability, and introduce no bias.

2. Actor-Critic Methods:
   This is an important milestone in the field of reinforcement learning, combining policy gradient (REINFORCE is a member) and value function estimation.

   • Actor: Responsible for learning and updating the policy, deciding what action to take in a given state (i.e., executing the core logic of REINFORCE).
   • Critic: Responsible for learning a value function to evaluate the quality of the action taken by the Actor. The Critic’s evaluation can replace the part of the episode total reward in REINFORCE, providing the Actor with more timely and lower variance feedback.
     Unlike REINFORCE which needs to wait for a complete episode to end before updating, the Actor-Critic algorithm can update after each step, which greatly improves sample efficiency and convergence speed. It is more stable during training than REINFORCE, and the jitter situation is significantly improved.

3. A2C (Advantage Actor-Critic) and PPO (Proximal Policy Optimization):
   These are currently very popular and efficient deep reinforcement learning algorithms, which are further developments of the Actor-Critic idea.

   • A2C (Advantage Actor-Critic) is a synchronous and deterministic version of the Actor-Critic algorithm. By introducing the “Advantage Function” for further optimization, the Critic not only evaluates the value but also measures whether an action is “better” or “worse” compared to the average level, thereby guiding the Actor to update the policy more effectively. A2C simplifies the complexity of asynchronous operations whilst maintaining the dual advantages of policy and value function learning.
   • PPO (Proximal Policy Optimization) is one of the most advanced and widely used policy gradient algorithms. It introduces a “Clipping” mechanism on the basis of A2C to limit the magnitude of policy updates, ensuring update stability and avoiding excessive policy changes, thus achieving a good balance between learning efficiency and stability.
     Recent research even points out that A2C can be considered a special case of PPO under certain conditions. These algorithms are widely used in complex fields such as robot control, game AI (such as OpenAI Five, AlphaStar, etc.), and autonomous driving, and have achieved remarkable achievements.

Conclusion

As the core cornerstone of policy gradient methods in reinforcement learning, REINFORCE’s “life mentor”-like learning philosophy—adjusting strategies by reviewing overall performance—has paved the way for subsequent more complex and efficient algorithms. Although it has disadvantages such as high variance and slow convergence, through the introduction of baselines, development into Actor-Critic, and then the evolution of advanced algorithms such as PPO, policy gradient methods have become indispensable tools for solving high-dimensional complex decision-making problems. Understanding REINFORCE is like understanding an evolutionary history of an agent from ignorant attempts to shrewd decision-making.

Precision-Recall Curve

In the vast world of AI, we often need to evaluate how well a model performs. For example, if we trained an AI to recognize cats, and it can tell me if there is a cat in a picture. So, how is this AI performing? Simple “accuracy” may not tell us the whole truth. At this time, we need some more refined tools to give the AI a “physical examination”, and the Precision-Recall Curve is one of the very important “medical reports”.

Why can’t we just look at “Accuracy”?

In daily life, we often say that high “accuracy” means doing well. For example, if an AI’s accuracy in recognizing cats reaches 99%, it sounds impressive, right? However, if this AI faces 10,000 pictures, and only 100 of them are cats, and it judges all pictures as “not cats”, then its “accuracy” is still as high as 99% (because it correctly judged 9900 non-cat pictures), but this is obviously a useless AI! It didn’t find a single cat.

This is the problem caused by Imbalanced Data. In many practical applications, the category of things we care about (such as diseases, fraudulent transactions, spam emails, etc.) is often the minority. Simply pursuing high accuracy may make the AI “turn a blind eye” to the minority we really want to find.

To better evaluate the performance of AI in dealing with such problems, we need to introduce two more professional concepts: Precision and Recall.

Precision: Quality Over Quantity, Don’t “Cry Wolf”

Imagine you are a “Spam Email Identification AI Assistant”. Your task is to find spam emails.

• Precision focuses on: Among the emails you identified as “spam”, what proportion are really spam?

If your precision is high, it means you rarely misjudge important work emails as spam. You are cautious in your “actions”; once you say it is spam, it is likely to be true. To use a common saying, it is “Better to lack than to have low quality“ (Ning Que Wu Lan), or “Don’t cry wolf easily“.

Recall: Leave No One Behind, Don’t Let “Fish Slip Through the Net”

Also as a “Spam Email Identification AI Assistant”, besides “not accidentally injuring”, you also have to “not let go”.

• Recall focuses on: Among all real spam emails, what proportion did you successfully identify?

If your recall is high, it means you can catch almost all spam emails and prevent them from entering your inbox. You guard the “pass” very strictly and won’t let too many fish slip through the net. To use a common saying, it is “Not one less“, or “Don’t let the wolf run away“.

Precision and Recall: You Can’t Have Your Cake and Eat It Too

Often, Precision and Recall are like two ends of a scale; it is difficult to make them both reach the maximum at the same time.

• If you want to increase Recall (block all potential spam emails), you may relax the criteria, which may result in accidentally injuring some normal emails (Precision decreases).
• If you want to increase Precision (ensure that every email judged as spam is really spam), you may tighten the criteria, which may result in letting go of some real spam emails (Recall decreases).

For example, in medical diagnosis, if an AI is to diagnose a rare disease:

• High Precision means doctors believe that patients diagnosed as “ill” by the AI are indeed ill, avoiding unnecessary panic and further checks.
• High Recall means the AI can discover the vast majority of sick patients, avoiding missed diagnoses and delayed treatment.

Different application scenarios have different preferences for Precision and Recall. For example, for spam emails, we would rather block a few more than receive too much junk (High Recall is more important, a little misjudgment is acceptable); while for terminal illness diagnosis, we would rather do more checks (misdiagnosis, lower Precision), than miss a real patient (High Recall is very important).

Precision-Recall Curve: A “Comprehensive Physical Examination Report” for AI Models

So, how can we see Precision and Recall, and their trade-off relationship, in one graph? This is where the Precision-Recall Curve comes into play.

Imagine that when our AI model decides whether an email is spam, it actually gives a score of “probability of being spam” (for example, between 0 and 1). We can set a Threshold:

• If the probability score is higher than this threshold, the AI judges it as spam.
• If the probability score is lower than this threshold, the AI judges it is not spam.

By changing this threshold, we get different combinations of Precision and Recall:

• Threshold set very high: The AI will be very cautious, and only those “certain” spam emails will be identified. At this time, Precision will be high (judgments are accurate), but Recall may be very low (missing a lot).
• Threshold set very low: The AI will be very lenient, considering it spam as long as there is a little suspicion. At this time, Recall will be high (almost all spam is blocked), but Precision may be very low (accidentally injuring many normal emails).

Taking Recall obtained under these different thresholds as the horizontal axis and Precision as the vertical axis, connecting all the points gives the Precision-Recall Curve.

The shape of this curve can tell us a lot of information:

• The closer the curve is to the upper right corner of the graph, the better the model’s performance. This means that at the same Recall, the model can maintain higher Precision; or at the same Precision, the model can achieve higher Recall.
• If one model’s PR curve completely “encloses” another model’s curve, then the former’s performance is better than the latter.
• The area under the curve (Called Average Precision, AP) can also be used to measure the overall performance of the model; the larger the area, the better the model performance.

Summary

The Precision-Recall Curve is not just a professional term in the AI field; it is more like a detailed and practical “AI physical examination report”. It reveals the performance and trade-off of AI models in the two important dimensions of “finding accurately” (Precision) and “finding completely” (Recall), especially when dealing with those “minority” data. It allows us to understand the value of AI more comprehensively and accurately. For non-professionals, remembering the two intuitive metaphors of “Quality Over Quantity” and “Not One Less” will help to well understand the core meaning of Precision and Recall.

Proximal Gradient Descent

The “Gold Medal Coach” in Model Optimization: An In-depth but Accessible Understanding of Proximal Gradient Descent

In the vast field of Artificial Intelligence, whether it’s training an image model to recognize cats and dogs, or a complex system to predict stock trends, the core cannot be separated from a basic task: Optimization. Simply put, optimization is finding a set of optimal parameters to make our model perform as well as possible with the lowest possible error rate. It’s like a mountaineer looking for the best path to the summit, or a chef tweaking a recipe to make the most delicious dish.

And Gradient Descent is like a “mountain guide” in the AI world, guiding the model parameters step by step towards the optimal solution. But this guide sometimes encounters some “special terrains”—this is when Proximal Gradient Descent (PGD), which we will discuss in depth today, shows its prowess.

1. Gradient Descent: “Rolling Stone Downhill” in the AI World

Imagine you are standing somewhere on a high mountain, and your goal is to find the lowest point of the valley. If you close your eyes and can only perceive the slope of the ground beneath your feet, the most natural thing to do is to take a step in the direction of the steepest descent. Walking step by step like this, you will eventually reach the lowest point of the valley.

In an AI model, this “mountain” is the Loss Function, which measures the “degree of error” of the model’s prediction; the “lowest point of the valley” is where the model performs best; and the direction and size of your parameter adjustment each time you “take a step” are determined by the Gradient. The gradient is like telling you the direction of the steepest slope currently. This is the basic principle of Gradient Descent: constantly adjusting parameters in the direction of gradient descent until the loss function reaches a minimum value.

The reason Gradient Descent is so powerful is that it can handle the vast majority of “smooth” loss functions, just like dealing with a smooth surfaced mountain.

2. When the “Mountain Road” Becomes Rugged: The Dilemma of Standard Gradient Descent

However, the world of AI is always full of challenges. Sometimes, we want the model not only to predict accurately but also to have some extra “good qualities”, such as:

• Simplicity/Sparsity: We want the model to focus only on the most important features and ignore irrelevant minor features, so that the model can be “slimmer”, easier to understand, and less prone to overfitting. It’s like cooking, we only choose a few key ingredients instead of adding everything in. Mathematically, this usually corresponds to introducing an L1 regularization term in the loss function, which encourages many model parameters to become zero.
• Constraints: Sometimes model parameters must satisfy specific restrictions, such as age cannot be negative, or total budget cannot exceed a certain limit.
• Adversarial Robustness: We hope the model can withstand subtle “attacks” (such as adding tiny noise invisible to the naked eye in pictures) and still make correct judgments.

These “good qualities” often cause the loss function to become “rugged”, that is, mathematically becoming non-differentiable, or requiring finding the optimal solution within a constraint region.

When the mountain road suddenly presents a sharp cliff, a deep ravine, or you are required to find the lowest point only on a narrow “trail”, the ordinary “rolling stone downhill” strategy fails. You don’t know what the gradient is at the edge of the cliff, nor do you know how to stay on the narrow trail.

3. The Wisdom of “Proximal”: Introducing the “Gold Medal Coach”

This is the moment when Proximal Gradient Descent (PGD) comes on stage. The word “Proximal” in PGD implies “nearest” or “neighboring”. Its core idea is: decompose a complex problem into two steps, each of which is relatively easy to solve.

We can imagine PGD as a “Gold Medal Mountaineering Coach”:

1. Free Exploration (Gradient Descent Step): The coach first lets you “roll the stone downhill” freely according to the current slope as usual, finding a new position that you think minimizes the loss. This step only temporarily ignores those “special terrains” or “rules”.

   • “Hey, ignore those troublesome rules for now, just take a step in the steepest downhill direction based on the slope under your feet!”

2. Forced Calibration (Proximal Operator Step): After reaching the new position, the coach will immediately intervene and “pull” you back to the “nearest” point that complies with all “special terrains” or “rules”.

   • “Stop! You went too far just now, or fell into a ditch! According to our preset rules, like you must walk on the paved path, or you must jump over that cliff, I’ll help you adjust to the point closest to your current position that complies with the rules.”

This “pulling back” operation is mathematically called the Proximal Operator. It calculates the point “closest” to your current position under conditions that satisfy specific constraints or penalties (such as sparsity, within certain sets).

For example, if after free exploration, you get a parameter value of 0.3, but the rule requires the parameter to be 0 or 1 (for sparsity), the Proximal Operator will automatically help you “pull” it to either 0 or 1 (usually closing to 0 becomes 0, closing to 1 becomes 1, depending on the specific threshold).

So, every step of Proximal Gradient Descent is:
First “let” gradient descent explore freely, then “correct” and “calibrate” with the Proximal Operator.

These two steps alternate, allowing PGD to handle those non-smooth terms or constraints that are very tricky for standard gradient descent gracefully.

4. Applications and Future of Proximal Gradient Descent

Due to its powerful capabilities, PGD plays an indispensable role in many AI fields:

• Sparse Models: In machine learning, we often use techniques like Lasso regression to encourage models to produce sparse weights, i.e., leaving only a few most important features. PGD is one of the core algorithms to solve such problems, helping models find concise and effective solutions.
• Image Processing and Compressed Sensing: In image denoising, image restoration, and compressed sensing fields that need to reconstruct signals from a small amount of data, PGD can effectively handle problems that impose constraints on image structure (such as Total Variation regularization) to reconstruct high-quality images and signals.
• Adversarial Robustness Training: In deep learning, the PGD algorithm is widely used to generate adversarial examples and enhance model robustness through adversarial training. By applying tiny, carefully designed perturbations to input data (this is what the “proximal” step of PGD does) to deceive the model, it finds the model’s vulnerabilities and improves them.
• Online Optimization and Reinforcement Learning: With the increasing demand for real-time data processing, the online version of PGD also provides new ideas for model optimization in dynamic environments.

In recent years, PGD has shown great potential in processing large-scale, high-dimensional data and combining with deep learning models. For example, it is applied to optimize deep neural networks with non-smooth regularization terms to achieve model pruning and sparsification, improving model efficiency.

In summary, Proximal Gradient Descent is like an “all-around gold medal coach” in the world of AI optimization. It not only knows how to move forward along smooth slopes but also knows how to skillfully guide the model to find the best path in rugged, rule-complex “special terrains”. Its elegance and robustness make it a key weapon for solving modern AI challenges.

Research on Sparsification of Deep Learning Models Based on Proximal Gradient Descent. (2023). Hebei University.
Iterative Shrinkage-Thresholding Algorithm. (2024). Wikipedia.
Towards the Robustness of Adversarial Examples in Deep Learning. (2018). arXiv.

QLoRA

The emergence of Large Language Models (LLMs) is like opening a door to a new world of artificial intelligence. They can write poetry, code, and converse, doing almost anything. However, these models often possess hundreds of billions or even trillions of parameters, bringing a huge “sweet burden”: the computational resources and memory required to train and fine-tune them are often only affordable by a few tech giants, and are out of reach for ordinary developers.

To give more people the opportunity to customize and control these powerful “digital brains”, the AI community has been exploring more efficient fine-tuning methods. Among them, QLoRA technology is undoubtedly a shining new star. It is like a clever “magician” that significantly lowers the threshold for fine-tuning large models without sacrificing much performance.

1. From “Encyclopedia” to “Loose-leaf Notes”: Understanding LoRA

Imagine that a large language model is like a vast, all-encompassing “Encyclopedia”—it systematically records almost all human knowledge. This “Encyclopedia” is huge and heavy, and every word and punctuation mark in it corresponds to a parameter in the model, collectively determining its knowledge reserve and inference ability.

When we need to adapt this “Encyclopedia” to a specific field, such as turning it into a specialized “Medical Encyclopedia” or “History Encyclopedia”, we face two choices:

• Full Fine-tuning: This is like completely revising the entire “Encyclopedia”. We need to rewrite it word by word to ensure that all relevant content meets the professional requirements of the new field. This work is hugely expensive, time-consuming, and requires massive amounts of paper (computing resources) and ink (memory). For a giant encyclopedia with dozens or even hundreds of volumes, this is almost an impossible task.
• LoRA (Low-Rank Adaptation): LoRA takes a smarter, more economical approach. It no longer directly modifies the “main text” of the “Encyclopedia”, but is like adding a large number of “loose-leaf notes” or “annotations” to it. These “loose-leaf notes” only supplement, correct, or emphasize a specific topic. For example, adding the latest research results next to a medical entry, or attaching new interpretations next to a historical event.

Specifically, LoRA’s working principle is: It freezes most of the parameters of the original model (just like freezing the main text of the “Encyclopedia”), and only adds a very small amount of extra, learnable “adapter” parameters to the model. These adapters are like those “loose-leaf notes”; they are small, very fast to train, and occupy much less resources. During training, we only update the content on these “loose-leaf notes” without touching the original “Encyclopedia”. When the model needs to generate a response in a specific field, these “loose-leaf notes” will come into play, guiding the model to give an answer that better meets the needs.

This method greatly reduces the number of parameters that need to be trained, thereby significantly reducing memory and computing requirements, and also shortening training time.

2. The Wisdom of “Compression”: The “Q” in QLoRA — Quantization

LoRA is clever enough, but QLoRA goes a step further by introducing the “magic” of “Quantization”. The “Q” here stands for “Quantized”.

What is quantization? We can use examples from daily life to understand:

• Photo Compression: A high-definition photo on your phone may have tens of millions of pixels and take up more than ten megabytes of space. But if you just share it on social media or view it on a small screen, you will usually compress it into a picture with hundreds of thousands of pixels and a few hundred KB in size. Although some details are lost, it is almost invisible to the naked eye, but it greatly saves storage space and transmission bandwidth.
• Income and Expenditure Records: You can record every income and expenditure accurately to two decimal places, such as 23.45 yuan, 1.78 yuan. But if you just want to quickly understand the total expenses for this month, you may only roughly record them as 23 yuan, 2 yuan, which is easier to calculate and remember, and does not affect your judgment of the overall situation.

The principle of “quantization” in AI models is similar. Model parameters are usually stored and calculated in the form of 32-bit floating-point numbers (Float32), which is like an extremely precise recording method. Quantization is converting these high-precision parameters (such as 32-bit or 16-bit floating-point numbers) into lower-precision representations, such as 8-bit or even 4-bit integers. This conversion greatly reduces the memory space occupied by the model and the resources required for calculation.

What is the brilliance of QLoRA’s quantization?
QLoRA adopts more advanced technology in quantization. It introduces a data type called 4-bit NormalFloat (NF4). This specially designed 4-bit quantization method is optimized for the normal distribution characteristics common to parameters in models, which means it can maximize the retention of the model’s original performance while significantly compressing data, reducing precision loss. In addition, it also adopts a “Double Quantization” mechanism, which quantizes the quantization constants again, further squeezing memory space.

3. Strong Combination: The Birth of QLoRA

The essence of QLoRA lies in ingeniously combining LoRA’s “loose-leaf notes” strategy with advanced “data compression” technology (quantization). This means:

1. First Compress the “Encyclopedia”: QLoRA first “compresses” the huge original large language model (“Encyclopedia”) into a 4-bit low-precision version. This greatly reduces the memory footprint of the entire model, just like condensing dozens of volumes of encyclopedia into a few thin booklets, which is no problem even if placed on an ordinary bookshelf (consumer-grade graphics card).
2. Then Add “Loose-leaf Notes”: On the basis of this already highly compressed model, QLoRA then applies LoRA technology to add a small amount of trainable “loose-leaf note” adapters. These adapters are still trained and updated with higher precision (usually BF16 or FP16) because they are the key to learning new knowledge.

Through this dual optimization of “Base Compression + Precise Supplement”, QLoRA can achieve amazing results:

• Memory Miracle: For example, a giant model with 65 billion parameters, with the support of QLoRA, can be fine-tuned on a GPU with only 48GB of video memory. You know, traditional 16-bit full-precision fine-tuning may require more than 780GB of video memory!
• Performance Guarantee: Although the model is compressed to 4 bits, QLoRA still achieves performance very close to 16-bit full-precision fine-tuning or LoRA fine-tuning on many benchmarks and tasks. Even in some cases, its optimized result model (such as Guanaco) can reach 99.3% of ChatGPT’s performance level in the Vicuna benchmark.
• Engineering Wisdom of “Paged Optimizers”: In addition to these core technologies, QLoRA also introduces “Paged Optimizers”. It’s like when a computer operating system runs out of memory, it temporarily stores infrequently used data on the hard disk (CPU memory) and quickly recalls it when needed (GPU memory). This mechanism ensures that even if the GPU video memory occasionally peaks, model training can proceed stably, avoiding Out of Memory (OOM) errors.

4. The Transformative Significance Brought by QLoRA

The emergence of QLoRA is undoubtedly an important milestone in the field of AI, bringing profound significance:

• True “Inclusive AI”: In the past, fine-tuning large models was an “exclusive game” for a few top laboratories and large enterprises. Today, QLoRA allows individual developers, researchers, and even small teams to use ordinary consumer-grade GPUs (such as RTX 3060 with 12GB video memory) to perform efficient large model fine-tuning. This greatly lowers the threshold and allows more innovations to emerge.
• Accelerating Innovation Ecosystem: Lower barriers mean faster iteration speeds and richer application scenarios. People can more easily customize efficient and practical exclusive large models for specific tasks, specific languages, or specific data.
• Balance between High Performance and High Efficiency: While significantly cutting resource requirements, QLoRA can still maintain excellent model performance, finding an excellent balance between performance and efficiency. It avoids the dilemma of “you can’t have your cake and eat it too”.
• Broad Application Prospects: QLoRA has shown huge application potential in many fields such as natural language generation, question answering systems, and personalized recommendations, improving the quality and efficiency of models in these tasks.

5. Summary and Outlook

QLoRA technology is like a bridge connecting the huge potential of AI large models with the limited resources of ordinary developers. Through ingenious quantization and low-rank adaptation technologies, it has turned the originally unattainable fine-tuning of AI large models into a civilian operation.

In the future, we expect QLoRA and its variant technologies to continue to develop, further optimizing the efficiency of compression and fine-tuning, so that the ability of AI large models can be integrated into every aspect of our lives like breathing, truly realizing the popularization and democratization of artificial intelligence.

Policy Gradient

The field of AI is full of mysterious terms, and “Policy Gradient” is a core concept in Reinforcement Learning that might seem abstract to non-professionals. However, understanding it does not require advanced mathematics; we can unveil it through some daily life metaphors.

What is Policy Gradient? — How to Teach AI to “Make Decisions”

Imagine you are teaching a child to ride a bicycle. The child needs to learn how to balance, pedal, and steer. You wouldn’t directly tell them “tilt your center of gravity 3 degrees to the left within 0.5 seconds”, but encourage them to try more. If they fall, tell them what to do differently next time; if they do well, praise them. Policy Gradient is such a method of “teaching” Artificial Intelligence (AI) to make decisions.

In AI, the core goal of Reinforcement Learning is for an AI to learn how to act in an environment to maximize rewards. And strictly speaking, a “Policy” is a set of “codes of conduct” or “decision schemes” in the AI’s brain, telling the AI what action to take in a specific situation. For example, in autonomous driving, the policy might be to choose “brake” when seeing a red light, or “swerve left” when there is an obstacle ahead.

The core philosophy of Policy Gradient is: Directly optimize this “decision scheme”. Unlike other methods that first evaluate the goodness (value) of each action, it directly adjusts the decision scheme itself, making actions that bring more rewards more likely to be chosen.

Vivid Metaphors: Master Chef and “Trial-and-Error Learning”

Metaphor 1: A Chef Learning to Cook

Suppose you are learning a brand-new dish without a recipe.

• Policy: This is the “cooking plan” for this dish in your mind—salt first or sugar first? High heat or low heat? How long to stir-fry? Your policy might be completely random at first, or based on some vague experience.
• Action: You actually operate according to the “cooking plan” in your head, such as adding 5 grams of salt and using medium heat.
• State: This is the current condition of the dish, such as color, smell, and at what stage of cooking it is.
• Reward: After the dish is done, the taster’s feedback is the reward. If they say “Delicious!”, that’s a big positive reward; if they say “Too salty!”, that’s a negative reward.

The learning process of Policy Gradient is like this:

1. Trying and Exploring: You try to cook according to the current “cooking plan” in your mind (AI performs a series of actions based on the current policy).
2. Getting Feedback: After the dish is done, you get feedback from the taster (AI gets rewards from the environment).
3. Summarizing and Adjusting: If a certain step led to “too salty”, you will slightly reduce the amount of salt next time; if a certain ingredient made the dish “delicious”, you will consider adding more next time. The direction of “slightly reducing” or “adding more” is the “gradient”.
4. Repeated Practice: You constantly cook, taste, and adjust until you master the best “cooking plan” and become a master chef.

Metaphor 2: Climbing Blindfolded to Find the Peak

Imagine you are blindfolded and placed on a hillside with the goal of finding the highest peak.

• Your Position: Like the AI’s “policy parameters”, it determines how decisions are made.
• Height of the Mountain: This is the “total reward” the AI gets, which you want to maximize.
• Policy Gradient: It tells you which direction to take a step in to climb higher faster. You can’t jump to the top all at once, but each time you can choose the direction with the steepest slope to take a small step forward.

Through repeated “trials” (generating a series of actions), the AI will discover which action sequences bring high rewards, and then Policy Gradient will tell it how to fine-tune its internal decision mechanism (policy parameters) so that it is more likely to take these high-reward actions in the future.

Core Elements of Policy Gradient

• Policy: Usually a function or neural network that inputs the current environmental state and outputs the probability distribution of taking various actions in that state. For example, in autonomous driving, input the current road condition image and output the probability of each action such as turn left, turn right, accelerate, brake, etc.
• Trajectory: The process of an AI performing a series of actions and experiencing a series of states from beginning to end. This is like the complete process of you cooking a dish, from preparation to serving.
• Reward: The environment’s feedback on the AI’s actions, which can be immediate rewards or cumulative rewards of the final result.
• Gradient: In mathematics, the gradient represents the direction in which a function increases fastest. In Policy Gradient, it indicates how we should adjust the parameters of the policy to maximize the expected reward obtained by the AI.

Mechanism: Monte Carlo and Parameter Update

Since we cannot exhaust all possible action combinations to calculate the optimal policy, Policy Gradient algorithms usually use the Monte Carlo method to estimate the policy gradient. This means that the AI will interact with the environment multiple times, generate multiple “trajectories”, and then estimate and update the policy based on the average reward of these trajectories.

Every time the policy parameters are updated, the Policy Gradient algorithm makes minor adjustments to the policy according to the gradient direction, ensuring that the adjusted policy increases the probability of high-reward actions and decreases the probability of low-reward actions.

Pros and Cons of Policy Gradient

Pros:

• Direct Policy Optimization: No need to calculate the value of each action first; optimal behavior patterns can be learned directly.
• Suitable for Continuous Action Spaces: Performs well in scenarios requiring fine movements such as robot control, where AI can choose any minute action intensity rather than just choosing from a few discrete options.
• Able to Learn Stochastic Policies: Allows AI to explore and discover new, possibly better behaviors, rather than always following a preset best path.

Cons:

• Slow Convergence Speed: Because only minor adjustments are made each time, it may take many attempts to find the best policy.
• High Variance: The randomness of each attempt result is large, which may lead to an unstable learning process.

Latest Progress and Application

To solve the above challenges, researchers have proposed many improved algorithms for Policy Gradient, such as the famous REINFORCE (the most basic policy gradient algorithm, estimating gradients directly based on Monte Carlo sampling, suitable for stochastic policies), the Actor-Critic family (combining policy gradient and value function estimation, where “Actor” is responsible for decision-making and “Critic” is responsible for evaluating the quality of decisions), TRPO (Trust Region Policy Optimization), and PPO (Proximal Policy Optimization). These algorithms improve learning efficiency and stability while maintaining the advantages of Policy Gradient.

Policy Gradient methods have broad applications in Game AI (such as Atari games), Robot Control (such as teaching robots to walk and grasp objects), and Autonomous Driving. For example, AI can be trained to choose the best action based on the screen in a game to get a high score. In the field of unmanned vehicles, Policy Gradient can help vehicles learn decision-making under complex road conditions, such as how to overtake safely and change lanes.

In short, Policy Gradient is like a patient coach who, through constant trials, feedback, and adjustments, directly teaches AI how to make better and better decisions, making it “smarter” and more “adaptive” in complex environments.

Phi

“Small but Strong” in the AI Field: Revealing Microsoft’s Phi Model Family

In the vast universe of Artificial Intelligence (AI), we often hear the term “Large Models,” which are like powerful supercomputers processing massive amounts of information. However, a group of “small and sophisticated” stars have also emerged in the AI world. With their compact size and excellent performance, they are changing the application landscape of AI. Today, we are going to explore in simple terms such a representative model—Microsoft’s Phi Model Family.

What is the Phi Model? “A Smart Assistant in Your Pocket”

Imagine having an extremely smart personal assistant who can not only understand you but also help you write reports and solve problems. If this assistant were a clumsy supercomputer that required running to a central computer room every time you used it, it would certainly not be very convenient. But what if he is a “mini brain” that can fit in your pocket, respond at any time, and even work without an internet connection?

Microsoft’s Phi model family is the “smart assistant in your pocket” in the AI field. It is a series of Small Language Models (SLMs) developed by Microsoft that emphasize “small but powerful.” Unlike Large Language Models (LLMs) with hundreds of billions or trillions of parameters (which can be understood as units of “knowledge points” or “processing power”), Phi models have much smaller parameter sizes. For example, the latest Phi-3 Mini has only 380 million parameters, and Phi-4 has only 14 billion parameters. But these “little guys” have shown surprising wisdom.

Why is “Small and Sophisticated” So Important? — The Advantage of “Pocket Computers”

Why are AI models more worthy of attention the smaller they are? We can use an analogy to understand:

1. Universality of “Mini Hosts”: In the past, computer processing of complex tasks required large servers. Now, your smartphone or laptop can do many things that only large computers could do in the past. The Phi model is like a “mini host” in the AI field. It can run on various “edge devices,” such as your mobile phone, tablet, smart watch, and even IoT devices in offline environments. This means that AI no longer relies on powerful cloud connections but can be closer to our daily lives and provide intelligent services anytime, anywhere.
2. Efficiency of “Specialized Expertise”: Large language models are often “all-around players” capable of handling various open-ended tasks. But like a jack-of-all-trades employee, they may not be as good as a specialized expert in certain specific fields. The Phi model is more like a “specialized engineer” or “domain expert.” Trained on carefully selected “high-quality textbook-style” data, it focuses on learning and mastering specific skills, such as powerful language understanding, reasoning, mathematics calculation, and even programming capabilities. This allows it to perform specific tasks not only excellently but also much faster and with less computing resources and power consumption.
3. Democratization of “Affordability”: Training and running large models requires huge financial investment and energy consumption—just like building and maintaining a power station. Lightweight models like Phi greatly lower the threshold for AI. It has lower deployment costs and is more energy-efficient, allowing more developers and enterprises to afford it, thereby integrating advanced AI capabilities into various innovative applications and realizing the “democratization” of AI.

How Capable is the Phi Model? “Not Only Able to Speak, But Also Understand the World”

Despite being “small,” the capabilities of the Phi model family cannot be underestimated. Microsoft’s research shows that the latest Phi models outperform many larger models, even including GPT-3.5 Turbo, in multiple benchmark tests.

• Language Understanding and Generation: Like a knowledgeable but concise friend, the Phi model can accurately understand your intentions and provide answers or complete writing tasks in a concise and efficient manner.
• Logical Reasoning and Mathematics: For complex logical problems and mathematical calculations, the Phi model also demonstrates strong solving capabilities. The latest Phi-4 model performs particularly well on math tasks, even surpassing some large models.
• Programming Assistance: The Phi model is also a qualified “programming assistant” that can help developers write, understand, and debug code.
• Multimodal Capability — “AI That Can See Pictures”: It is worth mentioning that members with multimodal capabilities, such as Phi-3 Vision and Phi-4 Multimodal, have also appeared in the Phi family. This means they are no longer limited to processing text information but can “read” pictures, charts, and even perceive audio input like us. For example, you can show a chart to Phi-3 Vision, and it can not only recognize the text and data in the chart but also analyze and provide insights and suggestions. It’s like equipping your “pocket assistant” with a pair of “eyes” so he can understand the world more comprehensively.

The Future of Phi Models: AI Available to Everyone

Microsoft released the Phi model family as an open-source project, and developers can obtain and use them for free on platforms like Azure AI Studio and Hugging Face. This open strategy has greatly promoted innovation, allowing developers around the world to build their own AI applications based on Phi models.

With the development of AI technology, our demand for AI will become increasingly diverse. The emergence of the Phi model family indicates that AI will no longer be the “patent” of a few large technology companies but will become more inclusive and penetrate into every aspect of our lives. From smart assistants on mobile phones to automated quality inspection in factories to intelligent analysis in agriculture, these “small but strong” AI models are building a smarter, more efficient, and accessible future step by step.