NeRF

AI technology is evolving with each passing day. One of the concepts that has received much attention and is highly disruptive in recent years is “Neural Radiance Fields”, or NeRF for short. This technology is like opening a “magic door” for the digital world, allowing computers to reconstruct and render three-dimensional scenes with unprecedented realism.

What is NeRF? — Digital Magic That “Brings Photos to Life”

Imagine you take several photos of an object or scene from different angles with your mobile phone. Traditionally, these photos are just flat memories. But NeRF can use these seemingly ordinary two-dimensional photos, like having magic power, to “understand” every detail and every ray of light in this three-dimensional scene, and even predict what it would look like if you looked at it from any angle that has never been photographed. It doesn’t simply piece photos together, but truly “constructs” a three-dimensional world in the computer that you can explore freely.

Metaphor:
If traditional 3D modeling is like carving a realistic model, requiring superb skills and a lot of time to depict every face and every edge, then NeRF is more like using a few photos as “clues” and asking a clever “painter” (neural network) to “imagine” and “redraw” the entire three-dimensional space. This “painter” does not carve the model directly, but learns what color and transparency each point in space should have, and can finally generate a realistic picture according to your perspective.

How Does NeRF Achieve This “Magic”?

The core of NeRF lies in using neural networks to implicitly represent a three-dimensional scene. This sounds a bit abstract, let’s break it down:

1. Input: Multi-angle Photos and Camera Information
   What you provide to NeRF are multiple two-dimensional photos of the same scene taken from different positions and directions, as well as the position and orientation of the camera when each photo was taken (like knowing where you stood and which direction the lens was facing when you took the photo).

2. Core “Painter”: Neural Network Modeling “Radiance Field”
   The key to NeRF is using a special neural network (usually a Multi-Layer Perceptron, MLP) to simulate a “neural radiance field.” This “radiance field” is not a physical model, but more like an “encyclopedia” about this three-dimensional scene. For any point in space, and any observation direction, this “encyclopedia” can tell you what color of light will be emitted there (color), and how much light will pass through (transparency or density).

   • Like a Transparent Jelly Box: You can imagine the entire three-dimensional space as a huge transparent jelly box, where each “jelly particle” inside that is too small to distinguish has its own color and transparency. NeRF’s neural network learns how to describe the properties of these “jelly particles.”
   • Implicit Representation: This representation method is called “implicit” representation because it does not directly build traditional 3D mesh models or point clouds, but “remembers” the geometric shape and lighting information in the scene through the mathematical functions of the neural network.

3. Learning and Training: “Reading” 3D from Photos
   This neural network “painter” is blank at the beginning, and it needs to become smart through learning. The process of learning is to compare against the photos you input: it will “look at” this “transparent jelly box” from a virtual perspective like a human eye. Based on the color and transparency of the “jelly particles” inside, it calculates the color that this line of sight should see. Then, it compares this calculated color with the actual photo taken. If different, it adjusts the internal parameters of the neural network until it can accurately “reproduce” the appearance seen in all input photos. Through repeated training, the neural network “masters” the color and transparency distribution of the entire three-dimensional space.

4. Rendering and Generation: Creating Unseen Perspectives
   Once the neural network training is completed, it becomes a powerful “scene generator.” You can let it “look” at this scene from any new angle that has never been photographed, and it can instantly render a highly realistic image based on the learned “radiance field” information.

What are the Advantages of NeRF?

• Photo-realistic Quality: New perspective images generated by NeRF have extremely high realism and detail restoration capabilities, making virtual scenes look almost identical to real photos.
• No Need for Traditional 3D Modeling: It gets rid of the tedious manual modeling process in traditional 3D modeling, and can reconstruct three-dimensional scenes with just multiple two-dimensional photos.
• Continuous Scene Representation: The implicit representation provided by the neural network is continuous, which means it can describe arbitrarily fine details in space without losing information due to discretization.

Application Scenarios of NeRF

The emergence of NeRF has brought new possibilities to many fields:

• Virtual Reality (VR) and Augmented Reality (AR): Creating realistic virtual environments and digital content, improving immersion.
• Movies and Games: Used to generate high-quality visual effects, scenes, and animations. Especially in film production, it can achieve more flexible scene reproduction and perspective switching.
• Medical Imaging: Reconstructing comprehensive anatomical structures from 2D scans (such as MRI), providing doctors with more useful visual information.
• Digital Twins and City Modeling: Capable of creating detailed digital replicas of buildings, cities, and even large scenes.
• Robotics and Autonomous Driving: Helping robots and autonomous vehicles better understand the surrounding three-dimensional environment.

Challenges and Latest Developments in NeRF

Although NeRF technology is amazing, it still faces some challenges:

• Computing Resources and Time: Training NeRF models requires a lot of computing resources and a long time.
• Static Scene Limitations: The original NeRF is mainly suitable for static scenes and has limited processing capabilities for rapidly changing dynamic scenes.
• Complexity of Processing Large-scale Scenes: Efficiency and accuracy will be affected when processing ultra-large-scale scenes.

To overcome these limitations, researchers have been continuously improving NeRF technology. For example:

• Efficiency Optimization: Variants like PixelNeRF, Mega-NeRD, and NSVF reduce required computing resources and training time and improve rendering speed by introducing more effective network architectures or sparse representations. Technologies such as “Gaussian Splatting” have also brought significant improvements in speed and quality, surpassing NeRF in some aspects, but NeRF still has advantages in memory efficiency and adaptability of implicit representation.
• Dynamic Scenes and Editability: Some new research directions are exploring how to let NeRF handle dynamic scenes, and how to directly edit the scene content generated by NeRF so that it can be modified like traditional 3D models.
• Combining Multi-modal Data: Future NeRF research may also combine other inputs such as text and audio to create richer interaction and content generation methods.
• Application Expansion: For example, at the CVPR conference in 2024, the SAX-NeRF framework was proposed, which can reconstruct three-dimensional X-ray scenes from sparse X-ray images without CT data. Tsinghua University’s GenN2N framework unified multiple NeRF-to-NeRF conversion tasks, improving editing quality and efficiency. 3D generative AI based on NeRF has also made breakthroughs, generating editable 3D objects from a single image, or creating 3D scenes through text prompts.

In summary, NeRF and its derivative technologies are evolving rapidly. Its powerful ability to convert two-dimensional photos into interactive three-dimensional scenes undoubtedly heralds a huge revolution in future digital content creation and interactive experiences. We can look forward to it bringing infinite possibilities in many fields such as the virtual world, media entertainment, medical health, etc.

ONNX Runtime

The “Universal Translator” and “High-Performance Engine” in the AI Field — A Detailed Explanation of ONNX Runtime

In the wave of artificial intelligence, every day we may inadvertently come into contact with various services driven by AI models: whether it is the intelligent assistants in mobile phones, recommendation systems, or the perception decision-making of autonomous vehicles. Behind these AI models lies a silently dedicated “unsung hero” — ONNX Runtime.

For non-professionals, the deployment of AI models may sound somewhat abstract. Imagine that you have created various exquisite design drawings (AI models) with different tools, some with pencils, some with pens, and some with software. Now you need to send these drawings to different factories to produce products. The problem arises: the machines and production processes used by each factory are different. They may only recognize drawings drawn by specific tools, or require you to redraw one specifically for their machines. This is not only troublesome but also time-consuming and laborious.

This is the challenge once faced in AI model deployment.

Chapter 1: The “Dialects” and “Mandarin” of the AI World — The Birth of ONNX

In the world of AI, the situation is very similar to the analogy above. There are many powerful deep learning frameworks on the market, such as TensorFlow, PyTorch, Keras, etc. Each framework has its own unique “language” and “syntax” to define and train AI models. A model trained in PyTorch may be “acclimatized” in the TensorFlow environment and difficult to run directly. It is like people from different countries or regions speaking different dialects, making communication difficult.

To solve this problem of “language barrier”, major technology companies and research institutions jointly launched an open standard called ONNX (Open Neural Network Exchange). You can understand ONNX as the “Mandarin” or “unified blueprint” of the AI model world. It defines a common way to describe the computational graph and various parameter weights of AI models.

Metaphor: ONNX is like the PDF file format in the field of data transmission. No matter whether your document was originally created with Word, Excel, or PowerPoint, as long as it is exported as PDF, it can be viewed and shared in a unified format on any device. The same is true for ONNX, which allows developers to convert models trained by different frameworks into a unified .onnx format file. In this way, everyone can communicate and transfer models using the same standard. It greatly promotes the interoperability of models between different frameworks.

Chapter 2: The “High-Speed Rail” of Model Deployment — The Debut of ONNX Runtime

After having ONNX as a unified “drawing standard”, the next step is how to “produce” products efficiently—that is, to make AI models run at high speed in various practical applications. At this time, it is not enough just to have a common format. We also need a “high-performance factory” or “dedicated engine” capable of executing these “drawings” quickly and efficiently.

This “engine” is ONNX Runtime.

ONNX Runtime is an open-source inference engine specifically designed to run models in ONNX format. Please note that it is not used to train AI models, but is responsible for putting already trained ONNX models into actual use (i.e., performing “inference” or “prediction”).

Another metaphor: If ONNX is the “Mandarin” standard file (PDF) of AI models, then ONNX Runtime is a universal player or processor capable of “reading” and “executing” this “Mandarin” file with the fastest speed and highest efficiency. It knows how to optimally assign this general drawing to various “machines” in the factory for processing.

Chapter 3: ONNX Runtime: Why is it so Powerful?

ONNX Runtime plays such an important role in AI model deployment thanks to its core advantages:

1. Extreme Performance Optimization, Like a “Smart Factory”
   The primary goal of ONNX Runtime is to accelerate model inference. It is like an efficiently operating smart factory equipped with advanced automated processes and management systems internally. It will not just execute tasks step by step like an ordinary worker, but will intelligently optimize the computational graph of the model. For example, it automatically performs “graph optimization” (merging multiple simple operations into a more efficient one), “memory optimization”, etc., ensuring that the model completes inference with the fastest speed and minimal resources. In practical applications, ONNX Runtime can significantly improve model inference performance. Microsoft’s large-scale services, such as Bing, Office, and Azure AI, have seen an average CPU performance increase of double after using ONNX Runtime.

2. Cross-Platform, Fully Compatible, Like a “Universal Adapter”
   Whether on personal computers (Windows, Linux, macOS), servers, mobile phones (Android, iOS), or even resource-constrained edge devices or IoT devices, ONNX Runtime works well. It supports a variety of hardware accelerators, such as TensorRT on NVIDIA GPUs, OpenVINO on Intel processors, and DirectML on Windows. This means that your trained model only needs to be converted into ONNX format, and then it can be easily deployed to almost any device you want via ONNX Runtime without repeatedly modifying or optimizing the model for different platforms.

3. Convenient Deployment, Achieving “Plug and Play”
   ONNX Runtime provides APIs in multiple programming languages (Python, C++, C#, Java, JavaScript, etc.), allowing developers to easily integrate it into various applications. It greatly simplifies the “last mile” of AI model from laboratory training completion to final practical application deployment. Developers can focus more on model innovation and training without worrying too much about compatibility and performance issues during deployment.

Chapter 4: What Practical Problems Does ONNX Runtime Solve?

ONNX and ONNX Runtime jointly solve several key pain points in AI development:

• Connecting the “Meridians” of Training and Deployment: In the past, moving a model from a training environment (such as PyTorch) to a deployment environment (such as deploying to mobile phones or edge devices) often required complex conversion and adaptation processes, like crossing a chasm. ONNX and ONNX Runtime built a “universal bridge”, greatly simplifying this process.
• reducing Development and Maintenance Costs: Developers no longer need to maintain multiple versions of models or complex codes for each deployment target (different hardware, different operating systems), saving a lot of time and resources.
• Accelerating the Speed of AI Landing: Performance optimization and convenient deployment allow AI models to be applied to actual products and services faster, whether it is intelligent customer service, image recognition, voice processing, or recommendation systems. For example, real-time applications such as autonomous vehicles and video analysis systems have extremely high requirements for low latency and high throughput, and ONNX Runtime can meet these needs very well.
• Opening the Road to AI “Inclusiveness”: As open standards and open-source projects, ONNX and ONNX Runtime encourage broader cooperation and innovation, making AI technology easier for everyone to access and use, promoting the prosperity of the AI ecosystem.

Chapter 5: Looking to the Future: The Road to AI “Inclusiveness”

ONNX and ONNX Runtime are continuing to develop. According to the latest roadmap, ONNX Runtime will be updated quarterly, continuously improving support for new platforms and new features. For example, version 1.21, to be released in February 2025, will include various bug fixes and performance improvements. ONNX Runtime not only supports traditional machine learning models but also deep neural networks. It has even begun to support running ML models like PyTorch on web browsers and mobile devices, as well as optimizations in large model training (ONNX Runtime Training).

In the future, with the increasingly complex and diverse AI scenarios, the requirements for the performance, compatibility, and convenience of model deployment will become higher and higher. ONNX and ONNX Runtime, as key hubs connecting AI model training and practical applications, will continue to play an important role, promoting AI technology to serve human society more efficiently and universally, allowing everyone to enjoy the convenience brought by AI.

OPT

In the field of Artificial Intelligence (AI), “OPT“ refers to the “Open Pre-trained Transformer“. It is a series of Large Language Models (LLMs) developed by Meta AI (the parent company of Facebook). Unlike many other large language models, Meta has open-sourced the OPT models and their training code, aiming to promote open research and development in the AI field.

What is a Large Language Model (LLM)?

Imagine you have a very diligent and knowledgeable student. This student has read most of the text materials on the earth: books, articles, web pages, conversations, and so on. He not only memorized (learned) the content but also understood the language patterns, logical relationships, and even some subtleties of human thinking. When asked a question, he can synthesize his knowledge and give a coherent, logical, and even creative answer. This “student” is a Large Language Model. By learning from massive text data, it masters the ability to generate human language, understand semantics, and perform various linguistic tasks.

OPT: An “Open” and Powerful Language Brain

OPT stands for “Open Pre-trained Transformer.” We can understand it from these words:

1. Open:
   Usually, training a large language model requires huge computing resources and investment, resulting in most of such models being held by a few large companies that do not publicly disclose their core code or complete model weights. It’s like only a few people can see the study notes and thought processes of that “knowledgeable student.” The highlight of Meta AI’s release of OPT lies in its “openness.” It provides models of different sizes ranging from 125 million to 175 billion parameters, as well as the code and logs for training these models, allowing researchers around the world to study, understand, and improve deeper. This openness promotes collaboration within the AI community and enables researchers to better identify and address biases and limitations that may exist in the model.

2. Pre-trained:
   “Pre-trained” means that the model has passed a “big exam” before performing specific tasks (such as answering questions, translating). This “big exam” is reading and learning massive amounts of text data. It learns the structure, grammar, and semantics of language by predicting the next word in a sentence or filling in missing words. Just like that student, he laid a solid foundation through extensive reading, rather than cramming for a specific exam. The OPT model is pre-trained on large-scale public datasets containing various texts from the Internet, thus equipping it with general language understanding and generation capabilities.

3. Transformer:
   This is a neural network architecture underlying the OPT model and is also the key to the current success of large language models. If you view the language model as a “brain,” then the Transformer is the “thinking mechanism” of this brain. It is particularly good at processing sequential data, such as text. Simply put, the Transformer uses a technique called “Self-Attention” to allow the model to pay attention to the importance of all other words in a sentence simultaneously when processing a word, thereby better understanding contextual relationships. This is like a student who doesn’t just stare at the current word when reading but connects the content of the whole sentence, the whole paragraph, and even the whole article to think.

What Can the OPT Model Do?

As a large language model, OPT possesses a variety of powerful capabilities, for example:

• Text Generation: Given a beginning, it can create coherent stories, articles, or poems.
• Q&A System: Understand user questions and provide relevant information.
• Language Translation: Convert text from one language to another.
• Text Summarization: Extract key information from long articles to generate concise summaries.
• Code Generation: It can even generate code based on descriptions.

The OPT model released by Meta AI, especially its largest version OPT-175B, is comparable in performance to OpenAI’s GPT-3, but the carbon footprint required during its development process is only one-seventh of that of GPT-3, showing higher energy efficiency.

Limitations and Challenges of OPT

Although OPT is powerful, it is not perfect. Like all large language models, OPT faces challenges:

• High Computational Cost: Although more efficient than GPT-3, training and running models like OPT still require huge computing resources.
• “Hallucination” Phenomenon: Models sometimes generate information that sounds reasonable but is actually false.
• Bias and Toxicity: Since the model is trained on a large amount of Internet data, it may inherit and amplify social biases, toxic or discriminatory language existing in the training data, and even generate harmful content. Meta AI also emphasized the importance of sharing its limitations, biases, and risks when releasing OPT. This is like a student; if the materials he reads contain biases themselves, the knowledge he learns may also contain these biases.

All in all, OPT represents an important milestone in the field of artificial intelligence regarding large language models. By open-sourcing its code, it lowers the research threshold and accelerates the entire community’s understanding and progress of such cutting-edge technologies. It is a powerful and versatile “language brain” capable of completing many complex text tasks, but it also reminds us that, like driving any powerful tool, we also need to understand its working principles and potential risks to achieve responsible and beneficial AI development.

MoCo

Technology in the field of artificial intelligence is changing with each passing day. A concept called MoCo (Momentum Contrast), provides an ingenious solution for how machines can learn “autodidactically” from massive amounts of data. For non-professionals, MoCo may sound a bit complicated, but through life examples, we can easily understand its core idea.

1. The Autodidactic Challenge: AI’s “Self-learning” Dilemma

Imagine that when humans learn new things, we often need the guidance of a teacher to tell us that this is an “apple” and that is a “banana.” This learning method with clear labels is called “Supervised Learning” in the AI field. But in the real world, most data (such as countless pictures and videos on the Internet) are unlabeled, relying on manual labeling one by one, which is costly and time-consuming.

So, is it possible for AI to learn to identify things through its own observation and comparison like a child? This is the goal of “Unsupervised Learning.” It is like a child who sees various kinds of fruits. No one tells him which is which, but he can slowly discover that “a red sphere is very similar to another red sphere, but not quite like a yellow crescent thing” by observing characteristics such as appearance, color, and shape. This method of learning through comparison is the core of “Contrastive Learning.”

2. Contrastive Learning: Learning from “Finding Similarities and Distinguishing Differences”

Core Idea: The goal of contrastive learning is to let AI learn to distinguish between “similar things” and “dissimilar things.” It no longer needs to know exactly what object this is, but only needs to know that A and B are very similar, and A and C are very dissimilar.

Metaphor in Life:
Suppose you are learning to identify different kinds of dogs. You have a photo A of a Golden Retriever on hand.

• “Similar Things” (Positive Samples): You processed this photo A, such as cropping it or adjusting the brightness, to get photo A’. Although the appearance is slightly different, they are essentially “variants” of the same Golden Retriever. Contrastive learning hopes AI can view A and A’ as “similar” so that their “distance” in its “inner” feature space is very close.
• “Dissimilar Things” (Negative Samples): At the same time, you also have a photo B of a Husky, or a photo C of a cat. These are completely different objects from the Golden Retriever photo A. Contrastive learning hopes AI can view A and B, A and C as “dissimilar” so that their “distance” from A in the feature space is as far as possible.

By continuously practicing such “finding similarities and distinguishing differences,” AI can gradually distill the essential characteristics of things, such as learning the fur color and body shape characteristics of a Golden Retriever, without knowing its name is “Golden Retriever.”

3. The Magic of MoCo: Momentum and Dynamic Dictionary

Contrastive learning sounds great, but implementing it has a big challenge: to enable AI to learn and distinguish better, it needs a large number of “dissimilar” samples for comparison. It’s like a learner who can easily distinguish if he has only seen a few dogs and cats, but if he wants to distinguish a Golden Retriever from thousands of animals, he needs a huge “library of dissimilar animals” as a reference.

Traditional contrastive learning methods either can only process a small number of dissimilar samples during each training (limited by computer memory) or encounter problems of instability and inconsistency in the “library of dissimilar animals.” MoCo was born to solve this problem. It cleverly introduced the mechanism of “Momentum” and the concept of “Dynamic Dictionary.”

MoCo’s “Three Magic Weapons”:

1. Query Encoder — The Active Learning Student:
   This is like a student who is studying hard. It receives a picture (such as Golden Retriever photo A’) and then tries to extract the features of this picture. Its parameters update rapidly during training, constantly learning.

2. Key Encoder — The Steady and Wise Teacher:
   This is one of the core designs of MoCo. It is also a neural network, similar in structure to the Query Encoder. But the difference is that its parameter update is not directly through gradient backpropagation, but slowly and controllably “learned” from the Query Encoder. This process is like “momentum,” possessing inertia.
   Metaphor: Imagine an experienced master (Key Encoder) taking a new apprentice (Query Encoder). The apprentice progresses quickly and absorbs new knowledge every day. As for the master, his knowledge is accumulated over years of experience and will not be drastically shaken by the apprentice’s performance today, but only slowly and steadily updates his own experience system. In this way, the master provides a very stable and reliable reference frame for the apprentice. It is the master’s “steadiness” that guarantees the quality of the “reference frame.”

3. Queue — The Never-Stopping Updating “Reference Library”:
   To provide massive “dissimilar” samples, MoCo established a special “Queue.” This queue stores many pictures processed in the past (their features are generated by the Key Encoder). When new picture features are generated and added to the queue, the oldest picture features will be removed.
   Metaphor: This is like a large library containing various “dissimilar” animal pictures in history. This library is not fixed. It is updated every day. New books (new dissimilar samples) are put into storage, and old books (the oldest dissimilar samples) are taken out of storage, always maintaining the “novelty” and “diversity” of its content. Moreover, all books in the library are cataloged and organized by that steady “master,” so they maintain consistency.

How Does MoCo Work?
When the “student” (Query Encoder) sees a picture (such as Golden Retriever photo A’), it generates a feature. Then, it compares this feature with two types of features:

• Positive Sample: The feature of the same Golden Retriever photo A processed by the master (obtained from the Key Encoder).
• Negative Sample: Large amounts of “dissimilar” animal picture features randomly taken from the “Reference Library” (Queue) (also generated by the Key Encoder).

In this way, AI can perform contrastive learning among massive and consistent “dissimilar” samples, greatly improving learning efficiency and effectiveness. This allows contrastive learning to break free from the dependence on huge computing resources and achieve good performance.

4. MoCo’s Profound Impact and Latest Progress

The proposal of MoCo greatly promoted the development of Self-Supervised Learning, allowing AI to learn very powerful image feature representations without manual annotation. These features learned through MoCo can be directly applied to various downstream tasks, such as image classification, object detection, and semantic segmentation, performing even better than traditional supervised learning methods in many cases. Versions like MoCo v1, v2, v3, etc., continue to iterate and optimize performance. For example, MoCo v2 introduced an MLP projection head and stronger data augmentation methods, further improving results.

By 2025, contrastive learning remains a hot spot in the AI field. New research directions such as MoCo++ are exploring “hard negative mining,” that is, specifically finding “dissimilar” samples that are “specious” to positive samples, thereby allowing the model to learn more finely. In addition, the scope of application of contrastive learning has also expanded from images and text to graph-structured data, such as graph representation learning through the SRGCL method.

5. Conclusion

MoCo is like designing an efficient and ingenious “self-learning system” for AI in the ocean of artificial intelligence. Through the cooperation of the “steady master” and the “dynamic library,” it enables AI to autodidactically learn the essential characteristics of things from unlabeled massive data. This ability not only saves a lot of manpower and material resources but, more importantly, provides a strong cornerstone for AI to move towards true intelligence. In the future, we look forward to MoCo and its derivative contrastive learning methods creating miracles in more fields.

MobileNet

Why is your smartphone so “smart”? Have you ever marveled at how your phone camera can accurately identify cats and dogs, recognize faces, or get information instantly by scanning a product? Behind these seemingly simple functions lies powerful artificial intelligence. However, AI models are often very “huge” and “power-hungry.” How to run these AI functions smoothly on mobile phones or smart devices with limited resources was once a huge challenge.

Against this background, an AI model family called MobileNet came into being. It is like a “smart brain” tailored for mobile phones, greatly reducing the requirements for mobile phone computing power and battery power while ensuring recognition accuracy.

1. Why Do We Need MobileNet? — Clumsy Brains vs. Dexterous Pocket Assistants

Imagine if you want to carry an encyclopedia with you so that you can look up various knowledge anywhere. Traditional AI models are like a vast “Encyclopedia Britannica”, detailed and knowledgeable. But the problem is that this set of books is too heavy, and you can’t put it in your backpack at all, let alone put it in your pocket for reading at any time.

Our smartphones, smart watches, IoT devices, etc., are like your “portable assistants.” Their storage space and battery capacity are very limited and cannot carry that “clumsy encyclopedia.” What they need is a “condensed essence manual”—which can find information quickly and is light and portable. MobileNet is such an “essence handbook” designed for mobile devices.

Its core mission is: To make deep learning models smaller, faster, and more power-efficient without sacrificing too much accuracy.

2. MobileNet’s “Slimming Secret”: Depthwise Separable Convolution

The key to MobileNet’s successful “slimming” lies in its ingenious improvement of the core operation of traditional Convolutional Neural Networks (CNN)—Convolution. This secret is called “Depthwise Separable Convolution“.

Let’s start with traditional convolution:

Traditional Convolution: The All-Around Chef Does It All

Suppose you are a chef with various ingredients in front of you (such as onions, tomatoes, green peppers), and you need to use these ingredients to make dishes with multiple flavors. Traditional convolution operations are like an “all-around chef.” He will mix all ingredients (each color channel or feature of the input image) together, and then use dozens or even hundreds of different “recipes” (convolution kernels) to process them at the same time, cooking dozens of different dishes (output features) at once.

This chef is highly skilled, but every time he makes a dish, he needs to process all ingredients again and match various spices (weights), which is a huge workload. This means a lot of calculation and parameters, and the model naturally becomes large and slow.

Depthwise Separable Convolution: Dismantling Tasks and Collaborating

MobileNet’s “Depthwise Separable Convolution” splits the work of this “all-around chef” into two steps, allowing multiple “specialized chefs” to collaborate, greatly improving efficiency.

1. Depthwise Convolution: The Specialized “Ingredient Processor”
   Imagine you have a team: each member focuses only on processing one ingredient. For example, one member specializes in processing onions, another in tomatoes, and another in green peppers. They each use their own method (an independent convolution kernel) to process the ingredients at hand without interfering with each other.

   In this stage, each input channel (such as the red channel, green channel, blue channel of the picture, or a specific feature learned in the previous layer) is processed by only one independent convolution kernel. It only focuses on “seeing clearly” the characteristics of this single channel, and then generates a corresponding output. The advantage of this is that the workload and storage space required to process each ingredient (each channel) are greatly reduced.

2. Pointwise Convolution: The Efficient “Flavor Blender”
   Now, various ingredients have been processed by their respective “processors.” Next, it’s the turn of the “flavor blender.” This blender no longer needs to process ingredients repeatedly. He only needs to mix and match these processed, independent ingredients (outputs of depthwise convolution) in different proportions and ways to create various final dishes (new output features).

   In AI, this corresponds to a 1x1 convolution kernel operation. It no longer changes the width and height of the image, but is only responsible for information integration between different channels. Since the convolution kernel size is only 1x1, its calculation amount is very small, but it can effectively combine all information from depthwise convolution.

Through this division of labor mode of “independent processing first, then efficient blending,” depthwise separable convolution significantly reduces the overall calculation amount and model parameters, allowing the model size to be reduced to 1/8 or even 1/9 of traditional convolutional networks while maintaining similar accuracy.

3. Evolution of MobileNet: The Pocket Brain Getting Smarter

MobileNet is not static; it is an evolving family. Currently, multiple versions have been launched, each becoming more efficient and accurate on the basis of the previous generation:

• MobileNetV1 (2017): Laid the foundation for depthwise separable convolution and proved the feasibility of this lightweight design.
• MobileNetV2 (2018): Introduced “Inverted Residuals” and “Linear Bottlenecks”. This is like a chef discovering that some processing steps can be simplified or even some unnecessary complex intermediate links can be skipped when processing ingredients, directly getting the result, further improving efficiency and performance.
• MobileNetV3 (2019): Combined Automated Machine Learning (AutoML) technology and the latest architecture optimization. This means that it no longer relies solely on human experience to design, but lets AI “explore” and “learn” how to build the most efficient model. The V3 version also provides “Large” and “Small” models according to different performance requirements, further adapting to high-resource and low-resource scenarios. On mobile CPUs, MobileNetV3-Large is even twice as fast as MobileNetV2 while maintaining the same accuracy.

The latest development trends show that the evolution of the MobileNet series continues, and there is even research regarding MobileNetV4, continuously optimizing mobile inference efficiency through more innovative technologies.

4. Application Scenarios of MobileNet: Ubiquitous “Edge Intelligence”

The emergence of the MobileNet model family has greatly promoted the application of AI in mobile devices and edge computing fields, which we call “Edge AI”. This means that AI no longer needs to send all data to the “cloud server” central kitchen for processing, but can think and judge locally on the device directly. This brings many benefits:

• Real-time: No need to wait for data upload and download, faster response speed. For example, mobile phone real-time face recognition unlocking can be completed in the blink of an eye.
• Privacy Protection: Personal data (such as face images, fingerprints) does not need to leave the device, providing better security guarantees.
• Low Power Consumption: Local computing is usually more power-efficient than frequent network communication.
• Offline Work: AI functions can run normally without network connection.

MobileNet is widely used in the following fields:

• Smartphones: Face recognition, object recognition, AR filters, smart assistants (such as faster smart assistants on Pixel 4).
• Smart Home and IoT: Smart cameras (real-time intruder identification), smart door locks (face recognition unlocking), smart speakers, etc.
• Autonomous Driving and Robotics: Real-time environmental perception and target detection locally on vehicles or robots without relying on high-speed networks.
• Industrial Inspection: Drones equipped with MobileNet models analyze equipment failures or crop diseases in real-time locally.

Summary

The MobileNet series of models is an important innovation in the field of artificial intelligence. Through unique “Depthwise Separable Convolution” technology, as well as continuous architecture optimization and automated search in subsequent versions, it successfully brings powerful and complex AI capabilities to resource-limited mobile and edge devices. It is not just a technical term but also a behind-the-scenes hero of many convenient and smart experiences in our daily lives. With the continuous evolution of MobileNet, we can expect to feel more surprises brought by ubiquitous and instantly responsive “Edge Intelligence” in the future smart world.

NASNet

The AI Architect: An In-Depth Look at NASNet

Imagine if you wanted to build a house. The traditional way is to hire an architect to draw detailed blueprints by hand based on experience and knowledge, including room layout, floor structure, water and power supply systems, and so on. This requires the architect to have years of professional knowledge and talent. But if you were to design an “intelligent building” like a neural network in the field of Artificial Intelligence (AI), the complexity might be much higher than building a house!

For a long time, building high-performance neural network models has been the exclusive “knack” of AI researchers and engineers. They need to rely on profound theoretical knowledge and repeated experiments to carefully select appropriate network layers (such as convolutional layers, fully connected layers), cleverly design the connections between layers (such as skip connections, residual connections), and determine the specific parameters of each layer (such as kernel size, number of filters). This process is not only time-consuming and laborious but also requires extremely high experience from AI experts, just like a skilled carpenter crafting exquisite furniture with a hammer and chisel. However, human energy is always limited. Faced with massive possibilities, it is difficult for us to ensure finding that “perfect” design.

Against this background, a revolutionary concept called “Neural Architecture Search“ (NAS) came into being. It is like an “automated architect” with unlimited energy and creativity, capable of automatically exploring and designing high-performance neural network structures. NASNet is a milestone representative among the many excellent “works” designed by this “automated architect”.

What is Neural Architecture Search (NAS): AI Designing AI

To understand NASNet, we first need to understand its “driving force”—Neural Architecture Search (NAS). Simply put, NAS is a set of algorithms that allows AI to design and optimize AI models by itself, thereby greatly expanding the possibilities of model design. This process can be vividly compared to hiring a “robot chef.” It no longer relies on the recipes of human chefs but can try various ingredients (various operation units of neural networks such as convolution, pooling) by itself, match different cooking methods (connection methods), then taste (evaluate performance) the dishes it makes, and continuously improve based on “taste” (model performance on specific tasks), finally finding delicious dishes (high-performance neural network architectures).

The core elements of the NAS “robot chef’s” work are three:

1. Search Space (The “Ingredient Warehouse”): This defines which basic ingredients the “robot chef” can use and the combination rules between ingredients. The innovation of NASNet is that it didn’t try to design the entire complex “feast” at once, but focused on designing reusable “dish modules”—called “Cells”—and then assembling these cells like Lego blocks. This greatly narrows the search scope and makes the problem easier to solve.
2. Search Strategy (The “Cooking Method”): This is the strategy of how the “robot chef” explores the “ingredient warehouse” to find the best combination. NASNet initially adopted Reinforcement Learning as its core strategy. You can imagine there is a “controlling brain” (usually a Recurrent Neural Network, RNN), which will “predict” a new set of “dish combinations” (generate a neural network architecture) based on past experience, then let it “cook” (train this architecture), “taste” (evaluate performance), and finally adjust the direction of the next “prediction” based on the “tasting result,” striving to do better. Besides reinforcement learning, there are various “cooking methods” such as Bayesian optimization, evolutionary algorithms, and gradient-based methods available.
3. Performance Evaluation Strategy (The “Taster”): Whenever the “robot chef” makes a new dish, a “taster” is needed to score it. In AI, this is scoring by testing the accuracy or efficiency of the model on a validation set. This is the most time-consuming and computationally expensive part of the entire process because every proposed architecture needs to be trained and evaluated.

NASNet: A “Star Architecture” Designed by AI Itself

NASNet is not a search algorithm, but a set of neural network architectures discovered and verified by the NAS search algorithm. It was proposed by the Google Brain team in 2017 to address challenges in the field of image recognition.

The most critical contribution of NASNet is that it discovered a series of high-performance transferable convolutional cells through NAS. Just like the “robot chef” didn’t directly design a complete banquet, but first designed two of the most core and useful “dish modules”:

• Normal Cell: The main function of this cell is to extract image features, but it will not change the spatial size of the image feature map, just like a dish, although the taste becomes richer, the portion remains unchanged.
• Reduction Cell: This cell can effectively reduce the spatial resolution of the image feature map, just like concentrating a large dish into an essence while maintaining its nutrition and flavor, which helps the network capture large-scale features more effectively and reduce computation.

Then, researchers, or further, the NAS algorithm itself, stack these “Normal Cells” and “Reduction Cells” in a specific way to form the complete NASNet network architecture. This modular design allows excellent cell structures searched on small datasets (such as CIFAR-10) to be very efficiently transferred to large datasets (such as ImageNet) and achieve equally outstanding performance, even surpassing the best models manually designed by human experts before.

The appearance of NASNet achieved the most advanced accuracy at the time in image classification tasks. For example, NASNet-A achieved 82.7% top-1 accuracy on ImageNet, 1.2% higher than the optimal architecture designed by humans. It also has variants like NASNet-B and NASNet-C, demonstrating the powerful capability of this automated design method.

Advantages of NASNet: AI’s Superpower

NASNet and the NAS technology it represents bring significant advantages in many aspects:

• Super-human Performance: NAS can discover excellent architectures that human experts can hardly imagine or discover, often surpassing human-crafted models on specific tasks, just like the outstanding performance of NASNet in the field of image recognition.
• Automation and Efficiency: It greatly reduces the time and energy for AI experts to manually design and debug neural network structures, lowering the threshold for AI model design and allowing more people to utilize high-performance AI models.
• Transferability: By searching for general cells or modules, structures learned on one task or dataset can be transferred to other tasks or datasets while maintaining excellent performance, which is one of the core contributions of NASNet.
• Wide Application: Models found by NAS like NASNet not only perform well in tasks such as image classification but also achieve better performance than manually designed networks in computer vision tasks such as object detection and image segmentation.

Challenges and Future Directions: The Continuously Evolving “Automated Architect”

Although NASNet has brought huge breakthroughs, Neural Architecture Search still faces some challenges:

• Huge Computational Cost: This is the biggest “pain point” of NAS. Early NAS methods might require thousands of GPU days to complete a search, and this “electricity bill” is not a small amount. Even though NASNet has accelerated training time by more than 7 times by searching for cell structures, it still requires a large amount of computing resources.
  • Improvement Directions: To solve this problem, researchers are exploring more efficient search algorithms, such as gradient-based methods, one-shot NAS, multi-fidelity methods, and techniques like weight sharing, reducing training epochs, using proxy models, or pre-searching on small datasets to accelerate the evaluation process. For example, recent progress includes using “differentiable model scaling” to optimize network width and depth more effectively.
• Model Interpretability: Automatically generated complex architectures are sometimes like a “black box,” and we can hardly fully understand their internal working principles, which may affect the reliability and credibility of the model.
• Design of Search Space: The quality of search space design directly affects the quality of the final result. How to design a more unexpected and reasonable search space remains a research focus.

NAS is an important part of the AutoML (Automated Machine Learning) field. Future research directions will continue to explore more efficient search algorithms, smarter search spaces, and improve the interpretability of NAS, so that the “automated architect” can not only build good houses but also clearly explain why building them this way is the best.

Summary

The emergence of NASNet marks an important step for the AI field from “human designing AI” to “AI designing AI”. It has not only achieved remarkable achievements in tasks such as image recognition, but more importantly, it verified the huge potential of Neural Architecture Search (NAS). Although NAS technology still faces challenges such as high computational costs, scientists are constantly working hard to make it more efficient, intelligent, and easy to understand. In the future, we can look forward to AI, the “automated architect,” designing more unexpected and superior intelligent “buildings,” promoting Artificial Intelligence to achieve new breakthroughs in various fields.

Mish Activation

In the world of Artificial Intelligence, especially Deep Learning, every calculation in a neural network relies on a core component — the activation function. They are like the brain of a neuron, deciding how information is transmitted and whether it is “activated.” Today, we will explore in simple terms a novel activation function that has received much attention in recent years: Mish. It not only outperforms many predecessors in performance but also brings new vitality to deep learning models with its unique “personality.”

What is an Activation Function? The “Decision Maker” of Neural Networks

Imagine you are training a robot to recognize cats. When the robot sees an image, it analyzes the picture through layers of “neurons.” Each neuron receives some information (digital signals) and then needs to decide whether to pass this information to the next neuron or simply “stop” it. The switch for this “decision” is the activation function.

Early activation functions, such as Sigmoid and Tanh, were like simple “on/off” or “yes/no” buttons, allowing neural networks to learn some simple patterns. But as the number of network layers increases and tasks become complex, these simple buttons become powerless, easily leading to the “gradient vanishing” problem, resulting in low learning efficiency or even stagnation.

To solve these problems, researchers introduced the ReLU (Rectified Linear Unit) activation function. Its operation is very simple: if the input is positive, output it as is; if it is negative, output 0. This is like a limiter, only letting “positive” information pass. ReLU’s advantage is fast calculation speed, effectively mitigating the gradient vanishing problem. But it also has a “dead zone”: if the input is always negative, the neuron will “die” and stop learning, which is called the “Dying ReLU” problem.

The Rise of Mish: A Smarter Decision Maker

On the basis of ReLU and its variants, researchers continued to explore more powerful activation functions. “Mish: A Self Regularized Non-Monotonic Neural Activation Function” was proposed by Diganta Misra in 2019, aiming to combine the advantages of existing activation functions while avoiding their disadvantages.

The mathematical expression of the Mish activation function is: f(x) = x * tanh(softplus(x)). Seeing this formula for the first time might seem complicated, but we can break it down into a few metaphors from daily life to understand.

1. Softplus: The Smooth “Dimmer”
   • First is softplus(x). Remember the “switch” metaphor of ReLU? ReLU is like a digital gate, pass if positive, zero if negative. Softplus is a gentler “dimmer” switch. When the input is negative, it doesn’t drop to zero directly but slowly approaches zero, never truly becoming zero. When the input is positive, it is almost as large as the input. It’s like when night falls, the light doesn’t turn off completely with a “click,” but gently dims until it is almost invisible.
2. Tanh: The “Compressor” of Information
   • Next is the tanh() function, which is a hyperbolic tangent function capable of compressing any input value to between -1 and 1. Imagine you have a pile of packages of various sizes. Tanh’s job is to neatly compress them so their volume is within a controllable range. In this way, no matter how large or small the original information is, it becomes easier to manage and transmit after being processed by Tanh.
3. x * tanh(softplus(x)): The “Skillful Processing” of Information
   • Finally, Mish multiplies the original input x by the result of tanh(softplus(x)). This is like a “skillful processing” procedure. softplus(x) provides a smooth, never fully closed “signal strength,” and tanh() “normalizes” this signal strength. Multiplying these two not only retains the information of the original input x but also introduces a clever non-linear transformation. This multiplication mechanism is related to a property called “Self-Gating,” allowing the neuron to adjust its output based on the input itself, thereby improving the efficiency of information flow.

Taken together, Mish is like a sophisticated signal processing center. It doesn’t simply let the signal pass or block it, but adjusts the signal strength through a smooth dimmer, then normalizes it with a compressor, and finally cleverly combines it with the original signal, making the transmitted information more detailed and expressive.

The Unique Charm of Mish: Why is it Better?

The reason why Mish activation function is considered the “next generation” activation function is due to its multiple key features:

• Smoothness: The Mish function is continuously differentiable everywhere, without the “sharp corners” like ReLU. This means that during the neural network optimization process, the gradient (can be understood as the direction and speed of learning) changes more smoothly, avoiding drastic oscillations, thereby making the training process more stable and easier to find the optimal solution.
• Non-monotonicity: Traditional activation functions like ReLU are monotonically increasing. Mish’s curve has a slight dip in some negative value areas before rising again. This non-monotonicity allows Mish to better handle and retain negative value information, avoiding “information loss,” especially performing well when facing subtle but important negative signals.
• Unbounded above, Bounded below: Mish can accept arbitrarily large positive inputs and output corresponding positive numbers, avoiding the problem of saturation after the output value reaches an upper limit (i.e., gradient approaching zero). At the same time, it has a lower bound of about -0.31. This feature helps maintain gradient flow and has a “Self-regularization” effect, just like a smart learner who can self-adjust during training to improve the model’s generalization ability.

Applications and Outlook: What Has Mish Brought?

Since Mish was proposed, it has demonstrated excellent performance in multiple deep learning tasks. Research shows that in tasks such as image classification (e.g., CIFAR-100, ImageNet-1k datasets) and object detection (e.g., YOLOv4 model), models using Mish activation function can surpass models using other activation functions like ReLU and Swish by more than 1% to 2% in accuracy. Especially when building deeper neural networks, Mish effectively prevents performance degradation, enabling models to learn more complex features.

For example, in the YOLOv4 object detection model, Mish was introduced as the activation function, helping it increase the average precision by 2.1% on the MS-COCO object detection benchmark. The FastAI team also set records on multiple leaderboards by combining Mish with the Ranger optimizer, proving the powerful potential of Mish in practical applications.

The emergence of Mish once again proves the indispensable status of activation functions in deep learning and their profound impact on model performance. It provides a smoother, more flexible, and more adaptive “neuron decision mechanism,” helping AI models better understand and learn complex data. Although the computational cost may be slightly higher than ReLU, the performance improvement it brings is often worth it. With the continuous development of deep learning technology, Mish is likely to become an important choice in future AI model design, continuing to drive artificial intelligence towards a smarter and more efficient future.

A New Perspective on AI Optimization Algorithms — Mirror Descent: Why Do We Need to “Walk in the Mirror” to Arrive Faster?

In the vast world of Artificial Intelligence (AI), optimization algorithms play a core role. They are like navigation systems, guiding AI models to find the best path in a complex “terrain” to learn to recognize images, understand language, or even play chess. Among them, Gradient Descent is the most famous, simple and effective. However, when facing certain special “terrains,” a more ingenious “mirror walking” method — Mirror Descent — often achieves better results.

1. Reviewing Gradient Descent: A Naive Way Downhill

Imagine you are blindfolded and placed on a rolling hill. Your goal is to find the lowest point (e.g., a lake in the valley). Your only strategy is: with each step, sense which direction is the steepest from your current position, and then take a small step in that direction. This is the core idea of Gradient Descent.

Mathematically, the “height” of this hill is the loss function we want to minimize, and your position is the parameters of the AI model. The steepest direction is guided by the Gradient. Gradient Descent updates parameters in the opposite direction of the gradient each time, just like you always walk down the steepest slope. This method is simple and intuitive, performing well in Euclidean geometry (the plane or 3D space we perceive daily).

However, if the terrain of the hill becomes very strange, for example, not smooth, or you are restricted to a special area (e.g., you can only walk on a narrow path on the top of the mountain, or only circle at the bottom of a bowl), the simple “steepest” strategy may no longer be the optimal choice.

2. Walking into the Mirror World: Why Do We Need to “Change Shoes”?

Now, let’s introduce some more complex challenges. In AI, we sometimes need to optimize some special quantities, for example:

• Probability Distribution: All probabilities must add up to 1 and cannot be negative. For example, when a model predicts the probability of a word appearing, these probabilities must sum to 1.
• Sparse Vector: A vector where most elements are zero. For example, we want the model to select only a few key features from numerous features.

In these cases, traditional Gradient Descent may encounter trouble. If we perform gradient updates directly in these special spaces, we may need extra processing, such as forcing probability values back to a “sum of 1” state after each update, or forcing non-negativity. It’s like wearing a pair of heavy hiking boots to attend an elegant dance. Although you can walk, it always feels awkward and even prone to mistakes.

Mirror Descent provides an elegant solution. Unlike Gradient Descent, which uses “one pair of shoes for everywhere,” it can “change into a pair of the most fitting shoes” according to the characteristics of the current “terrain.” This pair of “special shoes” is realized through a tool called “Mirror Map.”

To use an analogy: You are not walking directly on the hill now, but first entering a “Mirror World.” In this mirror world, the originally strange hill terrain becomes very flat and regular, and you can easily find the corresponding position of the lowest point here. After finding it, you return to the real world through reverse “mirror transformation,” and at this time, you are already standing at the lowest point of the original hill.

3. Mirror Descent: Disassembling the Principle

Mirror Descent can achieve this mainly depending on several core concepts:

3.1 Mirror Map

Mirror Map, also known as “Potential Function,” is a bridge from the original space (the space where we want to optimize parameters) to the “Mirror Space” (a mathematically more regular space). It is usually a convex function whose gradient maps points in the original space to the mirror space.

For example, for the probability distribution optimization problem we mentioned earlier, a commonly used mirror map is the negative entropy function. Through this mapping, the optimization of the probability vector is transformed into the optimization of log-probability in another space, making the constrained probability problem easier to handle.

Through the mirror map, we “hide” the complex geometric constraints in the original space and perform unconstrained optimization in the mirror space, just like unfolding a distorted sphere into a plane for processing.

3.2 Strolling in the “Mirror Space”

After entering the mirror space through the mirror map, we can execute standard gradient descent steps here. Because the geometric structure of the mirror space is usually “friendlier” than the original space, this step becomes simpler and more direct. It’s like moving forward in the steepest direction on flat ground without extra obstacles.

3.3 Mapping Back to the “Real World”

After completing a gradient update step in the mirror space, we cannot stay here. We need to return to the original space through the “inverse operation” (inverse mapping) of the mirror map to get the new values of our model parameters. This new parameter value is a step we took in the original space, but this step considers the unique geometric structure of the original space, so it is more effective and reasonable than simple gradient descent. This update method of shuttling back and forth between the original space and the mirror space is exactly the origin of the name “Mirror Descent.”

3.4 A Special Ruler Measuring Distance: Bregman Divergence

In traditional Gradient Descent, we usually use Euclidean distance (the straight-line distance in our daily life) to measure how close two points are. But in Mirror Descent, since we introduce non-Euclidean geometric structures, we use a more generalized concept of “distance,” called Bregman Divergence.

Bregman Divergence is defined based on a specific mirror map function, and it can better reflect “distance” and “difference” in non-Euclidean spaces. For example, in probability distribution problems, if negative entropy is used as the mirror map, then the corresponding Bregman Divergence becomes Kullback-Leibler Divergence (KL Divergence), a common method for measuring differences between two probability distributions. This special “ruler” allows Mirror Descent to move more accurately along the “correct” direction when dealing with certain problems.

4. What Are the Powers of Mirror Descent? Application Scenarios

Mirror Descent has extensive applications in the AI field, showing unique advantages especially in the following scenarios:

• Online Learning and Game Theory: In these scenarios, the model needs to constantly adjust strategies as new data arrives. Mirror Descent can effectively handle these dynamic optimization problems that often have special structures (such as probability distributions summing to 1).
• Reinforcement Learning (RL): In recent years, Mirror Descent has also been applied to policy optimization in reinforcement learning, producing algorithms such as “Mirror Descent Policy Optimization (MDPO).” Such methods help the model balance exploration and stability when updating policies by introducing Bregman divergence as a trust-region constraint.
• Large-Scale and High-Dimensional Data Optimization: When the dimension of data is very high and the optimization problem has non-Euclidean constraints, Mirror Descent can help algorithms converge faster and obtain better solutions.
• Implicit Regularization: Research shows that Mirror Descent has an implicit regularization effect. When applied to classification problems, it can converge to a generalized maximum-margin solution, which helps improve the generalization ability of the model.

5. Latest Dynamics and Future Outlook

In recent years, the importance of Mirror Descent has become increasingly prominent in the machine learning field, with continuous new research progress:

• Efficient Implementation: Researchers are developing more efficient algorithms based on Mirror Descent, such as p-GD, which can be implemented in deep learning models with almost no extra computational overhead. This allows the advantages of Mirror Descent to be better applied to practical deep learning tasks.
• Meta-Learning Optimizers: A study called “Meta Mirror Descent (MetaMD)” proposes that the best Bregman divergence can be learned through meta-learning to accelerate the optimization process and provide better generalization guarantees. This means future optimizers may be able to automatically choose the most suitable “shoes” for different tasks.
• Stochastic Incremental Mirror Descent: When dealing with large-scale datasets, stochastic algorithms are essential. Researchers are exploring Stochastic Incremental Mirror Descent algorithms with Nesterov smoothing to improve efficiency in large-scale convex optimization problems.

In short, Mirror Descent is a powerful and elegant optimization tool. It teaches us that when solving complex problems, sometimes we don’t have to stick to the “straightforward” way, but can find simpler and more effective solutions in the “Mirror World” through ingenious “perspective shifting” and “tool switching,” ultimately achieving faster and more robust development of AI.

Unveiling AI’s Rising Star: Mistral AI — Making Intelligent AI Accessible

In today’s rapid development of artificial intelligence, Large Language Models (LLMs) have become an indispensable part of our lives. They are like “super brains” with massive knowledge, capable of understanding, generating human language, and even writing code. However, these powerful “super brains” often require huge computing resources and are mostly controlled by a few technology giants. Against this background, a French startup called Mistral AI has stood out, becoming a dazzling new star in the AI field with its innovative spirit and the concept of “openness and efficiency.”

What is a Large Language Model (LLM)?

Before diving into Mistral AI, let’s briefly understand what a Large Language Model (LLM) is. Imagine you have a very learned friend who has read almost all books, articles, and internet information in the world. When you ask him any question, he can quickly give a clear and rich answer, and even help you write articles, translate text, and write program code. Large language models are such “digital friends.” By learning massive amounts of text data, they master the laws of language and knowledge, thus being able to perform various complex language tasks.

Mistral AI: A Paradigm of “Small and Beautiful” Wisdom

Mistral AI was founded in 2023 by former researchers from Meta and DeepMind. From the beginning, they held an ambitious goal: to provide top-notch AI performance while making models more lightweight, efficient, and as open as possible. This is in sharp contrast to the “bigger is better” philosophy of some mainstream AI companies.

You can compare Mistral AI to a well-designed, energy-saving, and environmentally friendly sports car manufacturer. Traditional sports cars may rely on piling up powerful engines to achieve extreme speed, but Mistral AI is committed to optimizing design, reducing body weight, and improving engine technology to achieve the same or even faster speed with smaller displacement and less fuel consumption.

Their core concepts are as follows:

1. Extreme Efficiency: Mistral AI challenges the traditional notion that “bigger models are better.” They focus on developing models that consume fewer computing resources (like less “fuel”) while maintaining or even surpassing top-notch performance.
2. Embracing Open Source: Unlike many companies that treat models as “trade secrets,” Mistral AI vigorously promotes open source. They have released many high-performance models, allowing developers to use, modify, and deploy them for free, just like providing a set of exquisite “advanced toolboxes” and “instructions,” allowing everyone to innovate and build on this basis.

Mistral AI’s Star Models: Each Has Its Own Magic

Mistral AI has launched a series of models that have caused a sensation in the AI community, the most famous of which include:

1. Mistral 7B: A Lightweight Miracle

“7B” represents that this model has 7 billion parameters. Parameters are the number of “neuron connections” in a large language model that determine its learning ability. Generally speaking, the more parameters, the more powerful the model. But Mistral 7B breaks the convention. It is like a lightweight but agile athlete. With unique skills and optimized training methods (such as “Sliding Window Attention” and “Grouped Query Attention”), it performs excellently in multiple benchmarks, even surpassing some models with twice or even four times its parameters, such as Llama 2 13B and Llama 1 34B.

This ability to “punch above its weight” means that developers can run and deploy high-performance AI models with lower costs and less computing power, allowing more people to enjoy the convenience brought by AI.

2. Mixtral 8x7B: The Wisdom of the Committee of Experts

The Mixtral 8x7B model introduces a more ingenious design — the “Mixture of Experts (MoE)” architecture. You can imagine it as a team of 8 experts in different fields. When you have a problem, the system will not ask all 8 experts to deal with it, but intelligently select only 2 to 3 most relevant experts to solve it based on the nature of the problem. In this way, although the knowledge volume of the entire team (model) is very large (total parameters reach 47 billion), the computing resources actually called when processing tasks are greatly reduced (only about 13 billion parameters are activated each time).

This design allows Mixtral 8x7B to maintain high performance while having faster inference speed and higher efficiency. In some tests, it even outperformed OpenAI’s GPT-3.5 and Meta’s Llama 2 70B models.

3. Mistral Large and Mistral Large 2: Flagship All-Rounders

Mistral Large is Mistral AI’s flagship commercial model, representing their most powerful capabilities. It possesses excellent logical reasoning capabilities, strong multilingual support (initially excelling in English, French, Spanish, German, and Italian), and performs excellently in complex tasks such as code generation and mathematical problem solving. You can think of it as a top-level learned consultant capable of handling various complex and professional tasks.

Mistral Large 2, released in July this year (2024), is the latest upgrade to this flagship model. It has up to 123 billion parameters, further improving performance in code, mathematics, reasoning, and multilingualism (including Chinese, Japanese, Korean, Russian, and many other languages), and supports a text content window of up to 128k. This means it can process and understand longer documents or conversations at once, like a wise man with superb memory and profound understanding.

4. Mistral Small 3.1: Balancing Performance and Accessibility

In March 2025, Mistral AI released its latest lightweight open-source model, Mistral Small 3.1. This model has 24 billion parameters and has made significant progress in improving text performance and multimodal understanding (i.e., understanding and processing more than one type of information, such as text and images), and also supports a 128k context window. More importantly, this model runs well even on relatively ordinary hardware devices (for example, a Mac laptop with 32GB of RAM or a single RTX 4090 graphics card), greatly increasing the accessibility of advanced AI technology.

Latest Dynamics: Sustainable Development of AI Ecosystem

Mistral AI maintained vigorous innovation vitality in 2025:

• Launching AI Studio: In October 2025, Mistral AI officially launched Mistral AI Studio, a production-oriented AI platform aimed at helping developers and enterprises build and deploy AI applications more conveniently.
• Huge Financing: In September 2025, Mistral AI successfully completed a round of financing of 1.7 billion euros, which will undoubtedly accelerate its technology research and development and market expansion.
• AI Coding Tool Stack: In July 2025, Mistral AI released Codestral 25.08 and its complete enterprise-level AI coding tool stack, aiming to solve the practical implementation problems of generative AI in enterprise software development, providing a secure, customizable, and efficient AI-native development environment.
• Le Chat Application: Mistral AI also launched its AI assistant application Le Chat and continuously added new features, such as “Memories” and connections with more than 20 enterprise platforms.

Conclusion

With its unique “efficiency and openness” strategy, Mistral AI has opened up a new path in the highly competitive AI field. They proved that high-performance AI is not only the “big and comprehensive” mode. Through ingenious architectural design and extreme pursuit of efficiency, even relatively lightweight models can unleash amazing capabilities. By open-sourcing its innovative models, Mistral AI is promoting the development of a more open and inclusive AI ecosystem, making cutting-edge AI technology no longer just the patent of a few technology giants, but can be mastered and utilized by a wider range of developers and enterprises, jointly promoting the progress of artificial intelligence.

Mask R-CNN: The “Fire Eyes” That Let AI See the World Clearly

In the world of artificial intelligence, the ability of machines to “understand” pictures is developing rapidly. From identifying what is in the image (classification), to finding out where the objects are (object detection), to what we are going to discuss in depth today — not only finding objects but also accurately outlining the contours of each object, this is the “Fire Eyes” (a Chinese idiom meaning sharp and penetrative eyesight, originating from the Monkey King) in the AI field: Mask R-CNN.

I. From “Rough Identification” to “Precise Outline”: The Evolution of AI Vision

Imagine you are taking a photo with your phone:

• Image Classification: Your phone tells you, “This is a picture of a cat.” (AI identifies the category of the entire photo)
• Object Detection: Your phone draws a box on the cat you photographed and tells you, “There is a cat here, and there is a dog there.” (AI finds all objects of interest in the picture and marks them with rough boxes)
• Instance Segmentation (Enter Mask R-CNN!): Your phone not only draws boxes on the cat and dog but also precisely outlines the complete contour of each cat and dog like a silhouette, and can even distinguish between ‘the first cat’ and ‘the second cat’. This is Mask R-CNN, which combines object detection and pixel-level image segmentation to achieve finer understanding.

Mask R-CNN was proposed in 2017 by a team led by Kaiming He, a Chinese scientist at Facebook AI Research. It was developed on the basis of Faster R-CNN (Faster Region-Convolutional Neural Network). If Faster R-CNN is compared to a “scout” who can accurately locate and box targets, then Mask R-CNN is an “artist” added on this basis who can accurately cut out a “silhouette” for each target.

II. Demystifying How Mask R-CNN Works: Seeing the World Clearly Step by Step

The power of Mask R-CNN lies in its ingenious multi-task collaborative working mechanism. We can imagine it as an AI system with multiple expert groups, each performing its own duties and finally completing fine image analysis tasks together.

1. “Image Understanding Expert”: Backbone Network and Feature Pyramid Network (FPN)

   • Metaphor: Like an experienced observer, first scan the entire room to understand the major features in the room (such as lighting, placement of main furniture, etc.), forming a “rough impression map.”
   • Principle: The input image first goes through a powerful Convolutional Neural Network (such as ResNet), which is called the “Backbone Network.” Its task is to extract features from the image and generate a series of “feature maps.” To better handle objects of different sizes, Mask R-CNN also incorporates the “Feature Pyramid Network” (FPN). FPN allows AI to understand images at different scales, for example, using high-level features to understand the overall semantics of the image (“This is a person”), and using low-level features to capture object details (“This person’s eyes, nose, and mouth”).

2. “Region Proposal Expert”: Region Proposal Network (RPN)

   • Metaphor: Based on the “rough impression map,” this expert begins to point out “regions that may hide interesting items” in the room (for example, “there may be a toy behind the sofa,” “there may be a bag under the table”), giving many candidate regions.
   • Principle: RPN slides on the feature map to generate a series of “Region Proposals” that may contain objects. These regions are initially judged by RPN as “foreground” (object) or “background,” and the box position is fine-tuned.

3. “Precise Focus Expert”: RoI Align (Region of Interest Align)

   • Metaphor: Traditional object detection might just roughly crop and scale those “regions that may hide items,” such as forcibly turning circular items into squares, leading to information distortion (imagine roughly cutting out an image with scissors). RoI Align is like a high-precision scanner that can accurately extract features of each candidate region based on the proportion and position information of the image to ensure pixel-level alignment and avoid information loss.
   • Principle: This is one of the most significant innovations of Mask R-CNN. RoI Pooling (Region of Interest Pooling) used by Faster R-CNN involves quantization operations (such as rounding) when dealing with non-integer coordinates, which causes slight deviations between features and object positions in the original image, especially affecting small objects and pixel-level segmentation tasks. RoI Align achieves more accurate feature extraction through methods like bilinear interpolation, solving this “misalignment” problem, thereby significantly improving mask accuracy.

4. “Multi-task Collaboration Expert”: Classification, Bounding Box Regression, and Mask Prediction Branch

   • Metaphor: After precise focusing, three expert groups start working simultaneously:
     • Classification Expert: “This item is a cat!” (Confirm what category the item is)
     • Bounding Box Regression Expert: “This cat’s box needs to be fine-tuned 2 pixels to the upper left and enlarged a bit to be more precise.” (Fine-tune the position and size of the box)
     • Mask Prediction Expert: “This is the precise outline of the cat!” (Outline the shape of the cat pixel by pixel)
   • Principle: For each region processed by RoI Align, Mask R-CNN outputs three results in parallel:
     • Classification: Judge which category the object in this region belongs to (e.g., cat, dog, car, etc.).
     • Bounding Box Regression: Further refine the specific position and size of the box to surround the object more tightly.
     • Mask Prediction: This is a Fully Convolutional Network (FCN) branch that generates a binary mask for each region of interest, which can indicate pixel by pixel which parts of the region belong to the object. This is the key for Mask R-CNN to achieve instance segmentation. Unlike previous methods, the mask branch of Mask R-CNN is parallel and decoupled from the classification branch, which allows the model to learn each task more effectively.

III. Applications and Future of Mask R-CNN

Due to its high precision and versatility in instance segmentation, Mask R-CNN has shown huge potential in many fields.

• Autonomous Driving: Vehicles need to accurately identify pedestrians, vehicles, and traffic signs on the road and accurately distinguish their boundaries to ensure driving safety.
• Medical Image Analysis: Doctors can use Mask R-CNN to accurately segment tumors and lesion areas to assist in diagnosis and treatment, such as detecting defects in industrial CT images.
• Robotic Manipulation: Robots need to accurately identify and grasp objects of specific shapes. Mask R-CNN can help robots “see clearly” the accurate contours of objects for finer operations.
• Smart Retail and Warehousing: Used for product identification, inventory management, and even precise placement of items on shelves.
• Image Editing and Enhancement: Automatically identify portraits and perform background separation to achieve functions like “one-click cutout.”

Although Mask R-CNN is excellent, it also has certain limitations, such as high computational requirements, making its real-time performance inferior to some specialized real-time detection models like the YOLO series. However, as a milestone model in the field of instance segmentation, Mask R-CNN has not only promoted the development of computer vision technology but also laid the foundation for the birth of subsequent more advanced models.

In summary, Mask R-CNN is like installing “fire eyes” on AI capable of accurately identifying and outlining object contours, moving machine understanding of images from fuzzy to fine. With the continuous evolution of technology, we look forward to it shining in more fields in the future, bringing more convenience and innovation to humanity.