Transformer in Vision

AI领域的概念层出不穷，每次技术的飞跃，都如同为我们打开一扇通往未来的窗户。今天，我们要聊的是一个近年在人工智能，特别是计算机视觉领域掀起巨浪的技术——Vision Transformer（视觉Transformer）。它就像一位新来的“超级阅卷老师”，用它独特的方式，理解和“批阅”我们眼前的世界。

一、引言：从“读懂文字”到“看懂世界”的革命

在人工智能的世界里，让机器“看懂”图片和视频，甚至理解其中的内容，一直是个核心挑战。过去很长一段时间，我们依赖的都是一种叫做“卷积神经网络”（CNN）的技术。想象一下，CNN就像一位传统的阅卷老师，擅长“局部观察，循序渐进”地批改试卷。它会一行一行、一段一段地看，然后从局部细节中总结出规律。

然而，近年来，另一位“老师”——Transformer，在自然语言处理（NLP）领域，也就是让机器理解和生成文字的领域，取得了突破性进展。它凭借其独特的“全局视角”和“注意力机制”，彻底改变了机器读懂文字的方式。现在，这位“文字大师”开始跨界挑战“视觉理解”任务，催生了我们今天要讲的Vision Transformer。它不再仅仅关注局部，而是试图一下子“纵览全局”，并根据重要性“分配注意力”，这带来了全新的思考方式。

二、传统视觉AI的“阅卷老师”：卷积神经网络（CNN）

要理解Vision Transformer的特别之处，我们先简单回顾一下它的“前辈”——卷积神经网络（CNN”。
CNN处理图像的方式，可以比喻为一名非常细致且有经验的“厨师”在处理食材。

  1. 局部感受野：就像厨师切菜，会先处理胡萝卜丝、土豆块等单个食材，CNN也是逐块、逐像素地扫描图像，捕捉局部纹理、边缘等细节信息。它有一个“感受野”，只专注于当前的小区域。
  2. 层层抽象：这些局部信息经过一层层处理，就像把切好的食材进行烹饪、调味，从简单的线条到复杂的形状，再到物体的整体轮廓，逐步提取出越来越高级的特征。
  3. 优点与局限：CNN擅长从局部特征中归纳模式，并在许多视觉任务中表现出色。但它的局限性在于，它很难直接捕捉图像中两个相距很远，但又相互关联的元素之间的关系。就像厨师切完菜，很难立刻知道所有菜品组合后会产生怎样的独特风味，需要一步步尝试。

三、新一代“阅卷老师”：Transformer登场

Transformer模型最初由Google在2017年提出，彻底革新了自然语言处理（NLP）领域。它摒弃了传统的循环神经网络（RNN）和卷积神经网络（CNN），完全基于一种叫做**自注意力机制（Self-Attention Mechanism）**的“全局焦点”技术构建。

想象一下，你面前有一份非常复杂的合同。传统的阅读方式是逐字逐句看，而Transformer的注意力机制，则像是在读合同之前，就先大致扫描一遍，然后根据合同条款之间的内在逻辑关系，自动判断哪些词句是最重要的，哪些词句只是辅助说明，让它能同时考虑所有文字，并理解它们之间的相互关联。这种“纵览全局，分清主次”的能力，让它在处理长文本依赖问题时尤其有效。

那么，当这份擅长“读懂文字”的“超级阅卷老师”，来到“看图识物”的计算机视觉领域时，它会如何工作呢？这就是**Vision Transformer (ViT)**的核心思想：把图片当成一段段文字来处理。

四、“视觉Transformer”如何工作？

Vision Transformer（ViT）的工作流程，可以形象地比喻为老师批改一份由许多小卡片组成的“图像考卷”：

  1. 图像“切分”（Patching）：首先，一张完整的图片被切割成许多个大小相同的小方块，我们称之为“图像块”（Image Patch）。就像一份完整的试卷，被平均分成了很多个小卡片，每张卡片上有一小部分图像。例如，一张224x224像素的图片，可以被切分成196个16x16像素的小块。
  2. “图像块”变为“词语”（Tokenization）：接着，每个图像块都会被数字化，转换为一个特殊的“词向量”（Patch Embedding）。你可以把这看作是把每张小卡片上的图像内容，总结成了一个简短的“标签”或“编码”。
  3. 位置编码（Positional Encoding）：光有“词语”还不够，我们还需要知道这些“词语”在原始图片中的位置关系。ViT会给每个图像块的向量添加一个“位置编码”，就像给每张小卡片盖上一个“位置章”，告诉模型这张卡片原来在图片的左上角还是右下角。这样，即使图像块被打乱了，模型也能知道它们原本的顺序。
  4. “自注意力”机制（Self-Attention）：这是整个Vision Transformer最核心、最神奇的部分。在进入Transformer的主体——“编码器”后，各个图像块（现在是带有位置信息的“词向量”）不再是孤立地被处理。模型会同时审视所有的图像块，并让每个图像块都去“关注”其他所有图像块。
    • “全局视野”：与CNN的局部观察不同，自注意力机制让ViT从一开始就拥有了“全局视野”，能够直接建立图像中任意两个像素区域之间的关系，无论它们相距多远。
    • “权重分配”：当模型在处理某个图像块时，它会计算这个图像块与图片中所有其他图像块的关联性强弱，并根据关联性赋予不同的“注意力权重”。例如，当识别一张猫的图片时，模型可能会发现猫的眼睛和猫的胡须之间的关联性很强，而猫的眼睛和背景中一棵树的关联性则很弱。模型会更“关注”那些关联性强的图像块。
    • “多头注意力”（Multi-Head Attention）：为了更全面地理解图像，Vision Transformer通常会采用“多头注意力”机制。这就像组织一个评审小组，由多位“阅卷老师”从不同的角度（不同的“头”）去审视图像块之间的关系。有的“头”可能关注颜色，有的“头”关注形状，有的“头”关注位置，最后综合大家的意见。
  5. 输出与应用：经过多层这样的“自注意力”和前馈神经网络处理后，模型就学习到了图像中各个部分之间复杂的相互关系和更高级的视觉特征。最后，这些特征会被用于各种视觉任务，如图像分类（识别图片是什么）、目标检测（找出图片中有哪些物体）、语义分割（精确地描绘出每个物体的边界）等。

五、为什么“视觉Transformer”很厉害？

Vision Transformer的出现，为计算机视觉领域带来了许多激动人心的优势：

  1. 捕捉长距离依赖：传统CNN在捕捉图像中相隔遥远但有联系的特征时比较费力，因为它受限于局部感受野。而Vision Transformer的自注意力机制天生就能处理这种**“长距离依赖”**，能更好地理解图像的整体结构和上下文信息。
  2. 泛化能力更强：由于Vision Transformer的“归纳偏置”（可以理解为模型对数据结构的先验假设）比CNN弱，这意味着它对数据的假设更少，能够从大规模数据中学习到更通用的视觉模式。一旦数据量足够大，它的表现往往优于CNN。
  3. 可扩展性：Transformer模型在处理大规模数据集和构建大规模模型时表现出强大的潜力，这在图像识别、特别是预训练大型视觉模型方面具有巨大优势。
  4. 统一性：它为图像和文本处理提供了一个统一的架构，这对于未来多模态AI（同时处理图像、文本、语音等多种数据）的发展具有重要意义。

当然，Vision Transformer也并非完美无缺。它通常需要非常庞大的数据集进行训练，才能发挥出其全部潜力。对于较小的数据集，传统的CNN可能表现更好。

六、日常生活中的应用

Vision Transformer及其衍生的模型，正在悄然改变我们与数字世界的互动方式：

  • 智能手机相册：当你用手机拍完照，相册能自动识别出照片中的人物、地点、事物，并进行分类管理，这背后可能就有Vision Transformer的功劳。
  • 医疗影像分析：在医学领域，辅助医生分析X光片、CT扫描或病理切片，帮助检测疾病，比如识别肿瘤或病变区域。
  • 自动驾驶：帮助车辆识别路标、行人、其他车辆以及各种复杂路况，是自动驾驶技术安全可靠运行的关键。
  • 安防监控：在人群密集的场所，识别异常行为、进行人脸识别、追踪可疑目标，提升公共安全水平。
  • AI绘画与内容生成：像DALL-E, Midjourney这样能通过文字描述生成逼真图像的AI模型，其内部的核心也离不开Transformer架构对图像和文本的深刻理解。
  • 视频分析：理解视频内容，进行行为识别、事件检测，例如在体育赛事中分析运动员动作，或在工业生产中监控设备运行状态。

七、未来展望

Vision Transformer自2020年提出以来，已成为计算机视觉领域的重要研究方向，并有望在未来进一步替代CNN成为主流方法。

最新的研究和发展趋势包括：

  • 混合架构（Hybrid Architectures）：结合CNN和Transformer的优点，利用CNN提取局部特征，再用Transformer进行全局建模，以达到更好的性能和效率。比如Swin Transformer通过引入“移位窗口机制”，在局部窗口内计算自注意力，同时降低了计算复杂度，优化了内存和计算资源消耗。
  • 轻量化和高效性：为了在移动设备和边缘计算场景中使用，研究者们正在努力开发更小、更快的Vision Transformer模型，例如MobileViT将轻量卷积与轻量Transformer结合。
  • 更广泛的应用：除了传统的图像分类、目标检测和分割，Vision Transformer还在持续探索更多领域，如三维视觉、图像生成、多模态理解（视觉-语言结合）等，展现出强大的通用性。例如，MambaVision结合了状态空间序列模型与Transformer，在某些任务上实现了性能提升和计算负载降低。

Vision Transformer的崛起，标志着人工智能在“看懂世界”的道路上迈出了重要一步。它以其独特的全局视角和注意力机制，为我们开启了理解和处理视觉信息的新篇章。未来，随着技术的不断演进，我们有理由相信，这位“超级阅卷老师”将帮助AI更好地感知和创造世界。

Title: Transformer in Vision
Tags: [“Deep Learning”, “CV”]

AI concepts emerge one after another, and every technological leap opens a window to the future for us. Today, we are going to talk about a technology that has made huge waves in Artificial Intelligence, especially in the field of Computer Vision in recent years — Vision Transformer. It is like a new “super grading teacher” who understands and “grades” the world before our eyes in its own unique way.

1. Introduction: The Revolution from “Reading Text” to “Understanding the World”

In the world of Artificial Intelligence, making machines “see” images and videos, and even understand their content, has always been a core challenge. For a long time in the past, we relied on a technology called “Convolutional Neural Network” (CNN). Imagine that CNN is like a traditional grading teacher who is good at “local observation and gradual progress” when correcting papers. It scans line by line, paragraph by paragraph, and then summarizes the rules from local details.

However, in recent years, another “teacher” — Transformer, has made breakthrough progress in the field of Natural Language Processing (NLP), which is the field of making machines understand and generate text. With its unique “global perspective” and “attention mechanism”, it has completely changed the way machines read text. Now, this “text master” has begun to cross over to challenge the task of “visual understanding”, giving birth to the Vision Transformer we are discussing today. It no longer focuses only on local parts, but tries to “survey the whole situation” at once and “allocate attention” according to importance, bringing a brand new way of thinking.

2. The Traditional “Grading Teacher” of Computer Vision: Convolutional Neural Network (CNN)

To understand the uniqueness of Vision Transformer, let’s briefly review its “predecessor” — Convolutional Neural Network (CNN).
CNN processes images in a way that can be likened to a very meticulous and experienced “chef” processing ingredients:

  1. Local Receptive Field: Just like a chef chopping vegetables will first process individual ingredients like shredded carrots and potato chunks, CNN also scans the image block by block and pixel by pixel, capturing details like local textures and edges. It has a “receptive field” that focuses only on the current small area.
  2. Layered Abstraction: These local information are processed layer by layer, just like cooking and seasoning the chopped ingredients, gradually extracting higher-level features from simple lines to complex shapes, and then to the overall outline of objects.
  3. Advantages and Limitations: CNN excels at inducing patterns from local features and performs well in many visual tasks. But its limitation lies in that it is difficult to directly capture the relationship between two elements in an image that are far apart but related. Just like a chef who finishes chopping vegetables, it is hard to immediately know what unique flavor will be produced after combining all the dishes; it requires step-by-step trial.

3. The New Generation “Grading Teacher”: Transformer Enters the Scene

The Transformer model was first proposed by Google in 2017, completely revolutionizing the field of Natural Language Processing (NLP). It abandoned traditional Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs), and was built entirely based on a “global focus” technology called Self-Attention Mechanism.

Imagine you have a very complicated contract in front of you. The traditional way of reading is to read word by word, while Transformer’s attention mechanism is like scanning roughly before reading the contract, and then automatically judging which sentences are the most important and which are just auxiliary explanations based on the internal logical relationship between contract terms, allowing it to consider all text simultaneously and understand their interrelationships. This ability to “survey the whole and distinguish the primary from the secondary” makes it particularly effective when dealing with long text dependency problems.

So, when this “super grading teacher” who is good at “reading text” comes to the field of Computer Vision for “image recognition”, how does it work? This is the core idea of Vision Transformer (ViT): Treat images as segments of text.

4. How Does “Vision Transformer” Work?

The workflow of Vision Transformer (ViT) can be vividly likened to a teacher grading an “image exam paper” composed of many small cards:

  1. Image “Patching”: First, a complete picture is cut into many small squares of the same size, which we call “Image Patches”. It’s like a complete test paper divided equally into many small cards, and each card has a small part of the image. For example, a 224x224 pixel image can be cut into 196 16x16 pixel small patches.
  2. “Patches” into “Words” (Tokenization): Next, each image patch is digitized and converted into a special “Patch Embedding”. You can think of this as summarizing the image content on each small card into a short “label” or “code”.
  3. Positional Encoding: Just having “words” is not enough; we also need to know the positional relationship of these “words” in the original picture. ViT adds a “positional encoding” to the vector of each image patch, just like stamping a “location stamp” on each small card, telling the model whether this card was originally in the upper left corner or the lower right corner of the picture. In this way, even if the image patches are shuffled, the model knows their original order.
  4. “Self-Attention” Mechanism: This is the most core and magical part of the entire Vision Transformer. After entering the main body of the Transformer — the “Encoder”, the image patches (now “word vectors” with position information) are no longer processed in isolation. The model examines all image patches simultaneously and lets each image patch “pay attention” to all other image patches.
    • “Global View”: Unlike CNN’s local observation, the self-attention mechanism gives ViT a “global view” from the start, enabling it to directly establish relationships between any two pixel regions in the image, regardless of how far apart they are.
    • “Weight Allocation”: When the model processes a certain image patch, it calculates the strength of expectation between this image patch and all other image patches in the picture, and assigns different “attention weights” based on the correlation. For example, when identifying a picture of a cat, the model might find a strong correlation between the cat’s eyes and the cat’s whiskers, while the correlation between the cat’s eyes and a tree in the background is weak. The model will pay more “attention” to those strongly correlated image patches.
    • “Multi-Head Attention”: To understand the image more comprehensively, Vision Transformer usually adopts a “Multi-Head Attention” mechanism. This is like organizing a review panel, with multiple “grading teachers” examining the relationship between image patches from different angles (different “heads”). Some “heads” may focus on color, some on shape, and some on position, and finally synthesize everyone’s opinions.
  5. Output and Application: After multiple layers of such “self-attention” and feed-forward neural network processing, the model learns the complex interrelationships and higher-level visual features between various parts of the image. Finally, these features are used for various visual tasks, such as image classification (identifying what the picture is), object detection (finding out what objects are in the picture), semantic segmentation (precisely depicting the boundary of each object), etc.

5. Why is “Vision Transformer” Powerful?

The emergence of Vision Transformer has brought many exciting advantages to the field of computer vision:

  1. Capturing Long-Range Dependencies: Traditional CNNs struggle to capture features that are far apart but related in an image because they are limited by local receptive fields. Vision Transformer’s self-attention mechanism is naturally capable of handling such “long-range dependencies”, allowing for a better understanding of the overall structure and context of the image.
  2. Stronger Generalization Ability: Since Vision Transformer has a weaker “inductive bias” (which can be understood as the model’s prior assumption about data structure) than CNN, this means it makes fewer assumptions about data and can learn more general visual patterns from large-scale data. Once the data volume is large enough, its performance often outperforms CNN.
  3. Scalability: Transformer models show great potential when dealing with large-scale datasets and building large-scale models, which has huge advantages in image recognition, especially in pre-training large visual models.
  4. Unification: It provides a unified architecture for image and text processing, which is of great significance for the development of future multi-modal AI (processing multiple data such as images, text, voice, etc., simultaneously).

Of course, Vision Transformer is not perfect. It usually requires very large datasets for training to unleash its full potential. For smaller datasets, traditional CNNs might perform better.

6. Applications in Daily Life

Vision Transformer and its derived models are quietly changing the way we interact with the digital world:

  • Smartphone Albums: When you finish taking photos with your mobile phone, the album can automatically identify people, places, and things in the photos and manage them by category. This may be due to Vision Transformer.
  • Medical Image Analysis: In the medical field, it assists doctors in analyzing X-rays, CT scans, or pathological slices to help detect diseases, such as identifying tumors or lesion areas.
  • Autonomous Driving: Helping vehicles identify road signs, pedestrians, other vehicles, and various complex road conditions is key to the safe and reliable operation of autonomous driving technology.
  • Security Monitoring: In crowded places, identifying abnormal behaviors, performing face recognition, and tracking suspicious targets to improve public safety.
  • AI Painting and Content Generation: AI models like DALL-E and Midjourney, which can generate realistic images through text descriptions, also rely on the core Transformer architecture for deep understanding of images and texts.
  • Video Analysis: Understanding video content, performing behavior recognition and event detection, such as analyzing athlete movements in sports events or monitoring equipment operating status in industrial production.

7. Future Outlook

Since its proposal in 2020, Vision Transformer has become an important research direction in the field of computer vision and is expected to further replace CNN as the mainstream method in the future.

The latest research and development trends include:

  • Hybrid Architectures: Combining the advantages of CNN and Transformer, using CNN to extract local features and then using Transformer for global modeling to achieve better performance and efficiency. For example, Swin Transformer calculates self-attention within local windows by introducing a “shifted window mechanism”, which reduces computational complexity and optimizes memory and computing resource consumption.
  • Lightweight and Efficiency: To use in mobile devices and edge computing scenarios, researchers are working hard to develop smaller and faster Vision Transformer models, such as MobileViT which combines lightweight convolution with lightweight Transformer.
  • Broader Applications: In addition to traditional image classification, object detection, and segmentation, Vision Transformer is continuously exploring more fields such as 3D vision, image generation, and multi-modal understanding (vision-language combination), showing strong versatility. For example, MambaVision combines state space sequence models with Transformer, achieving performance improvements and reduced computational load in certain tasks.

The rise of Vision Transformer marks an important step for artificial intelligence on the road to “understanding the world”. With its unique global perspective and attention mechanism, it opens a new chapter for us to understand and process visual information. In the future, with the continuous evolution of technology, we have reason to believe that this “super grading teacher” will help AI better perceive and create the world.