分词

在人工智能（AI）领域，尤其是大型语言模型（LLM）的飞速发展中，有一个看似简单却至关重要的概念——分词（Tokenization）。它像是连接人类语言和机器理解之间的一座桥梁。想象一下，我们人类交流时，大脑会自然地将一句话分解成一个个有意义的词语或概念来理解。但对于不理解人类语言的计算机来说，它需要一套规则来完成这个“分解”过程。分词正是这项任务。

1. 分词：语言的“乐高积木”

什么是分词？

简单来说，分词就是将一段连续的文本序列切分成一个个独立的、有意义的单元，这些单元我们称之为“token”（令牌）。就好比我们建造乐高模型，不能直接使用一大块塑料，而是需要一块块预先设计好的积木（token）来拼搭。这些积木可以是单个的字符、常见的词语，甚至是词语的一部分。

为什么AI需要分词？

计算机不直接理解文字本身，它们只理解数字。为了让AI模型能够处理和学习文本数据，我们需要将文本转换成模型能够识别的数字表示。分词就是这转换过程的第一步，它决定了模型“看到”的语言基本单位是什么。分词后的每个token会被赋予一个唯一的ID，然后这些ID再被映射成模型可以处理的数值向量。

如果没有分词，AI模型就像一个不懂单词的孩子，面对着一长串没有间断的字母，根本无从下手。只有把文字切割成有意义的“积木”，模型才能搭建起对语言的理解。

2. 不同种类的“乐高积木”：分词方法的演变

分词的方式有很多种，就像乐高积木有各种形状和大小，各有各的用处。

2.1 字符级分词：最细小的“珠子”

思路： 将每个独立的字符都视为一个token。
比喻： 就像将一串项链上的每个珠子都分开。
优点： 灵活性高，不存在“未知词”（Out-of-Vocabulary, OOV）问题，因为任何文本都能分解成已知的字符集。
缺点： 会导致模型的上下文窗口被拉得很长，因为一个词可能需要十几个字符来表示，模型难以学习高层级的语义信息。

2.2 词级分词：常见的“单词”积木

思路： 将文本按照词语（通过空格或词典）进行分割。
比喻： 就像一本为儿童设计的拼图书，每个词语都已被预先剪好。
优点： 易于理解和实现，尤其对于英文这类词语间有空格分隔的语言。
缺点：

  • 新词问题（OOV）： 如果遇到词典中没有的新词、网络流行语或专业术语，模型就无法识别。
  • 中文分词的挑战： 中文与英文不同，词语之间没有天然的空格分隔，使得中文分词成为一项更具挑战性的任务。例如，“我爱北京天安门”这句话，到底是“我/爱/北京/天安门”还是“我爱/北京/天安门”？这需要依靠上下文和语义来判断。

2.3 子词级分词：更智能的“可拆卸”积木

为了解决词级分词的OOV问题和字符级分词效率低的问题，现代大型语言模型普遍采用了子词级分词方法。

思路： 这种方法介于字符级和词级之间。它会学习一个词汇表，其中包含常见的词语和一些常见的词语片段（子词）。如果遇到词汇表中没有的词，它能将其拆分成更小的、已知的子词单元。
比喻： 这就像一个智能乐高套装。它不仅有常见的完整积木，还有一些可以进行拼装或拆解的特殊积木块，比如“连接件”、“转角件”等。当你遇到一个新的、复杂的结构（一个不认识的词），它能智慧地将其分解成已知的小片段。例如，“unhappiness”这个词，它可能会被拆分成“un”、“happi”、“ness”。
主要的算法有： 字节对编码 (BPE)、WordPiece、Unigram LM等。
优点：

  • 平衡性： 既能有效处理常见词，又能将未知词分解为有意义的子单元，减少了OOV问题。
  • 降低词汇表大小： 相比词级分词，子词级分词可以在不牺牲太多语义信息的情况下，显著减小模型需要学习的词汇表规模。
  • 高效利用上下文窗口： 在有限的“上下文窗口”（模型一次能处理的token数量）内，可以编码更多的信息。

3. 分词在大型语言模型中的作用与挑战

分词是大型语言模型理解和生成文本的基石。

  • 文本输入的处理： 当你向ChatGPT提问时，你的问题首先会被分词器处理成一个个token序列，然后这些token才会被模型读取和理解。
  • 生成文本： 模型在生成回答时，也是一个token一个token地预测和生成。
  • 成本与效率： 许多大型语言模型的API是按照token数量计费的，因此高效的分词能够帮助用户更经济地使用服务。同时，能将更多内容塞入模型的“上下文窗口”也依赖于高效的分词。

然而，分词并非完美无缺，它也带来了模型的一些独特挑战：

  • “颠倒单词”难题： 研究发现，大型语言模型有时在执行看似简单的任务（如颠倒一个单词的字母顺序）时会遇到困难。原因在于，模型“看到”的是整体的token，而不是token内部的单个字符。如果“elephant”是一个token，模型就无法轻易地操作其中的单个字母。
  • 中文场景的复杂性： 中文分词的挑战尤为突出。由于词语间的无空格特性，“错误分词是阻碍LLM精确理解输入从而导致不满意输出的关键点，这一缺陷在中文场景中更为明显”。
  • 对抗性攻击： 研究人员甚至构建了专门的对抗性数据集（如ADT），通过挑战LLM的分词方式，来揭示模型的漏洞并导致不准确的响应。这意味着，即使人眼看起来无差别的文本，一旦分词不同，可能会让模型产生截然不同的理解。

4. 分词的未来：持续演进的“积木”工艺

随着AI技术的不断发展，分词技术也在持续演进：

  • 领域和语言定制化： 针对不同语言（如中文）和特定领域（如法律、医疗）的需求，会出现更加优化和专业的定制化分词器。
  • 优化算法： 研究人员正不断改进分词算法和流程，以提升LLM的整体能力，例如融合预训练语言模型、多标准联合学习等。
  • 可能超越文本分词： 一些前沿探索甚至开始质疑传统文本分词作为AI核心输入的地位。例如，DeepSeek-OCR模型尝试以像素的形式处理文本，将文字直接转化为视觉信息，这可能“终结分词器时代”。特斯拉前AI总监、OpenAI创始团队成员Karpathy也曾表示，或许所有LLM输入都应该是图像，即使纯文本也最好先渲染成图像再喂给模型，因为分词器“丑陋、独立”、“引入了Unicode和字节编码的所有糟粕”，带来了安全和越狱风险。

总而言之，分词是AI，特别是大型语言模型，理解和处理人类语言的基石。它就像是为机器打造语言“乐高积木”的工艺，它的精度和效率直接影响着AI模型的性能和智能程度。理解分词，能让我们更好地认识AI的优点和局限，并期待未来更智能的语言处理方式。

In the field of Artificial Intelligence (AI), especially with the rapid development of Large Language Models (LLMs), there is a concept that seems simple but is crucial — Tokenization. It acts like a bridge connecting human language and machine understanding. Imagine when we humans communicate, our brains naturally decompose a sentence into meaningful words or concepts to understand it. However, for computers that do not understand human language, they need a set of rules to complete this “decomposition” process. Tokenization is exactly this task.

1. Tokenization: The “Lego Bricks” of Language

What is Tokenization?

Simply put, tokenization is the process of splitting a continuous text sequence into independent, meaningful units, which we call “tokens”. Just like building a Lego model, we cannot directly use a large block of plastic, but need pre-designed bricks (tokens) to assemble it. These bricks can be individual characters, common words, or even parts of words.

Why does AI need Tokenization?

Computers do not directly understand text itself; they only understand numbers. To enable AI models to process and learn from text data, we need to convert text into numerical representations that the model can recognize. Tokenization is the first step in this conversion process, determining what the basic units of language the model “sees” are. Each token after tokenization is assigned a unique ID, and these IDs are then mapped into numerical vectors that the model can process.

Without tokenization, an AI model is like a child who doesn’t understand words, facing a long string of uninterrupted letters, with no way to start. Only by cutting the text into meaningful “bricks” can the model build an understanding of language.

2. Different Types of “Lego Bricks”: The Evolution of Tokenization Methods

Tokenization methods vary, just like Lego bricks come in various shapes and sizes, each with its own use.

2.1 Character-level Tokenization: The Smallest “Beads”

Idea: Treat each independent character as a token.
Metaphor: Like separating every bead on a necklace.
Pros: High flexibility, no “Out-of-Vocabulary” (OOV) problem, because any text can be decomposed into a known character set.
Cons: Results in the model’s context window being stretched very long, as a word might need a dozen characters to represent, making it difficult for the model to learn high-level semantic information.

2.2 Word-level Tokenization: Common “Word” Bricks

Idea: Split text according to words (via spaces or a dictionary).
Metaphor: Like a puzzle book designed for children, where every word has been pre-cut.
Pros: Easy to understand and implement, especially for languages like English where words are separated by spaces.
Cons:

  • New Word Problem (OOV): If it encounters a new word, internet slang, or technical term not in the dictionary, the model cannot recognize it.
  • Chinese Tokenization Challenge: Unlike English, Chinese words are not naturally separated by spaces, making Chinese tokenization a more challenging task. For example, is the sentence “我爱北京天安门” (I love Beijing Tiananmen) segmented as “我/爱/北京/天安门” or “我爱/北京/天安门”? This requires reliance on context and semantics to judge.

2.3 Subword-level Tokenization: Smarter “Detachable” Bricks

To solve the OOV problem of word-level tokenization and the inefficiency of character-level tokenization, modern large language models universally adopt Subword-level Tokenization.

Idea: This method lies between character-level and word-level. It learns a vocabulary containing common words and some common word fragments (subwords). If it encounters a word not in the vocabulary, it can split it into smaller, known subword units.
Metaphor: This is like a smart Lego set. It not only has common complete bricks but also some special bricks that can be assembled or disassembled, such as “connectors,” “corner pieces,” etc. When you encounter a new, complex structure (an unrecognized word), it can intelligently decompose it into known small fragments. For example, the word “unhappiness” might be split into “un”, “happi”, “ness”.
Main Algorithms: Byte Pair Encoding (BPE), WordPiece, Unigram LM, etc.
Pros:

  • Balance: Can effectively handle common words and decompose unknown words into meaningful sub-units, reducing the OOV problem.
  • Reduced Vocabulary Size: Compared to word-level tokenization, subword-level tokenization can significantly reduce the size of the vocabulary the model needs to learn without sacrificing too much semantic information.
  • Efficient Use of Context Window: Within a limited “context window” (the number of tokens a model can process at once), more information can be encoded.

3. The Role and Challenges of Tokenization in Large Language Models

Tokenization is the cornerstone of large language models for understanding and generating text.

  • Processing Text Input: When you ask ChatGPT a question, your question is first processed by the tokenizer into a sequence of tokens, and then these tokens are read and understood by the model.
  • Generating Text: When the model generates an answer, it also predicts and generates one token after another.
  • Cost and Efficiency: Many large language model APIs charge based on the number of tokens, so efficient tokenization helps users use the service more economically. At the same time, fitting more content into the model’s “context window” also relies on efficient tokenization.

However, tokenization is not flawless, and it also brings some unique challenges to models:

  • The “Reversing Words” Puzzle: Research has found that large language models sometimes struggle with seemingly simple tasks (such as reversing the letters of a word). The reason is that the model “sees” the whole token, not the individual characters inside the token. If “elephant” is a token, the model cannot easily manipulate the individual letters within it.
  • Complexity in Chinese Scenarios: The challenge of Chinese tokenization is particularly prominent. Due to the lack of spaces between words, “incorrect tokenization is a key point hindering LLMs from accurately understanding input, leading to unsatisfactory output, and this defect is even more obvious in Chinese scenarios.”
  • Adversarial Attacks: Researchers have even built specialized adversarial datasets (such as ADT) to reveal model vulnerabilities and cause inaccurate responses by challenging LLM tokenization methods. This means that even text that looks identical to the human eye, once tokenized differently, might lead the model to a completely different understanding.

4. The Future of Tokenization: Continuously Evolving “Brick” Craftsmanship

With the continuous development of AI technology, tokenization technology is also constantly evolving:

  • Domain and Language Customization: Custom tokenizers that are more optimized and professional will appear for different languages (such as Chinese) and specific domains (such as law, medical).
  • Optimization Algorithms: Researchers are constantly improving tokenization algorithms and processes to enhance the overall capability of LLMs, such as integrating pre-trained language models, multi-criteria joint learning, etc.
  • Possibility of Transcending Text Tokenization: Some frontier explorations have even begun to question the status of traditional text tokenization as the core input of AI. For example, the DeepSeek-OCR model attempts to process text in the form of pixels, converting text directly into visual information, which may “end the era of tokenizers”. Former Tesla AI Director and OpenAI founding team member Karpathy also stated that perhaps all LLM inputs should be images, and even pure text is best rendered into images before being fed to the model, because tokenizers are “ugly, separate,” “introduce all the cruft of Unicode and byte encodings,” and bring security and jailbreak risks.

In summary, tokenization is the cornerstone for AI, especially large language models, to understand and process human language. It is like the craft of creating language “Lego bricks” for machines; its precision and efficiency directly affect the performance and intelligence level of AI models. Understanding tokenization allows us to better recognize the strengths and limitations of AI and look forward to smarter language processing methods in the future.