FLOPs

AI世界的“燃料”：深入浅出理解FLOPs

在人工智能（AI）的浩瀚宇宙中，我们常常听到“算力”、“计算量”这些词汇，它们如同支撑一座座摩天大楼的地基，决定着AI模型能走多远，能变得多强大。而在这片地基之下，有一个核心的衡量单位，叫做FLOPs。它不仅是衡量AI模型“力气”大小的关键，也在不断演进中驱动着整个AI领域飞速向前。

到底什么是FLOPs？为什么它对AI如此重要？对于非专业人士来说，我们可以通过一些日常生活的比喻来形象地理解它。

一、FLOPs：AI世界的“浮点数食谱”与“速度计”

当我们提到FLOPs，其实是指两个相关但略有不同的概念：

  1. FLOPs (Floating Point Operations)：小写的“s”代表复数，指的是“浮点运算次数”，也就是一次AI计算任务（比如让AI识别一张图片）中，总共需要进行多少次这样的数学运算。它衡量的是计算量。
  2. FLOPS (Floating Point Operations Per Second)：大写的“S”代表“每秒”（Per Second），指的是“每秒浮点运算次数”，也就是计算机硬件一秒钟能完成多少次浮点运算。它衡量的是计算速度或硬件性能。

为了方便理解，整个文章中我们主要聚焦于大写的 FLOPS 来解释其在衡量算力上的意义（硬件性能）。

日常比喻：一场复杂的烹饪盛宴

想象一下，你正在准备一顿极其丰盛、步骤繁琐的晚餐。这顿晚餐需要进行大量的切菜、搅拌、称重、加热等操作。

  • 浮点运算 (Floating Point Operations)：就像是食谱中的每一个具体操作，比如“将2.5克盐加入1.5升水中混合”、“将面粉和水以3.14:1的比例搅拌”。这些操作都涉及小数，是比较精密的计算。 AI模型，特别是神经网络，在处理数据时，会进行大量的涉及小数的加减乘除运算，这就是浮点运算。

  • FLOPs (浮点运算次数)：就是完成这顿晚餐所需的所有切菜、搅拌、称重等操作的总次数。一道越复杂的菜（比如一个参数量庞大的AI模型），需要的总操作次数就越多。 比如，GPT-3这样的大模型，单次推理的FLOPs可达到约2000亿次。

  • FLOPS (每秒浮点运算次数)：就是你（或你的厨房帮手）一秒钟能完成多少个这样的操作。如果你是米其林大厨，一秒能切好几片菜，搅拌好几次酱料，那么你的FLOPS就很高，你的烹饪效率就很快。 反之，如果你的动作很慢，FLOPS就很低。计算机的CPU、GPU等硬件，它们一秒能完成的浮点运算次数就是它们的FLOPS指标。

所以，简单来说，FLOPs（小写s）告诉你完成任务“需要多少工作量”，而FLOPS（大写S）告诉你你的“工具能多快完成工作”。

二、FLOPs在AI领域的“核心引擎”作用

AI，尤其是深度学习，其训练和推理过程本质上就是进行海量的浮点运算。 无论是图像识别、语音识别还是大型语言模型（如ChatGPT），都离不开巨大的计算量。

1. 衡量AI模型的“胃口”和“效率”

FLOPs是衡量机器学习模型计算复杂度的基本指标。

  • 模型复杂度：一个更复杂的AI模型，比如参数量巨大的大语言模型（LLM），在处理一个任务时，需要的总浮点运算次数（FLOPs）会非常高。 这好比一道菜的工序越多，所需的总操作次数就越多。
  • 模型效率：较低的FLOPs通常意味着模型运行速度更快，所需的计算能力更少。这对于资源有限的设备（如手机、边缘AI设备）尤其重要。 研究人员常常努力设计出FLOPs更低但性能依然强大的模型，例如EfficientNet等架构就致力于在不牺牲性能的情况下降低计算成本。

2. 评估硬件的“马力”和“速度”

电脑的CPU、特别是用于AI的图形处理器（GPU）或专用AI芯片（如TPU），它们的强大之处就在于能以极高的FLOPS进行浮点运算。

  • 训练模型：训练大型AI模型，就像是教一个学生学习海量的知识。这需要极其强大的FLOPS硬件，才能在合理的时间内完成。数据中心的大型硬件算力（TFLOPS级别）是训练模型的关键。
  • 推理应用：当模型训练好后，让它实际去“工作”（比如识别一张图片或回答一个问题），这个过程叫推理。推理也需要计算能力，但通常比训练所需的FLOPS低，更侧重于低延迟和高吞吐量。 移动设备上的AI应用（如人脸识别），就需要选择FLOPs较低的模型，以确保其在有限的硬件FLOPS下快速且不耗电地运行。

三、FLOPs单位：从个位数到“宇宙级别”

为了表示巨大的浮点运算次数和速度，FLOPs常以以下单位表示：

  • 单个FLOPs：一次浮点运算。
  • KFLOPS：千次浮点运算每秒 (10^3)。
  • MFLOPS：百万次浮点运算每秒 (10^6)。
  • GFLOPS：十亿次浮点运算每秒 (10^9)。
  • TFLOPS：万亿次浮点运算每秒 (10^12)。 许多高性能AI芯片的算力都以TFLOPS计。例如，苹果M2 GPU有3.6 TFLOPS的性能，而RTX 4090提供82.58 TFLOPS。
  • PFLOPS：千万亿次浮点运算每秒 (10^15)。
  • EFLOPS：百亿亿次浮点运算每秒 (10^18)。
  • ZFLOPS：十万亿亿次浮点运算每秒 (10^21)。

最新的信息显示，GPU的峰值算力已超过3000 TFLOPS (FP8)，而某些AI专用ASIC（如华为昇腾910）在FP16精度下可达640 TFLOPS。 这种巨大的算力，让AI模型训练能够在“月”级别的时间内完成万亿级模型的训练。

四、FLOPs与AI发展：算力即生产力

“算力是人工智能时代的‘核心引擎’”，它既是模型训练的“发动机”，也是推理落地的“变速器”。 没有强大的算力，再精妙的算法、再庞大的数据也只能停留在理论阶段。

  • 大模型时代：随着GPT-3、GPT-4等大型语言模型的崛起，AI模型的参数量呈指数级增长，其训练和运行对算力的需求也达到了前所未有的高度。例如，OpenAI训练GPT-4可能使用了2.5万块A100等效卡，总算力接近2.1×10²⁵ FLOPs。 这种庞大的计算需求直接推动了GPU等AI专用芯片以及高性能计算集群的发展。
  • 算力竞赛：当前各大科技公司在全球范围内展开“算力军备竞赛”，争相推出更高FLOPS的AI芯片和服务器。 例如，英伟达在AI芯片市场占据主导地位，其GPU凭借强大的并行计算能力和CUDA生态，成为AI训练的“绝对主力”。 AMD、谷歌TPU等也在不断发力，甚至云计算巨头也纷纷自研芯片以应对庞大的算力需求。
  • 效率优化：除了追求更高的FLOPS，如何在有限的算力下更高效地运行AI模型也成为关键。条件计算（Conditional Computation）如MoE（Mixture-of-Experts）架构，通过激活模型中的部分“专家”网络，可以在总参数量不变的情况下，显著降低单次推理的计算成本（FLOPs）。 这就像在同一个厨房里，不是所有厨师都同时做每一道菜，而是根据菜品需求，由擅长不同菜品的厨师协作完成，大大提高了整体效率。

五、结语

就像蒸汽机驱动了第一次工业革命，电力驱动了第二次工业革命一样，强大的算力，特别是以FLOPs为衡量核心的AI算力，正在成为推动人工智能甚至整个数字经济发展的“新引擎”。 理解FLOPs，就理解了AI世界最底层的动力源泉之一。它告诉我们，每一次AI的进步，都离不开背后成千上万、乃至于天文数字般的浮点运算的支撑。随着算力技术的不断突破，AI的未来也将拥有无限可能。

The “Fuel” of the AI World: Understanding FLOPs in Simple Terms

In the vast universe of Artificial Intelligence (AI), we often hear terms like “computing power” and “calculation volume”. They are like the foundations supporting skyscrapers, determining how far AI models can go and how powerful they can become. Beneath these foundations lies a core unit of measurement called FLOPs. It is not only the key to measuring the “strength” of AI models but also drives the entire AI field forward in its constant evolution.

What exactly is FLOPs? Why is it so important for AI? For non-professionals, we can understand it vividly through some daily life analogies.

I. FLOPs: The “Floating Point Recipe” and “Speedometer” of the AI World

When we mention FLOPs, we are actually referring to two related but slightly different concepts:

  1. FLOPs (Floating Point Operations): The lowercase “s” stands for plural, referring to the “number of floating-point operations”. It basically means the total number of such mathematical operations required in a single AI calculation task (such as letting AI recognize a picture). It measures calculation volume (workload).
  2. FLOPS (Floating Point Operations Per Second): The uppercase “S” stands for “Per Second”, referring to the “number of floating-point operations per second”. It means how many floating-point operations computer hardware can complete in one second. It measures computing speed or hardware performance.

To make it easier to understand, throughout this article, we mainly focus on the uppercase FLOPS to explain its significance in measuring computing power (hardware performance).

Daily Analogy: A Complex Cooking Feast

Imagine you are preparing an extremely sumptuous, multi-step dinner. This dinner requires a lot of chopping, stirring, weighing, heating, and other operations.

  • Floating Point Operations: It’s like every specific operation in the recipe, such as “mix 2.5 grams of salt into 1.5 liters of water”, or “mix flour and water in a ratio of 3.14:1”. These operations involve decimals and are relatively precise calculations. AI models, especially neural networks, perform a large number of addition, subtraction, multiplication, and division operations involving decimals when processing data. This is floating-point operation.

  • FLOPs (Total Operations): It is the total number of all chopping, stirring, weighing, and other operations required to complete this dinner. A more complex dish (such as an AI model with a huge number of parameters) requires more total operations. For example, for a large model like GPT-3, the FLOPs for a single inference can reach about 200 billion.

  • FLOPS (Operations Per Second): It is how many such operations you (or your kitchen helper) can complete in one second. If you are a Michelin chef who can chop several slices of vegetables and stir sauces several times in a second, then your FLOPS is very high, and your cooking efficiency is very fast. Conversely, if your movements are slow, FLOPS is low. Computer CPUs, GPUs, and other hardware, the number of floating-point operations they can complete in one second is their FLOPS indicator.

So, simply put, FLOPs (lowercase s) tells you “how much workload is needed” to complete the task, while FLOPS (uppercase S) tells you “how fast your tools can complete the work”.

II. The “Core Engine” Role of FLOPs in the AI Field

AI, especially deep learning, essentially involves massive floating-point operations in its training and inference processes. Whether it is image recognition, speech recognition, or large language models (such as ChatGPT), they are inseparable from huge calculation volumes.

1. Measuring the “Appetite” and “Efficiency” of AI Models

FLOPs is a basic indicator for measuring the computational complexity of machine learning models.

  • Model Complexity: A more complex AI model, such as a Large Language Model (LLM) with huge parameters, will require a very high total number of floating-point operations (FLOPs) when processing a task. This is like a dish with more procedures requiring more total operations.
  • Model Efficiency: Lower FLOPs usually mean that the model runs faster and requires less computing power. This is especially important for resource-constrained devices (such as mobile phones and edge AI devices). Researchers often strive to design models with lower FLOPs but still powerful performance. For example, architectures like EfficientNet are dedicated to reducing computational costs without sacrificing performance.

2. Assessing Hardware “Horsepower” and “Speed”

Computer CPUs, especially Graphics Processing Units (GPUs) or dedicated AI chips (such as TPUs) used for AI, are powerful because they can perform floating-point operations at extremely high FLOPS.

  • Training Models: Training large AI models is like teaching a student to learn massive amounts of knowledge. This requires extremely powerful FLOPS hardware to complete in a reasonable time. The large-scale hardware computing power (TFLOPS level) in data centers is key to training models.
  • Inference Application: After the model is trained, letting it actually “work” (such as recognizing a picture or answering a question), this process is called inference. Inference also requires computing power, but usually requires lower FLOPS than training, focusing more on low latency and high throughput. AI applications on mobile devices (such as face recognition) need to choose models with lower FLOPs to ensure they run quickly and without consuming too much power under limited hardware FLOPS.

III. FLOPs Units: From Single Digits to “Cosmic Level”

To represent huge floating-point operation counts and speeds, FLOPs are often expressed in the following units:

  • Single FLOPs: One floating-point operation.
  • KFLOPS: Kilo Floating Point Operations Per Second (10310^3).
  • MFLOPS: Mega Floating Point Operations Per Second (10610^6).
  • GFLOPS: Giga Floating Point Operations Per Second (10910^9).
  • TFLOPS: Tera Floating Point Operations Per Second (101210^{12}). Many high-performance AI chips calculate computing power in TFLOPS. For example, the Apple M2 GPU has a performance of 3.6 TFLOPS, while the RTX 4090 offers 82.58 TFLOPS.
  • PFLOPS: Peta Floating Point Operations Per Second (101510^{15}).
  • EFLOPS: Exa Floating Point Operations Per Second (101810^{18}).
  • ZFLOPS: Zetta Floating Point Operations Per Second (102110^{21}).

Latest information shows that the peak computing power of GPUs has exceeded 3000 TFLOPS (FP8), while certain AI-specific ASICs (such as Huawei Ascend 910) can reach 640 TFLOPS at FP16 precision. This huge computing power allows AI model training to complete the training of trillion-level models within a “month” level timeframe.

IV. FLOPs and AI Development: Computing Power is Productivity

“Computing power is the ‘core engine’ of the artificial intelligence era.” It is both the “engine” for model training and the “transmission” for inference deployment. Without powerful computing power, no matter how exquisite the algorithm or how huge the data is, it can only stay at the theoretical stage.

  • The Era of Large Models: With the rise of large language models like GPT-3 and GPT-4, the number of parameters of AI models has grown exponentially, and the demand for computing power for training and running them has reached unprecedented heights. For example, OpenAI may have used 25,000 A100 equivalent cards to train GPT-4, with a total computing power close to 2.1×10252.1 \times 10^{25} FLOPs. This huge computing demand has directly promoted the development of AI-specific chips such as GPUs and high-performance computing clusters.
  • Computing Power Race: Currently, major technology companies are launching a “computing power arms race” globally, scrambling to launch AI chips and servers with higher FLOPS. For example, NVIDIA dominates the AI chip market, and its GPUs have become the “absolute main force” for AI training with powerful parallel computing capabilities and the CUDA ecosystem. AMD, Google TPU, etc., are also constantly exerting force, and even cloud computing giants are developing their own chips to cope with huge computing power demands.
  • Efficiency Optimization: In addition to pursuing higher FLOPS, how to run AI models more efficiently with limited computing power has also become key. Conditional Computation such as MoE (Mixture-of-Experts) architecture, by activating part of the “expert” networks in the model, can significantly reduce the computational cost (FLOPs) of a single inference while the total number of parameters remains unchanged. This is like in the same kitchen, not all chefs cook every dish at the same time, but chefs who are good at different dishes collaborate according to the needs of the dishes, greatly improving overall efficiency.

V. Conclusion

Just as the steam engine drove the First Industrial Revolution and electricity drove the Second Industrial Revolution, powerful computing power, especially AI computing power measured by FLOPs, is becoming the “new engine” driving the development of artificial intelligence and even the entire digital economy. Understanding FLOPs means understanding one of the most fundamental power sources in the AI world. It tells us that every progress in AI is inseparable from the support of thousands, even astronomical numbers of floating-point operations behind it. With the continuous breakthrough of computing power technology, the future of AI will also have infinite possibilities.