神经网络的层级结构和非线性激活函数：为什么能学习复杂函数？
层级结构：逐层抽象，构建复杂映射
想象一下，我们想让计算机识别一张猫的图片。我们可以把这张图片看作是一个巨大的数字矩阵，每个数字代表一个像素点的颜色值。要让计算机理解这张图片，我们不能直接把这些数字一股脑地丢给它，而是需要逐步提取出图片中的关键特征。
- 输入层：最底层，接收原始数据（比如图片的像素值）。
- 隐藏层：中间层，对数据进行逐层抽象。第一层隐藏层可能提取出一些简单的特征，比如边缘、颜色块；第二层隐藏层则可能基于这些简单特征，提取出更复杂的特征，比如眼睛、鼻子等。
- 输出层：最后一层，给出最终的预测结果（比如“猫”或“狗”）。
通过这种层级结构，神经网络可以逐步从原始数据中提取出越来越抽象的特征，最终实现对复杂数据的分类或回归。
非线性激活函数：打破线性限制，增强表达能力
如果神经网络的每一层都只进行线性变换，那么无论叠加多少层，整个网络也只能表达线性函数。这显然不能满足我们对复杂函数的拟合需求。
- 线性变换：简单的加权求和，只能表示直线或平面。
- 非线性激活函数：在加权求和之后，引入一个非线性函数，将线性空间映射到非线性空间。常见的激活函数有ReLU、Sigmoid、Tanh等。
非线性激活函数的作用：
- 引入非线性：使得神经网络能够拟合任意复杂的非线性函数。
- 增加模型的表达能力：让模型能够学习到更复杂的特征。
- 提高模型的拟合能力：使得模型能够更好地拟合训练数据。
总结
- 层级结构：通过逐层抽象，将复杂问题分解为一系列简单的子问题，逐步提取出数据的深层特征。
- 非线性激活函数：打破线性限制，增强模型的表达能力，使得模型能够拟合任意复杂的函数。
两者结合，使得神经网络具有强大的学习能力，能够从大量数据中学习到复杂的模式，并应用于各种任务，如图像分类、自然语言处理、语音识别等。
形象比喻
我们可以把神经网络想象成一个工厂。输入层是原材料，隐藏层是加工车间，每一层都对原材料进行加工，提取出更精细的部件。最终，输出层将这些部件组装成一个完整的产品。非线性激活函数就像是加工车间的机器，它们为产品增加了多样性和复杂性。
进一步思考
- 深度：神经网络的层数越多，表示它能够提取的特征越抽象，模型的表达能力越强。
- 宽度：每一层神经元的数量越多，表示模型能够学习到的特征越丰富。
- 超参数：学习率、优化器等超参数对模型的性能有重要影响。
- 正则化：L1正则化、L2正则化等可以防止过拟合，提高模型的泛化能力。
title: How neural networks learn complex functions
date: 2025-03-04 10:33:34
tags:
- Neural Network
Hierarchical Structure of Neural Networks and Non-linear Activation Functions: Why Can They Learn Complex Functions?
Hierarchical Structure: Layer-by-Layer Abstraction, Constructing Complex Mappings
Imagine we want a computer to recognize a picture of a cat. We can view this picture as a huge matrix of numbers, where each number represents the color value of a pixel. To let the computer understand this picture, we can’t directly throw these numbers at it all at once; instead, we need to extract key features from the picture step by step.
- Input Layer: The lowest layer, receiving raw data (such as pixel values of an image).
- Hidden Layers: Intermediate layers, performing layer-by-layer abstraction on data. The first hidden layer might extract some simple features, such as edges and color patches; the second hidden layer might extract more complex features based on these simple features, such as eyes, noses, etc.
- Output Layer: The last layer, providing the final prediction result (such as “cat” or “dog”).
Through this hierarchical structure, neural networks can gradually extract increasingly abstract features from raw data, ultimately achieving classification or regression of complex data.
Non-linear Activation Functions: Breaking Linear Limitations, Enhancing Expressive Power
If every layer of a neural network only performs linear transformations, then no matter how many layers are stacked, the entire network can only express linear functions. This obviously cannot meet our needs for fitting complex functions.
- Linear Transformation: Simple weighted summation, can only represent straight lines or planes.
- Non-linear Activation Function: After weighted summation, a non-linear function is introduced to map the linear space to a non-linear space. Common activation functions include ReLU, Sigmoid, Tanh, etc.
Role of Non-linear Activation Functions:
- Introducing Non-linearity: Enabling neural networks to fit arbitrarily complex non-linear functions.
- Increasing Model’s Expressive Power: Allowing the model to learn more complex features.
- Improving Model’s Fitting Ability: Enabling the model to better fit training data.
Summary
- Hierarchical Structure: By layer-by-layer abstraction, complex problems are decomposed into a series of simple sub-problems, gradually extracting deep features of data.
- Non-linear Activation Functions: Breaking linear limitations, enhancing the model’s expressive power, enabling the model to fit arbitrarily complex functions.
Combining Both gives neural networks powerful learning capabilities, enabling them to learn complex patterns from large amounts of data and apply them to various tasks, such as image classification, natural language processing, speech recognition, etc.
Vivid Metaphor
We can imagine a neural network as a factory. The input layer is the raw material, and the hidden layers are processing workshops. Each layer processes the raw materials, extracting finer parts. Finally, the output layer assembles these parts into a complete product. Non-linear activation functions are like machines in the processing workshops; they add diversity and complexity to the products.
Further Thinking
- Depth: The more layers a neural network has, the more abstract the features it can extract, and the stronger the model’s expressive power.
- Width: The more neurons in each layer, the richer the features the model can learn.
- Hyperparameters: Hyperparameters like learning rate and optimizer significantly impact model performance.
- Regularization: L1 regularization, L2 regularization, etc., can prevent overfitting and improve the model’s generalization ability.