更新
This commit is contained in:
@@ -0,0 +1,204 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9ed6d9cb",
|
||||
"metadata": {
|
||||
"origin_pos": 0
|
||||
},
|
||||
"source": [
|
||||
"# 前向传播、反向传播和计算图\n",
|
||||
":label:`sec_backprop`\n",
|
||||
"\n",
|
||||
"我们已经学习了如何用小批量随机梯度下降训练模型。\n",
|
||||
"然而当实现该算法时，我们只考虑了通过*前向传播*（forward propagation）所涉及的计算。\n",
|
||||
"在计算梯度时，我们只调用了深度学习框架提供的反向传播函数，而不知其所以然。\n",
|
||||
"\n",
|
||||
"梯度的自动计算（自动微分）大大简化了深度学习算法的实现。\n",
|
||||
"在自动微分之前，即使是对复杂模型的微小调整也需要手工重新计算复杂的导数，\n",
|
||||
"学术论文也不得不分配大量页面来推导更新规则。\n",
|
||||
"本节将通过一些基本的数学和计算图，\n",
|
||||
"深入探讨*反向传播*的细节。\n",
|
||||
"首先，我们将重点放在带权重衰减（$L_2$正则化）的单隐藏层多层感知机上。\n",
|
||||
"\n",
|
||||
"## 前向传播\n",
|
||||
"\n",
|
||||
"*前向传播*（forward propagation或forward pass）\n",
|
||||
"指的是：按顺序（从输入层到输出层）计算和存储神经网络中每层的结果。\n",
|
||||
"\n",
|
||||
"我们将一步步研究单隐藏层神经网络的机制，\n",
|
||||
"为了简单起见，我们假设输入样本是 $\\mathbf{x}\\in \\mathbb{R}^d$，\n",
|
||||
"并且我们的隐藏层不包括偏置项。\n",
|
||||
"这里的中间变量是：\n",
|
||||
"\n",
|
||||
"$$\\mathbf{z}= \\mathbf{W}^{(1)} \\mathbf{x},$$\n",
|
||||
"\n",
|
||||
"其中$\\mathbf{W}^{(1)} \\in \\mathbb{R}^{h \\times d}$\n",
|
||||
"是隐藏层的权重参数。\n",
|
||||
"将中间变量$\\mathbf{z}\\in \\mathbb{R}^h$通过激活函数$\\phi$后，\n",
|
||||
"我们得到长度为$h$的隐藏激活向量：\n",
|
||||
"\n",
|
||||
"$$\\mathbf{h}= \\phi (\\mathbf{z}).$$\n",
|
||||
"\n",
|
||||
"隐藏变量$\\mathbf{h}$也是一个中间变量。\n",
|
||||
"假设输出层的参数只有权重$\\mathbf{W}^{(2)} \\in \\mathbb{R}^{q \\times h}$，\n",
|
||||
"我们可以得到输出层变量，它是一个长度为$q$的向量：\n",
|
||||
"\n",
|
||||
"$$\\mathbf{o}= \\mathbf{W}^{(2)} \\mathbf{h}.$$\n",
|
||||
"\n",
|
||||
"假设损失函数为$l$，样本标签为$y$，我们可以计算单个数据样本的损失项，\n",
|
||||
"\n",
|
||||
"$$L = l(\\mathbf{o}, y).$$\n",
|
||||
"\n",
|
||||
"根据$L_2$正则化的定义，给定超参数$\\lambda$，正则化项为\n",
|
||||
"\n",
|
||||
"$$s = \\frac{\\lambda}{2} \\left(\\|\\mathbf{W}^{(1)}\\|_F^2 + \\|\\mathbf{W}^{(2)}\\|_F^2\\right),$$\n",
|
||||
":eqlabel:`eq_forward-s`\n",
|
||||
"\n",
|
||||
"其中矩阵的Frobenius范数是将矩阵展平为向量后应用的$L_2$范数。\n",
|
||||
"最后，模型在给定数据样本上的正则化损失为：\n",
|
||||
"\n",
|
||||
"$$J = L + s.$$\n",
|
||||
"\n",
|
||||
"在下面的讨论中，我们将$J$称为*目标函数*（objective function）。\n",
|
||||
"\n",
|
||||
"## 前向传播计算图\n",
|
||||
"\n",
|
||||
"绘制*计算图*有助于我们可视化计算中操作符和变量的依赖关系。\n",
|
||||
" :numref:`fig_forward` 是与上述简单网络相对应的计算图，\n",
|
||||
" 其中正方形表示变量，圆圈表示操作符。\n",
|
||||
" 左下角表示输入，右上角表示输出。\n",
|
||||
" 注意显示数据流的箭头方向主要是向右和向上的。\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":label:`fig_forward`\n",
|
||||
"\n",
|
||||
"## 反向传播\n",
|
||||
"\n",
|
||||
"*反向传播*（backward propagation或backpropagation）指的是计算神经网络参数梯度的方法。\n",
|
||||
"简言之，该方法根据微积分中的*链式规则*，按相反的顺序从输出层到输入层遍历网络。\n",
|
||||
"该算法存储了计算某些参数梯度时所需的任何中间变量（偏导数）。\n",
|
||||
"假设我们有函数$\\mathsf{Y}=f(\\mathsf{X})$和$\\mathsf{Z}=g(\\mathsf{Y})$，\n",
|
||||
"其中输入和输出$\\mathsf{X}, \\mathsf{Y}, \\mathsf{Z}$是任意形状的张量。\n",
|
||||
"利用链式法则，我们可以计算$\\mathsf{Z}$关于$\\mathsf{X}$的导数\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial \\mathsf{Z}}{\\partial \\mathsf{X}} = \\text{prod}\\left(\\frac{\\partial \\mathsf{Z}}{\\partial \\mathsf{Y}}, \\frac{\\partial \\mathsf{Y}}{\\partial \\mathsf{X}}\\right).$$\n",
|
||||
"\n",
|
||||
"在这里，我们使用$\\text{prod}$运算符在执行必要的操作（如换位和交换输入位置）后将其参数相乘。\n",
|
||||
"对于向量，这很简单，它只是矩阵-矩阵乘法。\n",
|
||||
"对于高维张量，我们使用适当的对应项。\n",
|
||||
"运算符$\\text{prod}$指代了所有的这些符号。\n",
|
||||
"\n",
|
||||
"回想一下，在计算图 :numref:`fig_forward`中的单隐藏层简单网络的参数是\n",
|
||||
"$\\mathbf{W}^{(1)}$和$\\mathbf{W}^{(2)}$。\n",
|
||||
"反向传播的目的是计算梯度$\\partial J/\\partial \\mathbf{W}^{(1)}$和\n",
|
||||
"$\\partial J/\\partial \\mathbf{W}^{(2)}$。\n",
|
||||
"为此，我们应用链式法则，依次计算每个中间变量和参数的梯度。\n",
|
||||
"计算的顺序与前向传播中执行的顺序相反，因为我们需要从计算图的结果开始，并朝着参数的方向努力。第一步是计算目标函数$J=L+s$相对于损失项$L$和正则项$s$的梯度。\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial J}{\\partial L} = 1 \\; \\text{and} \\; \\frac{\\partial J}{\\partial s} = 1.$$\n",
|
||||
"\n",
|
||||
"接下来，我们根据链式法则计算目标函数关于输出层变量$\\mathbf{o}$的梯度：\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial J}{\\partial \\mathbf{o}}\n",
|
||||
"= \\text{prod}\\left(\\frac{\\partial J}{\\partial L}, \\frac{\\partial L}{\\partial \\mathbf{o}}\\right)\n",
|
||||
"= \\frac{\\partial L}{\\partial \\mathbf{o}}\n",
|
||||
"\\in \\mathbb{R}^q.\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"接下来，我们计算正则化项相对于两个参数的梯度：\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial s}{\\partial \\mathbf{W}^{(1)}} = \\lambda \\mathbf{W}^{(1)}\n",
|
||||
"\\; \\text{and} \\;\n",
|
||||
"\\frac{\\partial s}{\\partial \\mathbf{W}^{(2)}} = \\lambda \\mathbf{W}^{(2)}.$$\n",
|
||||
"\n",
|
||||
"现在我们可以计算最接近输出层的模型参数的梯度\n",
|
||||
"$\\partial J/\\partial \\mathbf{W}^{(2)} \\in \\mathbb{R}^{q \\times h}$。\n",
|
||||
"使用链式法则得出：\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial J}{\\partial \\mathbf{W}^{(2)}}= \\text{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{o}}, \\frac{\\partial \\mathbf{o}}{\\partial \\mathbf{W}^{(2)}}\\right) + \\text{prod}\\left(\\frac{\\partial J}{\\partial s}, \\frac{\\partial s}{\\partial \\mathbf{W}^{(2)}}\\right)= \\frac{\\partial J}{\\partial \\mathbf{o}} \\mathbf{h}^\\top + \\lambda \\mathbf{W}^{(2)}.$$\n",
|
||||
":eqlabel:`eq_backprop-J-h`\n",
|
||||
"\n",
|
||||
"为了获得关于$\\mathbf{W}^{(1)}$的梯度，我们需要继续沿着输出层到隐藏层反向传播。\n",
|
||||
"关于隐藏层输出的梯度$\\partial J/\\partial \\mathbf{h} \\in \\mathbb{R}^h$由下式给出：\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial J}{\\partial \\mathbf{h}}\n",
|
||||
"= \\text{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{o}}, \\frac{\\partial \\mathbf{o}}{\\partial \\mathbf{h}}\\right)\n",
|
||||
"= {\\mathbf{W}^{(2)}}^\\top \\frac{\\partial J}{\\partial \\mathbf{o}}.\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"由于激活函数$\\phi$是按元素计算的，\n",
|
||||
"计算中间变量$\\mathbf{z}$的梯度$\\partial J/\\partial \\mathbf{z} \\in \\mathbb{R}^h$\n",
|
||||
"需要使用按元素乘法运算符，我们用$\\odot$表示：\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial J}{\\partial \\mathbf{z}}\n",
|
||||
"= \\text{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{h}}, \\frac{\\partial \\mathbf{h}}{\\partial \\mathbf{z}}\\right)\n",
|
||||
"= \\frac{\\partial J}{\\partial \\mathbf{h}} \\odot \\phi'\\left(\\mathbf{z}\\right).\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"最后，我们可以得到最接近输入层的模型参数的梯度\n",
|
||||
"$\\partial J/\\partial \\mathbf{W}^{(1)} \\in \\mathbb{R}^{h \\times d}$。\n",
|
||||
"根据链式法则，我们得到：\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial J}{\\partial \\mathbf{W}^{(1)}}\n",
|
||||
"= \\text{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{z}}, \\frac{\\partial \\mathbf{z}}{\\partial \\mathbf{W}^{(1)}}\\right) + \\text{prod}\\left(\\frac{\\partial J}{\\partial s}, \\frac{\\partial s}{\\partial \\mathbf{W}^{(1)}}\\right)\n",
|
||||
"= \\frac{\\partial J}{\\partial \\mathbf{z}} \\mathbf{x}^\\top + \\lambda \\mathbf{W}^{(1)}.\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"## 训练神经网络\n",
|
||||
"\n",
|
||||
"在训练神经网络时，前向传播和反向传播相互依赖。\n",
|
||||
"对于前向传播，我们沿着依赖的方向遍历计算图并计算其路径上的所有变量。\n",
|
||||
"然后将这些用于反向传播，其中计算顺序与计算图的相反。\n",
|
||||
"\n",
|
||||
"以上述简单网络为例：一方面，在前向传播期间计算正则项\n",
|
||||
" :eqref:`eq_forward-s`取决于模型参数$\\mathbf{W}^{(1)}$和\n",
|
||||
"$\\mathbf{W}^{(2)}$的当前值。\n",
|
||||
"它们是由优化算法根据最近迭代的反向传播给出的。\n",
|
||||
"另一方面，反向传播期间参数 :eqref:`eq_backprop-J-h`的梯度计算，\n",
|
||||
"取决于由前向传播给出的隐藏变量$\\mathbf{h}$的当前值。\n",
|
||||
"\n",
|
||||
"因此，在训练神经网络时，在初始化模型参数后，\n",
|
||||
"我们交替使用前向传播和反向传播，利用反向传播给出的梯度来更新模型参数。\n",
|
||||
"注意，反向传播重复利用前向传播中存储的中间值，以避免重复计算。\n",
|
||||
"带来的影响之一是我们需要保留中间值，直到反向传播完成。\n",
|
||||
"这也是训练比单纯的预测需要更多的内存（显存）的原因之一。\n",
|
||||
"此外，这些中间值的大小与网络层的数量和批量的大小大致成正比。\n",
|
||||
"因此，使用更大的批量来训练更深层次的网络更容易导致*内存不足*（out of memory）错误。\n",
|
||||
"\n",
|
||||
"## 小结\n",
|
||||
"\n",
|
||||
"* 前向传播在神经网络定义的计算图中按顺序计算和存储中间变量，它的顺序是从输入层到输出层。\n",
|
||||
"* 反向传播按相反的顺序（从输出层到输入层）计算和存储神经网络的中间变量和参数的梯度。\n",
|
||||
"* 在训练深度学习模型时，前向传播和反向传播是相互依赖的。\n",
|
||||
"* 训练比预测需要更多的内存。\n",
|
||||
"\n",
|
||||
"## 练习\n",
|
||||
"\n",
|
||||
"1. 假设一些标量函数$\\mathbf{X}$的输入$\\mathbf{X}$是$n \\times m$矩阵。$f$相对于$\\mathbf{X}$的梯度维数是多少？\n",
|
||||
"1. 向本节中描述的模型的隐藏层添加偏置项（不需要在正则化项中包含偏置项）。\n",
|
||||
" 1. 画出相应的计算图。\n",
|
||||
" 1. 推导正向和反向传播方程。\n",
|
||||
"1. 计算本节所描述的模型，用于训练和预测的内存占用。\n",
|
||||
"1. 假设想计算二阶导数。计算图发生了什么？预计计算需要多长时间？\n",
|
||||
"1. 假设计算图对当前拥有的GPU来说太大了。\n",
|
||||
" 1. 请试着把它划分到多个GPU上。\n",
|
||||
" 1. 与小批量训练相比，有哪些优点和缺点？\n",
|
||||
"\n",
|
||||
"[Discussions](https://discuss.d2l.ai/t/5769)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"required_libs": []
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user