{
"cells": [
{
"cell_type": "markdown",
"id": "3b3e0a72",
"metadata": {
"origin_pos": 0
},
"source": [
"# 线性回归的从零开始实现\n",
":label:`sec_linear_scratch`\n",
"\n",
"在了解线性回归的关键思想之后，我们可以开始通过代码来动手实现线性回归了。\n",
"在这一节中，(**我们将从零开始实现整个方法，\n",
"包括数据流水线、模型、损失函数和小批量随机梯度下降优化器**)。\n",
"虽然现代的深度学习框架几乎可以自动化地进行所有这些工作，但从零开始实现可以确保我们真正知道自己在做什么。\n",
"同时，了解更细致的工作原理将方便我们自定义模型、自定义层或自定义损失函数。\n",
"在这一节中，我们将只使用张量和自动求导。\n",
"在之后的章节中，我们会充分利用深度学习框架的优势，介绍更简洁的实现方式。\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c6f4cd71",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:01.462670Z",
"iopub.status.busy": "2023-08-18T07:02:01.461918Z",
"iopub.status.idle": "2023-08-18T07:02:04.547486Z",
"shell.execute_reply": "2023-08-18T07:02:04.546281Z"
},
"origin_pos": 2,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import random\n",
"import torch\n",
"from d2l import torch as d2l"
]
},
{
"cell_type": "markdown",
"id": "18283191",
"metadata": {
"origin_pos": 5
},
"source": [
"## 生成数据集\n",
"\n",
"为了简单起见，我们将[**根据带有噪声的线性模型构造一个人造数据集。**]\n",
"我们的任务是使用这个有限样本的数据集来恢复这个模型的参数。\n",
"我们将使用低维数据，这样可以很容易地将其可视化。\n",
"在下面的代码中，我们生成一个包含1000个样本的数据集，\n",
"每个样本包含从标准正态分布中采样的2个特征。\n",
"我们的合成数据集是一个矩阵$\\mathbf{X}\\in \\mathbb{R}^{1000 \\times 2}$。\n",
"\n",
"(**我们使用线性模型参数$\\mathbf{w} = [2, -3.4]^\\top$、$b = 4.2$\n",
"和噪声项$\\epsilon$生成数据集及其标签：\n",
"\n",
"$$\\mathbf{y}= \\mathbf{X} \\mathbf{w} + b + \\mathbf\\epsilon.$$\n",
"**)\n",
"\n",
"$\\epsilon$可以视为模型预测和标签时的潜在观测误差。\n",
"在这里我们认为标准假设成立，即$\\epsilon$服从均值为0的正态分布。\n",
"为了简化问题，我们将标准差设为0.01。\n",
"下面的代码生成合成数据集。\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "54efeafe",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.553500Z",
"iopub.status.busy": "2023-08-18T07:02:04.552544Z",
"iopub.status.idle": "2023-08-18T07:02:04.560226Z",
"shell.execute_reply": "2023-08-18T07:02:04.559125Z"
},
"origin_pos": 6,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def synthetic_data(w, b, num_examples): #@save\n",
" \"\"\"生成y=Xw+b+噪声\"\"\"\n",
" X = torch.normal(0, 1, (num_examples, len(w)))\n",
" y = torch.matmul(X, w) + b\n",
" y += torch.normal(0, 0.01, y.shape)\n",
" return X, y.reshape((-1, 1))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1e60261c",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.564932Z",
"iopub.status.busy": "2023-08-18T07:02:04.564190Z",
"iopub.status.idle": "2023-08-18T07:02:04.575309Z",
"shell.execute_reply": "2023-08-18T07:02:04.574216Z"
},
"origin_pos": 8,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"true_w = torch.tensor([2, -3.4])\n",
"true_b = 4.2\n",
"features, labels = synthetic_data(true_w, true_b, 1000)"
]
},
{
"cell_type": "markdown",
"id": "772256cb",
"metadata": {
"origin_pos": 9
},
"source": [
"注意，[**`features`中的每一行都包含一个二维数据样本，\n",
"`labels`中的每一行都包含一维标签值（一个标量）**]。\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ec13e4f8",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.580067Z",
"iopub.status.busy": "2023-08-18T07:02:04.579449Z",
"iopub.status.idle": "2023-08-18T07:02:04.587391Z",
"shell.execute_reply": "2023-08-18T07:02:04.586306Z"
},
"origin_pos": 10,
"tab": [
"pytorch"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"features: tensor([1.4632, 0.5511]) \n",
"label: tensor([5.2498])\n"
]
}
],
"source": [
"print('features:', features[0],'\\nlabel:', labels[0])"
]
},
{
"cell_type": "markdown",
"id": "6b8c624b",
"metadata": {
"origin_pos": 11
},
"source": [
"通过生成第二个特征`features[:, 1]`和`labels`的散点图，\n",
"可以直观观察到两者之间的线性关系。\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "53ef493c",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.592131Z",
"iopub.status.busy": "2023-08-18T07:02:04.591402Z",
"iopub.status.idle": "2023-08-18T07:02:04.829190Z",
"shell.execute_reply": "2023-08-18T07:02:04.827927Z"
},
"origin_pos": 12,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"d2l.set_figsize()\n",
"d2l.plt.scatter(features[:, (1)].detach().numpy(), labels.detach().numpy(), 1);"
]
},
{
"cell_type": "markdown",
"id": "b032f500",
"metadata": {
"origin_pos": 13
},
"source": [
"## 读取数据集\n",
"\n",
"回想一下，训练模型时要对数据集进行遍历，每次抽取一小批量样本，并使用它们来更新我们的模型。\n",
"由于这个过程是训练机器学习算法的基础，所以有必要定义一个函数，\n",
"该函数能打乱数据集中的样本并以小批量方式获取数据。\n",
"\n",
"在下面的代码中，我们[**定义一个`data_iter`函数，\n",
"该函数接收批量大小、特征矩阵和标签向量作为输入，生成大小为`batch_size`的小批量**]。\n",
"每个小批量包含一组特征和标签。\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "3da34ac6",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.839342Z",
"iopub.status.busy": "2023-08-18T07:02:04.838682Z",
"iopub.status.idle": "2023-08-18T07:02:04.846329Z",
"shell.execute_reply": "2023-08-18T07:02:04.845247Z"
},
"origin_pos": 14,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def data_iter(batch_size, features, labels):\n",
" num_examples = len(features)\n",
" indices = list(range(num_examples))\n",
" # 这些样本是随机读取的，没有特定的顺序\n",
" random.shuffle(indices)\n",
" for i in range(0, num_examples, batch_size):\n",
" batch_indices = torch.tensor(\n",
" indices[i: min(i + batch_size, num_examples)])\n",
" yield features[batch_indices], labels[batch_indices]"
]
},
{
"cell_type": "markdown",
"id": "52e08a78",
"metadata": {
"origin_pos": 16
},
"source": [
"通常，我们利用GPU并行运算的优势，处理合理大小的“小批量”。\n",
"每个样本都可以并行地进行模型计算，且每个样本损失函数的梯度也可以被并行计算。\n",
"GPU可以在处理几百个样本时，所花费的时间不比处理一个样本时多太多。\n",
"\n",
"我们直观感受一下小批量运算：读取第一个小批量数据样本并打印。\n",
"每个批量的特征维度显示批量大小和输入特征数。\n",
"同样的，批量的标签形状与`batch_size`相等。\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "1dce0726",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.851066Z",
"iopub.status.busy": "2023-08-18T07:02:04.850456Z",
"iopub.status.idle": "2023-08-18T07:02:04.859860Z",
"shell.execute_reply": "2023-08-18T07:02:04.858756Z"
},
"origin_pos": 17,
"tab": [
"pytorch"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ 0.3934, 2.5705],\n",
" [ 0.5849, -0.7124],\n",
" [ 0.1008, 0.6947],\n",
" [-0.4493, -0.9037],\n",
" [ 2.3104, -0.2798],\n",
" [-0.0173, -0.2552],\n",
" [ 0.1963, -0.5445],\n",
" [-1.0580, -0.5180],\n",
" [ 0.8417, -1.5547],\n",
" [-0.6316, 0.9732]]) \n",
" tensor([[-3.7623],\n",
" [ 7.7852],\n",
" [ 2.0443],\n",
" [ 6.3767],\n",
" [ 9.7776],\n",
" [ 5.0301],\n",
" [ 6.4541],\n",
" [ 3.8407],\n",
" [11.1396],\n",
" [-0.3836]])\n"
]
}
],
"source": [
"batch_size = 10\n",
"\n",
"for X, y in data_iter(batch_size, features, labels):\n",
" print(X, '\\n', y)\n",
" break"
]
},
{
"cell_type": "markdown",
"id": "d86e62d7",
"metadata": {
"origin_pos": 18
},
"source": [
"当我们运行迭代时，我们会连续地获得不同的小批量，直至遍历完整个数据集。\n",
"上面实现的迭代对教学来说很好，但它的执行效率很低，可能会在实际问题上陷入麻烦。\n",
"例如，它要求我们将所有数据加载到内存中，并执行大量的随机内存访问。\n",
"在深度学习框架中实现的内置迭代器效率要高得多，\n",
"它可以处理存储在文件中的数据和数据流提供的数据。\n",
"\n",
"## 初始化模型参数\n",
"\n",
"[**在我们开始用小批量随机梯度下降优化我们的模型参数之前**]，\n",
"(**我们需要先有一些参数**)。\n",
"在下面的代码中，我们通过从均值为0、标准差为0.01的正态分布中采样随机数来初始化权重，\n",
"并将偏置初始化为0。\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "12c51289",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.864457Z",
"iopub.status.busy": "2023-08-18T07:02:04.863853Z",
"iopub.status.idle": "2023-08-18T07:02:04.869983Z",
"shell.execute_reply": "2023-08-18T07:02:04.868859Z"
},
"origin_pos": 20,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"w = torch.normal(0, 0.01, size=(2,1), requires_grad=True)\n",
"b = torch.zeros(1, requires_grad=True)"
]
},
{
"cell_type": "markdown",
"id": "c59d5d68",
"metadata": {
"origin_pos": 23
},
"source": [
"在初始化参数之后，我们的任务是更新这些参数，直到这些参数足够拟合我们的数据。\n",
"每次更新都需要计算损失函数关于模型参数的梯度。\n",
"有了这个梯度，我们就可以向减小损失的方向更新每个参数。\n",
"因为手动计算梯度很枯燥而且容易出错，所以没有人会手动计算梯度。\n",
"我们使用 :numref:`sec_autograd`中引入的自动微分来计算梯度。\n",
"\n",
"## 定义模型\n",
"\n",
"接下来，我们必须[**定义模型，将模型的输入和参数同模型的输出关联起来。**]\n",
"回想一下，要计算线性模型的输出，\n",
"我们只需计算输入特征$\\mathbf{X}$和模型权重$\\mathbf{w}$的矩阵-向量乘法后加上偏置$b$。\n",
"注意，上面的$\\mathbf{Xw}$是一个向量，而$b$是一个标量。\n",
"回想一下 :numref:`subsec_broadcasting`中描述的广播机制：\n",
"当我们用一个向量加一个标量时，标量会被加到向量的每个分量上。\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b8b29b19",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.874642Z",
"iopub.status.busy": "2023-08-18T07:02:04.874004Z",
"iopub.status.idle": "2023-08-18T07:02:04.879521Z",
"shell.execute_reply": "2023-08-18T07:02:04.878471Z"
},
"origin_pos": 24,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def linreg(X, w, b): #@save\n",
" \"\"\"线性回归模型\"\"\"\n",
" return torch.matmul(X, w) + b"
]
},
{
"cell_type": "markdown",
"id": "6b7765ef",
"metadata": {
"origin_pos": 25
},
"source": [
"## [**定义损失函数**]\n",
"\n",
"因为需要计算损失函数的梯度，所以我们应该先定义损失函数。\n",
"这里我们使用 :numref:`sec_linear_regression`中描述的平方损失函数。\n",
"在实现中，我们需要将真实值`y`的形状转换为和预测值`y_hat`的形状相同。\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "3dda15c7",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.884156Z",
"iopub.status.busy": "2023-08-18T07:02:04.883559Z",
"iopub.status.idle": "2023-08-18T07:02:04.889065Z",
"shell.execute_reply": "2023-08-18T07:02:04.887964Z"
},
"origin_pos": 26,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def squared_loss(y_hat, y): #@save\n",
" \"\"\"均方损失\"\"\"\n",
" return (y_hat - y.reshape(y_hat.shape)) ** 2 / 2"
]
},
{
"cell_type": "markdown",
"id": "56969029",
"metadata": {
"origin_pos": 27
},
"source": [
"## (**定义优化算法**)\n",
"\n",
"正如我们在 :numref:`sec_linear_regression`中讨论的，线性回归有解析解。\n",
"尽管线性回归有解析解，但本书中的其他模型却没有。\n",
"这里我们介绍小批量随机梯度下降。\n",
"\n",
"在每一步中，使用从数据集中随机抽取的一个小批量，然后根据参数计算损失的梯度。\n",
"接下来，朝着减少损失的方向更新我们的参数。\n",
"下面的函数实现小批量随机梯度下降更新。\n",
"该函数接受模型参数集合、学习速率和批量大小作为输入。每\n",
"一步更新的大小由学习速率`lr`决定。\n",
"因为我们计算的损失是一个批量样本的总和，所以我们用批量大小（`batch_size`）\n",
"来规范化步长，这样步长大小就不会取决于我们对批量大小的选择。\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "8f92242d",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.893665Z",
"iopub.status.busy": "2023-08-18T07:02:04.892999Z",
"iopub.status.idle": "2023-08-18T07:02:04.899100Z",
"shell.execute_reply": "2023-08-18T07:02:04.898003Z"
},
"origin_pos": 29,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def sgd(params, lr, batch_size): #@save\n",
" \"\"\"小批量随机梯度下降\"\"\"\n",
" with torch.no_grad():\n",
" for param in params:\n",
" param -= lr * param.grad / batch_size\n",
" param.grad.zero_()"
]
},
{
"cell_type": "markdown",
"id": "89067f86",
"metadata": {
"origin_pos": 32
},
"source": [
"## 训练\n",
"\n",
"现在我们已经准备好了模型训练所有需要的要素，可以实现主要的[**训练过程**]部分了。\n",
"理解这段代码至关重要，因为从事深度学习后，\n",
"相同的训练过程几乎一遍又一遍地出现。\n",
"在每次迭代中，我们读取一小批量训练样本，并通过我们的模型来获得一组预测。\n",
"计算完损失后，我们开始反向传播，存储每个参数的梯度。\n",
"最后，我们调用优化算法`sgd`来更新模型参数。\n",
"\n",
"概括一下，我们将执行以下循环：\n",
"\n",
"* 初始化参数\n",
"* 重复以下训练，直到完成\n",
" * 计算梯度$\\mathbf{g} \\leftarrow \\partial_{(\\mathbf{w},b)} \\frac{1}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} l(\\mathbf{x}^{(i)}, y^{(i)}, \\mathbf{w}, b)$\n",
" * 更新参数$(\\mathbf{w}, b) \\leftarrow (\\mathbf{w}, b) - \\eta \\mathbf{g}$\n",
"\n",
"在每个*迭代周期*（epoch）中，我们使用`data_iter`函数遍历整个数据集，\n",
"并将训练数据集中所有样本都使用一次（假设样本数能够被批量大小整除）。\n",
"这里的迭代周期个数`num_epochs`和学习率`lr`都是超参数，分别设为3和0.03。\n",
"设置超参数很棘手，需要通过反复试验进行调整。\n",
"我们现在忽略这些细节，以后会在 :numref:`chap_optimization`中详细介绍。\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "9163db58",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.903791Z",
"iopub.status.busy": "2023-08-18T07:02:04.903216Z",
"iopub.status.idle": "2023-08-18T07:02:04.908499Z",
"shell.execute_reply": "2023-08-18T07:02:04.907341Z"
},
"origin_pos": 33,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"lr = 0.03\n",
"num_epochs = 3\n",
"net = linreg\n",
"loss = squared_loss"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "ad5c2cd1",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:04.913061Z",
"iopub.status.busy": "2023-08-18T07:02:04.912436Z",
"iopub.status.idle": "2023-08-18T07:02:05.067276Z",
"shell.execute_reply": "2023-08-18T07:02:05.066107Z"
},
"origin_pos": 35,
"tab": [
"pytorch"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 1, loss 0.042790\n",
"epoch 2, loss 0.000162\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 3, loss 0.000051\n"
]
}
],
"source": [
"for epoch in range(num_epochs):\n",
" for X, y in data_iter(batch_size, features, labels):\n",
" l = loss(net(X, w, b), y) # X和y的小批量损失\n",
" # 因为l形状是(batch_size,1)，而不是一个标量。l中的所有元素被加到一起，\n",
" # 并以此计算关于[w,b]的梯度\n",
" l.sum().backward()\n",
" sgd([w, b], lr, batch_size) # 使用参数的梯度更新参数\n",
" with torch.no_grad():\n",
" train_l = loss(net(features, w, b), labels)\n",
" print(f'epoch {epoch + 1}, loss {float(train_l.mean()):f}')"
]
},
{
"cell_type": "markdown",
"id": "427d8cda",
"metadata": {
"origin_pos": 38
},
"source": [
"因为我们使用的是自己合成的数据集，所以我们知道真正的参数是什么。\n",
"因此，我们可以通过[**比较真实参数和通过训练学到的参数来评估训练的成功程度**]。\n",
"事实上，真实参数和通过训练学到的参数确实非常接近。\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "a4c3d525",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:02:05.072546Z",
"iopub.status.busy": "2023-08-18T07:02:05.071769Z",
"iopub.status.idle": "2023-08-18T07:02:05.079203Z",
"shell.execute_reply": "2023-08-18T07:02:05.078107Z"
},
"origin_pos": 39,
"tab": [
"pytorch"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"w的估计误差: tensor([-1.3804e-04, 5.7936e-05], grad_fn=)\n",
"b的估计误差: tensor([0.0006], grad_fn=)\n"
]
}
],
"source": [
"print(f'w的估计误差: {true_w - w.reshape(true_w.shape)}')\n",
"print(f'b的估计误差: {true_b - b}')"
]
},
{
"cell_type": "markdown",
"id": "9f3d71ee",
"metadata": {
"origin_pos": 40
},
"source": [
"注意，我们不应该想当然地认为我们能够完美地求解参数。\n",
"在机器学习中，我们通常不太关心恢复真正的参数，而更关心如何高度准确预测参数。\n",
"幸运的是，即使是在复杂的优化问题上，随机梯度下降通常也能找到非常好的解。\n",
"其中一个原因是，在深度网络中存在许多参数组合能够实现高度精确的预测。\n",
"\n",
"## 小结\n",
"\n",
"* 我们学习了深度网络是如何实现和优化的。在这一过程中只使用张量和自动微分，不需要定义层或复杂的优化器。\n",
"* 这一节只触及到了表面知识。在下面的部分中，我们将基于刚刚介绍的概念描述其他模型，并学习如何更简洁地实现其他模型。\n",
"\n",
"## 练习\n",
"\n",
"1. 如果我们将权重初始化为零，会发生什么。算法仍然有效吗？\n",
"1. 假设试图为电压和电流的关系建立一个模型。自动微分可以用来学习模型的参数吗?\n",
"1. 能基于[普朗克定律](https://en.wikipedia.org/wiki/Planck%27s_law)使用光谱能量密度来确定物体的温度吗？\n",
"1. 计算二阶导数时可能会遇到什么问题？这些问题可以如何解决？\n",
"1. 为什么在`squared_loss`函数中需要使用`reshape`函数？\n",
"1. 尝试使用不同的学习率，观察损失函数值下降的快慢。\n",
"1. 如果样本个数不能被批量大小整除，`data_iter`函数的行为会有什么变化？\n"
]
},
{
"cell_type": "markdown",
"id": "193224b1",
"metadata": {
"origin_pos": 42,
"tab": [
"pytorch"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/1778)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}