{
"cells": [
{
"cell_type": "markdown",
"id": "8bb0bdbd",
"metadata": {
"origin_pos": 0
},
"source": [
"# 自注意力和位置编码\n",
":label:`sec_self-attention-and-positional-encoding`\n",
"\n",
"在深度学习中，经常使用卷积神经网络（CNN）或循环神经网络（RNN）对序列进行编码。\n",
"想象一下，有了注意力机制之后，我们将词元序列输入注意力池化中，\n",
"以便同一组词元同时充当查询、键和值。\n",
"具体来说，每个查询都会关注所有的键－值对并生成一个注意力输出。\n",
"由于查询、键和值来自同一组输入，因此被称为\n",
"*自注意力*（self-attention）\n",
" :cite:`Lin.Feng.Santos.ea.2017,Vaswani.Shazeer.Parmar.ea.2017`，\n",
"也被称为*内部注意力*（intra-attention） :cite:`Cheng.Dong.Lapata.2016,Parikh.Tackstrom.Das.ea.2016,Paulus.Xiong.Socher.2017`。\n",
"本节将使用自注意力进行序列编码，以及如何使用序列的顺序作为补充信息。\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "1f68f3c6",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:34.234618Z",
"iopub.status.busy": "2023-08-18T07:01:34.233587Z",
"iopub.status.idle": "2023-08-18T07:01:37.175197Z",
"shell.execute_reply": "2023-08-18T07:01:37.174050Z"
},
"origin_pos": 2,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"import math\n",
"import torch\n",
"from torch import nn\n",
"from d2l import torch as d2l"
]
},
{
"cell_type": "markdown",
"id": "b917fbb9",
"metadata": {
"origin_pos": 5
},
"source": [
"## [**自注意力**]\n",
"\n",
"给定一个由词元组成的输入序列$\\mathbf{x}_1, \\ldots, \\mathbf{x}_n$，\n",
"其中任意$\\mathbf{x}_i \\in \\mathbb{R}^d$（$1 \\leq i \\leq n$）。\n",
"该序列的自注意力输出为一个长度相同的序列\n",
"$\\mathbf{y}_1, \\ldots, \\mathbf{y}_n$，其中：\n",
"\n",
"$$\\mathbf{y}_i = f(\\mathbf{x}_i, (\\mathbf{x}_1, \\mathbf{x}_1), \\ldots, (\\mathbf{x}_n, \\mathbf{x}_n)) \\in \\mathbb{R}^d$$\n",
"\n",
"根据 :eqref:`eq_attn-pooling`中定义的注意力汇聚函数$f$。\n",
"下面的代码片段是基于多头注意力对一个张量完成自注意力的计算，\n",
"张量的形状为（批量大小，时间步的数目或词元序列的长度，$d$）。\n",
"输出与输入的张量形状相同。\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "91993c5f",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:37.181087Z",
"iopub.status.busy": "2023-08-18T07:01:37.180270Z",
"iopub.status.idle": "2023-08-18T07:01:37.209854Z",
"shell.execute_reply": "2023-08-18T07:01:37.208705Z"
},
"origin_pos": 7,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"text/plain": [
"MultiHeadAttention(\n",
" (attention): DotProductAttention(\n",
" (dropout): Dropout(p=0.5, inplace=False)\n",
" )\n",
" (W_q): Linear(in_features=100, out_features=100, bias=False)\n",
" (W_k): Linear(in_features=100, out_features=100, bias=False)\n",
" (W_v): Linear(in_features=100, out_features=100, bias=False)\n",
" (W_o): Linear(in_features=100, out_features=100, bias=False)\n",
")"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"num_hiddens, num_heads = 100, 5\n",
"attention = d2l.MultiHeadAttention(num_hiddens, num_hiddens, num_hiddens,\n",
" num_hiddens, num_heads, 0.5)\n",
"attention.eval()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "05a56888",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:37.214732Z",
"iopub.status.busy": "2023-08-18T07:01:37.214099Z",
"iopub.status.idle": "2023-08-18T07:01:37.231099Z",
"shell.execute_reply": "2023-08-18T07:01:37.229941Z"
},
"origin_pos": 10,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([2, 4, 100])"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"batch_size, num_queries, valid_lens = 2, 4, torch.tensor([3, 2])\n",
"X = torch.ones((batch_size, num_queries, num_hiddens))\n",
"attention(X, X, X, valid_lens).shape"
]
},
{
"cell_type": "markdown",
"id": "7c242dc9",
"metadata": {
"origin_pos": 12
},
"source": [
"## 比较卷积神经网络、循环神经网络和自注意力\n",
":label:`subsec_cnn-rnn-self-attention`\n",
"\n",
"接下来比较下面几个架构，目标都是将由$n$个词元组成的序列映射到另一个长度相等的序列，其中的每个输入词元或输出词元都由$d$维向量表示。具体来说，将比较的是卷积神经网络、循环神经网络和自注意力这几个架构的计算复杂性、顺序操作和最大路径长度。请注意，顺序操作会妨碍并行计算，而任意的序列位置组合之间的路径越短，则能更轻松地学习序列中的远距离依赖关系 :cite:`Hochreiter.Bengio.Frasconi.ea.2001`。\n",
"\n",
"\n",
":label:`fig_cnn-rnn-self-attention`\n",
"\n",
"考虑一个卷积核大小为$k$的卷积层。\n",
"在后面的章节将提供关于使用卷积神经网络处理序列的更多详细信息。\n",
"目前只需要知道的是，由于序列长度是$n$，输入和输出的通道数量都是$d$，\n",
"所以卷积层的计算复杂度为$\\mathcal{O}(knd^2)$。\n",
"如 :numref:`fig_cnn-rnn-self-attention`所示，\n",
"卷积神经网络是分层的，因此为有$\\mathcal{O}(1)$个顺序操作，\n",
"最大路径长度为$\\mathcal{O}(n/k)$。\n",
"例如，$\\mathbf{x}_1$和$\\mathbf{x}_5$处于\n",
" :numref:`fig_cnn-rnn-self-attention`中卷积核大小为3的双层卷积神经网络的感受野内。\n",
"\n",
"当更新循环神经网络的隐状态时，\n",
"$d \\times d$权重矩阵和$d$维隐状态的乘法计算复杂度为$\\mathcal{O}(d^2)$。\n",
"由于序列长度为$n$，因此循环神经网络层的计算复杂度为$\\mathcal{O}(nd^2)$。\n",
"根据 :numref:`fig_cnn-rnn-self-attention`，\n",
"有$\\mathcal{O}(n)$个顺序操作无法并行化，最大路径长度也是$\\mathcal{O}(n)$。\n",
"\n",
"在自注意力中，查询、键和值都是$n \\times d$矩阵。\n",
"考虑 :eqref:`eq_softmax_QK_V`中缩放的”点－积“注意力，\n",
"其中$n \\times d$矩阵乘以$d \\times n$矩阵。\n",
"之后输出的$n \\times n$矩阵乘以$n \\times d$矩阵。\n",
"因此，自注意力具有$\\mathcal{O}(n^2d)$计算复杂性。\n",
"正如在 :numref:`fig_cnn-rnn-self-attention`中所讲，\n",
"每个词元都通过自注意力直接连接到任何其他词元。\n",
"因此，有$\\mathcal{O}(1)$个顺序操作可以并行计算，\n",
"最大路径长度也是$\\mathcal{O}(1)$。\n",
"\n",
"总而言之，卷积神经网络和自注意力都拥有并行计算的优势，\n",
"而且自注意力的最大路径长度最短。\n",
"但是因为其计算复杂度是关于序列长度的二次方，所以在很长的序列中计算会非常慢。\n",
"\n",
"## [**位置编码**]\n",
":label:`subsec_positional-encoding`\n",
"\n",
"在处理词元序列时，循环神经网络是逐个的重复地处理词元的，\n",
"而自注意力则因为并行计算而放弃了顺序操作。\n",
"为了使用序列的顺序信息，通过在输入表示中添加\n",
"*位置编码*（positional encoding）来注入绝对的或相对的位置信息。\n",
"位置编码可以通过学习得到也可以直接固定得到。\n",
"接下来描述的是基于正弦函数和余弦函数的固定位置编码\n",
" :cite:`Vaswani.Shazeer.Parmar.ea.2017`。\n",
"\n",
"假设输入表示$\\mathbf{X} \\in \\mathbb{R}^{n \\times d}$\n",
"包含一个序列中$n$个词元的$d$维嵌入表示。\n",
"位置编码使用相同形状的位置嵌入矩阵\n",
"$\\mathbf{P} \\in \\mathbb{R}^{n \\times d}$输出$\\mathbf{X} + \\mathbf{P}$，\n",
"矩阵第$i$行、第$2j$列和$2j+1$列上的元素为：\n",
"\n",
"$$\\begin{aligned} p_{i, 2j} &= \\sin\\left(\\frac{i}{10000^{2j/d}}\\right),\\\\p_{i, 2j+1} &= \\cos\\left(\\frac{i}{10000^{2j/d}}\\right).\\end{aligned}$$\n",
":eqlabel:`eq_positional-encoding-def`\n",
"\n",
"乍一看，这种基于三角函数的设计看起来很奇怪。\n",
"在解释这个设计之前，让我们先在下面的`PositionalEncoding`类中实现它。\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a1520381",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:37.236150Z",
"iopub.status.busy": "2023-08-18T07:01:37.235749Z",
"iopub.status.idle": "2023-08-18T07:01:37.246341Z",
"shell.execute_reply": "2023-08-18T07:01:37.245419Z"
},
"origin_pos": 14,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"#@save\n",
"class PositionalEncoding(nn.Module):\n",
" \"\"\"位置编码\"\"\"\n",
" def __init__(self, num_hiddens, dropout, max_len=1000):\n",
" super(PositionalEncoding, self).__init__()\n",
" self.dropout = nn.Dropout(dropout)\n",
" # 创建一个足够长的P\n",
" self.P = torch.zeros((1, max_len, num_hiddens))\n",
" X = torch.arange(max_len, dtype=torch.float32).reshape(\n",
" -1, 1) / torch.pow(10000, torch.arange(\n",
" 0, num_hiddens, 2, dtype=torch.float32) / num_hiddens)\n",
" self.P[:, :, 0::2] = torch.sin(X)\n",
" self.P[:, :, 1::2] = torch.cos(X)\n",
"\n",
" def forward(self, X):\n",
" X = X + self.P[:, :X.shape[1], :].to(X.device)\n",
" return self.dropout(X)"
]
},
{
"cell_type": "markdown",
"id": "b2f91685",
"metadata": {
"origin_pos": 17
},
"source": [
"在位置嵌入矩阵$\\mathbf{P}$中，\n",
"[**行代表词元在序列中的位置，列代表位置编码的不同维度**]。\n",
"从下面的例子中可以看到位置嵌入矩阵的第$6$列和第$7$列的频率高于第$8$列和第$9$列。\n",
"第$6$列和第$7$列之间的偏移量（第$8$列和第$9$列相同）是由于正弦函数和余弦函数的交替。\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "2530db11",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:37.253441Z",
"iopub.status.busy": "2023-08-18T07:01:37.251675Z",
"iopub.status.idle": "2023-08-18T07:01:37.511460Z",
"shell.execute_reply": "2023-08-18T07:01:37.510281Z"
},
"origin_pos": 19,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"encoding_dim, num_steps = 32, 60\n",
"pos_encoding = PositionalEncoding(encoding_dim, 0)\n",
"pos_encoding.eval()\n",
"X = pos_encoding(torch.zeros((1, num_steps, encoding_dim)))\n",
"P = pos_encoding.P[:, :X.shape[1], :]\n",
"d2l.plot(torch.arange(num_steps), P[0, :, 6:10].T, xlabel='Row (position)',\n",
" figsize=(6, 2.5), legend=[\"Col %d\" % d for d in torch.arange(6, 10)])"
]
},
{
"cell_type": "markdown",
"id": "28698b9d",
"metadata": {
"origin_pos": 22
},
"source": [
"### 绝对位置信息\n",
"\n",
"为了明白沿着编码维度单调降低的频率与绝对位置信息的关系，\n",
"让我们打印出$0, 1, \\ldots, 7$的[**二进制表示**]形式。\n",
"正如所看到的，每个数字、每两个数字和每四个数字上的比特值\n",
"在第一个最低位、第二个最低位和第三个最低位上分别交替。\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "07196b9a",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:37.516113Z",
"iopub.status.busy": "2023-08-18T07:01:37.515203Z",
"iopub.status.idle": "2023-08-18T07:01:37.523367Z",
"shell.execute_reply": "2023-08-18T07:01:37.520554Z"
},
"origin_pos": 23,
"tab": [
"pytorch"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0的二进制是：000\n",
"1的二进制是：001\n",
"2的二进制是：010\n",
"3的二进制是：011\n",
"4的二进制是：100\n",
"5的二进制是：101\n",
"6的二进制是：110\n",
"7的二进制是：111\n"
]
}
],
"source": [
"for i in range(8):\n",
" print(f'{i}的二进制是：{i:>03b}')"
]
},
{
"cell_type": "markdown",
"id": "bc8f0fc5",
"metadata": {
"origin_pos": 24
},
"source": [
"在二进制表示中，较高比特位的交替频率低于较低比特位，\n",
"与下面的热图所示相似，只是位置编码通过使用三角函数[**在编码维度上降低频率**]。\n",
"由于输出是浮点数，因此此类连续表示比二进制表示法更节省空间。\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "fb689860",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:01:37.528541Z",
"iopub.status.busy": "2023-08-18T07:01:37.527891Z",
"iopub.status.idle": "2023-08-18T07:01:37.784120Z",
"shell.execute_reply": "2023-08-18T07:01:37.782997Z"
},
"origin_pos": 26,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"P = P[0, :, :].unsqueeze(0).unsqueeze(0)\n",
"d2l.show_heatmaps(P, xlabel='Column (encoding dimension)',\n",
" ylabel='Row (position)', figsize=(3.5, 4), cmap='Blues')"
]
},
{
"cell_type": "markdown",
"id": "6c836c1d",
"metadata": {
"origin_pos": 29
},
"source": [
"### 相对位置信息\n",
"\n",
"除了捕获绝对位置信息之外，上述的位置编码还允许模型学习得到输入序列中相对位置信息。\n",
"这是因为对于任何确定的位置偏移$\\delta$，位置$i + \\delta$处\n",
"的位置编码可以线性投影位置$i$处的位置编码来表示。\n",
"\n",
"这种投影的数学解释是，令$\\omega_j = 1/10000^{2j/d}$，\n",
"对于任何确定的位置偏移$\\delta$，\n",
" :eqref:`eq_positional-encoding-def`中的任何一对\n",
"$(p_{i, 2j}, p_{i, 2j+1})$都可以线性投影到\n",
"$(p_{i+\\delta, 2j}, p_{i+\\delta, 2j+1})$：\n",
"\n",
"$$\\begin{aligned}\n",
"&\\begin{bmatrix} \\cos(\\delta \\omega_j) & \\sin(\\delta \\omega_j) \\\\ -\\sin(\\delta \\omega_j) & \\cos(\\delta \\omega_j) \\\\ \\end{bmatrix}\n",
"\\begin{bmatrix} p_{i, 2j} \\\\ p_{i, 2j+1} \\\\ \\end{bmatrix}\\\\\n",
"=&\\begin{bmatrix} \\cos(\\delta \\omega_j) \\sin(i \\omega_j) + \\sin(\\delta \\omega_j) \\cos(i \\omega_j) \\\\ -\\sin(\\delta \\omega_j) \\sin(i \\omega_j) + \\cos(\\delta \\omega_j) \\cos(i \\omega_j) \\\\ \\end{bmatrix}\\\\\n",
"=&\\begin{bmatrix} \\sin\\left((i+\\delta) \\omega_j\\right) \\\\ \\cos\\left((i+\\delta) \\omega_j\\right) \\\\ \\end{bmatrix}\\\\\n",
"=& \n",
"\\begin{bmatrix} p_{i+\\delta, 2j} \\\\ p_{i+\\delta, 2j+1} \\\\ \\end{bmatrix},\n",
"\\end{aligned}$$\n",
"\n",
"$2\\times 2$投影矩阵不依赖于任何位置的索引$i$。\n",
"\n",
"## 小结\n",
"\n",
"* 在自注意力中，查询、键和值都来自同一组输入。\n",
"* 卷积神经网络和自注意力都拥有并行计算的优势，而且自注意力的最大路径长度最短。但是因为其计算复杂度是关于序列长度的二次方，所以在很长的序列中计算会非常慢。\n",
"* 为了使用序列的顺序信息，可以通过在输入表示中添加位置编码，来注入绝对的或相对的位置信息。\n",
"\n",
"## 练习\n",
"\n",
"1. 假设设计一个深度架构，通过堆叠基于位置编码的自注意力层来表示序列。可能会存在什么问题？\n",
"1. 请设计一种可学习的位置编码方法。\n"
]
},
{
"cell_type": "markdown",
"id": "3d4bb49e",
"metadata": {
"origin_pos": 31,
"tab": [
"pytorch"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/5762)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}