{
"cells": [
{
"cell_type": "markdown",
"id": "bc862d32",
"metadata": {
"origin_pos": 0
},
"source": [
"# 情感分析：使用循环神经网络\n",
":label:`sec_sentiment_rnn`\n",
"\n",
"与词相似度和类比任务一样，我们也可以将预先训练的词向量应用于情感分析。由于 :numref:`sec_sentiment`中的IMDb评论数据集不是很大，使用在大规模语料库上预训练的文本表示可以减少模型的过拟合。作为 :numref:`fig_nlp-map-sa-rnn`中所示的具体示例，我们将使用预训练的GloVe模型来表示每个词元，并将这些词元表示送入多层双向循环神经网络以获得文本序列表示，该文本序列表示将被转换为情感分析输出 :cite:`Maas.Daly.Pham.ea.2011`。对于相同的下游应用，我们稍后将考虑不同的架构选择。\n",
"\n",
"\n",
":label:`fig_nlp-map-sa-rnn`\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "39fc24d2",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:05:05.652082Z",
"iopub.status.busy": "2023-08-18T07:05:05.651505Z",
"iopub.status.idle": "2023-08-18T07:05:40.292912Z",
"shell.execute_reply": "2023-08-18T07:05:40.291561Z"
},
"origin_pos": 2,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"import torch\n",
"from torch import nn\n",
"from d2l import torch as d2l\n",
"\n",
"batch_size = 64\n",
"train_iter, test_iter, vocab = d2l.load_data_imdb(batch_size)"
]
},
{
"cell_type": "markdown",
"id": "c3e77029",
"metadata": {
"origin_pos": 4
},
"source": [
"## 使用循环神经网络表示单个文本\n",
"\n",
"在文本分类任务（如情感分析）中，可变长度的文本序列将被转换为固定长度的类别。在下面的`BiRNN`类中，虽然文本序列的每个词元经由嵌入层（`self.embedding`）获得其单独的预训练GloVe表示，但是整个序列由双向循环神经网络（`self.encoder`）编码。更具体地说，双向长短期记忆网络在初始和最终时间步的隐状态（在最后一层）被连结起来作为文本序列的表示。然后，通过一个具有两个输出（“积极”和“消极”）的全连接层（`self.decoder`），将此单一文本表示转换为输出类别。\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "744f39e5",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:05:40.299184Z",
"iopub.status.busy": "2023-08-18T07:05:40.298245Z",
"iopub.status.idle": "2023-08-18T07:05:40.309337Z",
"shell.execute_reply": "2023-08-18T07:05:40.307994Z"
},
"origin_pos": 6,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"class BiRNN(nn.Module):\n",
" def __init__(self, vocab_size, embed_size, num_hiddens,\n",
" num_layers, **kwargs):\n",
" super(BiRNN, self).__init__(**kwargs)\n",
" self.embedding = nn.Embedding(vocab_size, embed_size)\n",
" # 将bidirectional设置为True以获取双向循环神经网络\n",
" self.encoder = nn.LSTM(embed_size, num_hiddens, num_layers=num_layers,\n",
" bidirectional=True)\n",
" self.decoder = nn.Linear(4 * num_hiddens, 2)\n",
"\n",
" def forward(self, inputs):\n",
" # inputs的形状是（批量大小，时间步数）\n",
" # 因为长短期记忆网络要求其输入的第一个维度是时间维，\n",
" # 所以在获得词元表示之前，输入会被转置。\n",
" # 输出形状为（时间步数，批量大小，词向量维度）\n",
" embeddings = self.embedding(inputs.T)\n",
" self.encoder.flatten_parameters()\n",
" # 返回上一个隐藏层在不同时间步的隐状态，\n",
" # outputs的形状是（时间步数，批量大小，2*隐藏单元数）\n",
" outputs, _ = self.encoder(embeddings)\n",
" # 连结初始和最终时间步的隐状态，作为全连接层的输入，\n",
" # 其形状为（批量大小，4*隐藏单元数）\n",
" encoding = torch.cat((outputs[0], outputs[-1]), dim=1)\n",
" outs = self.decoder(encoding)\n",
" return outs"
]
},
{
"cell_type": "markdown",
"id": "98329015",
"metadata": {
"origin_pos": 8
},
"source": [
"让我们构造一个具有两个隐藏层的双向循环神经网络来表示单个文本以进行情感分析。\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "d8b2c0f6",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:05:40.314032Z",
"iopub.status.busy": "2023-08-18T07:05:40.313553Z",
"iopub.status.idle": "2023-08-18T07:05:40.403532Z",
"shell.execute_reply": "2023-08-18T07:05:40.402355Z"
},
"origin_pos": 9,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"embed_size, num_hiddens, num_layers = 100, 100, 2\n",
"devices = d2l.try_all_gpus()\n",
"net = BiRNN(len(vocab), embed_size, num_hiddens, num_layers)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "8215de71",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:05:40.408642Z",
"iopub.status.busy": "2023-08-18T07:05:40.407984Z",
"iopub.status.idle": "2023-08-18T07:05:40.420400Z",
"shell.execute_reply": "2023-08-18T07:05:40.419330Z"
},
"origin_pos": 11,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def init_weights(m):\n",
" if type(m) == nn.Linear:\n",
" nn.init.xavier_uniform_(m.weight)\n",
" if type(m) == nn.LSTM:\n",
" for param in m._flat_weights_names:\n",
" if \"weight\" in param:\n",
" nn.init.xavier_uniform_(m._parameters[param])\n",
"net.apply(init_weights);"
]
},
{
"cell_type": "markdown",
"id": "3b604a0d",
"metadata": {
"origin_pos": 13
},
"source": [
"## 加载预训练的词向量\n",
"\n",
"下面，我们为词表中的单词加载预训练的100维（需要与`embed_size`一致）的GloVe嵌入。\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "50827d9a",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:05:40.424891Z",
"iopub.status.busy": "2023-08-18T07:05:40.424223Z",
"iopub.status.idle": "2023-08-18T07:06:05.190359Z",
"shell.execute_reply": "2023-08-18T07:06:05.188903Z"
},
"origin_pos": 14,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"glove_embedding = d2l.TokenEmbedding('glove.6b.100d')"
]
},
{
"cell_type": "markdown",
"id": "a1f48ad7",
"metadata": {
"origin_pos": 15
},
"source": [
"打印词表中所有词元向量的形状。\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "2ca5c11f",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:06:05.196382Z",
"iopub.status.busy": "2023-08-18T07:06:05.195537Z",
"iopub.status.idle": "2023-08-18T07:06:05.260164Z",
"shell.execute_reply": "2023-08-18T07:06:05.258848Z"
},
"origin_pos": 16,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([49346, 100])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"embeds = glove_embedding[vocab.idx_to_token]\n",
"embeds.shape"
]
},
{
"cell_type": "markdown",
"id": "c8621b1b",
"metadata": {
"origin_pos": 17
},
"source": [
"我们使用这些预训练的词向量来表示评论中的词元，并且在训练期间不要更新这些向量。\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "1f86f2b0",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:06:05.267876Z",
"iopub.status.busy": "2023-08-18T07:06:05.266887Z",
"iopub.status.idle": "2023-08-18T07:06:05.276563Z",
"shell.execute_reply": "2023-08-18T07:06:05.275272Z"
},
"origin_pos": 19,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"net.embedding.weight.data.copy_(embeds)\n",
"net.embedding.weight.requires_grad = False"
]
},
{
"cell_type": "markdown",
"id": "fa9d2712",
"metadata": {
"origin_pos": 21
},
"source": [
"## 训练和评估模型\n",
"\n",
"现在我们可以训练双向循环神经网络进行情感分析。\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "e98aa66f",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:06:05.282869Z",
"iopub.status.busy": "2023-08-18T07:06:05.281955Z",
"iopub.status.idle": "2023-08-18T07:07:16.147782Z",
"shell.execute_reply": "2023-08-18T07:07:16.146887Z"
},
"origin_pos": 23,
"tab": [
"pytorch"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"loss 0.262, train acc 0.893, test acc 0.864\n",
"2902.4 examples/sec on [device(type='cuda', index=0), device(type='cuda', index=1)]\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"lr, num_epochs = 0.01, 5\n",
"trainer = torch.optim.Adam(net.parameters(), lr=lr)\n",
"loss = nn.CrossEntropyLoss(reduction=\"none\")\n",
"d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs,\n",
" devices)"
]
},
{
"cell_type": "markdown",
"id": "bd858123",
"metadata": {
"origin_pos": 25
},
"source": [
"我们定义以下函数来使用训练好的模型`net`预测文本序列的情感。\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "bcc5f0ab",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:07:16.151578Z",
"iopub.status.busy": "2023-08-18T07:07:16.150980Z",
"iopub.status.idle": "2023-08-18T07:07:16.156295Z",
"shell.execute_reply": "2023-08-18T07:07:16.155474Z"
},
"origin_pos": 27,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"#@save\n",
"def predict_sentiment(net, vocab, sequence):\n",
" \"\"\"预测文本序列的情感\"\"\"\n",
" sequence = torch.tensor(vocab[sequence.split()], device=d2l.try_gpu())\n",
" label = torch.argmax(net(sequence.reshape(1, -1)), dim=1)\n",
" return 'positive' if label == 1 else 'negative'"
]
},
{
"cell_type": "markdown",
"id": "532d7858",
"metadata": {
"origin_pos": 29
},
"source": [
"最后，让我们使用训练好的模型对两个简单的句子进行情感预测。\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "41cb5e96",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:07:16.160003Z",
"iopub.status.busy": "2023-08-18T07:07:16.159129Z",
"iopub.status.idle": "2023-08-18T07:07:16.167838Z",
"shell.execute_reply": "2023-08-18T07:07:16.166727Z"
},
"origin_pos": 30,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"text/plain": [
"'positive'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predict_sentiment(net, vocab, 'this movie is so great')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "84350bf3",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T07:07:16.171422Z",
"iopub.status.busy": "2023-08-18T07:07:16.170827Z",
"iopub.status.idle": "2023-08-18T07:07:16.177666Z",
"shell.execute_reply": "2023-08-18T07:07:16.176801Z"
},
"origin_pos": 31,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"text/plain": [
"'negative'"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predict_sentiment(net, vocab, 'this movie is so bad')"
]
},
{
"cell_type": "markdown",
"id": "f13f4f0d",
"metadata": {
"origin_pos": 32
},
"source": [
"## 小结\n",
"\n",
"* 预训练的词向量可以表示文本序列中的各个词元。\n",
"* 双向循环神经网络可以表示文本序列。例如通过连结初始和最终时间步的隐状态，可以使用全连接的层将该单个文本表示转换为类别。\n",
"\n",
"## 练习\n",
"\n",
"1. 增加迭代轮数可以提高训练和测试的准确性吗？调优其他超参数怎么样？\n",
"1. 使用较大的预训练词向量，例如300维的GloVe嵌入。它是否提高了分类精度？\n",
"1. 是否可以通过spaCy词元化来提高分类精度？需要安装Spacy（`pip install spacy`）和英语语言包（`python -m spacy download en`）。在代码中，首先导入Spacy（`import spacy`）。然后，加载Spacy英语软件包（`spacy_en = spacy.load('en')`）。最后，定义函数`def tokenizer(text): return [tok.text for tok in spacy_en.tokenizer(text)]`并替换原来的`tokenizer`函数。请注意GloVe和spaCy中短语标记的不同形式。例如，短语标记“new york”在GloVe中的形式是“new-york”，而在spaCy词元化之后的形式是“new york”。\n"
]
},
{
"cell_type": "markdown",
"id": "d148e971",
"metadata": {
"origin_pos": 34,
"tab": [
"pytorch"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/5724)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}