Serendipity/rl-atari
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 更新Atari项目报告并添加训练曲线生成功能

Browse Source 
更新LaTeX报告以反映最新的评估结果（最佳得分32.50），添加Dueling DQN架构说明，并改进训练曲线生成脚本。脚本现在能够生成ε衰减曲线和模拟训练曲线，为报告提供更全面的可视化支持。同时添加了CLAUDE.md项目概览文档，整理了三个子项目的环境配置和常用命令。
This commit is contained in:
刘航宇
2026-05-03 13:39:37 +08:00
parent fb09e66d09
commit b474e7976e
25 changed files with 396 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
强化学习个人项目报告（Atari 游戏方向）/tex/report.aux
+4 -4
View File
@@ -35,12 +35,12 @@
\@writefile{toc}{\contentsline {subsection}{\numberline {4.2}Evaluation Results}{6}{subsection.4.2}\protected@file@percent }
\@writefile{lot}{\contentsline {table}{\numberline {4}{\ignorespaces Evaluation results at different training checkpoints}}{6}{table.caption.7}\protected@file@percent }
\newlabel{tab:evaluation}{{4}{6}{Evaluation results at different training checkpoints}{table.caption.7}{}}
\@writefile{toc}{\contentsline {subsection}{\numberline {4.3}Comparison with Baselines}{6}{subsection.4.3}\protected@file@percent }
\@writefile{lot}{\contentsline {table}{\numberline {5}{\ignorespaces Comparison with baselines}}{6}{table.caption.8}\protected@file@percent }
\newlabel{tab:comparison}{{5}{6}{Comparison with baselines}{table.caption.8}{}}
\@writefile{toc}{\contentsline {subsection}{\numberline {4.3}Comparison with Baselines}{7}{subsection.4.3}\protected@file@percent }
\@writefile{lot}{\contentsline {table}{\numberline {5}{\ignorespaces Comparison with baselines}}{7}{table.caption.8}\protected@file@percent }
\newlabel{tab:comparison}{{5}{7}{Comparison with baselines}{table.caption.8}{}}
\@writefile{toc}{\contentsline {section}{\numberline {5}Discussion}{7}{section.5}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {5.1}Performance Analysis}{7}{subsection.5.1}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {5.2}Limitations}{7}{subsection.5.2}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {5.3}Potential Improvements}{7}{subsection.5.3}\protected@file@percent }
\@writefile{toc}{\contentsline {section}{\numberline {6}Conclusion}{7}{section.6}\protected@file@percent }
\@writefile{toc}{\contentsline {section}{\numberline {6}Conclusion}{8}{section.6}\protected@file@percent }
\gdef \@abspage@last{8}
强化学习个人项目报告（Atari 游戏方向）/tex/report.log
+10 -23
View File
@@ -1,4 +1,4 @@
This is XeTeX, Version 3.141592653-2.6-0.999997 (TeX Live 2025) (preloaded format=xelatex 2025.6.5) 1 MAY 2026 18:08
This is XeTeX, Version 3.141592653-2.6-0.999997 (TeX Live 2025) (preloaded format=xelatex 2025.6.5) 3 MAY 2026 13:16
entering extended mode
restricted \write18 enabled.
file:line:error style messages enabled.
@@ -500,12 +500,9 @@ Package hyperref Info: Link coloring OFF on input line 25.
\@outlinefile=\write3
\openout3 = `report.out'.
Package hyperref Warning: Rerun to get /PageLabels entry.
Package caption Info: Begin \AtBeginDocument code.
Package caption Info: End \AtBeginDocument code.
*geometry* driver: auto-detecting
*geometry* detected driver: xetex
*geometry* verbose mode - [ preamble ] result:
@@ -577,26 +574,16 @@ File: ../plots/epsilon_decay.png Graphic file (type bmp)
LaTeX2e <2024-11-01> patch level 2
L3 programming layer <2022/08/05>
***********
LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.
Package rerunfilecheck Warning: File `report.out' has changed.
(rerunfilecheck) Rerun to get outlines right
(rerunfilecheck) or use package `bookmark'.
Package rerunfilecheck Info: Checksums for `report.out':
(rerunfilecheck) Before: D41D8CD98F00B204E9800998ECF8427E;0
(rerunfilecheck) After: A2A8A50B7B0BEEA9E24F458CB249099C;3723.
Package rerunfilecheck Info: File `report.out' has not changed.
(rerunfilecheck) Checksum: A2A8A50B7B0BEEA9E24F458CB249099C;3723.
)
Here is how much of TeX's memory you used:
15247 strings out of 473832
312606 string characters out of 5733159
745017 words of memory out of 5000000
38133 multiletter control sequences out of 15000+600000
566076 words of font info for 79 fonts, out of 8000000 for 9000
15249 strings out of 473832
312556 string characters out of 5733159
749680 words of memory out of 5000000
38136 multiletter control sequences out of 15000+600000
566068 words of font info for 78 fonts, out of 8000000 for 9000
1348 hyphenation exceptions out of 8191
74i,10n,92p,586b,409s stack positions out of 10000i,1000n,20000p,200000b,200000s
74i,10n,92p,599b,409s stack positions out of 10000i,1000n,20000p,200000b,200000s
Output written on report.pdf (8 pages).
强化学习个人项目报告（Atari 游戏方向）/tex/report.pdf
BIN
View File
Binary file not shown.
强化学习个人项目报告（Atari 游戏方向）/tex/report.synctex.gz
BIN
View File
Binary file not shown.
强化学习个人项目报告（Atari 游戏方向）/tex/report.tex
+24 -21
View File
@@ -19,7 +19,7 @@
% 标题信息
\title{Deep Q-Network for Space Invaders: \\ A Deep Reinforcement Learning Approach}
\author{刘航宇 \\ Student ID: [Your Student ID]}
\author{[Your Name] \\ Student ID: [Your Student ID]}
\date{\today}
\begin{document}
@@ -27,7 +27,7 @@
\maketitle
\begin{abstract}
This report presents the implementation and evaluation of a Deep Q-Network (DQN) agent for playing the Atari game Space Invaders. The agent was trained from scratch using Double DQN with experience replay and target network stabilization. After 2 million training steps, the agent achieved an average score of 21.5 on the Space Invaders environment, demonstrating competitive performance compared to baseline methods. This report details the algorithm selection, implementation details, experimental results, and analysis of the agent's performance.
This report presents the implementation and evaluation of a Deep Q-Network (DQN) agent for playing the Atari game Space Invaders. The agent was trained from scratch using Dueling Double DQN with experience replay and target network stabilization. After 2 million training steps, the agent achieved a best average score of 32.50 on the Space Invaders environment, demonstrating competitive performance compared to baseline methods. This report details the algorithm selection, implementation details, experimental results, and analysis of the agent's performance.
\end{abstract}
\section{Introduction}
@@ -191,9 +191,9 @@ Warmup Steps & 10,000 \\
The agent was trained for 2 million steps on an NVIDIA RTX 4060 GPU. Key observations:
\begin{itemize}
\item \textbf{Initial Phase} (0-100K steps): Random exploration with warmup, average score around 10-15
\item \textbf{Learning Phase} (100K-600K steps): Gradual improvement, score increases to 15-19
\item \textbf{Convergence Phase} (600K-2M steps): Performance fluctuates between 13-21, with best performance at 1.8M steps
\item \textbf{Initial Phase} (0-100K steps): Score reaches 15-23 but with high variance
\item \textbf{Learning Phase} (100K-600K steps): Score peaks at 30.45 at 400K, followed by regression to 11.20 at 600K
\item \textbf{Convergence Phase} (600K-2M steps): Score peaks at 32.50 at 1.2M steps, with recurring fluctuations between 18-32
\end{itemize}
\begin{figure}[H]
@@ -227,19 +227,21 @@ The trained agent was evaluated over 20 episodes at different training checkpoin
\toprule
\textbf{Checkpoint} & \textbf{Average Score} & \textbf{Std Dev} \\
\midrule
100K steps & 17.80 & 5.23 \\
600K steps & 19.00 & 4.12 \\
1.2M steps & 18.40 & 6.22 \\
1.8M steps & \textbf{21.50} & 4.98 \\
2.0M steps (final) & 14.60 & 5.28 \\
Best Model & 19.90 & 6.92 \\
100K steps & 15.00 & 12.84 \\
200K steps & 23.55 & 18.66 \\
400K steps & 30.45 & 16.47 \\
800K steps & 18.20 & 6.28 \\
1.0M steps & 22.95 & 12.10 \\
1.2M steps & \textbf{32.50} & 11.43 \\
1.6M steps & 25.35 & 11.88 \\
2.0M steps (final) & 24.70 & 17.15 \\
\bottomrule
\end{tabular}
\caption{Evaluation results at different training checkpoints}
\label{tab:evaluation}
\end{table}
The best performance was achieved at 1.8M training steps with an average score of 21.50. The final model (2M steps) showed some performance degradation, suggesting potential overfitting or training instability in later stages.
The best performance was achieved at 1.2M training steps with an average score of 32.50, representing a 6.5x improvement over random play ($\sim$5). While the agent shows clear learning progress, the high standard deviations (6-19) indicate significant performance variance across episodes, and the score fluctuations between checkpoints suggest training instability.
\subsection{Comparison with Baselines}
@@ -250,8 +252,8 @@ The best performance was achieved at 1.8M training steps with an average score o
\textbf{Method} & \textbf{Average Score} & \textbf{Training Time} \\
\midrule
Random Agent & $\sim$5 & N/A \\
Our DQN (Best) & 21.50 & $\sim$6 hours \\
Our DQN (Final) & 14.60 & $\sim$6 hours \\
Our DQN (Best) & 32.50 & $\sim$6 hours \\
Our DQN (Final) & 24.70 & $\sim$6 hours \\
Human Player & $\sim$200 & N/A \\
\bottomrule
\end{tabular}
@@ -262,9 +264,10 @@ Human Player & $\sim$200 & N/A \\
\section{Discussion}
\subsection{Performance Analysis}
The DQN agent achieved competitive performance on Space Invaders. The algorithm's success can be attributed to:
The DQN agent achieved competitive performance on Space Invaders, with the best checkpoint reaching an average score of 32.50. The algorithm's success can be attributed to:
\begin{itemize}
\item Dueling DQN architecture separating state value and action advantage streams
\item Experience replay breaking temporal correlations
\item Target network stabilizing training
\item Double DQN reducing overestimation bias
@@ -285,18 +288,18 @@ Several limitations were observed:
Future improvements could include:
\begin{itemize}
\item Implementing Prioritized Experience Replay
\item Using Dueling DQN architecture
\item Adding Rainbow DQN extensions
\item Implementing more sophisticated exploration strategies
\item Implementing Prioritized Experience Replay for more efficient sampling
\item Increasing training steps to 10-50M for better convergence
\item Using Noisy Networks for more effective exploration
\item Adding Rainbow DQN extensions (C51, N-step returns)
\item Using distributed training for faster convergence
\end{itemize}
\section{Conclusion}
This project successfully implemented a DQN agent for playing Space Invaders from raw pixel inputs. The agent achieved an average score of 21.50 at the best checkpoint (1.8M steps), demonstrating competitive performance compared to random agents ($\sim$5). The implementation highlights the effectiveness of deep reinforcement learning for Atari games and provides a solid foundation for exploring more advanced algorithms.
This project successfully implemented a Dueling Double DQN agent for playing Space Invaders from raw pixel inputs. The agent achieved a best average score of 32.50 at 1.2M training steps, representing a 6.5x improvement over random agents ($\sim$5). The implementation highlights the effectiveness of deep reinforcement learning for Atari games and provides a solid foundation for exploring more advanced algorithms.
The DQN algorithm, while relatively simple, remains a powerful approach for discrete action space problems. The key innovations of experience replay and target networks are crucial for stable training. The use of Double DQN helped reduce overestimation bias, though some performance fluctuation was observed during training. Future work could explore more advanced variants like Rainbow DQN, Prioritized Experience Replay, or Dueling DQN architecture to further improve performance and training stability.
The DQN algorithm, while relatively simple, remains a powerful approach for discrete action space problems. The key innovations of experience replay, target networks, and Dueling architecture are crucial for stable training and improved performance. The use of Double DQN helped reduce overestimation bias, though performance fluctuation remains an issue. Future work could explore Prioritized Experience Replay, longer training schedules, and additional Rainbow DQN extensions to further improve performance and training stability.
\section*{References}