XJTLU Entrepreneur College (Taicang) Cover Sheet Module code and Title DTS307TC Reinforcement Learning School Title School of AI and Advanced Computing Assignment Title Coursework 1 Submission Deadline 04/May/2026 23:59 Final Word Count If you agree to let the university use your work anonymously for teaching and learning purposes, please type “yes” here. I certify that I have read and understood the University’s Policy for dealing with Plagiarism, Collusion and the Fabrication of Data (available on Learning Mall Online). With reference to this policy I certify that: • My work does not contain any instances of plagiarism and/or collusion. My work does not contain any fabricated data. By uploading my assignment onto Learning Mall Online, I formally declare that all of the above information is true to the best of my knowledge and belief. Scoring – For Tutor Use Student ID Stage of Marker Learning Outcomes Achieved （F/P/M/D） Final Marking Code (please modify as appropriate) Score A B C 1st Marker – red pen Moderation The original mark has been accepted by the moderator Y/N IM (please circle as appropriate): – green pen Initials Data entry and score calculation have been checked by Y another tutor (please circle): 2nd Marker if needed – green pen For Academic Office Use Possible Academic Infringement (please tick as appropriate) Date Days Late ☐ Category A Received late Penalty Total Academic Infringement Penalty ☐ Category B (A,B, C, D, E, Please modify where necessary) _____________________ ☐ Category C ☐ Category D ☐ Category E School of Artificial Intelligence and Advanced Computing Xi’an Jiaotong-Liverpool University DTS307TC Reinforcement Learning Coursework - Individual Report Due: 04/May/2026 23:59 Weight: 40% Maximum score: 40 marks Overview The purpose of this assignment is to gain experience in Python programming and the design of reinforcement leaning algorithms. You are expected to implement an RL algorithm that solves a specific environment and provide an explanation of the algorithm’s methodology. You are expected to analyse your results, including challenges and your solutions. Learning Outcomes Assessed A: Systematically understand the fundamental concepts and principles of reinforcement learning B: Critically analyse real-life problem situations and expertly map them as reinforcement learning tasks. C: Mastery of Monte Carlo Methods and Temporal Difference Learning D: Proficiency in Deep Reinforcement Learning algorithms Late policy 5% of the total marks available for the assessment shall be deducted from the assessment mark for each working day after the submission date, up to a maximum of five working days Avoid Plagiarism • Do not submit work from other students. • Do not share code/work with other students • Do not use open-source code as it is or without proper reference. 2 Risks • Please read the coursework instructions and requirements carefully. Not following these instructions and requirements may result in a loss of marks. • The assignment must be submitted via Learning Mall. Only electronic submission is accepted and no hard copy submission. • All students must download their file and check that it is viewable after submission. Documents may become corrupted during the uploading process (e.g. due to slow internet connections). However, students are responsible for submitting a functional and correct file for assessments. • Academic Integrity Policy is strictly followed. Individual Report (40 marks) The primary objective of this coursework is to familiarize students with the PPO algorithm using basic deep learning libraries, enabling them to improve their capability in transferring mathematical and theoretical knowledge into Python implementation, and further their understanding of the actor- critic algorithm. Algorithm Overview Proximal Policy Optimization (PPO) is a state-of-the-art reinforcement learning algorithm that optimizes a stochastic policy in an on-policy manner. To ensure stable training and avoid catastrophic performance collapse, PPO utilizes a clipped surrogate objective to prevent the policy update from stepping too far from the current behavior. The Environment: CarRacing-v3 We will be using the Car Racing environment from the OpenAI Gymnasium. This environment features a top-down racing track where the agent must learn to navigate through tiles based on pixel inputs. You can find more details about this environment on their website.(https://gymnasium. farama.org/environments/box2d/car_racing/) Here’s a code snippet for you to get started: import gymnasium as gym env = gym . make ( " CarRacing - v3 " , render_mode = " rgb_array " ) env . reset () Since CarRacing-v3 is quite computationally expensive for a standard laptop (due to the pixel processing), you might want to consider using a gray-scaling or frame-stacking wrapper to speed up training. Alternatively, you can also use the lab computers, which have GPUs and have all the environment already set up. The PPO Agent You will implement an RL agent using PPO to play the CarRacing-v3 environment. The agent will use the standard observation and actions provided by the environment. You may edit the 3 environment to speed up your training, but your agent must still perform well in the standard environment. (i.e, removing the camera zoom at the beginning is allowed during training, but your agent should still be tested in the original environment.) You should record your training and evaluation process using Tensorboard. You should also record important losses and other data for your analysis later. The Report Upon completion of your implementation, you are required to submit a comprehensive technical report. The report should document your engineering decisions, the theoretical grounding of your code, and a critical analysis of the agent’s performance. 1. Introduction • Provide a brief overview of Reinforcement Learning in the context of the CarRacing-v3 environment. • Define the state space (pixels), action space (discrete commands), and the reward structure of the task. 2. Methodology • Mathematical Foundation: Formulate the PPO objective function. Explain the significance of the clipping parameter and the probability ratio. • Advantage Estimation: Describe your method for calculating advantages (e.g., standard advantage vs. Generalized Advantage Estimation (GAE)). 3. Implementation Details • Describe your implementation, including any challenges faced and how you addressed them. • Explain the structure of your policy and value networks. • Detail the training process and hyperparameters used. 4. Results and Analysis • Present your results (use graphs for better clarity). • Discuss the performance of your agent and any trends observed. • Briefly compare your custom implementation’s stability and sample efficiency against baseline benchmarks (e.g., Stable-Baselines3). 5. Conclusion • Summarize your key findings regarding the sensitivity of PPO to hyperparameter tuning and the effectiveness of the actor-critic framework in continuous-input environments. Note: All figures and plots must be clearly labeled with axes titles and legends. Raw code snippets should be kept to a minimum in the report; focus on high-level logic and pseudo- code where necessary. 4 Important Note • Do NOT use Stable-baselines libraries or any other reinforcement learning specific libraries in your implementation (You may use tensorboard for recording your results). • Do NOT exceed the word count limit of 3000 words for each report, reference and appendix excluded. • Although you are allowed to use any generative AI tools to assist your work, please keep in mind that you should be using them responsibly. (Good use: Improve your report after writing it and always review its output to ensure that it is correct. Bad use: Copy-pasting an entire report from AI without any effort of your own. ) Submission Requirements Please prepare and submit the following documents: • A cover page featuring your student ID. This page should be the first page of your report. • A zip file containing all the source codes and your trained agent model, which should be named using your full name and student ID in the following format: CW1_ID_Name.zip • One PDF file for your report. The file should be separated from the zip file, which contains your code. The files should be named in the following format: CW1_ID_Name.pdf Note that the quality of the code, the clarity of your writing, and the format/style of your report will be taken into consideration during the evaluation. The detailed rubric is outlined below. Rubric CW1 (40 makrs) Criteria Marks Code Performance Code runs without errors and performs tasks as specified. 6 Code Quality Code is well-organized, includes meaningful comments, and uses appropriate variable names. 6 Methodology Comprehensive coverage of topics with detailed explanations of approaches and methodologies. 6 Result analysis Insightful analysis of results. 6 Report Quality Report is well-structured, formatted, and free of grammatical errors. 6 Evidence of Work All required elements are included and correct. 6 Submission Follows all requirements for submission 4 5