CourseWare Wiki
Switch Term
Summer 2023 / 2024
Summer 2022 / 2023
Summer 2021 / 2022
Summer 2020 / 2021
Summer 2019 / 2020
Summer 2018 / 2019
Summer 2017 / 2018
Search
Log In
b232
courses
smu
lectures
Differences
This shows you the differences between two versions of the page.
View differences:
Side by Side
Inline
Go
Link to this comparison view
Both sides previous revision
Previous revision
2024/05/29 15:07 kuzelon2 [Lecture 3 - Reinforcement Learning 3]
2024/05/20 15:11 zelezny [Literature]
2024/05/20 14:36 tothjan2
2024/05/20 14:35 tothjan2
2024/04/29 15:15 souregus
2024/04/22 15:03 souregus
2024/04/15 14:57 souregus [Lecture 8 - Natural Language Processing 2]
2024/04/08 10:29 souregus [Lecture 7 - Natural Language Processing 1]
2024/03/26 09:52 tothjan2 [Literature]
2024/03/25 15:51 kuzelon2 [Lecture 6 - Reinforcement Learning 6]
2024/03/18 15:00 kuzelon2 [Lecture 5 - Reinforcement Learning 5]
2024/03/18 14:57 kuzelon2 [Lecture 4 - Reinforcement Learning 4]
2024/03/18 14:57 kuzelon2 [Lecture 4 - Reinforcement Learning 4]
2024/03/18 14:54 kuzelon2 [Lecture 3 - Reinforcement Learning 3]
2024/03/11 15:59 kuzelon2 [Lecture 4 - Reinforcement Learning 4]
2024/03/04 14:47 kuzelon2 [Lecture 3 - Reinforcement Learning 3]
2024/02/26 15:02 kuzelon2 [Lecture 1 - Reinforcement Learning 1]
2024/02/26 15:01 kuzelon2 [Lecture 1 - Reinforcement Learning 1]
2024/02/26 14:49 kuzelon2 [Lecture 1 - Reinforcement Learning 1]
2024/02/26 14:45 kuzelon2 [Lecture 2 - Reinforcement Learning 2]
2024/02/16 15:13 souregus
2024/02/16 15:12 souregus [Lecture 7 - Natural Language Processing 1]
2024/02/16 14:07 kuzelon2
2024/02/09 10:14 external edit
Go
Previous revision
2024/05/29 15:07 kuzelon2 [Lecture 3 - Reinforcement Learning 3]
2024/05/20 15:11 zelezny [Literature]
2024/05/20 14:36 tothjan2
2024/05/20 14:35 tothjan2
2024/04/29 15:15 souregus
2024/04/22 15:03 souregus
2024/04/15 14:57 souregus [Lecture 8 - Natural Language Processing 2]
2024/04/08 10:29 souregus [Lecture 7 - Natural Language Processing 1]
2024/03/26 09:52 tothjan2 [Literature]
2024/03/25 15:51 kuzelon2 [Lecture 6 - Reinforcement Learning 6]
2024/03/18 15:00 kuzelon2 [Lecture 5 - Reinforcement Learning 5]
2024/03/18 14:57 kuzelon2 [Lecture 4 - Reinforcement Learning 4]
2024/03/18 14:57 kuzelon2 [Lecture 4 - Reinforcement Learning 4]
2024/03/18 14:54 kuzelon2 [Lecture 3 - Reinforcement Learning 3]
2024/03/11 15:59 kuzelon2 [Lecture 4 - Reinforcement Learning 4]
2024/03/04 14:47 kuzelon2 [Lecture 3 - Reinforcement Learning 3]
2024/02/26 15:02 kuzelon2 [Lecture 1 - Reinforcement Learning 1]
2024/02/26 15:01 kuzelon2 [Lecture 1 - Reinforcement Learning 1]
2024/02/26 14:49 kuzelon2 [Lecture 1 - Reinforcement Learning 1]
2024/02/26 14:45 kuzelon2 [Lecture 2 - Reinforcement Learning 2]
2024/02/16 15:13 souregus
2024/02/16 15:12 souregus [Lecture 7 - Natural Language Processing 1]
2024/02/16 14:07 kuzelon2
2024/02/09 10:14 external edit
Go
courses:smu:lectures [2019/06/11 00:11]
courses:smu:lectures [2024/05/29 15:07]
(current)
kuzelon2
[Lecture 3 - Reinforcement Learning 3]
Line 1:
Line 1:
+
====== Lectures ======
+
Lectures are given in person. Videos will be provided, but not necessarily directly from the lectures (the form is at the discretion of the individual lecturers).
+
===== Literature =====
+
+
In the lecture descriptions below, we refer to this **supplementary course material**:
+
+
**Relevant**
+
+
* [[https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf|RL]]: // R. S. Sutton, A. G. Barto: Reinforcement learning: An introduction. MIT press, 2018. //
+
* [[https://web.stanford.edu/~jurafsky/slp3/|NLP]]: // D. Jurafsky & J. H. Martin: Speech and Language Processing - 3rd edition draft //
+
* [[https://direct.mit.edu/books/book/2604/An-Introduction-to-Computational-Learning-Theory|COLT]]: // M. J. Kearns, U. Vazirani: An Introduction to Computational Learning Theory, MIT Press 1994//
+
+
+
RL & NLP are available online.
+
+
The RL part of the course is heavily based on the RL course of prof Emma Brunskill. The relevant lectures from prof Brunskill's course are: [[https://youtu.be/FgzM3zpZ55o|Lecture 1]],[[https://youtu.be/E3f2Camj0Is|Lecture 2]], [[https://youtu.be/dRIhrn8cc9w|Lecture 3]], [[https://youtu.be/j080VBVGkfQ|Lecture 4]], [[https://youtu.be/buptHUzDKcE|Lecture 5]], [[https://youtu.be/gOV8-bC1_KU|Lecture 6]], [[https://youtu.be/RN8qpSs8ozY|Lecture 11]].
+
+
/* There are nice materials by Volodymyr Kuleshov and Stefano Ermon on probabilistic graphical models (for the Bayesian networks part of the course): [[https://ermongroup.github.io/cs228-notes/]]. The relevant chapters are: [[https://ermongroup.github.io/cs228-notes/representation/directed/]], [[https://ermongroup.github.io/cs228-notes/inference/ve/]], [[https://ermongroup.github.io/cs228-notes/inference/sampling/]]. */
+
+
The NLP part of the course is heavily based on the NLP course(s) from Dan Jurafsky (Stanford), following his book: Speech and Language Processing (see NLP above) - particularly its 3rd edition draft (2nd ed. is insufficient!). The relevant chapters for us are 3, 6, 7 and 9. There are also some nice related [[https://web.stanford.edu/~jurafsky/slp3/|materials]] and [[https://www.youtube.com/channel/UC_48v322owNVtORXuMeRmpA|videos]]
+
+
For the COLT part: besides the monograph by Kearns et al linked above, the [[https://en.wikipedia.org/wiki/Computational_learning_theory#|Wikipedia page]] has pointers to two COLT survey papers (Angluin, Haussler) which are relevant to the PAC part. There are also external courses with lecture material available; for example, [[http://www.cs.cmu.edu/~ninamf/ML11/index.html|8803 Machine Learning Theory]] at Georgia Tech covers all COLT topics of SMU (there are subtle differences in the algorithms and proofs). Video footage of the lectures available [[https://www.youtube.com/playlist?list=PLQL6z4JeTTQlgt77fhOe2Jovjjz4THF_G|here]].
+
+
----
+
+
==== Lecture 1 - Reinforcement Learning 1 ====
+
+
**Slides:** {{ :courses:smu:lecture_1_video_version_2024.pdf |Slides}}
+
+
**Videos:** [[https://www.dropbox.com/s/hxbej4u8k4ni3zn/lecture_1_part_1.mp4?dl=0|Markov Processes]], [[https://www.dropbox.com/s/o7vbas1z6tuar3i/lecture_1_part_2.mp4?dl=0|Markov Reward Processes]], [[https://www.dropbox.com/s/ebotxr9e7l0enx8/lecture_1_part_3.mp4?dl=0|Markov Decision Processes]], Proofs are not part of the videos - they are not a compulsory part, although they are in the slides and will be discussed in the lecture (time permitting// - update after the lecture: we did not have time to go over the proofs//).
+
+
**A Note about organization of the first six lectures on reinforcement learning:** //These first four lectures will be about reinforcement learning. You will have three different sources of materials to choose from://
+
+
//1. Lectures given in-person on Mondays.//
+
+
//2. Videos that will be uploaded before each lecture on the course web page.//
+
+
//3. Course videos by Professor Emma Brunskill (the relevant lectures from her course are linked to above).//
+
+
//Option 1 is the traditional one. Option 2 is kind of a minimal version. I will try to make the videos concise and shorter than the lectures but I will still aim for them to cover all the important material - we will not use lecture videos from the last year, there will be new ones recorded. Option 3 will probably be most rewarding if you want to learn reinforcement learning in greater depth (and I recommend it - Prof. Brunskill’s course is great).//
+
+
//Now regarding the first lecture, it is a brief recap of Markov decision processes (MDPs) because not all students of the course have the same background from their previous studies. If you feel confident about your knowledge of MDPs, feel free to skip the lecture (but still come to the exercise session).//
+
+
**Note:** //This lecture is heavily based on a lecture by Prof Emma Brunskill (all potential errors are likely mine).//
+
+
**Relevant videos from Prof Brunskill's course:** [[https://youtu.be/FgzM3zpZ55o|Lecture 1]],[[https://youtu.be/E3f2Camj0Is|Lecture 2]].
+
+
**Typos in the videos:** There is a typo on the slide 27 titled "Computing Value Function (2/3)". There is a missing gamma by which the state transition matrix should be multiplied. This has been corrected in the slides but not in the video.
+
+
----
+
==== Lecture 2 - Reinforcement Learning 2 ====
+
+
**Slides:** {{ :courses:smu:lecture_2_2023.pdf |Slides}}
+
+
**Videos:** [[https://www.dropbox.com/s/11wwqttnzd8cxdk/lecture_2_part_1.mp4?dl=0|Short Recap]], [[https://www.dropbox.com/s/22qgeuz75ha8d38/lecture_2_part_2.mp4?dl=0|Problem Statement and Statistical Properties of Estimators]], [[https://www.dropbox.com/s/eqpwyy24rycwztr/lecture_2_part_3.mp4?dl=0|Monte-Carlo Policy Evaluation Methods]], [[https://www.dropbox.com/s/25oc27xnlbmz1ba/lecture_2_part_4.mp4?dl=0|Temporal Difference Policy Evaluation Methods]].
+
+
**Important:** //There is a typo on several of the slides in the video in the pseudocode of First-Visit and Every-Visit Monte-Carlo algorithms. On the line where G(s) is updated, there should be g_{i,t} instead of g_{i,1}. This has been fixed in the slides, but not in the videos.//
+
+
In this lecture, we will talk about model-free policy evaluation methods. We will end the lecture with a short discussion of model-free control with which we will start next week.
+
+
**Note:** //This lecture is heavily based on a lecture by Prof Emma Brunskill (all potential errors are likely mine). //
+
+
**Relevant videos from Prof Brunskill's course:** [[https://youtu.be/dRIhrn8cc9w|Lecture 3]]
+
+
----
+
==== Lecture 3 - Reinforcement Learning 3 ====
+
**Slides:** {{ :courses:smu:lecture_3_2024.pdf |Slides}}
+
+
**Videos:** [[https://www.dropbox.com/s/qczd4ihbgp52o8l/lecture_3_part_1.mp4?dl=0|Short Recap]], [[https://www.dropbox.com/s/gbbfge8ki6fwjhc/lecture_3_part_2.mp4?dl=0|Model-Free Control - Problem Statement]], [[https://www.dropbox.com/s/27gvsdz2m503xlv/lecture_3_part_3.mp4?dl=0|Monte-Carlo On Policy Iteration]], [[https://www.dropbox.com/s/kgzq785gph2th9r/lecture_3_part_4.mp4?dl=0|SARSA]], [[https://www.dropbox.com/s/mg6z730j7kjeghr/lecture_3_part_5.mp4?dl=0|Q-Learning]].
+
+
**Note:** //This lecture is heavily based on a lecture by Prof Emma Brunskill (all potential errors are likely mine). There was a typo in the pseudocode of SARSA - the action sampled inside the loop should be sampled from \pi(s_{t+1}) instead of \pi(s_t). The error remains in the video. //
+
+
**Erratum:** There is an error on slide 73, where we should be updating Q(b,right) - we are always updating the state with the "small" ladybug symbol, Thanks to M. Komínek for discovering this error.
+
+
**Relevant videos from Prof Brunskill's course:** [[https://youtu.be/j080VBVGkfQ|Lecture 4]]
+
----
+
==== Lecture 4 - Reinforcement Learning 4 ====
+
+
+
**Slides:** {{ :courses:smu:lecture_4_2024.pdf |Slides}}
+
+
**Videos:** [[https://www.dropbox.com/s/t9ilpbbm65tsenb/lecture_4_part_1.mp4?dl=0|Intro + Short Recap]], [[https://www.dropbox.com/s/2mgxg2yzhdzvejb/lecture_4_part_2.mp4?dl=0|RL with Value Function Approximation - Problem Description]], [[https://www.dropbox.com/s/ut1l8altlnrrqi2/lecture_4_part_3.mp4?dl=0|Some More Background (mostly SGD)]], [[https://www.dropbox.com/s/cqi1hf7j0bofq3v/Lecture%204%2C%20Part%204.mp4?dl=0|Policy Approximation with VFA]], [[https://www.dropbox.com/s/y6v058n7hfqca6x/lecture_4_part_5.mp4?dl=0|Control with VFA]].
+
+
**Note:** //This lecture is heavily based on a lecture by Prof Emma Brunskill (all potential errors are likely mine). //
+
+
**Relevant videos from Prof Brunskill's course:** [[https://youtu.be/buptHUzDKcE|Lecture 5]], [[https://youtu.be/gOV8-bC1_KU|Lecture 6]], [[https://youtu.be/RN8qpSs8ozY|Lecture 11]].
+
----
+
==== Lecture 5 - Reinforcement Learning 5 ====
+
+
**Slides:** {{ :courses:smu:lecture_5_2024.pdf |Slides}}
+
+
**Videos:** [[https://www.dropbox.com/s/e6pzzo97brnzuo1/lecture_4_part_6.mp4?dl=0|Bandits]]. There is no video for the proofs part of the lecture, which is not compulsory material and will not appear in the exam.
+
+
**Note:** //This lecture is heavily based on a lecture by Prof Emma Brunskill (all potential errors are likely mine). //
+
+
**Relevant videos from Prof Brunskill's course:** [[https://youtu.be/RN8qpSs8ozY|Lecture 11]].
+
+
----
+
==== Lecture 6 - Reinforcement Learning 6 ====
+
+
**Slides:** {{ :courses:smu:lecture_6_2024.pdf |Slides}}
+
+
**Videos:** Coming soon.
+
+
**Note:** //This lecture is heavily based on a lecture by Prof Emma Brunskill (all potential errors are likely mine). //
+
+
**Relevant videos from Prof Brunskill's course:** [[https://youtu.be/jJ7JbQBTChM?si=r_DCDP8Y8p7HdZXh|Lecture 12]].
+
+
----
+
==== Lecture 7 - Natural Language Processing 1 ====
+
+
**Probabilistic models:** [[https://drive.google.com/file/d/1EXwaraCccUibzDv3YHmiYTL6aNBF-JsQ/view?usp=share_link | slides]]
+
+
**Video:** [[https://drive.google.com/file/d/1EVKHjgz2CAzBHLYbkc7TVjTs3_mgh5Nz/view?usp=share_link |google-drive]]
+
+
----
+
==== Lecture 8 - Natural Language Processing 2 ====
+
+
** Vector models: ** [[https://drive.google.com/file/d/1EiAeLQLxy2F_fE-ZPxuIMVFi-CUUl0AA/view?usp=share_link| slides]]
+
+
**Video:** [[https://drive.google.com/file/d/1EhQ0bVdZlbhHkUFRZ1Gcd8U_woDmelDN/view?usp=share_link |google-drive]]
+
+
----
+
==== Lecture 9 - Natural Language Processing 3 ====
+
+
** Matrix models: ** [[https://drive.google.com/file/d/1GuHzTu5XWBj9vqvfdkfCIL2AeFg6ao-L/view?usp=share_link | slides]]
+
+
**Video:** [[https://drive.google.com/file/d/1EoEjaxDXBwLj_Lctiwel9fS402v0JD8K/view?usp=sharing | google-drive]]
+
+
----
+
==== Lecture 10 - Natural Language Processing 4 ====
+
+
** Neural models: ** [[https://drive.google.com/file/d/1HufWicAqTYJvJLel8mEGbEEEdSoak-Dr/view?usp=share_link | slides]]
+
+
**Video:** [[https://drive.google.com/file/d/1HvOZf9kvidmgGtmQcAHDUnp_R9MCdWyb/view?usp=share_link | google-drive]]
+
+
----
+
==== Lecture 11 - Computational Learning Theory 1 ====
+
**Slides:** {{ :courses:smu:colt-1.pdf | COLT - lecture 1}}
+
+
**Video:** [[https://www.youtube.com/watch?v=e6XCe84AYEc&list=PLQL6z4JeTTQlgt77fhOe2Jovjjz4THF_G&index=2 | COLT - lecture 1]]
+
----
+
+
==== Lecture 12 - Computational Learning Theory 2 ====
+
**Slides:** {{ :courses:smu:colt-2.pdf |COLT - lecture 2}}
+
+
**Video:** [[https://www.youtube.com/watch?v=1oK1zcl7lpA&list=PLQL6z4JeTTQlgt77fhOe2Jovjjz4THF_G&index=3 | COLT - lecture 2]]
+
----
+
+
==== Lecture 13 - Computational Learning Theory 3 ====
+
**Slides:** {{ :courses:smu:colt-3.pdf |COLT - lecture 3}}
+
+
**Video:** [[https://www.youtube.com/watch?v=6DIJdKXoggY&list=PLQL6z4JeTTQlgt77fhOe2Jovjjz4THF_G&index=4 | COLT - lecture 3]]