===== Syllabus ===== ^Lecture/ //Practice// ^Date ^Topic ^Lecturer ^Materials ^Notes ^ Feedback / discuss^ ^ 1. ^ 19.2 ^Recap: linear classifiers, linear regression, logistic regression, loss function, empirical risk minimisation, regularisation ^ BF ^ {{:courses:bev033dle:dl-recap.pdf| slides}}^ ^ [[https://cw.felk.cvut.cz/forum/thread-4727.html|thread]]^ | ::: | 20.2 | (no lab) | - | | || | || ||||| ^2.^ 26.2 ^Artificial neurons, activation functions, network architectures; sidestep: stochastic neurons; sidestep: biological neurons ^ BF ^ {{:courses:bev033dle:dl-neurons-nets.pdf| slides}}^ ^ [[https://cw.felk.cvut.cz/forum/thread-4728.html|thread]] ^ | ::: | 27.2 |Lab1: generative vs. discriminative learning I | BF | {{:courses:bev033dle:gen-bounds-1.pdf| task}} {{:courses:bev033dle:model.tgz| model}}| due 9.04|| | || ||||| ^3.^ 4.3 ^Neural networks as classifiers, empirical risk minimisation, loss functions, model complexity and generalisation bounds; neural networks as nonlinear regression models ^ BF ^ {{:courses:bev033dle:dl-losses-gen-bounds.pdf| slides}}^ ^ [[https://cw.felk.cvut.cz/forum/thread-4753.html|thread]] ^ | ::: | 5.3 |Seminar | BF | {{:courses:bev033dle:sem-recap-neurons-ss20.pdf| problems }}| || | || Reading: generalization bounds, VC dimension, large margin: [[https://winvector.github.io/margin/margin.pdf| (J. Mount)]], [[https://link.springer.com/content/pdf/10.1007%2F978-3-319-21852-6_6.pdf | (V. V. V’yugin, Theorem 6.8)]] ||||| ^4.^ 25.3 ^Backpropagation ^ AS ^ {{ :courses:bev033dle:backprop.pdf | slides}} \\ [[https://bbb.felk.cvut.cz/playback/presentation/2.0/playback.html?meetingId=6a90950eb4364003afb0597c4e2fcb1bb7333ec7-1585129801757|bbb]] [[http://bbb.felk.cvut.cz/download/presentation/6a90950eb4364003afb0597c4e2fcb1bb7333ec7-1585129801757/6a90950eb4364003afb0597c4e2fcb1bb7333ec7-1585129801757.mp4| mp4]] ^ ^ [[https://cw.felk.cvut.cz/forum/thread-4754.html|thread]] ^ | ::: | 26.3 |Lab2: generative vs. discriminative learning II | BF | {{ :courses:bev033dle:gen-bounds-2.pdf | assignment }} {{ :courses:bev033dle:lab-2-template.zip | template}} [[https://bbb.felk.cvut.cz/playback/presentation/2.0/playback.html?meetingId=7b032019efb3fbddc26968b07bbfbb0d55d8c6de-1585209901199|record 9:00]] | due 7.05 | [[https://cw.felk.cvut.cz/forum/thread-4819.html|thread]] | | || ||||| ^5.^ 1.4 ^Stochastic Gradient Descent ^ AS ^ {{ :courses:bev033dle:sgd.pdf | slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=51e761349ed16675151f2fb879e1fe839c6d1a14-1585731001126|bbb]] [[https://bbb04.felk.cvut.cz//presentation/51e761349ed16675151f2fb879e1fe839c6d1a14-1585731001126/51e761349ed16675151f2fb879e1fe839c6d1a14-1585731001126.mp4|mp4]] ^ ^ ^ | ::: | 2.4 |Seminar | AS | {{ :courses:bev033dle:sem-2-backprop-loss-functions.pdf | problems}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=21badc354d03f31f0b6792a6530b6e255a916779-1585817401982|record]] | || | ||Reading: Practical Recommendations for Gradient-Based Training [[https://arxiv.org/pdf/1206.5533.pdf | Y. Bengio]] ||||| ^6.^ 8.4 ^Convolutional Neural Networks ^ AS ^ {{ :courses:bev033dle:cnn.pdf | slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=f13f45d2dbf9a2963bfdda769aadcb7bdbcc741b-1586335801622 | bbb]] [[https://bbb04.felk.cvut.cz//presentation/f13f45d2dbf9a2963bfdda769aadcb7bdbcc741b-1586335801622/f13f45d2dbf9a2963bfdda769aadcb7bdbcc741b-1586335801622.mp4 | mp4]] ^ ^ ^ | ::: | 9.4 | No meeting -- Friday schedule | - | | || | || ||||| ^7.^ 15.4 ^Training neural networks 0: project pipeline, data collection, training/validation/test set, model selection (architecture), overfitting, early stopping ^ BF ^ {{ :courses:bev033dle:training-nns-0.pdf | slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=8348bce51958dc7b248ba8f981095572f5104b4f-1586940601419| bbb]] [[https://bbb04.felk.cvut.cz//presentation/8348bce51958dc7b248ba8f981095572f5104b4f-1586940601419/8348bce51958dc7b248ba8f981095572f5104b4f-1586940601419.mp4| mp4]]^ ^ [[https://cw.felk.cvut.cz/forum/thread-4926.html|thread]]^ | ::: | 16.4 | Lab3: PyTorch, training pipeline | AS | {{ :courses:bev033dle:lab-pytorch0.pdf | assignment}} \\ {{ :courses:bev033dle:mnist.py | template}}| due 21.05 || | || ||||| ^8.^ 22.4 ^Training neural networks 1: Data augmentation, Weight initialisation, Batch normalisation ^ BF ^ {{ :courses:bev033dle:training-nns-1.pdf | slides}} \\ [[ https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=fa3c02f31ba6c6336b55b86d292d182aef959ddc-1587545401830| bbb]] [[ https://bbb04.felk.cvut.cz//presentation/fa3c02f31ba6c6336b55b86d292d182aef959ddc-1587545401830/fa3c02f31ba6c6336b55b86d292d182aef959ddc-1587545401830.mp4| mp4]]^ ^ ^ | ::: | 23.4 | Seminar | AS | {{ :courses:bev033dle:sem-3.pdf | problems}} | | [[https://cw.felk.cvut.cz/forum/thread-4932.html|thread]] | | || ||||| ^9.^ 29.4 ^Training neural networks 2: Adaptive SGD methods ^ AS ^ {{ :courses:bev033dle:adaptive3.pdf | slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=a747a95490d0370e97afa7979fccf1c8656a3925-1588150201187|bbb]] [[https://bbb04.felk.cvut.cz//presentation/a747a95490d0370e97afa7979fccf1c8656a3925-1588150201187/a747a95490d0370e97afa7979fccf1c8656a3925-1588150201187.mp4|mp4]] ^ ^ ^ | ::: | 30.4 |Lab4: Pretrained CNN Finetuning | AS | [[courses:bev033dle:lab4| task ]] | due 28.05|| | || Reading: [[https://ruder.io/optimizing-gradient-descent | Overview of GD Optimization]], [[http://www.princeton.edu/~yc5/ele522_optimization/lectures/mirror_descent.pdf| Mirror Descent]] ||||| ^10.^ 6.5 ^Training neural networks 3: L2 weight regularization, dropout, and a bit beyond ^ AS ^ {{ :courses:bev033dle:regularizers.pdf | slides}}\\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=fc4f170103a8a407e0d628260143dbd28a2052c4-1588755001613|bbb]] [[https://bbb04.felk.cvut.cz//presentation/fc4f170103a8a407e0d628260143dbd28a2052c4-1588755001613/fc4f170103a8a407e0d628260143dbd28a2052c4-1588755001613.mp4|mp4]] ^ ^ ^ | ::: | 7.5 |Seminar | AS | {{ :courses:bev033dle:sem-4.pdf | problems}} \\ {{ :courses:bev033dle:seminar4-notes.pdf |notes}}| || | || ||||| ^11.^ 13.5 ^Training neural networks 4: adversarial patterns, robust learning approaches ^ BF ^ {{ :courses:bev033dle:adversarial.pdf | slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=d0df46bb1a938be59590584e55e65cb1dd4776f4-1589359801190|bbb]] [[https://bbb04.felk.cvut.cz//presentation/d0df46bb1a938be59590584e55e65cb1dd4776f4-1589359801190/d0df46bb1a938be59590584e55e65cb1dd4776f4-1589359801190.mp4|mp4]]^ ^ ^ | ::: | 14.5 |Lab5: Network visualisation, adversarial patterns | BF | {{ :courses:bev033dle:lab5.pdf | assignment}} \\ {{ :courses:bev033dle:template.zip | template}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=406d8772ab5c490a2017282b648fe5e8739fc0c5-1589446201992|record]] | due 11.06 || | || ||||| ^12.^ 20.5 ^Generative models: VAE, GANs (introductory level) ^ BF ^ {{ :courses:bev033dle:generative.pdf | slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=cad201d781a601a90e5094601b9ce78ff95ac0ac-1589964602097 | bbb]] [[https://bbb04.felk.cvut.cz//presentation/cad201d781a601a90e5094601b9ce78ff95ac0ac-1589964602097/cad201d781a601a90e5094601b9ce78ff95ac0ac-1589964602097.mp4|mp4]]^ ^ ^ | ::: | 21.5 |Lab6: Generative networks (optional/bonus lab) | BF | {{ :courses:bev033dle:vae.pdf | assignment}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=40774865bb4e0aa49783f8383dc02eb7a4b73872-1590051001676|bbb]] | submit before 18.6. || | || ||||| ^13 ^ 27.5 ^ Supervised representation and similarity learning ^ [[ http://cmp.felk.cvut.cz/~toliageo/ | Giorgos Tolias]] ^ {{ :courses:bev033dle:srsl.pdf|slides}} \\ [[https://bbb04.felk.cvut.cz//playback/presentation/2.0/playback.html?meetingId=76c01098dc012352ee48d35b084dd26dc20817a9-1590569401711|bbb]] [[https://bbb04.felk.cvut.cz//presentation/76c01098dc012352ee48d35b084dd26dc20817a9-1590569401711/76c01098dc012352ee48d35b084dd26dc20817a9-1590569401711.mp4|mp4]]^ ^ ^ | ::: | 28.5 | Seminar | BF | {{ :courses:bev033dle:sem-5.pdf | problems}}| || | || ||||| ^14.^ 3.6 ^Recurrent neural networks: recurrent back-propagation, RNN, GRU, LSTM ^ BF ^ {{ :courses:bev033dle:recurrent.pdf | slides}} ^ ^ ^ | ::: | 4.6 | (no lab) | | | || | || |||||