B0M33BDT – Technologie pro velká data

Classes will be held in a full-time manner, or until further notice. In the case of online teaching, teaching will be done through faculty MS Teams. We use the group B0M33BDT+BE0M33BDT.

Flash news:

• Computational cluster Metacentrum – please to the application form mention that you belongs to the group CVUT:FEL:B0M33BDT, ev. CVUT:FEL:A4M33BDT
• Guide to create your Hadoop in Metacentrum
• Our repository with support materials

It is necessary to have a basic knowledge of following technologies:

• Linux (basic commands)
• SQL (create table, SELECT, agg SELECT, JOIN)
• Python
  • typy list, tuple, dict, set
  • string manipulation
  • flow control (if, while, for)
  • function definition (def), lambda function
• basic regular expressions

We recommend bringing your own laptop that can connect to the internet. It is necessary to have applications for SCP connection (e.g. WinSCP) and for SSH connection (e.g. PuTTY). A smart text editor for writing Python and SQL scripts (Notepad++, PSPad, etc.) is also useful.

Classes are always held on Wednesdays. Classes were planned to be held in the building on Charles Square. For the duration of distance learning, links to the online classes will be listed with the respective week.

• odd week (L):
  • lecture 9:15–10:45, room KN:E-126
• even week, option A (S-A):
  • lecture 9:15–10:45, room KN:E-126
  • practice 2 parallels 11:00–12:30 a 12:45–14:15, room KN:E-307

The first week of the semester is odd, see below. In the case of full-time classes, the only limitation is the capacity of the room for the exercises (22 seats, others must sit next to, on a chair without a table or squeeze in with someone).

• 1. týden (S, 21. 9.): Organization, classification, motivation, overview
• presentation - PDF
• 2. týden (L, 28. 9.):
  • Státní svátek
• 3. týden (S, 5. 10.): Hadoop, cluster architecture
• presentation - PDF
  • Hadoop
• 4. week (L, 12. 10.): Storage, MapReduce, Hive
• presentation - PDF
  • Practices - Intro
• 5. week (S, 19. 10.): Ingest, architectures, security
• presentation - PDF
• 6. week (L, 26. 10.): Spark
• presentation - PDF
  • Practices - HDFS + Hive
• 7. week (S, 2. 11.): Streaming
• presentation - PDF
• 8. week (L, 9. 11.): Kafka
• presentation - PDF
  • Practices - Spark + midterm test + definition of homework
• 9. week (S, 16. 11.): Cloud intro
• presentation - PDF
• 10. week (L, 23. 11.): Cloud - Azure
• presentation - PDF
  • Practices - Azure + midterm test 2nd round
  • presentation - PDF - practice instructions CZ
  • presentation - PDF - practice instructions EN
• 11. week (S, 30. 11.): Azure workshop
• presentation - workshop - Intro PDF
• presentation - workshop - Spark PDF
• 12. week (L, 7. 12.): Serverless
• presentation - PDF
  • Practices

If someone does not want to be listed in the table, contact us and we anonymize your row.

• Obtaining at least 30 points out of 60 possible for the continuous test, homework and practical test.
• A maximum of 20 points can be earned on the mid-semester midterm test.
• homework maximum of 20, optionally successful completion of AZ-900 certification within the duration of the course
• max. 20 points for the theory and practice test at the end of the semester

• The content of the practical test includes tasks on data processing using Hadoop and its technologies. Everyone works independently, but it is allowed to use your own and internet materials: notes, sample codes from lectures and exercises, Google, Stack Overflow, etc.
• The practical test can be repeated once in a make-up period by agreement with the instructor. The result of the first test is cancelled and the result of the second test is valid, even if it is worse than the first test.

Check following pdf with tasks and check the list where you can see your assignment. Homeworks - PDF

Send your homework via e-mail - source code and the output is expected or you can send a link to your repository - source and the output is required there as well. There is no deadline for the homework but you have to get more than 0 points from it. If your delivery will not fulfill the task then the result is 0 points.

To get a free voucher for AZ-900 check this guide: https://www.microsoftpartnercommunity.com/t5/Partner-Skilling/How-to-get-free-Microsoft-Fundamentals-Certification-exam/m-p/52544

To prepare for it:

• Use MS Learn
• Nice video stream: https://marczak.io/az-900/

It has a written part for 20 points and an oral part for 20 points. Both are compulsory and may lead to the need to retake the exam. Location: IBC centrum - 6th floor, building C, Pobřežní 620, Karlín, 186 00 Praha 8 - from 09:00-11:30. In the first 15 minutes there will be a “moodle-like test” (ABCD multichoice) and right after the talk with an examiner. Slots will be available in KOS:

• 18.1.
• 25.1.
• 1.2.
• 8.2.

Marek Sušický

Hadoop: The Definitive Guide, 4th Edition, by Tom White