Benchmarking
Task assignment
Your task is to measure time performance of three different implementations of matrix multiplication and compare them:
Standard multiplication
Multiplication with the second matrix transposed before the multiplication
Multiplication of the matrices in 1D representation
You probably need to do the following steps:
Open the source code in IDE
Use Java Microbenchmark Harness (JMH) to rigorously compare the implementations
Do the measurements on each of the implementations of the matrix multiplication according to the methodology described in [1] - summary in
PDF
Determine the warm-up period for each implementation
You do not have to calculate the number of repetitions
Use sufficiently large number (e.g. 40 iterations and 30 executions/forks)
If the resulting confidence interval is too wide, you need to add more repetitions.
Measure the time performance for each implementation and compute the average performance and 95% confidence interval of the measurements (Section 9.3)
Compute the comparison ratios of the implementations and 95% confidence intervals of the ratios (Section 10.1)
Upload the report (PDF) with the results together with your implementation of the benchmark (MatrixMultiplicationBenchmark.java) and the JSON file with the measurements generated by JMH.
The confidence interval you are supposed to calculate are NOT the confidence interval that is shown by JMH
Report structure
The report should include the following parts:
Report templates: doc LaTeX
JMH
Materials
[2] Kalibera, T. and Jones, R. E. (2012) Quantifying performance changes with effect size confidence intervals. Technical Report 4–12, University of Kent
[3] The
slides from the third lecture