Lab 2: CNN Finetuning & Visualization

The main task is to fine-tune a pretrained CNN for a new classification task (transfer learning).

Skills: data loader from an image folder, data preprocessing, loading pretrained models, remote GPU servers, training part of the model. Insights: convolutional filters, error case analysis

In this lab we start from a model already pretrained on the ImageNet classification dataset (1000 categories and 1.2 million images) and try to adjust it for solving a small-scale but otherwise challenging classification problem.

• This will allow to work with a large-scale model at moderate computational expenses, since our fine-tuning dataset is small.
• We will see that a pretrained network has already learned powerful visual features, which will greatly simplify our task.
• We will consider two fine-tuning variants: adjusting the last layer or all layers.

It is a good time now to start working with GPU servers. Check How To page. The recommended setup is as follows:

1. SSH authentication with pre-shared keys
2. VS Code “Remote - SSH” extension
3. Lmod configuration loaded via `direnv` config

SOTA pretrained architectures are available in PyTorch. We will use the following models:

1. Squeezenet1_0 https://pytorch.org/hub/pytorch_vision_squeezenet/, which is a fast and small model but achieves somewhat lower accuracy on ImageNet.
2. ResNet-18 https://pytorch.org/hub/pytorch_vision_resnet/, which achieves better performance on ImageNet and is more stable in training.

import torchvision.models
model1 = torchvision.models.squeezenet1_0(weights=torchvision.models.SqueezeNet1_0_Weights.DEFAULT)
model2 = torchvision.models.resnet18(pretrained=True)

You can see the structure of the loaded model by calling print(model). You can also open the source defining the network architecture https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py. Usually it is defined as a hierarchy of Modules, where each Module is either an elementary layer (e.g. Conv2d, Linear, ReLU) or a container (e.g. Sequential).

The dataset is available on the gpu cluster at the path /mnt/datasets/PACS_cartoon/PACS_cartoon. You can also download it from here: PACS_cartoon (84Mb)

The PACS_cartoon dataset contains colored images of cartoons with 227×227 pixels and of 7 categories:

01: Dog
02: Elephant
03: Giraffe	
04: Guitar	
05: Horse
06: House
07: Person

Before we leverage the power of pre-trained models (Transfer Learning), we must first establish a baseline. In any machine learning project, a baseline serves as a “control group.” It answers the question: “How hard is this dataset to learn from scratch?”

By training a small CNN from scratch on your specific dataset, you will likely encounter the challenge of data scarcity. Without the millions of images used to train models like ResNet or VGG, your custom model may struggle to generalize or may start overfitting (memorizing the training data) quickly. This section effectively demonstrates the problem that Transfer Learning is designed to solve.

Steps:

1. Load the dataset.
2. Split the training data into validation and training.
3. Make sure you are using an appropriate pre-processing scheme.
4. Train a small CNN of your choice until validation accuracy begins to decrease for 50 epochs.
5. Find a good enough learning rate (justify it).
6. Report validation and training metrics (loss and accuracy) as graphs with epochs on x axis and metrics on y axis.
7. Report which model, what learning rate, and optimizer you used.

Now that you have a baseline, we will use Transfer Learning to improve performance. We will compare three distinct strategies:

• Linear Probing: We freeze the pre-trained backbone. The CNN acts as a fixed feature extractor (like a complex edge/texture detector), and we only train the final classification layer.
• Bias tuning: The first instance of parameter-efficient fine-tuning (PEFT). We keep the pre-trained backbone frozen while unfreezing all the biases and the last classification layer, as in Linear Probing.
• Tuning Last $N$ Layers: We unfreeze the final few convolutional blocks. This allows the model to adapt its high-level features (e.g., specific shapes or object parts) to your specific dataset while keeping the low-level features (edges, corners) intact.

Strategy A: Linear Probing

1. Load a pre-trained model (e.g., ResNet18, MobileNet, VGG16) trained on ImageNet.
2. Freeze all parameters in the convolutional base (requires_grad = False).
3. Replace the final classification head with a new layer matching your number of classes.
4. Train the model until convergence, similarly to the previous assignment.
5. Report training and validation curves as in the previous assignment.
6. Report final accuracy on test set.

Strategy B: Bias tuning

1. Take strategy A and unfreeze biases also.
2. Report training and validation curves as in the previous assignment.
3. Report final accuracy on test set.

Strategy C: Tuning Last $N$ Layers

1. Load the same pre-trained model.
2. Unfreeze the last convolutional block ($N$) (e.g., the last Sequential block in ResNet or the last 3-4).
3. Train the model until convergence, similarly to the previous assignment.
4. Save final accuracy on test set.
5. Unfreeze also the previous layer before the last $N-1$. Repeat steps 2-4 for $N-1, N-2, ... N-K $ until the beginning of the network.
6. Report curve, which will have test error on the y axis, and on the x axis, you will have the number of unfreezed layers from 1 to K

Visually prove that your training process actually organized the chaos of the initial random weights into structured feature detectors.

Steps:

1. Take one image from the test set.
2. Preprocess it.
3. Visualize activation maps after the first convolution layer as single-channel images (use some good colormap, e.g., cmap=“viridis”):
   1. The best model you get from the previous assignments.
   2. Randomly initialized model with the same architecture.
4. Describe what you see.

You should upload your results as a pdf file to BRUTE. The report is free-form. You can use LaTeX, docs, pdf export from Jupyter Notebook, etc. Make sure to include everything bolded in assignments in your report.