Main topic of today's lab is the problem of image fusion by means of image registration. There are often multiple images acquired for a single patient – this can be MR, CT at one session or a control MR few months after a previous scan, etc. For (clinical) evaluation of these data, we want the corresponding structures in the images to overlap in the best possible way. But the patient is not completely fixed in the scanner, so the position (in world coordinate system(!)) will in general be different for different scans. Restoring overlap of two image $I, J$ can be seen as finding a transform $T$ for one of the images such that an overlap error $E$ is minimized $$ T^{*} = \mathop{\arg\min}_T E(I, T(J))$$

Thus, the key concepts in image registration are the choice of transformation and the choice of the error function, typically called (image) metric.

Transformation Knowing the simplest type of transformation needed for aligning the two input images helps us to restrict the search only to a class of transformations. We may search only for

• translation if the images are only shifted to each other,
• tr. and rotation (rigid transform) for transforming rigid objects (objects not changing in size)
• tr., rot. and scaling, shearing (affine transform) for arbitrary linear transforms
• deformation vector field the most general case, the image may be locally deformed

Metric Optimal image registration metric $E(I, J)$ has its global minimum when the image $I, J$ overlap perfectly. If both $I$ and $J$ are of same modality and have similar intensity values, the sum of squared distances (SSD) between pixels pairs $(\mathbf{x}, t(\mathbf{x}))$ for all pixels $\mathbf{x}$ of image $I$ is such optimal metric.

The SSD metric will not work in case the images $I, J$ have different intensities, which occurs often in medical imaging. For instance when $I$ – CT image, $J$ – MR image, or when $I, J$ are acquired with different MR sequences (T1-weighted, T2-weighted, FLAIR, etc). In this cases, the (normalized) mutual information metric can be applied.

If the intensity / color information is not suitable for defining an image metric, we can formulate the registration problem as alignment of two set of points (corresponding landmarks in both images). The criterion in this case is usually the mean distance between the transformed moving image landmarks and the fixed image landmarks.

In today's homework, we will try out different registration techniques on several pairs of images – download them in a zip-archive.

[1.5 pt] Landmark registration (Matlab) We will use MATLAB for this part. Load the images histology.HEStain.tif and histology.PanCytokeratin.tif that show similar histology slices in two different stain colouring. The task is to find the best transform, that will map one image on each other. Due to the differences in the image intensities, we will use manually set landmarks to compute the transform.

1. call the cpselect tool and set corresponding landmarks, when closing the tool, you will be prompted to store the points – we will need them for computing the transformation. Let the PanCytokreatin image Ipc be the moving, HEStain image Ihe the fixed one.
2. compute the transformed moving image by first estimating the transform fitgeotrans and then warping the moving image with imwarp

       nrefsim_transform = fitgeotrans(moving_points, fixed_points, 'nonreflectivesimilarity');
       imwarp(Ipc, hist_nrfsim, 'OutputView', imref2d(size(Ihe)))

3. repeat step 2 for (at least 3) different choices of transforms available in fitgeotrans
4. use imshowpair function or the provided image_checkerboard to make joint visualization of the fixed and the transformed (warped) moving image. Place the visualizations for all tested transform types into your report.
5. Comment on the results and decide which of the tested transforms provided the best alignment of the two images

[2 pt] Multi-modal registration (MITK) Align the images T1.nii and FLAIR.nii, visualize the result and evaluate the histogram of intensities in image T1 in the lesion area (marked by LesionSeg-labels.nii). The segmentation is aligned with the FLAIR image so pay attention to which image is set as fixed/moving. Put screenshots into your report.

Due to contrast differences of the two MR sequences, we need to use one of the MultiModal algorithms. To also correct for acquisition artifacts, we want to search for affine transform, even though the images are from the same subject and the rigid transform would seem to be enough.

• Pipeline
  1. Load images, create a joint visualization before registration, put screenshot into report
  2. Select the proper MultiModal registration algorithm and perform registration
  3. Visualize the result by a method of your choice (checkerboard, color blend, …) and insert screenshot into report
  4. Compute the value histogram of T1 image in the lesion region (MITK plug-in Statistics)

[1.5 pt] Deformable registration (MITK) The most complex registration task is the mapping of images with different geometries. The transform in such case is a deformation (vector) field. The task is to applyit on two pairs of images.

• Input data
  1. take two of the US_heart images, which show two different time-steps of a 3D+time acquisition of the mitral valve
• Pipeline
  1. Load images and create a joint visualization before registration (screenshot into report)
  2. Select the deformable registration algorithm (Demons) and apply to the images
  3. Visualize the result with the joint visualization
  4. Inspect the result: Did the registration work perfectly? Are there some areas with high error (mismatch)? Does the resulting image
  5. Create a visualization of the deformation: open the MatchPoint Registration Visualizer plugin, select the registration object and visualize also the 3D deformation (either as grid or as vector field glyphs)

We will use the MatchPoint plug-ins from MITK Workbench for registration. We start with loading the two images and visualizing their differences with the MatchPoint Registration Evaluator. After opening the plugin, select the two images and hit Start Evaluation. You can select different visualization styles – Blend, Checkerboard, Color blend etc.

We then need to select the registration algorithm.

1. Open the MatchPoint Algorithm Browser plugin, select a suitable registration algorithm from the list.
2. Open the MatchPoint Algorithm Control plugin, the Selection tab shows the algorithm selected in previous step, hit Load selected algorithm. The plugin switches automatically to the tab Execution after the algorithm is loaded.
3. Select the two images in Data Manager. The first selected image is set as moving, the second as fixed.
4. Now hit Start algorithm button to run the registration
5. Two new data objects should appear, the registration object Reg #n and the transformed moving image Reg #n mapped moving data

Inspect the registration results in the MatchPoint Registration Evaluator plugin. This time, select the registration object and Start Evaluation.