Surface Reconstruction

Now we assume that all cameras are calibrated with respect to the common (global) coordinate system. The last phase of 3D scene reconstruction is (dense) reconstruction of surface.

The task is to reconstruct dense point cloud. We will use stereo-matching, which provides dense correspondences in a pair of images. Prior to stereo, the images must be epipolarly rectified. The implementation of rectification (the rectify package) as well as the implementation of stereo-matching (the gcs package) is available in the code repository.

The images of a single pair are modified by homographies Ha and Hb during rectification. Using the original camera matrices of the pair, we need to compute the camera matrices belonging to these new images, thus belonging to the set of established correspondences.

Let P1 and P2 are the two cameras (full, including K) of the image pair before rectification, and F12 is the fundamental matrix between the images. Then:

[Ha, Hb, img1_r, img2_r] = rectify( F12, img1, img2 ); 
D = gcs( img1_r, img2_r, [] );
Par = Ha * P1;
Pbr = Hb * P2;

Now D contains disparities in image img1_r. This means that pixel [x;y] in the first rectified image corresponds to the pixel [x-D(y,x);y] in the second rectified image. This correspondence should be then triangulated using the cameras Par, Pbr and gives a single point of dense point cloud. All pixels with computed disparity (i.e. not NaN) thus give a point cloud corresponding to a part of scene surface.

This dense reconstruction should be done for different image pairs (img_i,img_j). Note that for each pair there is different fundamental matrix Fij, different homographies Ha, Hb and different rectified cameras Par, Pbr, i.e., single camera Pi gives different rectified cameras when part of different pairs. The triangulated points are all in the common coordinate system, so they can be simply merged together.

The choice of pairs to be used in stereo is not unequivocal. We recommend to use horizontal and vertical pairs constructed from neighbouring cameras. In the case of 4×3 image capturing scheme this leads to 9 horizontal and 8 vertical pairs. However, it may be needed to cover parts of surface by additional pairs.

1. Compute disparity maps between chosen image pairs. Show the results.

1. Construct the dense point cloud.
2. Export the point cloud and the cameras into vrml file.

Having the disparity maps and corresponding dense image clouds (one for every pair, common coordinate system), we can proceed to surface reconstruction. The implementation of Poisson surface reconstruction is available (code repository, psr package).

The dense points must be prepared for the reconstruction. For each pair store the 3D points into 3D matrix X of size (image_height x image_width x 3 ), such that a 3D point [x;y;z] corresponding to the pixel [i;j] in the first rectified image of the pair is stored as

X(i,j,1) = x
X(i,j,2) = y
X(i,j,3) = z.

Store NaN for pixels without 3D point (disparity is NaN). (Thus the matrix X should be pre-allocated as X=nan(H,W,3) for each pair). Then join all these matrices from all pairs into the cell array

Xall = { ... X, ... }

This cell array thus contains dense clouds from all the pairs and it is the input of the surface reconstruction algorithm:

psr( Xall )

The algorithm creates a surface model in a file psr.ply in the current directory. This model can be inspected and adjusted using e.g. Meshlab, or older Scanalyze.

Note: the psr package uses the attached external binary (available for Win and Lx, 32b and 64b), it must be present in the current directory.

Optionally, the resulting surface can be coloured using the input images. For this purpose, the surface must be read back to matlab, particular vertexes must be re-projected into input images using the known camera matrices. Then the colour for each vertex can be set from proper pixel(s) in the image(s). Some helper functions are available in the ply_color package (code repository).

1. Create a model of scene surface using Poisson surface reconstruction. Trim the model in some 3D editor (e.g. meshlab) and export to vrml.