Using curvedness and spin images by Ahmed Ayadi

Using curvedness and spin images to find correspondences between two point clouds representing human faces

Institute for Human-Machine Communication Technische UniversitÂ¨at MÂ¨ unchen, Germany Univ.-Prof. Dr.-Ing. habil. G. Rigoll

Ingenieurpraxis

Author: Advisor:

Ahmed Ayadi Daniel Merget, M.Sc.

Started on: Handed in on:

23.02.2015 27.04.2015

Abstract In this work, we present an efficient algorithm to extract a set of corresponding points between two point clouds representing two different human faces. At first, a curvature based downsamling algorithm has been implemented and used on the 3D data to extract the most siginificant points for a common registration problem i.e. nose, mouth, eyes and ears points, but also so that the computational complexity caused by the large number of points can be reduced. Once this has been done, the algorithm computes a spin image for each oriented point of the downsampled data. A spin image is a sort of mapping between a three and a two dimensional space. To each point of the source data, we compute then the correlation between its spin image and some spin images of the target data. If some threshold of the linear correlation is reached, a corresponding point has thus been found : namely the point with the highest correlation. However, a small set of false positives has been detected. Their elimination can be easily implemented. A graphical interface has been also created. It allows a user to manually choose a point of the source cloud, and then to give how many corresponding points in the target cloud are to be shown. These points are then computed, plotted and labeled in an ascending order of the correlation of their respective spin images with the original spin image of the already selected point.

Contents 1 Mathematics 1.1 Shape index and curvedness . . . . . . . . . . . . . . . . . . . . . . . 1.2 Spin images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 1

2 Related Work 2.1 Principal curvature estimation . . . . . . . . . . . . . . . . . . . . . . 2.2 Spin images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 3 3

3 Own Work 3.1 The downsampling method . . 3.2 The generation of spin images 3.3 The searching method . . . . 3.4 The graphical interface . . . .

. . . .

4 4 4 5 6

4 Results 4.1 Downsampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Simulation and Matching . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Optimisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7 7 7 9

. . . .

References

. . . .

1 Mathematics 1.1

Shape index and curvedness

At each point p of a differentiable surface in 3-dimensional Euclidean space one may choose a unit normal vector (unique) and a unit tangent vector. We obtain therefore a normal plane i.e. the plane spanned by the two chosen vectors. This plane will then cut the surface in a plane curve, called normal section. This curve will in general have different curvatures for different normal planes i.e. for different tangent vectors at p. The principal curvatures at p, denoted k1 and k2 , are respectively the maximum and minimum values of this curvature. Shape Index and Curvedness are two functions R2 → R defined as follows : 1. SI : R2 → [0, 1] 2. CV : R2 → R

1.2

k1 + k2 1 − arctan( ) 2 k1 − k2 p k12 + k22 (k1 , k2 ) → 2 (k1 , k2 ) →

Spin images

The spin image is a surface representation technique that was initially introduced by Andrew E. Johnson in [1] and is used for surface matching and object recognition in 3-D scenes. An oriented point at a surface mesh vertex is defined by the 3-D position of the surface vertex (denoted as p) and a surface normal (denoted as n ). The other points of the cloud are then mapped into a 2D space using the map function So defined as follows :

1. Mathematics

So : R3 → R2

  x1 p α x = x2  → := kx − pk2 − n.(x − p)2 n.(x − p) β x3

Therefore, to generate a spin image at a point p, the algorithm takes every point x of the point cloud and computes its coordinates in the oriented point basis (p; n). To better undestand the meaning of (α, β) see the figure 1. The next step is to create a 2-D array that is going to represent the spin image generated using the 2-D basis. In order to achieve this goal, we have implemented the method descriped by Johnson in [1] and then modified it to best fit with our data.

Figure 1.1: Two coordinates can be calculated given an oriented point : α and β

2 Related Work 2.1

Principal curvature estimation

Principal curvatures and principal directions are fundamental local geometric properties. Once computed, the curvedness at each point can be easily found as shown in chapter 1. Various algorithms to estimate curvature have been proposed in literatures in recent years. They can be simply classified as two categories: direct curvature fitting and indirect curvature fitting though surface fitting. Goldfeather presented in his work [2] a surface fitting method : he tried to fit the neighbor points with a cubic polynomial surface. We have implemented this method but the time complexity was too large, and we have then opted for another method developed by Zhang et al [3].

2.2

Spin images

Spin images are a very useful tool in face matching operations. Many researches have already been presented on the matter. The thesis of Johnson in [1] is the text that came with the original idea of spin images. An article of Papadimitriou [4] was also very helpful by choosing the many parametrs (image width, bin size, resolution...) to generate such an image. We have tried to combine the two literatures and to take the ameliorations that Papadimitriou has presented.

3 Own Work 3.1

The downsampling method

The downsamling method takes a point cloud with a large number of vertices and then tries in two steps to extract the ones that are the most significant for the registration : 1. We use at first an already implemented function that computes mean curvature values and use them as weights for a random datasampling operation, so that the probability of being selected is proportional to the elements of weights. Thanks to this function, the size of the input data was dramatically reduced to n vertices with however a quite large number of outliers, where n is a number given by the user. 2. To eliminate this outliers, the curvedness values of the downsampled point cloud are computed. Because of the low size of this cloud, the operation is also very fast. The points are then sorted in an ascending order of their curvedness. Then the first m vertices are extracted, where m is as well another number given by the user. A possible call of the downsampling function is as shown below : 1 2 3 4

%f i r s t f a c e with n=2000 , m=1200 [ Ë&#x153; , i n d i c e s 1 , p o i n t s 1 , i n d e x 1 ]= s i s i S a m p l e 2 ( pC1 , 2 0 0 0 , 1 2 0 0 ) ; %second f a c e with n=3000 , m=1500 [ Ë&#x153; , i n d i c e s 2 , p o i n t s 2 , i n d e x 2 ]= s i s i S a m p l e 2 ( pC2 , 3 0 0 0 , 1 5 0 0 ) ;

3.2

The generation of spin images

Two important parameters had to be computed before the spin images can be generated : the bin size and the image width.

3. Own Work 1. We have set the bin size to be equal the resolution of the spin images. The resolution is defined as the median of all edge lengths in a mesh. We had therefore to triangulise our data so that we can compute this parameter from the obtained mesh. 2. We have used the formula in [3] to compute the image width. It is however important to note that we have opted for a quadratic image size. Once this parameters computed for every set of points (after downsamling), the spin images are then generated. Because of the low number of vertices in each downsampled set, we had to add points to get better results. For instance we call : 1 %r e a c h a t o t a l o f 15000 p o i n t s 2 i=a d d p o i n t s ( pC1 , i n d i c e s 1 , 1 5 0 0 0 ) ; 3 sampled1=pC1 ( i , : ) ; Finally, one can call : 1 spinImage1=SpinImages ( sampled1 , index1 , 1 , normals1 ) ; to generate the spin images. This is a much faster way than to use all the available vertices (more than 50000). It took approximately 60 seconds on Matlab to generate 2700 images using 15000 vertices for each.

3.3

The searching method

We do not expect to find a corresponding point for each vertex of the source data. That is why we have set a threshold, that tells us if such a point very probably could be found. First, a point of the source data is selected. Its spin image is computed or easily read if already generated. Second, either 1. The linear correlations between its spin image and every spin image of the target data are computed. or : 2. this correlation is only then computed, if the two images are likely to be similar, i.e. if the absolute difference of the number of zero elements between them is not too large. That is : we compute at first this differences, then we sort the points in an ascending order of it. The correlation is then computed with the m first points on the top of the list. The points(s) with the highest value(s) of correlation is/are then the best candidate(s) to be the corresponding point(s), if the correlation is larger than the mentioned threshold. A threshold of 0,5 gave thereby a good accuracy.

3. Own Work

3.4

The graphical interface

A graphical interface has been also designed. It allows a user to manually choose a point of the source cloud, and then to give how many corresponding points in the target cloud are to be shown. These points are then plotted. (see figure belows)

Figure 3.1: The graphical interface

4 Results 4.1

Downsampling

In figure 4.2, an overview of the downsamling with the two mentioned steps can be seen. One can rapidly deduce that the first step is not sufficient to accurately extract only the points we need. It must also be said, that for better results, we had to choose more points on the target face than on the source. Thus, the probability to find a good corresponding point will be higher, and the accuracy of the algorithm increase.

Figure 4.1: Downsampling : (1) original data (2) intermediate result (3) final result

4.2

Simulation and Matching

We took randomly two faces of the eight we had. For each face we have computed a set of feature points using the downsamling algorithm. Its spin images have been then generated. The whole operation (downsamling and generation) took approximately 70 seconds for a total of 4500 vertices (2000 source, 2500 target). Although the algorithm is best appropriate for one point, we have computed a corresponding point for each of the source vertices. This has taken good 10 minutes. We obtained

4. Results then two sets of 2000 points each : the source set, and the corresponding set. Besides, we had computed a vector of 2000 correlation factors to each pair of points. As said in the previous chapters, we had not expected to find an accurate correspondent for each source point. So we have eliminated pairs of points, whose correlations are lower than 0,5. Half of the points remained. (let n be their number). We wanted then to verify, if the corresponding points have been well chosen, i.e. if we can count on this method to get an accurate corresponding cloud. For this purpose, a matrix M of n colours (gradation) is generated. Then the source cloud was sorted by the x-coordinate and obtained two newly sorted clouds : 1 2 3 4

x=chosen1 ( : , 1 ) ; [ ˜ , i ]= sort ( x ) ; chosen1=chosen1 ( i , : ) ; chosen2=chosen2 ( i , : ) ; We plotted then the two clouds using the colors matrix as follows :

1 showPointCloud ( chosen1 , c o l o r s , ’ MarkerSize ’ , 1 2 0 ) ; 2 showPointCloud ( chosen2 , c o l o r s , ’ MarkerSize ’ , 1 2 0 ) ; and obtained the plots in figure 4.2.

Figure 4.2: Matching : (1) original points (2) corresponding points Some outliers can be detected, such as the green point in figure 4.2. Clearly, this point is supposed to be corresponding to one point of the left ear. We conjecture that the symmetry of human faces is the cause of this error. One can remove this outliers for example with inspecting the neighborhood of the original point. It is clear that this point is the only among his neighbours to have the green as color. Therefore, a possible method would be to detect such points and to delete them.

4. Results

4.3

Optimisation

The work we presented is to be optimised. For instance, the accuracy of the matching method would improve if other features in addition to spin images could be implemented. A voting scheme using spin images but also some other descriptors would give better results. So a corresponding point is only then one, if it satisfies some number of conditions (for instance correlation > 0.5 and so on..). The time complexity of the algorithm could be also ameliorated, so that the matching operation and then eventually the registration could be run in real-time. Especially the matching method is concerned.

Bibliography [1]

A.E. Johnson: Spin-Images: A Representation for 3-D Surface Matching. Doctoral dissertation, tech. report CMU-RI-TR-97-47, Robotics Institute, Carnegie Mellon University, August, 1997

[2]

J. Goldfeather: A Novel Cubic-Order Algorithm for Approximating Principal Direction Vectors. ACM Transactions on Graphics (TOG), 2004.

[3]

X. Zhang: Curvature Estimation of 3D Point Cloud Surfaces Through the Fitting of Normal Section Curvatures. Asiagraph 2008.

[4]

G. Papadimitriou: Spin Images. CVonline (Online Compendium of Computer Vision) 2010.