Computer Aided Process as 3D Data Provider in Footwear Industry Bita Ture Savadkoohi and Raffaele De Amicis
Abstract One of the new emerging of ubiquitous manufacturing is Mass Customization. The combination of 3D scanning systems with mathematical technique makes possible the development of CAD system which can help the selection of good footwear for a given customer. During the surface reconstruction process, a mesh is calculated from points cloud. This mesh may have holes corresponding to deficiencies in the original point data. Although these data are given in an arbitrary position and orientation in 3D space. Moreover, there is a need to mesh segmentation in order to shape retrieval form shoe last data base. Thus to apply sophisticated modeling operation on data set, substantial preprocessing is usually required. In this paper first, we describe an algorithm for filling hole. Then all of the models are aligned and at the end the models are segmented in order to use for further analysis. Keywords Mass customization ⋅ Filling hole ⋅ Alignment ⋅ Segmentation
1 Introduction Given the changing characteristics of consumer’s foot and industrial competition, it is imperative to adapt mass customization. The design of new shoes starts with the design of the new shoe last which is a mechanical form of human foot model. In such industry the shoe last should be designed rapidly from the individual foot model [1, 2]. Due to both of the object complexity and scanning process, some areas of the object outer surface may never be accessible, thus obtained data typically contains B. Ture Savadkoohi (✉) Department of Computer and Electrical Engineering, Seraj Higher Education Institute, Next to Municipal Museum, Maghsoodieh Avenue, Nowbar Alley, Tabriz, Iran e-mail:
[email protected] R. De Amicis Fondazione GraphiTech, Via Alla Cascata 56/c, 38123 Povo, Trento, Italy e-mail: Raff
[email protected] © Springer International Publishing Switzerland 2016 G. De Pietro et al. (eds.), Intelligent Interactive Multimedia Systems and Services 2016, Smart Innovation, Systems and Technologies 55, DOI 10.1007/978-3-319-39345-2_41
471
472
B.Ture Savadkoohi and R. De Amicis
missing pieces and holes. However this deficiency is not acceptable, where the geometric models are using in design processing in manufacturing. Although data from 3D scanning are given in arbitrary position in 3D scape and moreover there is need meaningful sub-meshes in order to best fit shape retrieval. Thus reverse engineering (RE), automatic alignment and segmentation have become an important step in the design, manufacture of new product and shape analysis [1]. An approach for preprocessing of 3D data for manufacture of new product and shape analysis is proposed in this paper. This paper in structures as follows: Sect. 2 is presented the filling hole in triangle mesh for building complete 3D model. An alignment of 3D foot model with shoe last data base is described in Sect. 3 while segmentation for shape retrieval is presented in Sect. 4. Finally, conclusion and remarks are summarized in Sect. 5.
2 Hole Filling Algorithm Since several factors such as occlusion, low reflectance or even missing pieces in the original geometry can lead to incomplete data and this deficiency is not acceptable when the 3D model is taking into actual application. Thus certain repairs must be done before taking these models into application. Existing approaches to fill holes in meshes can be distinguished in two main categories: the geometric and nongeometric approaches [3, 4]. A hole filling process that is applied here [5] is summarized as it follows: first, identify the holes automatically by looking close loop of boundary edges. Next, cover the holes with Advanced Front Mesh technique [6]. Then, estimate desirable normal instead of relocating them directly for modification of triangles in initial patch mesh. Finally, rotate triangles by local rotation and reposition these coordinate by solving the Poisson equation according to desirable normal and boundary vertices of the hole in order to make algorithm more accurate. As shown in Fig. 1a, all triangles that share one common vertex are called 1-ring triangle of vertex, all edges that share one common vertex are called 1-ring edges of the vertex and all vertices on 1-ring edges of a vertex (except itself) are called 1-ring vertex. Two triangles that share a common edge, they called adjacent triangle.
Fig. 1 a Preliminaries related to triangle mesh and hole. b Angles opposite to edge vi , vj
Computer Aided Process as 3D Data Provider in Footwear Industry
473
Fig. 2 Initial patch mesh generation: 𝜃 ≤ 75◦ , 75◦ ≤ 𝜃 ≤ 135◦ , 𝜃 ≥ 135◦
A boundary edge is adjacent to exactly one triangle and a boundary vertex is a vertex used to define a boundary edge. Thus, a closed cycle of boundary edges defies a hole. A boundary triangle is a triangle that owns one or two boundary vertices. The Advanced Front Mesh technique is applied over the hole to generate an initial patch mesh as it follows: First, for each vertex vi on the front calculate the angle 𝜃 between two adjacent boundary edges. Next, depending on the angle between ei and ei+1 build the new triangles on the plane, see Fig. 2. Then, calculate the distance between new vertex and related boundary vertices, if distance between them is less than given threshold, then they should merge. Then, the triangles involved in the initial patch mesh are modified by estimating their desirable normals instead of relocating them directly. The most important task of discrete harmonic function is to map a given disk-like ST onto a plane S∗ which is introduced by Eck et al. [7]. The goal is to find a suitable piecewise linear mapping such as f ∶ ST → S∗ to minimize Dirichlet energy as follows: E=
1 ‖gradST f ‖2 ds 2 ∫ST
(1)
Consider one triangle T = {v1 , v2 , v3 } in the surface ST . The Dirichlet energy can be expressed as follow: 2
∫ST
1 ‖gradST f ‖2 = (cot 𝜃3 ‖f (v1 ) − f (v2 )‖2 + cot 𝜃2 ‖f (v1 ) − f (v3 )‖2 2 + cot 𝜃1 ‖f (v2 ) − f (v3 )‖2 )
(2)
where 𝜃3 , 𝜃2 and 𝜃1 are the angles between edges (v3 v1 , v3 v2 ), (v2 v1 , v2 v3 ) and (v1 v3 , v1 v2 ). The normal equation for the minimization problem can be defined as the following linear system of equations: ∑ wi,j (f (vj ) − f (vi )) = 0 vi ∈ VI (3) vj 𝜖Ni
where wi,j = cot 𝛼i,j + cot 𝛽i,j and angles are showed in Fig. 1b. The associated matrix is symmetric and positive, so the linear system is uniquely solvable with iterative methods such a conjugate gradient method. Moreover that system has to be solved three time, once for x-, once for y- and once for z-coordinate.
474
B.Ture Savadkoohi and R. De Amicis
Poisson equation is able to reconstruct a scalar function from a guidance vector filed and boundary condition. Although it can be viewed as an alternative formulation of a least squares minimization [8]. Since Poisson equation requires a discrete guidance field defined on the triangles of the patch mesh. The guidance vector field is constructed by local rotation of each triangle in initial patch mesh. Local rotation is applied to each triangle of initial patch mesh. Let n be the original normal of triangle and n′ be the new normal of triangle that is calculated with desirable normal of vertices of triangle and c be the center of triangle. The rotation can obtain by rotating n to n′ around c. After rotation, the original patch mesh is torn apart and triangles are not connected anymore and these torn triangles are used to construct a guidance vector filed for Poisson equation. Finally, the disconnected triangles are stitched by solving Poisson equation. Consider an unknown scalar function, f, the Poisson equation with Dirichlet boundary condition is defined by: ∇2 f = ∇.w
with f |𝜕𝛺 = f ∗ |𝜕𝛺
over 𝛺,
where w is Guidance Vector Filed, ∇.w =
𝜕wx 𝜕x
+
𝜕wy 𝜕y
+
𝜕wz 𝜕z
(4)
is the divergence of w = 2
𝜕 (wx , wy , wz ), f ∗ provides the desirable values on the boundary 𝜕𝛺, ∇2 = ( 𝜕x + 2 𝜕2
𝜕z2
𝜕2 𝜕y2
+
) is Laplacian operator. Thus it can define as least-squares minimization problem: min f
∫𝛺
|∇2 f − ∇w|2
with f |𝜕𝛺 = f ∗ |𝜕𝛺
(5)
A discrete vector field on a triangle mesh is defined to be a piecewise constant vector function whose domain is the set of point on the mesh surface. A constant vector is defined for each triangle, and this vector is coplanar with the triangle. For a discrete vector filed w on the mesh, its divergence at vertex vi can be defined with: (div w)(vi ) =
∑
∇Bik .w|Tk |
(6)
Tk ∈Ni
where |Tk | is the area of triangle Tk , ∇Bi,k is the gradient vector of the vertices within Tk , Ni is the 1-ring vertices of vi . The discrete gradient of the scalar function f on a discrete mesh is expressed as: ∑ fi ∇∅i (v) (7) ∇f (v) = i
where ∅i (.) piecewise linear basis function valued 1 at vertex vi and 0 at all other vertices and fi is a scalar vector value attached to vertex vi and it is one of the coordinate of vi . The discrete Laplacian operator can determine as:
Computer Aided Process as 3D Data Provider in Footwear Industry
𝛥f (vi ) =
1 ∑ (cot 𝛼i,j + cot 𝛽i,j )(fi − fj ) 2 v ∈N j
475
(8)
i
where 𝛼i,j and 𝛽i,j are the two angles opposite to edge in the two triangles sharing edge (vi and vj ) and Ni is the set of the 1-ring vertexes of vertex vi . Finally discrete Poisson equation is expressed as follows: ∇2 f = div(∇f ) = ∇w. Discrete Poisson equation with known boundary condition can be defined by linear system such as: Ax = b. So that the coefficients matrix A is determined by Eq. 8 and the vector b is determined by Eq. 6 and unknown vector x is the coordinate of all vertices on the patch mesh. The smooth and accurate patch mesh is constructed as follow: First, compute the gradient of each new vertex on the adjacent triangle by using Eq. 7. Next, calculate the divergence of every boundary vertex by using Eq. 6. Then, determine the coefficient matrix A by Eq. 8 and vector b in this equation is determined by using divergence of all boundary vertices. Finally, solve the Poisson equation and obtain the new coordinate of all vertices of the patch mesh.
3 Alignment of 3D Foot with Shoe Last Data Base Principle Component Analysis (PCA) is a method that has been extensively used for analysis, modeling and recognition [9]. In analysis, instead of applying the PCA in classical way (sets of 3D point-clouds), for taking into account different size of triangles Weighted Principle Component Analysis(WPCA) is applied [1]. To achieve the alignment we describe the main steps and details of WPCA in the next steps. First, we apply step 1 through step 6 for the first models in shoe last data base, see Fig. 3. Then, for alignment of another models with the first model we apply step 1 through step 5 and steps 7 and 8. Let A be the total sum of the areas of all triangles in the mesh, Ai be the area of triangle i within the mesh, Tci be barycenter of each triangle, Tc be the total sum of barycenter of all triangles in mesh, matrix OME be Origin Matrix of
Fig. 3 a Input 3D model. b Translated barycenter of the 3D model to the origin. The red, green and blue lines are eigenvectors. c Rotated 3D model with it’s eigenvectors. d Target 3D models
476
B.Ture Savadkoohi and R. De Amicis
Eigenvectors that is included the position of eigenvectors of model as column and CM be Covariance Matrix type (3 × 3). The main steps of WPCA are as it follows: Step 1. Accomplish the translation invariance by finding the barycenter of the model as: n n n 1∑ 1∑ 1∑ xc = xi , yc = yi , zc = z (9) n i=0 n i=0 n i=0 i Step 2. Translate 3D model to origin. That is to say, for each coordinate, a corre′ sponding transformation is done by vi = {xi − xc , yi − yc , zi − zc } and define the new ′ ′ ′ ′ vertex set v = {v1 , v2 , ...vn }. Step 3. Calculate matrix CM as follows: ⎡ Covxx Covxy Covxz ⎤ CM = ⎢ Covyx Covyy Covyz ⎥ ⎥ ⎢ ⎣ Covzx Covzy Covzz ⎦ ∑n
Tc
∑n
(10)
Ai (Tc x−Tc x)(Tc x−Tc x)
i i . Obviously the matrix CM is where Tc = i=0n i and Covxx = i=0 A a symmetric real matrix, there for its eigenvalues are non negative real numbers and orthogonal. Step 4. Sort the eigenvectors in a decreasing order and find the corresponding eigenvectors. The eigenvectors are scaled to Euclidean unit length and form the rotation matrix R which has scaled eigenvectors as rows. Step 5. Apply matrix R to all of the vertices of a triangle and form a new vertex sets ′ ′ called, v′′ = {R × v1 , ..., R × vn }. Step 6. Rotate first shoe last with its eigenvectors in Fig. 3c up to a position where the 3D model becomes parallel with x-y plane, see Fig. 3d and build matrix OME with the new position of these three eigenvectors as Columns. Step 7. Set Transpose of a Matrix R as TM. The alignment is accomplished by ′ constructing a rotation matrix through R = OME × TM. ′ Step 8. Get the matrix R and apply it to all v′′ and calculate new vertex sets VA as ′ ′ the alignment of the model VA = {R × v′′1 , ..., R × v′′n }.
4 Segmentation of 3D Foot and Shoe Last Mesh segmentation of 3D models into meaningful parts is fundamental to model processing and has become challenging problem with application in 3D model retrieval. Different criteria of mesh segmentation methods has been summarized in [10, 11]. Our overall segmentation algorithm is inspired from [12]. In order to achieve robustness and effectiveness of segmentation, first plane feature is recognized by applying normal based initial decomposing. Then further segmentation based on curvature criteria and Gauss mapping, followed by the detection of quadric
Computer Aided Process as 3D Data Provider in Footwear Industry
477
surfaces is performed. Finally, the segmentation is refined by B-spline surface fitting technology. Let F = {f1 , f2 , ..., fn } be the surrounding triangular faces, N = {n1 , n2 , ..., nn } and AN = {a1 , a2 , ..., an } be normal vector and the area of each subset of F, di,j be the control points, Ni,3 (u) and Nj,3 (v) be stand for the univariate B-spline basis functions of 3-order in the u- and v-direction, defined over the knot vector vectors U = {u0 , u1 , ..., um+4 } and V{v0 , v1 , ..., vn+4 }. For planer segmentation the initial normal vectors of all vertices of the mesh is calculated as: ∑n j=1 aj .nfj (11) ni = ∑n a j=1 j Then the normal filtering method is applied in order to obtain more accurate normal vector [13]. So that instead of moving the original vertices of the mesh, the normal vectors of the vertices are re-computed. As the first seed, the region is initialized with an arbitrary, unsegmented triangular face. Next, when the normal vector angle between the added face and the seed face is smallest, (less than a pre-defined threshold), three neighboring faces of the seed is added and the region is growing. Then, the seed is updated with the latest added face. Finally, the region growing is iterated until no more faces can be added into the current region. So, this region is referred to as a segment. The above process is repeated until all faces of mesh are segmented. Thus, the initial segmentation is achieved. The first seed face is stored for each segment and the normal vector angles between this seed face and all other faces within the segment are calculated. When all those vector angles are less than a small value, this segment is remarked as a plane, otherwise, further curvature-based segmentation is performed. Curvature-based segmentation is applied for all non-plane segments. In order to determine the curvature for each vertex, the quadric surface fitting technology is applied. As shown in Fig. 4, a local surface patch around a vertex is approximated with a quadric surface: S(u, v) = (u, v, h(u, v)). This principle is applied to a discrete mesh and the Local Coordinate System (LGS) of each vertex is constructed to fit a quadric surface on the neighboring vertices of the vertex. Fig. 4 An illustration of the local coordinate system on a surface: v is the origin, h-axis is along the normal vector n at p on S, and u-, v-axes are two arbitrary, orthogonal unit vectors in the tangent plane at v
478
B.Ture Savadkoohi and R. De Amicis
Relevant to the surface theory [14], a boundary of a model is the intersection of adjacent surfaces, which mentions that the geometric property of each vertex around a boundary should be calculated from its surrounding neighborhood within one surface. On the other hand the anisotropic neighborhood is applied in order to eliminate the error. Moreover based on the initial segmentation result, the neighborhood of a vertex are exclusively searched from the segment of the vertex, i.e., the neighborhood of a vertex will never cross the segment of this vertex. At the same time, in order to guarantee that neither the local sharp edges nor the global sharp edges would be included in the neighborhood of the vertex such as the underlying surface of the vertex, the constraint that the normal vectors of all triangular faces of the neighborhood are closed to the normal vector of the vertex is applied. Thus more accurate curvature would be obtained. Then the quadric surface is fitted by transforming the neighboring vertices from the Global Coordinate System(GCS) to Local Coordinate System(LCS) as: (12) h(u, v) = au2 + buv + cv2 + eu + fv + g The least squares methods is utilized here. The Gaussian and mean curvature of a vertex are determined by 2 ⎧ K = 24ac−b (e +f 2 +1)2 ⎪ (13) ⎨ 2 2 ⎪ H = c+ce +a+af −bef 3 ⎩ (e2 +f 2 )+1) 2 The normal-weighted averaging method is applied to compute the curvature of a triangular face as follows: ⎧ ⎪K = ⎪ f ⎪ ⎨ ⎪ ⎪H = ⎪ f ⎩
−‖nni −nf ‖2
∑3
2𝜎 2 i=1 Kvi .e −‖nni −nf ‖2 ∑3 2 i=1 e 2𝜎
∑3 i=1
Hvi
∑3 i=1
e
−‖nni −nf ‖2 2𝜎 2 .e
(14)
−‖nni −nf ‖2 2𝜎 2
where nf is a normal vector of triangular face f, nv1 , nv2 , nv3 are the normal vectors of v1 , v2 , v3 . Since the surface can be classify with eight fundamental types in terms of the signs of Gaussian and mean curvatures, including peak, ridge, saddle ridge, flat, minimal surface, pit, saddle valley [15]. Thus the triangular faces of the mesh are labeled with these eight types and divided into different patches. In order to perform further segmentation using region growing method statistics methods and Gauss mapping are applied to recognize the quadric surface feature, including cylinder, sphere and cone. The main steps of procedure to detect cylinder include:
Computer Aided Process as 3D Data Provider in Footwear Industry
479
Step 1. As a seed face, select a triangular face and initialize the current region with this seed. Step 2. Add each connected neighboring faces of the seed face into region when the minimal average and variances of the minimal and maximal curvature are less than a small positive value. Otherwise, skip this face and end the growing at this point by setting a flag on the face. Step 3. Stop the growing when all neighboring faces of the seed face are skipped. Otherwise, add this face into the region and update the seed face with this face and go to step 2. In the case of sphere surface the Gaussian and mean curvature are constant. Since, a cone surface has no particular property in term of curvature, so Gauss mapping method is applied to segment and consist of the following three steps: Step 1. As a seed face, select a triangular face and initialize the current region with the seed face and map its normal vector onto the Gauss sphere. Step 2. Add each connected neighboring faces of the seed face into region when that face’s normal vector mapping is located around a small circle of the Gauss sphere, thus that face is on the cone surface. Step 3. Stop the growing when all neighboring faces of the seed face are skipped. Otherwise, update the seed face with an added neighboring face of the seed face within the current region and go to step 2. The Gauss mapping of the normal vectors of faces on the cone surface form a small circle of the Gauss sphere. Due to the noises and computational errors the mapping can not exactly located on a small circle. Thus, the mappings are fitted with a circle and the fitting error is calculated. The mapping are considered to be around a small circle and the region is seen as a cone surface segment when the error is less than a small value and the radius of the fitted circle is less than the radius of Gauss sphere, i.e., 1. In the neither planer nor quadric surface, B-spline surface fitting method is performed to refine the segmentation. A bicubic B-spline surface S(u, v) is determined by m n ∑ ∑ di,j Ni,3 (u)Nj,3 (v), 0 ≤ u, v ≤ 1 (15) S(u, v) = i=0 j=0
The surface is fitted when the control points di,j are found, so that the distance of the data points P to S(u, v) is minimized: E=
N ∑
dist2 (vk , S)
(16)
k=0
A few connected triangular faces within the surface segment are chosen as a seed region and its vertices are fitted with a bicubic B-spline surface. Taking the fitted B-spline surface as an input the vertices which are not included in but connected with the current region are used for growing the region. For each added vertex, three
480
B.Ture Savadkoohi and R. De Amicis
condition should be fulfilled: (i), the projection distance of the vertex onto the fitted face is less than a small value. (ii) the normal vector of the projection point on the fitted surface should be close to that of the vertex. (iii) the principle curvature of the projection point should be close to those of the vertex. Followed by fitting a new fitted B-spline surface on the region, when no more vertices can be added into region, the region growing process terminates. Then all faces in the current region are cleared and updated with origin seed region. By taking new fitted B-spline surface as input the region growing procedure above is applied again. This is repeated until the maximal region is achieved and a final segment is obtained.
5 Conclusions In this paper preprocessing algorithms of 3D scanning data before taking 3D models to the real application is proposed. For this aim, the hole in triangle mesh is filled by generating initial patch mesh. Then the initial patch mesh is modified with calculation the desirable normal based on Harmonic function. Finally accurate and smooth triangle mesh are obtained by solving the Poisson equation according to desirable normals and the boundary vertices of the hole. After obtaining complete 3D models, we applied the Weighted Principle Component Analysis technique for alignment of 3D foot with shoe last data base. Finally, in order to shape retrieval within the shoe last data base, segmentation method based is demonstrated. At first, to recognize the plane feature, normal based initial decomposition is applied. Then, by detecting the quadric surface feature, curvature criteria and Gauss mapping is performed. Finally, the segmentation is refined by using B-spline surface fitting technology.
References 1. Ture Savadkoohi, B., De Amicis, R.: Mass customization strategy in footwear industry. J. Adv. Comput. Sci. Appl. IJCSIA 5(2), 2250–3765 (2015) 2. Ture Savadkoohi, B., De Amicis, R.: 3D foot scanning in flexible manufacturing: comparisation and deformation solution. In: International Conference on Computer Graphics, Visualization, Computer Vision and Image processing. Las Palmas de Gran Canaria, Spain, 22–24 July 2015 3. Liepa, P.: Filling holes in meshes. In: Proceedings of the Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, pp. 200–205. Granada, Spain, 1–6 Sept 2003 4. Nooruddin, F.S., Turk, G.: Simplification and repair of polygonal models using volumetric techniques. J. IEEE Trans. Visual. Comput. Graph. 9(2), 191–205 (2003) 5. Zhao, W., Gao, S., lin, H.: A robust hole-filling algorithm for triangular mesh. J. Visual Comput. 23(12), 987–997 (2007) 6. George, L.P., Seveno, E.: The advancing-front mesh generation method revisited. J. Numer. Methods Eng. 37(7), 3605–3619 (1994) 7. Eck, M., DeRose, T.D., Kuchamp, T., Hoope, H., Lounsbery, M., Stuetzle, W.: Multi-resolution analysis of arbitrary meshes. In: Processing of SIGGRAPH, pp.173–182. Los Angeles, CA, USA, 6–11 Aug 1995
Computer Aided Process as 3D Data Provider in Footwear Industry
481
8. Perez, P., Gangnet, P., Blake, A.: Poisson image editing. In: Processing of SIGGRAPH, pp. 313–318. San Diego, California, USA, 27–31 July 2003 9. Paquet, E., Rioux, M.: Nefertiti: a query by content system for three-dimensional model and image databases management. J. Image Vis. Comput. 17(2), 157–166 (1999) 10. Shamir, A.: A survey on mesh segmentation techniques. J. Comput. Graph. Forum 27(6), 1539– 1556 (2008) 11. Chen, X., Golovinskiy, A., Funkhouser, T.: A benchmark for 3D mesh segmentation. J. ACM Trans. Graph. 28(3) (2009) 12. Wang, J., Yu, Z.: Surface feature based mesh segmentation. J. Comput. Graph. 35(3), 661–667 (2011) 13. Sun, X., Rosin, P., Martin, R., Langbein, F.: Fast and effective feature-preserving mesh denoising. J. IEEE Trans. Visual. Computer Graph. 13(5), 925–938 (2007) 14. do Carmo, M.: Differential Geometry of Curves and Surfaces. Prentice-Hall (1976) 15. Besl, P.J., Jain, R.: Segmentation through variable-order surface fitting. J. IEEE Trans. Pattern Anal. Mach. Intell. 10(2), 167–192 (1998)
