We analyze the optimal ring width, and develop formulas to ... treadmills without the same physical movements. ... same amount of movement in both worlds.
Multi-ring Color Fiducial Systems for Scalable Fiducial Tracking Augmented Reality Youngkwan Cho
Ulrich Neumann
{ykcho | uneumann}@graphics.usc.edu
Computer Science Department Integrated Media Systems Center University of Southern California Abstract In Augmented Reality (AR), a user can see a virtual world as well as the real world. To avoid the registration problem, the user’s pose in both worlds should be exactly the same. Therefore, to explore a large virtual world, the user should move correspondingly in the real world, and wide-area tracking is essential. Most of the developed fiducial tracking AR systems have very limited tracking ranges. In this paper, we propose multi-ring color fiducial systems for scalable fiducial tracking AR systems. We analyze the optimal ring width, and develop formulas to obtain the optimal fiducial set with system specific inputs. It is a simple and low-cost way to achieve wide-area tracking.
1 Introduction In Virtual Reality, where all scenes are computer-generated images, a virtual world could be explored by flying or steering treadmills without the same physical movements. In Augmented Reality (AR), a user sees a virtual world as well as the real world. Because the user’s pose in the real world is directly related to the user’s pose in the virtual world, AR requires the same amount of movement in both worlds. To apply AR to large scale applications, wide-area tracking is essential. The overlapped virtual world contains virtual objects to help understand the real world. To make the virtual world useful, the virtual objects are aligned with the real objects. Registration is one of the major issues in AR [Azuma94][Bajura95] [Tuceryan95]. The registration problem requires high accuracy or error-correction mechanisms in tracking. Fiducial tracking has been gaining interest as a solution to the registration problem.
1.1 Motivation The tracking range of a fiducial tracking AR system is confined by the detectability of fiducials in the input images. Most of the developed systems use their own single-size fiducials. A single-size fiducial might help fast fiducial detection, but the system will have a narrow tracking range because all fiducials have the same detection range. Different size fiducials will have different detection ranges, and by combining a series of different detection ranges from multi-size fiducials, the whole tracking range can be extended seamlessly.
1.2 Contribution We contribute the multi-ring multi-size concept to concentric circular fiducials and introduce a fiducial system concept. This fiducial system concept provides great scalability to fiducial tracking AR. It also provides a convenient way to unify multiple local coordinate systems. These fiducial systems introduce a much larger number of unique fiducials than a single-size fiducial system, and make fiducial identification much easier.
We develop formulas to calculate the optimal fiducial set for any size of applications with some system specific parameters. Users conveniently get the optimal fiducial set by just plugging in those parameters according to their systems. We provide a simple and low-cost way to achieve wide-area tracking, and we hope that this work trigger many research activities in large-scale applications.
2 Previous works 2.1 Fiducial tracking There are several developed fiducial tracking AR systems. Most of them have used solid or concentric circular fiducials [Mellor95][Neumann96][State96][Cho97], and a few of them have used the corners of big rectangles as fiducials [Kutulakos96][Koller97]. All of them have adopted single-size fiducials or rectangles, and their tracking areas are limited to just around a desktop or the opposite wall of a room. None of them seems to support full room range (~30x30 feet) tracking from arm-length distance to wall-to-wall distance.
2.2 Optical ceiling tracking UNC has a long history for wide-area head-tracking using active fiducials like infrared LEDs [Ward92]. Recently they developed the SCAAT method, and achieved excellent update rates and accuracy [Welch97]. LED systems, however, require power supply. They hide complex wiring and circuits on the ceiling. Each LED requires a hole in the ceiling tile. LEDs are all the same, and identification is a big problem. They solved it by controlling which LEDs are on. To achieve high update rates, an LED is on for less than one msec. It is hard to catch such an instance flashing if the distance is a little far. It also requires a custom made camera set. It has nice properties but it is a very expensive method.
3 Multi-ring color fiducials The 2D image location (u,v) of a 3D point (x,y,z) is determined by the following perspective projection. U fu V 0 w 0 u U v V
0 U 0 0 fv V 0 0
1
R
0 0
0
0 I 1 0
x T y 1 z , 1
/ w
/ w
where fu and fv are the effective focal lengths of a camera in the u and v direction, respectively, and (U0,V0) is the image center. R is the rotation matrix and T is the translation vector for the camera pose. The major axis length d of the ellipse, the projected image of a circular fiducial, is
d Df / w , where D is the diameter of the fiducial, f = fu = fv is the effective focal length, and w is the distance between the camera and the fiducial. When a camera is too far or too close to a fiducial, the projected image of the fiducial in the input image is too small or too large to detect it correctly. Even though each fiducial has a fixed trackable range, the whole tracking range could be extended by combining different trackable ranges of multi-size fiducials. Multi-ring color fiducials have different number of rings at different levels. The first level fiducials have one core circle and one outer ring. As the level goes up, one extra ring is added outside of the previous fiducial. The number of rings in a fiducial tells the fiducial level it belongs to. It introduces many unique fiducials and fiducial identification becomes easier. The core circle and rings are painted with six colors: red, green, blue, yellow, cyan, and magenta. There are two possible regular rules for the ring width: constant width ring and proportional width ring. Figure 1 shows some examples of multi-ring color fiducials. Proportional width ring fiducials Constant width ring fiducials First level Second level Third level Figure 1. Multi-ring multi-size color fiducials
Zfar Znear dfar, D1 dnear f f n 1 Dn c D1 Zfar Znear dfar c n 1 dnear f f Zfar dnear c n 1 cn Znear dfar log( Zfar / Znear ) n(c) log c Dn
Figure 2. Fiducial levels
Fiducial size at each level When the camera is at Zfar,1 from a level n fiducial, the major axis length of the level 1, 2, and j rings in an input image are 2w c dfar c -1 2w 2 d2 c dnear ,2 dnear c -1 2w j dj c ,(1 j n) c -1
dfar ,1
The diameters of the level 1 and n fiducials are Zfar Zfar 2w dfar c f f c 1 Znear Znear 2 w 2 D1 dnear c f f c 1 Zfar 2 w 2 n c c 1 n Dn f c 1
Dn
Figure 3 shows the major axis lengths of some fiducial levels, and the minimum and maximum fiducial sizes as a function of c.
(a) Fiducial sizes in an image
(b) Real fiducial sizes Figure 3. Fiducial sizes
3.1 Proportional width ring fiducials
Fiducial distribution
In proportional width ring fiducial system, the size ratio c between adjacent levels is the same. Di cDi 1 (c 1) c i 1D1 where Di is the fiducial diameter of the level i. The outer ring width is wider than the inner ring, and the outer ring is more easily detectable from a distance. The higher level fiducials have farther trackable range. By combining those trackable ranges, we can extend the whole tracking range. This fiducial system is good for wide area tracking with arbitrary camera movements. Let the desired tracking range be Znear ~ Zfar, and the camera focal length be f. Let w be the minimum detectable ring width in an input image. w depends on the camera, the digitizer, and fiducial detection algorithm.
When a camera is close to fiducials, it can see only a small region and works with small fiducials. When the camera is far from fiducials, it can see a large area and works with large fiducials. Therefore lower level fiducials have dense distributions and higher level fiducials have sparse distributions. Fiducials are usually distributed around interesting regions without any regular pattern. For analysis purposes, we use a regular grid distribution. We assume that the input image has WxH resolution (W H). To determine the camera pose, three or more non-collinear fiducials are required in an input image [Linnainmaa88] [Horaud89][Haralick94]. Figure 4 shows an input image of level i fiducials at distance Znear,i. With this configuration, any camera pose can see three or more fiducials in the valid tracking range of level i. The inter-fiducial distance for level i fiducials Li is H dnear Di L Li(c)
Number of required levels Let the tracking range of a level i fiducial be Znear,i ~ Zfar,i with the conditions Znear = Znear,i and Zfar,n = Zfar. The largest detectable fiducial size in an image is dnear ( Dif/Znear,i), and the smallest detectable fiducial size is dfar ( Dif/Zfar,i). To combine the trackable ranges smoothly, there should be no gap between adjacent work ranges. Dif Di 1 f Dif cDif Dif dnear ( c) dfar dnear dfar dnear dnear dfar dnear Zfar , i c and c dfar Znear , i
0 Zfar , i Znear , i 1
The required levels of fiducials can be expressed as a function of the size ratio c (Fig. 2).
i
dnear 2 H (c - 1) - 2c 2 w Di = 4c 2 w
Di
The ratio of the fiducial diameter over the inter-fiducial distance shows the fiducial density.
Figure 4. Input image of the level i fiducials at distance Znear,i Di 4c 2 w f (c ) Li H (c - 1) - 2c 2 w
Figure 5 shows the inter-fiducial distances and fiducial density in percentage form.
Optimal size ratio c All results come out as functions of the size ratio c. Which c gives the optimal results? We can think of the optimality in two aspects: system performance and fiducial sizes. For high system performance, Figure 5. Inter-fiducial distance fiducial detection process at level 1 and n, and fiducial should be fast. We can density f(c) concentrate on the performance of finding the smallest detectable ring in a fiducial, because the larger rings could be found easily by predicting their boundaries with c. The fiducial detection algorithm should look for rings whose diameters are in the range from dfar (
2w 2w 2 c) to dnear ( c ) . To minimize the processing c -1 c -1
time for fiducial detection, f(c)=(dnear - dfar)dfar should be minimized. (dnear - dfar) is the searching range, and dfar is the searching size. The bigger dfar is, the more processing time for boundary detection. 2w 2 w 2 2 w 4w 2 2 c( c c) c c -1 c -1 c -1 c -1 2 2 2 2 df (c ) 8w c 4 w c 4 w c( 2 c ) dc (c - 1)2 (c - 1)2 f (c) dfar (dnear dfar )
f(c)=(dnear - dfar)dfar has minimum when c = 2. The other aspect is fiducial size. If fiducials are too big, it is not easy to find proper places to put them on. They also occupy a large area in the input images. So, we can get the optimality by minimizing the fiducial density. df (c ) 8cw( H (c - 1) - 2c 2 w ) 4c 2 w( H 4cw ) 2 2 dc
( H (c - 1) - 2c w ) 4cwH (c - 2) ( H (c - 1) - 2c 2 w )2
f(c) has the minimum 16w/(H - 8w) when c=2. From both aspects, the optimal size ratio c is the same value 2.
3.2 Constant width ring fiducials In the constant width ring fiducial system, all rings have the same width c. Di Di 1 2c (c 0 ) D1 2(i 1)c The tracking ranges of all size levels are the same, because the ring widths are the same. Since all rings in all fiducials are recognizable in the tracking range, there are lots of uniquely identifiable fiducials, and fiducial identification is trivial. This fiducial system is good for wide-area tracking where a camera moves parallel to the fiducial plane, such as a ceiling tracker. Since this fiducial system has one tracking range for all size levels, the detectable range is not a control parameter. We can concentrate on the fiducial identification problem. The number of required fiducial levels is determined by the required number of unique fiducials.
Number of unique fiducials in the level i With six colors, the number of unique fiducials at the level i is Ni = 6*5i. This requires that the fiducial detection use all three color channels. We can improve the fiducial detection by scarifying the number of unique fiducials. We divide six colors into two groups
by a test channel: the colors whose test channel is non-zero belong to group 1, and the rest of the colors belong to group 2. For example, if we choose the red channel as the test channel, then red, yellow, and magenta belong to group 1, and green, blue, and cyan belong to group 2. Let the first ring have any color in group 1, and the core and the second ring have any colors in group 2. The other rings can have any colors from group 1 or group 2 as long as they are different from their neighbors. The first ring can be detected by examining only the test channel. The other rings can be found easily, because the ring widths are the same. This reduces the number of unique fiducials but it improves the performance. 3 i 1 , i2 Ni i 2 27 5 , i3
Fiducial size at each level The rings should be detectable from the farthest view point. The projected ring width should be greater than or equal to the minimum detectable ring width w. wZ far f cw c Z far f
To minimize the fiducial size, the diameter of the core circle is set to c. Then the size of fiducials at the level i is Di (2i 1)c (2i 1)
wZ far f
Fiducial distribution The calculation of the inter-fiducial distance is similar to that of the proportional width ring. The only difference is that the whole images of the first rings are required. The other rings could be recognized with their showing parts. The inter-fiducial distance L is H dnear ,1 D1 L(c ) 2 dnear ,1 HZnear 3wZ far 2f 2f
Number of required levels Let fiducials be distributed on area A’xB’. The number of required fiducials are roughly A*B/L(c)*L(c). When the fiducials are located randomly, all of them should be unique fiducials to minimize the fiducial identification problem. A* B Ni L(c) * L(c ) i
If they are distributed in a grid pattern, every second row or column requires unique fiducials, and the other row or column can have one common fiducial, that is the position holder. With the fiducial distribution in Fig. 4, any camera image contains at least two unique fiducials. With those unique fiducials, the position holders can be uniquely identified. It reduces the required number of fiducials by half. Even with reserving some fiducials for place holder and changing just place holders, the required number of fiducials could be reduced further or the tracking area could be extended in future.
4. Case studies Let’s assume the tracking area is 20’x20’ room. The camera is a NTSC video camera with FOVu 41.7 and FOVv 32. Let the minimum detectable ring width w be 5 pixels.
4.1 Fiducial tracking in video see-through
In this application, the AR system should work from very close distances to far distances. The proportional width ring fiducial system is appropriate. The closest distance is arm-length 2’ and the farthest distance is the corner-to-corner length 20 2 ’. The required number of fiducial levels is log( 20 2 / 2) / log2 4. The fiducial sizes in each level are D1 = 1.01”, D2 = 2.03”, D3 = 4.06”, and D4 = 8.1”. The inter-fiducial distances are L1 = 5.6”, L2 = 11.2”, L3 = 22.3”, and L4 = 44.5”.
4.2 Pose of moving objects Large fiducials are located on the walls and their positions are measured in the world coordinate system. Four or more small fiducials are on a moving object and their positions are measured in the object coordinate system. When an input image has four or more fiducials on the wall and four fiducials on the object, the camera pose can be calculated in the world coordinate and in the object coordinate. Based on these camera poses, the world-tocamera coordinate transformation matrix A and the object-tocamera coordinate transformation matrix B can be obtained. Pcw = APw, where Pw is a point in the world coordinate and Pcw is the corresponding point in the camera coordinate system. Pco = BPo, where Po is a point in the object coordinate and Pco is the corresponding point in the camera coordinate system. Because A is a rigid motion transformation matrix, it has the inverse matrix A-1. Any point defined in the object coordinate can be converted to its corresponding point in the world coordinate system. Pw = A-1BPo, The same method can be used for multiple coordinate transformations. The fiducial positions are usually measured with a mechanical 3D digitizer. When the fiducials are distributed wider than the range of the mechanical digitizer, the fiducials are divided into several groups and they are measured separately. Groups use different coordinate systems, and they are transformed to a common coordinate system. When two fiducial groups are far apart and when it is hard to find some common points, the above method can be used to unify the coordinates using the large fiducials in the groups.
4.3 Optical ceiling tracking In the optical ceiling tracking, the camera motion is usually parallel to the ceiling (fiducial plane). The constant width ring system is appropriate. Let the height of ceiling be 8’ and the camera be mounted on a user’s head at height 6’. Znear is 2’ and Zfar is
Znear , where cos( FOVu / 2)
Zfar
Znear
is the maximum allowable angle that a user can look down or look up (Fig. 6). If we allow 90, Zfar is roughly 5.5’. The Figure 6. Calculation of Zfar minimum ring width c is 0.40”. The inter-fiducial distance L is 6.43”. The required number of fiducials is 1,393. If we use the first level fiducials as place holders and the level 2 and 3 fiducials as unique fiducials, they can fill out 9*(27 + 135)*2 = 2916 fiducial positions. Three fiducial levels are enough for this case. The fiducial sizes are D1 = 1.2”, D2 = 2”, and D3 = 2.8”. FOV/2
5 Conclusion Concentric circular fiducials are not a new idea but they have been mainly used for detection purposes. They are used as
single-size fiducials with fixed number of rings (mostly one or two rings). We extend concentric circular fiducials to the multiring multi-size fiducial systems. Because these fiducial systems are incremental, the tracking range can be easily extended by adding new fiducials. They introduce a much larger number of unique fiducials than a single-size fiducial system, and make fiducial identification easier. They give great scalability to fiducial tracking AR. These fiducial systems help in building large-scale applications. They provide a convenient way to unify multiple local coordinate systems. They make it easy to determine fiducial positions in a large-scale application with a small range 3D digitizer. We analyze optimality of ring width and develop formulas to get an optimal set of fiducials easily for any size of applications by plugging in some system specific parameters. We provide a simple and low-cost way to achieve wide-area tracking. Large-scale applications can be implemented with some paper color fiducials, a commercial video camera, and a computer with a video digitizer board. We hope that this work trigger many research activities in large-scale applications.
References [Azuma94] Ronald Azuma and Gary Bishop, Improved Static and Dynamic Registration in an Optical See-through HMD, SIGGRAPH 1994, pp. 197-203 [Bajura95] Michael Bajura and Ulrich Neumann, Dynamic Registration Correction in Augmented Reality Systems, VRAIS 1995, pp. 189-196 [Cho97] Youngkwan Cho, Jun Park, and Ulrich Neumann, Fast Color Fiducial Detection and Dynamic Workspace Extension in Video Seethrough Augmented Reality, To be appear in the Proceedings of the Fifth Pacific Conference on Graphics and applications, October 1997 [Haralick94] R. Haralick, C. Lee, K. Ottenberg, and M. Nolle, Review and Analysis of Solutions of the Three Point Perspective Pose Estimation Problem, IJCV, Vol. 13, No. 3, 1994, pp. 331-356 [Horaud89] Radu Horaud, Bernard Conio, and Olivier Leboulleux, An Analytic Solution for the Perspective 4-Point Problem, CVGIP 47, 1989, pp. 33-44 [Koller97] D. Koller, G. Klinker, E. Rose, D. Breen, R. hitaker, M. Tuceryan. To appear in the Proceedings of the Symposium on VRST, Lausanne, Switzerland, September, 1997 [Kutulakos96] K. N. Kutulakos and J. Vallino, Affine Object Representations for Calibration-free Augmented Reality, VRAIS 1996 [Linnainmaa88] Seppo Linnainmaa, David Harwood, and Larry S. Davis, Pose Determination of a Three-Dimensional Object Using Triangle Pairs, PAMI, Vol. 10, No. 5, September 1988, pp. 634-647 [Mellor95] J. P. Mellor, Enhanced Reality Visualization in a Surgical Environment, AI Technical Report 1544, 1995 [Neumann96] Ulrich Neumann and Youngkwan Cho, A Self-Tracking Augmented Reality System, VRST 1996, pp. 109-115, July 1996 [State96] Andrei State, Gentaro Hirota, Davod T. Chen, Bill Garrett, and Mark Livingston, Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking, SIGGRAPH 1996, pp. 429-438 [Tuceryan95] M. Tuceryan, D. S. Greer, P. T. Whitaker, D. Breen, C. Crampton, E. Rose, K. H. Ahlers, Calibration Requirements and Procedures for Augmented Reality, [Ward92] M. Ward, R. Azuma, R. Bennett, S. Gottschalk, and H. Fuchs, A demonstrated optical tracker with scaleable work area for headmounted display systems, Proceedings of 1992 Symposium on Interactive 3D Graphics, 1992, pp. 43-52 [Welch97] Greg Welch and Gary Bishop, SCAAT: Incremental Tracking with Incomplete Information, SIGGRAPH 1997, pp..333-344
