Spatial orientation in translumenal surgery - Semantic Scholar

Correspondence:

Kurt Höller Central Institute of Healthcare Engineering (ZiMT) Friedrich-Alexander-Universitaet Erlangen-Nuernberg (FAU) Henkestr. 91, D-91052 Erlangen Tel.: +49 9131 85-26868 Fax: +49 9131 85-26862 [email protected]

K. Höller, A. Schneider, J. Jahn, J. Gutierrez, T. Wittenberg, H. Feußner, and J. Hornegger

Spatial orientation in translumenal surgery

Minimally Invasive Therapy & Allied Technologies, ISSN 1364-5706, October 2010.

Minim Invasive Ther Allied Technol Downloaded from informahealthcare.com by 93.135.59.119 on 09/26/10 For personal use only.

Minimally Invasive Therapy. 2010;19:262–273

REVIEW ARTICLE

Spatial orientation in translumenal surgery KURT HÖLLER1,2, ARMIN SCHNEIDER2, JASPER JAHN3, JAVIER GUTIÉRREZ3, THOMAS WITTENBERG1,3, HUBERTUS FEUßNER2, JOACHIM HORNEGGER1 1

Central Institute of Healthcare Engineering (ZIMT) and Pattern Recognition Lab (LME), Friedrich-Alexander University Erlangen-Nuremberg, Germany, 2Research Group for Minimally Invasive Surgery (MITI), Klinikum rechts der Isar, Technical University Munich, Germany, and 3Fraunhofer-Institute for Integrated Circuits IIS, Erlangen, Germany

Abstract “Natural Orifice Translumenal Endoscopic Surgery” (NOTES) is assumed to offer significant benefits to patients, such as reduced trauma as well as reduced collateral damage. But the potential advantages of this new technology can only be achieved through safe and standardized operation methods. Several barriers, which have been identified during clinical practice in flexible intra-abdominal endoscopy, can only be solved with computer-assisted surgical (CAS) systems. In order to assist the surgeon during the intervention and enhance his visual possibilities, some of these CAS systems require 3-D information of the intervention site, for others 3-D information is even mandatory. Therefore it is evident that the definition and design of new technologies for CAS systems must be strongly considered. A 3-D endoscope, called “Multisensor-Time-of-Flight” (MUSTOF) endoscope, is actually being developed. Within these developments, an optical 3-D time-of-flight (TOF) sensor is attached to the proximal end of a common endoscope. The 3-D depth information obtained by this enhanced endoscope can furthermore be registered with preoperatively acquired 3-D volumetric datasets such as CT or MRI. These enhanced or augmented 3-D data volumes could then be used to find the transgastric or transcolonic entry point to the abdomen. Furthermore, such acquired endoscopic depth data can be used to provide better orientation within the abdomen. Moreover it can also prevent intra-operative collisions and provide an optimized field of view with the possibility for off-axis viewing. Furthermore, providing a stable horizon on video-endoscopic images, especially within non-rigid endoscopic surgery scenarios (particularly within NOTES), remains an open issue. Hence, our recently presented “endorientation” approach for automated image orientation rectification could turn out as an important contribution. It works with a tiny micro-electro-mechanical systems (MEMS) tri-axial inertial sensor that is placed on the distal tip of an endoscope. By measuring the impact of gravity on each of the three orthogonal axes the rotation angle can be estimated with some calculations out of these three acceleration values, which can be used to automatically rectify the endoscopic images using image processing methods. Using such enhanced, progressive endoscopic system extensions proposed in this article, translumenal surgery could in the future be performed in a safer and more feasible manner.

Key words: MUSTOF, 3-D, Time-of-Flight, endoscopic image rectification, orientation correction, NOTES

Introduction NOTES (Natural Orifice Translumenal Endoscopic Surgery) offers a great area of active research in experimental endoscopy and has the potential to significantly advance the field of minimally invasive surgery (MIS). In July 2005, leading surgeons and gastroenterologists met in New York City to

coordinate further research activities in NOTES by founding the Natural Orifice Surgery Consortium for Assessment and Research (NOSCAR). In a white paper this consortium has addressed fundamental challenges to the safe introduction of NOTES and discussed potential barriers (1). Not only the need for a new collaboration between surgeons and gastroenterologists was ascertained, but also the demand on

Correspondence: K. Höller, Central Institute of Healthcare Engineering (ZIMT), Friedrich-Alexander University Erlangen-Nuremberg, Krankenhausstr. 2-4, D-91054 Erlangen, Germany. E-mail: [email protected] ISSN 1364-5706 print/ISSN 1365-2931 online ! 2010 Informa Healthcare DOI: 10.3109/13645706.2010.510762


Spatial orientation in translumenal surgery new advanced imaging and guidance technologies became clear (Figure 1). From the challenges provided in this white paper, several points seem to be solvable from the technological side with the integration of additional sensors.

263

NOTES, then orientation as well as triangulation within the site will be two fundamental requirements for any computer-assisted surgical (CAS) NOTES system (1).

Development of a multitasking platform Peritoneal access

Because some of the NOTES procedures will require an interdisciplinary team to guide, navigate and manipulate the instruments, devices with multiple ports are likely to be important. The role of robotics in this area seems promising, though a great deal of development work remains to be done. Thus development should focus on manual tools that ultimately can be modified for robotic control (1). Several approaches with computer-assisted surgical systems (3) require additional 3-D information to prevent injury or navigation errors. Multiple instruments with orientation differences between the camera view and the real world can cause coordination difficulties for the operating specialists. Therefore an image rectification method is desirable.

Translumenal surgery offers challenging possibilities for trauma reduction. Current investigations are mainly focused upon optimization of the access routes and their safe closure (2). However, especially in case of large hollow organs the optimal location for a secure introduction of the instrument into the abdominal cavity is hard to find (Figure 2). To visualize the vessels on the reverse side of the hollow organ or other organs behind the planned incision site and hence reduce the risk of lacerations would be a great improvement.

Maintaining spatial orientation In contrast to gastroenterologists, who are accustomed to working with their instruments in the same line of sight as with the sensor on the distal end of the video scope and light source, laparoscopic surgeons normally use multiple instruments and access ports with a focus to obtain optical correctness and a stable horizon. NOTES procedures, however, will often be performed with the video scope in a retroflexed position where the images are obtained and depicted upside down and an off-axis manipulation is required. Potential solutions to perform advanced procedures with two or more instruments and assistants include the incorporation of modern visualization systems as well as electronic image stabilization or inversion. If the principles learned from advanced laparoscopic operations are applicable to

Material and methods 3-D measurement by Time-of-Flight cameras In medical applications 3-D information can be acquired by endoscopic ultrasound (4), electromagnetic tracking (5) or optical approaches. In addition passive optical methods such as stereo vision (6), structure from motion (SfM) (7) or shape from shading (SfS) (8) are known. Active optical methods however, make use of pattern projection approaches (9) or consecutive illumination with varied colors (SPARC) (10). Since the last decade, a new emerging imaging technology called photonic mixer device (PMD),

Advanced imaging and guidance

Notes Open surgery

Laparoscopic surgery

Interventional endoscopy

Figure 1. Interdisciplinarity of NOTES and the role of imaging and guidance engineering.

Diagnostic endoscopy


264

K. Höller et al.

Figure 2. NOTES access routes to the peritoneal cavity.

better known as Time-of-Flight (ToF) camera, has been available. In contrast to all previous approaches, ToF cameras (Figure 3) provide surface information in real-time nearly independent of calculating capacity or feature clearness. Currently, ToF chips achieve lateral resolutions of up to 204 ! 204 pixels, frame rates of 10-100 frames per second (fps) and z-resolutions within 1 mm. These sensors enhance many approaches

for a third dimension: With knowledge of the treated person’s or object’s surface many tasks in vision, visualization and automation are possible, whereas 2-D data could not provide enough information. Since this 3-D data are available with more than 30 fps a new dimension of applications is opened also for challenging medical purposes. One of the most interesting approaches is a hybrid imaging system that combines a conventional CCD camera for color information and a ToF camera for depth information. This new technology uses a single sensor which is able to acquire a 3-D surface model in real-time. ToF cameras illuminate the scene actively with an optical reference signal. Usually, the emitted light is part of the non-visible area of the spectrum in the near infrared spectral range (Figure 4). Assuming constant speed of light c and amplitude modulation frequency fmod the distance d is proportional to the phase shift wd of emitted and reflected wave:

d=

Figure 3. PMD Cam Cube, with a dimension of 42 ! 45 ! 52 mm and a resolution of 204 ! 204 pixels (PMD Technologies GmbH, Siegen, Germany).

c ⋅ jd 4p ⋅ fmod

(1)

To get a distance value for each ToF pixel the modulation phase shift of the reflected optical wave s(t) has to be compared with an electrical reference signal g(t) which has basically the same phasing as the optical signal at the moment of emission. The

jd = atan2 ( ∆U 24 , ∆U 31 ) + π

TOF PMD

Distance

Figure 4. Principle of emitted and reflected wave with distance depending time of flight and phase shift.

reflected wave that generates electrons in the photoactive zone of each pixel has therefore a varying phasing depending on the respective distance. To compute this phase difference N ‡ 3 (average 4) measurement cycles are carried out with a certain equal duration (integration time) and a stepwise increased phase shift wtk of the electrical reference signal:

wtk =

265

between 0 and 2p one finally can compute the phase shift wd:

IR



2π ⋅ ( k − 1) N

with k = 1, 2,..., N

(2)

Each variation of wtk causes a different charge generated within the integration time. With c (tk ) = s (t ) ⊗ g (t ) = the correlation function a and the resulting voltages .cos (jd + tk ) 2 U k ~ K + c (tk ) (K: background illumination influence) a pair of phase shift depending voltage differences DU31 and DU24 can be built for N = 4:

−a ⋅ sin(jd ) ∆U 24 U2 − U 4 = = U 3 − U1 −a ⋅ cos(jd ) ∆U 31

(3)

Using a two-argument function to handle the ambiguity of the tangens function for the range of values

Practically this mathematically described procedure is done electrically with a phase mixing device (PMD). Modulated light is sent out, a part is reflected and collected by the sensor. Similar to a CCD camera chip, each pixel collects a countable number of photons. It represents the intensity of incoming light along a special line collected at the lens entrance pupil and focused down from the exit pupil onto a pixel on the PMD image plane (yellow intensity modulated beam in Figure 5). A swinging voltage g(t) with the same frequency as the emitted light s(t) divides the incoming photons after the delay tk onto two drains. This effect is called “charge swing”. For several periods (“integration time”) the generated charge is accumulated in both drains simultaneously. The integration time has to be chosen short enough to avoid a capacity overflow. An automatic approach has been developed for adjusting the integration time to the reflected light intensity (14). To avoid overflow by superposed daylight, some models incorporate a so-called Suppression of Background Illumination (SBI) circuit. At the end the charge depending voltage difference between both drains is measured and results in DU31 and after a time shift of 41f in DU24. Using equation 4 the phase shift and therefore the distance according to equation 1 is now computable. Registered with preoperative CT or MR data it may provide information on position and orientation of the robotic device or endoscope. It would be possible to show hidden organs or vessels by augmented reality, extend and virtually rotate the field of view and enable efficient collision prevention. An approach to face this

Sd(t)

CK(TK + 2) -g K(t + TK + 2)

( 4)

t

CK(TK) g K(t + TK)

Figure 5. Principle of a “charge swing” with a photonic mixing device for real-time distance measurement.

K. Höller et al.

challenge is the acquisition of 3-D information directly via the endoscope with a hybrid imaging system. Parallel to the CCD camera a Time-of-Flight (ToF) system is integrated (Figure 6). Accordingly, the name Multisensor-Time-of-Flight endoscope (MUSTOF endoscope) was chosen (15). For this device, sensor calibration, image reconstruction, feature extraction and volume registration are required (16). Since MUSTOF endoscopy provides a third dimension to endoscopic images in real-time, it could be possible to process those algorithms in realtime, too. To compensate the high optical attenuation of endoscopic systems, a much more efficient illumination unit had to be designed. However, to generate a sufficient light intensity, many LEDs are required in parallel. But it is very impractical to couple such an illumination unit with parallel circuits to the illumination fiber guide of an endoscope. Therefore, we attached a single fiber-coupled high-power laser diode to the endoscope. With a maximum output power of 2 W emitted from a single 200 mm diameter optical fiber, this laser diode can easily be coupled to the endoscope and provides sufficient light power to overcome the transmission loss of the endoscope illumination. The high-frequency characteristics of the laser diode have been studied thoroughly to design the required high-speed driver electronics. Using a single RF metal-oxide semi-conductor field effect transistor (MOSFET) for laser current modulation, it was possible to reach frequencies up to 50 MHz, with potential for up to 100 MHz for future ToF sensors. This results in an improved distance resolution. The modulation is synchronized with the ToF sensor for accurate phase-delay measurements. Thus, a powerful and versatile illumination light source for adopting standard 3-D ToF sensors to endoscopes was realized (17, 18).

Gravity-based endoscopic image rectification A still unsolved problem in flexible endoscopy during NOTES interventions (1) is the missing information about the image orientation (19). Thus, tip retro-flexion of a non-rigid video-endoscope may cause image rotation angles up to ±180" (20). Our so-called “EndoSens” approach for measuring this orientation angle relies on the integration of a Micro Electro-Mechanical System (MEMS) based inertial sensor device at the distal end of the videoscope (21). This MEMS-device is capable to measure influencing forces in three orthogonal directions. The x axis is defined along the endoscope direction. The y axis is orthogonal to the x axis and in the plane containing the endoscope’s tip. And finally the z axis is defined to build up a right-handed coordinate system. If the endoscope is not moving, only the acceleration of gravity g has an effect on the three axes. The basic idea is to build up a relation between the direction of gravity and the orientation of the endoscope. Equation 5 expresses how rotation parameters F, Q and C of the IMU (Inertial Measurement Unit, Figure 7) have to be chosen to get a correct spatial orientation. Therefore the z axis in the orientation of the endoscope has to be parallel to the direction of gravity g, which means a congruence regarding the orientation of visualized and real world scenery:

 Fx  F  = R ⋅R ⋅R F Q Y  y  Fz 

 0  − sin (Q ) g  ⋅  0 =  sin (F ) cos (Q ) g      os (F ) cos (Q ) g   g   co

(5)

With Fx, y, z: measured accelerations (specific forces) and RF, RY, RC rotation matrices over axes x, y and z. Using an atan2 function the rotation angle F can be computed out of acceleration values Fy and Fz on the

CCD


266

TOF

Figure 6. White-light and modulated IR light are coupled in the illumination channel of the endoscope. The reflected portions attain through the image channel to a narrow band beam splitter. There the IR fraction is deflected to the ToF camera whereas the remaining part passes to the CCD camera.



267

Pitch axis y θ Roll axis x φ ψ Yaw axis z

g

Figure 7. Measuring axis description of the attached IMU at the endoscope tip.

two axes y and z orthogonal to the endoscopic line of view in x-direction (19) (Equation 6, 7):

is only applied if the superposed acceleration to gravity g is below a boundary value DFabsmax (Equation 8):

F = atan2( Fy ,Fz )

(6)

g − Fxi 2 + Fyi 2 + Fzi 2 < ∆Fabsmax

Q = arcsin( − Fx / g )

(7)

As the vector of gravity g defines only two degrees of freedom it is not possible to calculate yaw C. Whenever Fx = ± g (→ Q = ± π → Fy = Fz = 0) it is not possible to calculate F either. With usual movements, gravity is at least by one magnitude higher than other accelerations. To avoid motion influence, correction

Figure 8. EndoSens hardware prototype in a bulk version.

(8 )

With the employed ST Microelectronics (Geneva, Switzerland) LIS331DL sensor (Figure 8) there is a uniform quantization of 8 bit for a range of ±2.3 g for each axis. This implies a quantization accuracy of 0.018g per step or 110 steps for the focused range of ±g. This is high enough to achieve a durable accuracy even to 1" within relatively calm movements. This is possible as roll angle F is calculated out of inverse trigonometric values of two orthogonal axes. Acceleration occurs only in the short moment of changing velocity or direction of the movement. For the special case of acceleration with the same order of magnitude as gravity, the upper acceleration limit can be chosen small enough to suppress calculation and to freeze the angle for this short period of time. By choosing a longer delay line for the used smoothing Hann filter and a higher minimum variation threshold on each axis, correction may be delayed by fractions of a second but will be stable even during fast movements. Actual video rate is 25–30 frames per second, according to the video signals PAL or NTSC. The accelerometer values are refreshed every 2.5 ms, which is equivalent to a rate of 400 values per second. Therefore the measuring frequency is up to 16 times higher than the image frame rate. For each image, just one angle value is needed for image rectification. As the down sampling step can be used to minimize impact of shocks or short movements and therefore to maximize accuracy an intelligent algorithm was found


268

K. Höller et al.

considering the following aspects: There are different approaches to obtain a triad of acceleration values to calculate the rectification angle for each new image. The simplest one is to pick the last triad out of the accelerometer data stream. For better noise reduction one can average over all samples. Another approach is to sort all values of each axis separately and to take the median of each. If the magnitude of all three axes is taken into account, one can choose the triad with the minimum superposed specific force which means the magnitude nearest to gravity. By comparing all these approaches different advantages such as noise reduction (by averaging several values) and less movement influence can be achieved (if samples have different impact with respect to superposed specific forces). Nevertheless, also disadvantages such as distortion (when chosen axis values are recorded at different measurement points) and delay (with averaging) are present. A maximization of the advantages and a minimization of the disadvantages of all approaches can be reached by summing up separately all n sensor values Fxi , Fyi and Fzi within an image frame with i = 1,. . ., n and weighting them with a weighting factor wi (Equation 9) with maximal weight w0:

wi =

1 1 + w0

By using the weighted sum (Equation 10, Figure 9) less movement influence combined with quite good noise reduction is provided. One main aspect in choosing the appropriate filter algorithm for rotation angle computation is the implementation characteristics of the target platform. In our approach, acquisition of acceleration measurements, filtering and down sampling is done on a small 8-Bit microcontroller (Figure 10). First it has to be considered whether it is useful to synchronize the measurements to the frame acquisition. We decided to keep the hardware design simple by using a free running data acquisition instead. In the case of using the last triad, only a simple last-in-first-out buffer is needed to provide down sampling. The use of mean values can be treated as a simplified way of finite impulse response (FIR) filtering. This results in very small computational requirements as long as the order of the filter (tap length) is not too big. In our case we limit the tap length to 16 taps. With a tap length based on a multiple of two the computational effort is reduced significantly. The implementation of the best triad or the weighted sum method requires much more attention. First it has to be kept in mind that all calculations should be done with fixed-point, e.g. integer math to avoid unneeded computational complexity. Squaring the acceleration vector components for magnitude calculation is done fast with the available hardware multiplier. The square root calculation must be done with dedicated integer math like a CORDIC (COordinate Rotation DIgital Computer) implementation (22). Another aspect for the best triad calculation and weighted sum approaches is intermediate result handling and data management. As a first step the magnitude of an incoming acceleration vector and the weighting factor for the weighted sum method has to be calculated. The data storage itself is realized with a

(9)

Fxi 2 + Fyi 2 + Fzi 2 − g

In the next step the sum has to be normalized by the sum of all weighting factors

  Fx   −1   i    n (wi )   Fyi  ⋅ wi  ⋅    i = 1  i =1      Fzi  

 Fx  F  =  y  Fz 

n

∑

x1 y1 z 1 0.8 g

∑

x 2 y 2 z2 0.85 g

x 3 y3 z3 0.95 g

(10)

x4 y4 z4 1.02 g

x5 y5 z5 1.1 g

x6 y6 z6 1.04 g

x7 y7 z7 0.8 g

x 8 y8 z 8 0.99 g

x9 y9 z9 0.9 g

x1

x2

x3

x4

x5

x6

x7

x8

x9

y1

y2

y3

y4

y5

y6

y7

y8

y9

z1

z2

z3

z4

z5

z6

z7

z8

z9

Single image frame to be rotated Figure 9. Summing up weighted values to calculate the rotation angle of the new image frame.



269

Pitch axis y θ Roll axis x φ ψ Yaw axis z g

Accelerometer measurements

Microcontroller downsampling

PC filtering GPU rotation

Dual monitor visualization

Figure 10. EndoSens sensor board with 3-axis accelerometer for measuring, EndoSens microcontroller with implemented Endorientation algorithm for downsampling, filtering and data conversion, PC for image rectification and second monitor for visualization.

ring buffer structure. To avoid the need of synchronization, we calculate the outgoing value on demand. This results in searching the ring buffer for the best triad every time the host software asks for an actual measurement. For the weighted sum method the calculation of equation 10 is done every time the host needs an actual measurement.

Before inserting the endoscope optic, two plastic cubes each of size 15 ! 15 ! 15 mm were inserted into the stomach. Thus it was possible to observe threedimensionally objects of known size and shape for assessment of accuracy (Figure 12). Determination of the cube size was possible with an average error precision of m = 0.89 mm based on 100 acquired distance maps (23).

Results Experimental MUSTOF evaluation An empty porcine stomach was manually insufflated with air. The ToF endoscope was inserted via the remaining parts of the esophagus (Figure 11).

Figure 11. Introducing the endoscope into the insufflated stomach.

Experimental EndoSens evaluation During a porcine animal study, the navigation complexity of a hybrid endoscopic instrument during a NOTES peritoneoscopy with the well-established trans-sigmoidal access (2) was compared with and without automated image rotation (Figure 13). The endoscopic inertial measurement unit was fixed on the tip of a flexible endoscope. Additionally a pulsed DC magnetic tracking sensor (Flock of Birds, Ascension, Burlington, VT, USA), was fixed on the hybrid instrument holder for recording of the movement of the transcutaneously inserted additional instrument. To evaluate the benefit of automated real-time MEMS-based image rectification, a special task had to be fulfilled under standardized conditions (20). First, only the original endoscopic view was presented to the surgeons, navigating the transcutaneous inserted instrument. In a second run, the image view with the automatically corrected image horizon was displayed on a control monitor, while the surgeons performed the same task again. For some test persons the order of original and rotated images was changed.


270

K. Höller et al.

A.

B.

Figure 12. Cubes in porcine stomach (l) made visible in 3-D (r) through an endoscope by our MUSTOF approach.

There was no learning effect. The second turn with the original view still took more time. During the study an unmanipulated image was available exclusively for the endoscopist to navigate the flexible scope. The time required to navigate the surgical instrument to the four markers was recorded and statistically evaluated. The participating test persons were surgeons with different levels of surgical experience and expertise, including beginners, well-trained surgeons and an expert. All of them considered the automated image rectification to be very useful to navigate the transcutaneously inserted instrument towards the markers, which were inserted in each quadrant of the abdomen. The navigation path of the electromagnetically tracked hybrid instrument, controlled by a welltrained test person, is displayed in a 3-D plot (Figure 14). Increased movement activities are visible at four distinct points. They represent accumulated movements of the surgeons hand. These were translated through the fixed point of the trocar to the tip of the rigid instrument inside the peretoneal cavity. With these translated movements the needles in each quadrant had to be grasped. In comparison to the

procedure based on the original image the movements based on rectified images are significantly more accurate with shorter paths as can be seen in Figure 15. Obviously the two parameters duration and path length are strongly correlated and can be regarded as a significant measure for the complexity of surgical procedures. Since duration and path length are decreased with the application of image rectification, the complexity of the complete procedure can be reduced.

Conclusions In this article we proposed two new approaches to enhance endoscopic 2-D images with a third dimension. The MUSTOF approach is technically more sophisticated than the Endorientation approach, but it provides versatile possibilities for combination with conventional modalities. Additional 3-D data will not be an unalterable precondition for performing NOTES. But it will help especially for a safer introduction of robotic devices and improve the visualization for surgeons who are not satisfied with the

Drop Dr op

Figure 13. Rectified view of falling water drop.

271

25 20

Z

15 10 5 0 25 20

25 20

15 10

y

10

5

5 0

15 x

0

Figure 14. A total path length of 16.51 m was recorded in the original images without horizon corrections. Additionally it is visible that the surgeon needed a lot of time to reach the markers at the four quadrants.

25 20 15 Z



10 5

0 25 20

25 20

15 10

10

5 y

15

5 0

0

x

Figure 15. Rectified images simplified the task of reaching markers and reduced the total path length to 8.05 m.


272

K. Höller et al.

in-line-view and loss of orientation from flexible endoscopy. Since gastroenterologists and surgeons are still not absolutely familiar with this new NOTES approach, they both will accept new technologies more likely than with established procedures.

Augmented reality Registration with preoperative volumes opens up lots of additional possibilities. The most promising one is to show hidden organs or vessels by augmented reality. They have to be segmented in the preoperative volumes and to be transformed by iteratively computed transformation parameters. Then these organs and vessels can be displayed by use of Augmented Reality (AR). This additional information could be helpful to avoid injuries, e.g. while finding the entry point to the peritoneal cavity, which requires a wall incision in the stomach or the colon without knowledge of the structures behind the visible wall.

estimation of the instrument’s position and the distance to tissue or organs.

Collision prevention The increasing demand for robotic devices to control multiple instruments through only one flexible endoscope requires additional control mechanisms to avoid unintentional injuries. To perform efficient collision prevention, real-time distance information is needed. Avoidance of impending tissue injury or collision with other instruments can be realized, as well as auto-positioning depending on respiration or other patient movements. For the spectrum of endoscopic surgery there is an urgent demand for a stable platform for secure movements and stabilization of the tissue during the operation in the peritoneal cavity.

Orientation Enhanced field of view Endoscopic axis in-line view and loss of spatial orientation are quite uncomfortable, especially for surgeons. To compensate this disadvantage, 3-D surface knowledge can be used to extend and enhance the field of view. Endorientation supports the reconstruction of the operation area by knowledge on the translation between subsequent images, whereas MUSTOF provides an additional feature set for registration. Using a 3-D mosaicking technique is especially useful if there are no features detectable in the color image. But another use of the real-time depth map could be much more important: Now a virtual rotation of the view on the introduced instruments can be performed orthogonal to the axis of the original line of sight (Figure 16). This allows a better intuitive

To provide more information on position and orientation of the robotic device or the endoscope, intraoperative 3-D data could be registered with preoperative CT or MR data. With the aid of the calculated transformation parameters, position and orientation can be represented, corrected and visualized. Alternatively the EndoSens hardware is able to measure the rotation angle directly. Therefore the original orientation can be represented, corrected and visualized as well. Thus, the Endorientation approach is much less challenging, but it works reliably and addresses a still unsolved problem directly. The hardware for both approaches is based on offthe-shelf components. Since automotive and consumer electronics industries are heavily interested in these emerging technologies, lot of research investment is being allocated to improve the chip design developed to date. Currently, a typical ToF camera is

Figure 16. Conventional color image (left), depth values (center), off-axis view (right).


Spatial orientation in translumenal surgery available for some 1.000 EUR. But if automotive and consumer applications enable large-scale production, prices will rapidly decline and innovation cycles will be quite short. Even if costs at the moment are too high to address the consumer market, the presented medical applications are auspicious approaches to improve actually provided health care. The EndoSens hardware is even much cheaper by now. The whole hardware is a few payable 100 EUR for the obtained benefits. And integration of this tiny MEMS chip in a new endoscope is not a difficult technical task. Acknowledgements We wish to thank our former colleague J. Penne for his work and ideas with the MUSTOF endoscope as well as R. Engelbrecht and T. Schrauder from the Chair of Microwave Engineering and High Frequency Technology, Friedrich-Alexander University, Erlangen-Nuremberg for their helpful support with developing the laser illumination unit of the MUSTOF endoscope. And we are thankful to G. Hager, P. Kazanzides, R. Kumar, D. Mirota and H. Girgis for their helpful suggestions and discussions regarding the EndoSens algorithms during Kurt’s research stay at the Engineering Research Center for Computer-Integrated Surgical Systems and Technology, The Johns Hopkins University, Baltimore, MD. Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper. References 1. Rattner D, Kalloo A. ASGE/SAGES working group on Natural Orifice Translumenal Endoscopic Surgery: White Paper October 2005. Surg Endosc. 2006;20:329–33. 2. Wilhelm D, Meining A, von Delius S, Burian M, Can S, Fiolka A, et al. Second generation sigmoid access for NOTES using the ISSA-system. Endosk heute. 2008;21:70. 3. Härtl F, Höller K, Beller S, Feußner H. Current status of the use of medical robots in Germany, Austria and Switzerland. In 3rd Russian-Bavarian Conference on Biomedical Engineering, Hornegger J, Mayr EW, Schookin S, Feußner H, Navab N, Gulyaev YV, Höller K, and Ganzha V (Eds.), Erlangen, 2007:2–4. 4. Fritscher-Ravens A, Mosse CA, Ikeda K, Swain P. Endoscopic transgastric lymphadenectomy using EUS for selection and guidance. Gastrointest Endosc. 2006;63:302–6. 5. Hummel J, Figl M, Kollmann C, Bergmann H, Birkfellner W. Evaluation of a miniature electromagnetic position tracker. Med Phys. 2002;29:2205–12. 6. Van Bergen P, Kunert W, Bessell J, Buess GF. Comparative study of two-dimensional and threedimensional vision systems for minimally invasive surgery. Surg Endosc. 1998;12:948–54.

273

7. Thormählen T, Broszio H, Meier PN. Three-dimensional endoscopy. In Falk Symposium No. 124, Medical Imaging in Gastroenterology and Hepatology, Hannover, September 2001 2002:124. 8. Okatani T, Deguchi K. Reconstructing shape from shading with a point light source at the projection center: Shape reconstruction from an endoscope image. In IEEE Proc. of ICPR ’96; 1996:830–4. 9. Hasegawa K, Noda K, Sato Y. Electronic endoscope system for shape measurement. In Proc. 16th ICPR 2002:761–4. 10. Fischer R. 3-D-Oberflächeninspektion in Echtzeit mit SPARC. Informationsdienst Wissenschaft 2008. 11. Heinol H. Untersuchung und Entwicklung von modulationslaufzeitbasierten 3D-Sichtsystemen. Ph.D. thesis, Department of Electrical Engineering and Computer Science at the University of Siegen; 2001. 12. Luan X. Experimental Investigation of Photonic Mixer Device and Development of TOF 3D Ranging Systems Based on PMD Technology. Ph.D. thesis, Department of Electrical Engineering and Computer Science at the University of Siegen; 2001. 13. Lange R: 3D Time-of-flight distance measurement with custom solidstate image sensors in CMOS/CCD-technology. Ph. D. thesis, Department of Electrical Engineering and Computer Science at University of Siegen; 2000. 14. Ritt P, Höller K, Penne J, Schneider A, Hornegger J, Feussner H. Auto-gain approach for use with time-of-flight examination in minimally invasive surgery. In: Hornegger J, et al. (Eds.): Proc. 3rd Russian-Bavarian Conference on Biomedical Engineering, Fraunhofer IIS, Erlangen, 2007:227–32. 15. Krüger S, Penne J, Hornegger J, Hohenberger W, Horbach T. “A novel approach to 3-D Endoscopy in Minimally Invasive Surgery,” in Abstract-band CURAC 2006, Lenarz T and Matthies HK, Eds.;2006:183. 16. Penne J, Höller K, Krüger S, Feussner H. NOTES 3D: Endoscopes learn to see 3D; basic algorithms for a novel endoscope. In Arajo H, Vitri J (Eds.): Proceedings of VISAPP 2007:134–9. 17. Tacke D. Ankopplung einer optischen Lichtlaufzeit-Kamera an ein Endoskop zur Aufnahme dreidimensionaler Oberflächenbilder. Diploma thesis, Chair of Pattern Recognition, FAU Erlangen-Nuremberg; 2008. 18. Schrauder T: Entwicklung einer Treiberschaltung für schnell modulierte Leistungslaserdioden zur Optimierung von Lichtlaufzeit-Kameras. Student thesis, Chair of Pattern Recognition, FAU Erlangen-Nuremberg; 2008. 19. Höller K, Petrunina M, Penne J, Schneider A, Wilhelm D, Feußner H, et al. Taking endoscopy to a higher dimension: Computer Aided 3-D NOTES. In: Proc. 4th Russian-Bavarian Conference on Bio-Medical Engineering, Zelenograd, Moscow, MIET 2008 Jul 8–9:33–7. 20. Höller K, Schneider A, Jahn J, Guttierrez J, Wittenberg T, Hornegger J, et L: Clinical evaluation of endorientation: Gravity related rectification for endoscopic images. In: Proc of the ISPA; 2009 Sep 16–18:713–7. 21. Höller K, Penne J, Schneider A, Jahn J, Guttierrez J, Wittenberg T, et al. Endoscopic orientation correction. In: Medical Image and Computer Assisted Intervention, 12th International Conference Proceedings MICCAI’09, London, UK; 2009 Sep 20–24:459–66. 22. Walther JS. The story of unified CORDIC. Journal of VLSI Signal Processing 2000;25:107–12. 23. Penne J, Höller K, Stürmer M, Schrauder T, Schneider A, Engelbrecht R, et al. Time-of-Flight 3-D Endoscopy. In: Medical Image and Computer Assisted Intervention. 12th International Conference Proceedings MICCAI’09, London, UK, 2009 Sep 20–24:467–74.