Article published in International Journal of Computer Assisted Radiology and Surgery, vol.10(2), pp. 217-229, February 2015 DOI 10.1007/s11548-014-1078-9
A vision-based system for fast and accurate laser scanning in robot-assisted phonomicrosurgery Giulio Dagnino · Leonardo S. Mattos · Darwin G. Caldwell
Abstract Purpose Surgical quality in phonomicrosurgery can be improved by open-loop laser control (e.g., high-speed scanning capabilities) with a robust and accurate closed-loop visual servoing systems. A new vision-based system for laser scanning control during robot-assisted phonomicrosurgery was developed and tested. Methods Laser scanning was accomplished with a dual control strategy, which adds a vision-based trajectory correction phase to a fast open-loop laser controller. The system is designed to eliminate open-loop aiming errors caused by system calibration limitations and by the unpredictable topology of real targets. Evaluation of the new system was performed using CO2 laser cutting trials on artificial targets and ex-vivo tissue. Results This system produced accuracy values corresponding to pixel resolution even when smoke created by the lasertarget interaction clutters the camera view. In realistic test scenarios, trajectory following RMS errors were reduced by almost 80 % with respect to open-loop system performances, reaching mean error values around 30 µm and maximum observed errors in the order of 60 µm.
G. Dagnino Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy Present Address G. Dagnino Bristol Robotics Laboratory, University of the West of England, Bristol, UK L. S. Mattos (B) · D. G. Caldwell Department of Advanced Robotics, Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genoa, Italy e-mail:
[email protected]
Conclusion A new vision-based laser microsurgical control system was shown to be effective and promising with significant positive potential impact on the safety and quality of laser microsurgeries.
Keywords Medical robotics · Laser microsurgery · Robot-assisted surgery · Vision-based control Introduction Phonomicrosurgeries are demanding surgical procedures. They involve delicate operations on vocal cords and require significant psychomotor skills from the surgeon [1,2]. These surgeries are performed to treat a range of benign and malignant pathologies, including polyps, cysts and cancer, all of which can adversely affect basic physiological functions such as deglutition, respiration and phonation [3]. Due to the location and small size of vocal cords (between 11 and 21 mm [4]), and also to the fact that lesions can be smaller than 1 mm [1], phonomicrosurgeries require the use of a surgical microscope, a specialized endoscope (the laryngoscope), and specialized surgical instruments such as a surgical laser, microforceps, microscissors and microlaryngeal knifes [5,6]. Currently, the major elective technology for phonomicrosurgery is the CO2 surgical laser coupled with the surgical microscope. In this case, the laser beam is manually controlled by the surgeon by means of a mechanical micromanipulator, which is connected to the microscope at a typical operating distance of 400 mm from the vocal cord [7]. This process uses a low-power visible laser (the aiming beam), which is coincident with the high-power surgical laser beam and is always active, giving continuous visual feedback on the laser aiming point. Nevertheless, a simple analysis of the
surgical setup and interviews with practitioners, as reported in [13], indicate this kind of manipulation is prone to errors resulting from ergonomics and inexperience. In laser phonomicrosurgery, accurate laser aiming is critical to minimize surgical impact on voice quality, while still guaranteeing total pathology removal [8]. At the same time, fast laser scanning motions are important to minimize tissue carbonization and thermal damage to surrounding tissue, which are side effects that negatively impact the quality of laser surgeries. These requirements, together with the issues identified above, lead to the believe that robotic assistance can have an extremely positive impact in this field. Robot assistance in the operating room has been increasing since the 1980s [9] and is currently helping surgeons pioneer new procedures and achieve unprecedented levels of performance (e.g., through motion scaling, tremor filtering, stereo vision interface) [10]. Major technological goals are to improve patient safety, reduce operating time, and improve the surgical outcome. However, commercial laser phonomicrosurgery devices such as the AcuBlade system (Lumenis Ltd), the SoftScan Plus (KLS Martin GmbH), and the HiScan Surgical (DEKA M.E.L.A. Srl) currently have little assistive capabilities. These commercial devices feature computerized laser scanning systems that permit uniform and user-defined con-
trol over the incision length, ablation area and penetration depth. However, the systems offer only a small set of predefined scan patterns and the surgeon still has to aim the laser beam by means of a traditional mechanical micromanipulator. Consequently, they do not address the controllability and precision issues mentioned above and the surgical outcome is still totally dependent on the surgeon dexterity and experience. At the Istituto Italiano di Tecnologia (IIT), research toward improving laser phonomicrosurgeries has resulted in the creation of the robot-assisted system shown in Fig. 1, [11]. This is a teleoperated system based on a motorized laser micromanipulator (Fig. 1a), which is coupled to a surgical microscope (Fig. 1b) and controlled in real time from a computer station (Fig. 1c). The system allows the surgeon to precisely aim the laser using a more ergonomic and intuitive user interface compared to other existing systems [13]. It also allows the definition of cutting paths for automatic execution using fast scanning motions, which effectively satisfy one of the basic requirements for improved surgical quality. Experiments performed with the IIT system have demonstrated that it is able to greatly reduce aiming errors when compared to traditional laser micromanipulators [12,13]. Nevertheless, further system improvements were still needed to guarantee good performance on 3D surfaces (i.e., on
Fig. 1 The IIT system for robot-assisted laser phonomicrosurgery: a the motorized laser micromanipulator; b the system setup showing the micromanipulator and the CO2 laser arm attached to a surgical microscope with built-in CCD camera; and c the system control software
real surgical scenarios). Experiments presented in [14] highlighted a performance degradation in those cases, which was mainly caused by two factors: 1) The laser was automatically controlled in open-loop, based on a static mapping from the control space to the task space; 2) this mapping was defined based on a simple model that assumed the target area as approximately planar and at a constant operating distance (OD) from the actuator mirror, as schematically represented in Fig. 2. A solution to the issues above is presented here. It consists of the creation of a robust laser tracker and a closedloop control system to correct the laser trajectory, effectively increasing system adaptability and precision for any given target. This solution was inspired by the concept of visual feedback, which is widely studied in the literature, e.g., [15– 19]. However, it differs from previous control approaches, such as traditional visual servoing [20,21], and high-speed visual servoing [22], by achieving high-speed laser scanning control using the slow medical-grade cameras (max. frame rate 30 fps) installed on surgical microscopes. Existing approaches are not readily applicable to the laser phonomicrosurgery because they would inevitably cause an unacceptable slowdown of the laser scan speed. This paper starts with a quick presentation of the IIT microsurgical system in Sect. 2 and of its control structure in Sect. 3. The laser visual feedback system responsible for obtaining real-time information on the laser aiming is introduced in Sect. 4. This system is the basis of the new vision-
based laser control system, which is presented in Sect. 5 and evaluated through the experiments described in Sect. 6. Experimental results are presented in Sect. 7 and discussed in Sect. 8. Finally, Sect. 9 offers concluding remarks and an overview of plans for future research developments.
Surgical system configuration The robot-assisted laser phonomicrosurgery system used and improved upon in this research consists of the following components: A CO2 surgical laser, the IIT laser micromanipulator, a surgical microscope with built-in CCD camera, and the system control software, which runs on a laptop computer (Dell Latitude XT2 tablet PC). This computer controls the laser micromanipulator through high-level commands and performs all the image processing routines described later in this paper. The system setup is showed in Fig. 1 and its main sub-systems are described below. Surgical laser system The CO2 laser source used in this study was an Opmilas CO2 25 (Carl Zeiss SMT GmbH, Germany), which emits light at a wavelength of 10.6 µm with a maximum power of 25 W. This system also features a visible red laser beam coupled coaxially with the optical axis of the CO2 laser. This second laser allows the surgeon to visually aim the surgical laser before activating it to start the tissue cutting or ablation. Both laser beams are delivered to the IIT laser micromanipulator via an articulated arm (Fig. 1b). IIT laser micromanipulator The laser micromanipulator is based on a single fast steering mirror driven by a parallel kinematics mechanism (Fig. 1a). This system offers high-speed operation (200 Hz) and 100 mrad optical beam deflection under closed-loop control. Its low-level controller can produce fast scanning motions with high precision, based on the position encoders. This guarantees 10 µrad optical beam deflection resolution with 10 µrad repeatability in both axes [11] provide more details. Surgical microscope
Fig. 2 Schematic for modeling the laser beam displacement from micromanipulator mirror rotations. The parameter, OD, is the operating distance; Rx and Ry are the linear displacements of the actuator respectively over the x and y directions; and θ x and θ y are the angular actuator displacements between P0 and P1
A stereo ENT surgical microscope (Leica Microsystems GmbH) is used in this study (Fig. 1b). This microscope has a focal length of 200 mm and 5 magnifications ranges: 6×, 10×, 16×, 25×, and 40×. The microscope also includes an integrated Leica M651 CCD camera, connected via USB to the control computer and providing live video streaming from the surgical area (Fig. 1c).
System operation The IIT surgical system is controlled from a computer terminal. Its control space is the 2D coordinates of the monitor where live video from the surgical site is displayed. Consequently, control of the laser aiming point on the target requires a mapping function that converts display coordinates to target coordinates. If the surgical area is assumed to lie on the microscope’s focal plane, the mapping mentioned above corresponds to a coordinate transformation between two 2D planes. This can be described by an affine transformation consisting of a translation, rotation, zoom and shear. The correct determination of this mapping guarantees a precise correspondence between a target point defined on the live video and the laser aiming point at the surgical site, which can be used for control purposes. At system initialization, the first requirement is calibration to determine this mapping. System calibration As detailed in [11], system calibration is performed in three steps: 1) data points acquisition; 2) calibration parameters estimation; and 3) optimization of the calibration parameters. During the first calibration step, the computer automatically positions the laser micromanipulator at n different positions and the laser tracker algorithm (described in the next section) provides the coordinates of the laser spot in the video frame. These image space coordinates are then matched to the micromanipulator coordinates (provided by its low-level controller), creating the calibration dataset. The second calibration step consists of the estimation of the affine transformation parameters. This is done using two points from the dataset, resulting in the definition of the following transformation: V
Pi = VR Rot · S · Z ·
R
Pt − RP0 + V P0
(1)
where Pi is a point of interest; P0 is a reference point (e.g., the first point of the calibration dataset); VR Rot is the rotation matrix from the target space to the video space; S is the scaling matrix, and Z is the shear matrix. Finally, the third calibration step consists of a refinement of the parameters in (1) using all points in the calibration dataset and an error minimization strategy. Open-loop control The mapping function identified above allows control of the laser aiming directly from a computer monitor. Target points are defined in video coordinates and used to generate commands that move the laser spot to the desired locations.
Although this is an open-loop control system, [12] demonstrated such systems can achieve good precision while also allowing the creation of a highly intuitive laser control interface. The control of the surgical system in open-loop works in real time as follows: When the surgeon clicks on a point of V the live video, the system acquires the pixel coordinates Pi of that point, defines it as a target and computes the corresponding position of the micromanipulator in the target frame RPi using the inverse of (1). This position is then sent to the micromanipulator controller, which precisely moves the actuator mirror to the commanded location. Intraoperative surgical planning Extrapolating the open-loop control concept from a single target point to a series of target points leads to the execution of trajectories planned in image space. The definition of such trajectories in real time is what we termed intraoperative surgical planning. Surgical plans are created directly on the live surgical video using graphics overlays. These are drawn using any computer pointing device (e.g., a mouse or a graphics tablet), making the system intuitive and very flexible, able to generate any desired laser scan trajectory. Once a plan is defined, it is converted to a trajectory in task space and submitted to the low-level laser micromanipulator controller. There it is automatically executed in a continuous high frequency loop giving rise to the required fast scanning laser motions. This planning system was previously evaluated in [11], demonstrating it affords high precision and can reduce trajectory following errors up to 80 % when compared to the traditional laser micromanipulator system. However, as mentioned previously, the system suffered from performance degradation on 3D targets. This problem led to the development of a visual feedback system to close the laser aiming control loop, as described in the next sections.
Laser visual feedback (LVF) Acquisition of the actual position of the laser spot on the surgical site is essential for closing the laser control loop, enabling automatic correction of aiming errors. This information is extracted from the live microscope video using a laser spot tracking system based on two image processing algorithms that work in parallel: one based on template matching and the other on intensity thresholding. This redundancy is important for the reliability and robustness of the tracking system, which should provide precise feedback information to guarantee the accuracy and stability of the final vision-based laser controller. Erroneous laser spot localizations caused by false positives generate wrong inputs to the
vision-based laser controller, ultimately producing erroneous motion commands to the laser micromanipulator and deteriorating the quality of the entire system. The first step to improve the reliability and processing speed of the laser spot tracking algorithms is the automatic selection of a guaranteed region of interest (gROI) within the live video frames [23]. This is done based on the system calibration and on the actuator mirror coordinates provided by its controller. This enables the laser spot coordinates to be estimated in the video frame using (1). The size of the gROI is different for each value of microscope magnification. It is predefined empirically to correspond to a small square with sides 5 times larger than the diameter of the observed laser spot. This keeps the gROI size small, helping to minimize the occurrence of tracking errors that can degrade the quality of the vision-based laser controller. The size of the gROI is defined by a single parameter that can be adjusted in run time if needed. Locating the laser spot within the gROI is primary performed using an algorithm based on template matching [24]. This algorithm starts with two filtering operations: (1) Color plane extraction The red plane is extracted from the input color image provided by the CCD camera. This color plane is the most significant for the task since the aiming laser is red in all CO2 surgical laser currently on the market. The output of this filtering step is a grayscale image. (2) Noise filtering A convolution linear filter is applied to smooth the grayscale image created in the previous step. In this case, a 3x3 Gaussian kernel is used. Subsequently, the algorithm performs the following additional steps: (A1) Image correction applies a brightness, contrast and gamma correction to the grayscale image. (A2) Template acquisition creates the template to search for the laser spot in the video images. The template is a copy of a small area of the input image that includes the laser spot. This is performed by the user using a pointing device (computer mouse or stylus). The process is very simple, involving only the drawing of a small square enclosing the laser spot on the live video. Template acquisition is performed only once during system initialization. However, if desired, a new template can be acquired in run time by repeating this process. (A3) Template processing This step extracts features from the acquired template image, which are used later to search for the laser spot. In this case, the edge of the laser spot is the information of interest since its high brightness produces strong and well-defined edge features. The traditional Canny edge detector algorithm
is used for this task. No further template processing is required. (A4) Template matching This step is responsible for actually searching for the laser spot within the gROI. A crosscorrelation template matching algorithm based solely on 2D translations is used to find the best matching position of the template over the gROI’s image. This position corresponds to the best estimated location of the laser spot. Assuming the template’s edge image is represented by w(x, y) of size K×L, and the gROI edge image by f (x, y) of size M × N (with K ≤ M and L ≤ N), the cross-correlation between w and f at a point (i, j) is given by C(i, j) =
−1 L−1 K
w(x, y) f (x + i, y + j)
(2)
x=0 y=0
The search algorithm moves the template w over the entire image f and computes their cross-correlation at each point, creating the correlation image C(x, y) of size M × N. The maximum value of C indicates the position where w best matches f . This position is output by the algorithm as the actual position of the laser spot on the video frame. This first laser tracking system was evaluated in [25] and proved to be very precise even during cutting procedures. Its position measurement accuracy was 0.072 ± 0.023 mm under 16× magnification, which corresponds to approximately 2 pixels in the video images. These small errors come from distortions of the laser spot caused by the 3D features on the target surface. These distortions negatively impact the performance of the template matching algorithm and in some cases may cause it to fail, i.e., it may not be able to locate the laser spot. When this happens, the algorithm returns an error message. A secondary laser tracking algorithm was developed to serve as a supervisory safety system and to maintain proper laser control if the primary algorithm failed. Its working principle is based on intensity thresholding and it is independent of shape features, so it is more robust against laser spot distortions than the template matching method. Its downside is reduced localization accuracy, nevertheless, erroneous laser position data can be detected and corrected using this secondary tracking algorithm, so it is an important feature for guaranteeing the safety levels needed for clinical applications. The first two processing steps of this second laser tracker are shared with the previous template matching algorithm: color plane extraction and noise filtering. The subsequent steps are described below: (B1) Thresholding image pixels with an intensity value above the threshold are set to 255 (maximum intensity
value); the other pixels are set to zero. The threshold value used here was empirically found through in vitro experiments using the surgical system setup described above, with constant external/microscope lightning and the surgical targets described in the following “Experimental validation” section. This processing step produces a binary output image. (B2) Erosion An erosion filter, applied to the binary image produced in the previous step, removes small particles from the image. This process produces a black image with a single white blob that corresponds to the laser spot. (B3) Centroid calculation The centroid of the white blob is computed. Its coordinates constitute the algorithm’s return value and represent the laser spot position in the video frame. One small limitation of this secondary laser tracking algorithm is the fact it is sensitive to changes in surgical site illumination. Consequently, at the beginning of the operation, the user must tune the threshold parameter. This is performed through the control GUI, which allows run-time adjustments of the threshold value and presents the results of the segmentation algorithm on a dedicated display. This way the user can manually select an appropriate threshold by verifying the laser spot is being correctly localized and segmented.
The input of this system corresponds to the points forming the desired trajectory, which is drawn by the surgeon on the live video. These points define the desired visual targets VP V V T i = ( P x T i , P y T i ) and their collection constitute the desired path array VP T . Once the desired path is defined, a path correction phase is started using the red aiming laser only. In this phase, the system automatically controls the laser to achieve VP T , correcting the laser aiming to precisely achieve each target V PT i . This process is performed according to the following algorithm: For each V PT i ∈ VP T , the laser is moved to the target position in open-loop using (1). Then, the actual position of the laser V PL = (V P x L , V P y L ) provided by the laser visual feedback (LVF) system is compared to the target position V P . This produces the following error: Ti E(n) = V PT − V PL
When case E¯ = |V PT i − V PL | ≤ 1 pixel, the laser position is acceptable (1 pixel ≈ 0.031 mm at 16× magnification) and the target is considered reached. Otherwise, the closed-loop vision-based laser control system is activated to correct the laser position (Fig. 3). This system is based on a PID control law [26,27] and generates the motion commands Rθ = ( Rθ x, Rθ y) to adjust the position of the laser micromanipulator mirror. These commands are sent to the lowlevel system controller at each processing time step n using (4):
Vision-based laser control (VLC)
θ (n) = k P · E(n) + k I · E(n) + k D · E D (n)
This system improves the accuracy and robustness of the robotic laser aiming system using visual feedback obtained from the laser tracker. It corrects errors in the actual laser path, automatically compensating for deviations caused by the 3D characteristics of the target or the limitations of the calibration system.
where
Fig. 3 Diagram of the vision-based laser servoing system. The system compares the actual position of the laser V PL to the target position V PT , generating the error E. If the error is not acceptable (E > 1 pixel), the
(3)
E(n) + E(n − 1) t 2 E(n) − E(n − 1) E D (n) = t E I (n) = E I (n − 1) +
(4)
(5) (6)
PID controller is activated to adjust the position of the laser micromanipulator mirror
Fig. 4 Target substrates used for the validation trials: a Chicken thigh; b Plaster block
In the equations, EI (n) is the integral of the error, ED (n) is the derivative of the error, K P , K I and K D are, respectively, the proportional, integral and derivative control gains, and t is the sampling time. In our system, the following constants were selected after tuning: K p = 10; K I = 8; K D = 0.18; t = 33 ms. The vision-based closed-loop controller drives the laser to each desired target point. At each of those points, the mirror position coordinates are recorded and stored in the array “Robot Positions.” Once all target points forming the desired path are tested and corrected, the “Robot Positions” array is downloaded to the low-level laser micromanipulator controller, which generates the fast scanning motion along the corrected path. This process satisfies the surgical requirements for fast and precise laser scanning motions. This control approach works well for laser phonomicrosurgery because the operating site is very stable. During these surgeries, the patient is fully anesthetized, intubated, and rigidly fixed in position by the laryngoscope. The only tissue motions observed are those caused by the surgeon while performing manipulations with a forceps (e.g., tissue stretching or retraction) and those caused by pressure changes in blood capillaries (≤50µm), which are considered acceptable for this application. In any case, if unexpected tissue motions occur between the laser trajectory correction phase and the completion of the therapy, the surgeon can easily turn off the surgical laser and program a new trajectory. Therefore, under the above hypothesis, surgical plans can be carefully defined, corrected and executed under relaxed time constraints and without affecting surgical precision.
Experimental validation The developed vision-based laser control system was assessed through two experimental series. The first one consisted of validation trials that assessed the automatic laser positioning performance on a predefined grid of points. The second series of experiments consisted of application trials
that assessed the system’s path following performance during real CO2 laser cutting. In both cases, the targets consisted of chicken tissue and small plaster blocks (Fig. 4). In addition, the microscope magnification was fixed at 16× and the CO2 laser power was set to 2W continuous mode, which correspond to typical settings used by surgeons during real laser phonomicrosurgeries. Video streaming from the microscope camera was digitalized at 30 fps at VGA resolution (640 × 480 pixels), resulting in an approximate distance of 0.03 mm/pixel for the objective used. Image latency in the system (i.e., the time delay between the real scene and that shown on the display) was approximately 100ms. Both experimental series aimed at characterizing the VLC system through quantitative measurements of performance, allowing the definition of specifications and comparisons with the previous open-loop (OL) control scheme. Consequently, the metrics chosen for system validation were the root-mean-squared-error (RMSE) and the maximum absolute error (MAE) measured during positioning and scanning trials. The experimental protocols are described below. In both cases, the experiments started with the system calibration as described earlier. The experimental results are presented and discussed in following sections.
System validation trials (positioning) In this case, the targets consisted of a grid of 300 points defined in image space over the live microscope video, as shown in Fig. 5. Four sets of 5 trials were conducted using only the low-power aiming laser and the following combinations of target substrate and controller: (plaster block, OL); (plaster block, VLC); (chicken tissue, OL); (chicken tissue, VLC). In the OL cases, the LVF system was used to acquire the actual laser aiming position at each target point, allowing the measurement of errors as the distance between the desired and actual laser positions.
Results Results from the experiments described above were used to assess the system’s calibration accuracy, laser positioning accuracy, and path following accuracy. The MATLAB Statistics Toolbox (MathWorks, Inc.) was used to analyze the experimental data, and the Mann–Whitney U test was used to establish significant differences between measurements. P ≤ 0.05 was regarded as significant [28].
System calibration accuracy
Fig. 5 Positioning grid overlaid on real-time video of the target substrates: a Plaster block; b Chicken thigh
In the closed-loop cases, the laser positioning on each target point was performed using the VLC system. Also here the final aiming positions were acquired using the LVF system to allow the measurement of positioning errors.
Application trials (path following) For these experiments, sets of 10 trials were performed with each of the four combinations of target substrate and controller described above. Here, the CO2 laser was used and the targets consisted of random target trajectories drawn in image space, as shown in Fig. 9. It should be stressed that these trials executed real laser cuts on the target substrates, creating smoke and changing the target morphology (when chicken tissue was used). During these experiments, the points forming the desired laser trajectory were saved for subsequent comparison with the actual laser aiming points obtained with the LVF system. These comparisons generated the performance metrics, RMSE and MAE, for each trajectory following trial.
The positioning errors recorded during the system validation trails (Fig. 5) were initially used to assess the system calibration accuracy. Considering all five trials in OL, the overall 1,500 positioning error measurements on the plaster blocks resulted in a RMSE of 0.09 ± 0.02 mm and an average MAE of 0.23 ± 0.04 mm. When using chicken tissue as the target, the 1,500 measurements resulted in a RMSE of 0.14 ± 0.02 mm and an average MAE of 0.31 ± 0.05 mm. Visual representations of the experimental results are presented in Fig. 6. The figure provides maps of positioning errors measured over the entire field of target points for sample trials with the two different substrates. This provides an intuitive indication of performance degradation on 3D targets (chicken tissue, plots B and D) with respect to more planar targets (plaster block, plots A and C).
Laser positioning accuracy A summary of the experimental results from the 20 system validation trials are presented in Tables 1 and 2. The tables report on the performance metrics obtained with both control systems (OL and VLC) and target substrates used. The results are also presented graphically in Fig. 7, which clearly illustrates the large accuracy improvements provided by the closed-loop vision-based laser controller. Results from trials on plaster blocks show a 67 % reduction in RMSE and 83 % reduction in average MAE when the VLC system was used instead of the OL system (Table 1). Similarly, the RMSE and average MAE measured during the trials with chicken tissue show, respectively, 79 and 77 % error reductions with the VLC system (Table 2). Figure 8 shows results from sample trials comparing the performance of VLC system to that of the OL system on both substrates tested: plaster block (OL in plot A, VLC in plot C) and chicken tissue (OL in plot B, VLC in plot D). The figure clearly demonstrates the new system is able to eliminate aiming errors within the entire test area.
Fig. 6 Results from open-loop positioning trials on plaster block (plots a and c) and chicken tissue (plots b and d). The error maps (plots c and d) show the absolute positioning error at each target point using a color scale ranging from blue (null positioning error) to red (maximum
Table 1 Results from validation trials on plaster blocks
absolute error). These values were calculated in image space as the distance, in pixels, between the target points (blue circles in a and b) and the laser position (red “+”). The position associated with the maximum absolute error measured during the trial is marked with the red “*”
OL
VLC
p
Reduction (%)
Number of targets
1,500
1,500
RMSE (mm)
0.09 ± 0.02
Average MAE (mm)
0.23 ± 0.04
0.03 ± 0.01
0.018
67
0.04 ± 0.01
0.008
83
OL
VLC
p
Reduction (%)
Number of targets
1,500
1,500
RMSE (mm)
0.14 ± 0.02
0.03 ± 0.01
0.012
79
Average MAE (mm)
0.31 ± 0.05
0.07 ± 0.05
0.007
77
Table 2 Results from validation trials on chicken tissue
Path following and cutting accuracy Path following results from the 40 application trials performed using real CO2 laser cutting are summarized in Tables 3 and 4. As above, the tables report the average RMSEs and MAEs measured during trials with both the OL and the VLC
systems, using plaster blocks and chicken tissue as targets. Representative pictures from these trials are presented in Fig. 9. The average correction time for each target point in a defined laser trajectory was 0.16 ± 0.02 s, which corresponds roughly to 5 ± 1 video frames at 30 fps. The total time
Fig. 7 Results from system validation trials demonstrating the accuracy improvements provided by the closed-loop vision-based laser controller
Fig. 8 Results from sample trials comparing the performance of the VLC system to that of the OL system on both substrates tested: Plaster block (OL in plot a, VLC in plot c) and chicken tissue (OL in plot b, VLC in plot d). On the plots, target points are marked with blue circles
required to correct an entire trajectory depended, therefore, on the trajectory length and number of waypoints used. For the cases shown in Fig. 9, where the trajectories were sampled with pixel resolution, the total average correction time using the VLC system was 15.7 ± 2.3 s. In practical future applications, this time can be greatly reduced without significant impact on the system’s performance by sub-sampling the desired trajectory and/or limiting its maximum allowed length. Data from the trials with plaster blocks demonstrated the average RMSE was reduced by 63 %, while the average MAE was reduced by 46 % when using the new VLC (Fig. 9b) system instead of the previous OL control (Fig. 9a), as summarized in Table 3. Similarly, trials with chickentissue (Fig. 9c,
and the achieved laser positions with red “+” symbols. The positions corresponding to the maximum absolute error measured during the trials are marked with a red “*”
Table 3 Results from application trials on plaster blocks
Table 4 Results from application trials on chicken tissue
OL
VLC
p
Reduction (%)
Number of targets
10
10
RMSE (mm)
0.08 ± 0.05
Average MAE (mm)
0.13 ± 0.10
0.03 ± 0.01
0.02
63
0.07 ± 0.04
0.03
46
OL
VLC
p
Reduction (%)
Number of targets
10
10
RMSE (mm)
0.13 ± 0.08
0.03 ± 0.01
0.001
77
Average MAE (mm)
0.26 ± 0.15
0.08 ± 0.02
0.003
69
Fig. 9 Sample path following trials with CO2 laser over plaster (a and b) and chicken tissue (c and d) using OL control (a and c) and the VLC system (b and d). The red lines represent the desired trajectory for laser cutting
d) showed respective reductions of RSME and MAE of 77 and 69 % with the VLC system, as reported in Table 4. In addition to the improved system accuracy on 3D targets, these application trials also showed the proposed VLC system is robust against smoke resulting from the laser–target interaction, indicating the system has potential to be used in real clinical scenarios. The data collected during this experimental set has also been used to evaluate the relationship between the quality of the calibration procedure and the final path following accuracy obtained both in OL and with the VLC system. The results of this analysis are presented in Fig. 10 using correlation trend lines.
Discussion The experimental results relating to the evaluation of the system calibration accuracy, shown in Tables 1 and 2, demonstrate the open-loop system can be fairly precise, achieving overall errors of as little as 90 µm on planar targets
Fig. 10 Trend lines showing dependency between the quality of the calibration procedure and the corresponding path following accuracy using the OL and the VLC systems. This dependency is eliminated when the VLC system is used
and approximately 140 µm on real tissue. However, the data also shows large errors can be observed at specific points of the trajectory when using this control system. Maximum absolute errors recorded during the trials reached, on average, 230 µm on planar targets and 310 µm on real tissue, corroborating the calibration and OL control system issues described previously. The same Tables 1 and 2, present data that validate the proposed vision-based system, showing it is able to eliminate the residual laser aiming errors coming from the OL system, effectively reducing both the overall mean errors and the maximum absolute errors to values roughly equivalent to pixel resolution (see Fig. 7). These results highlight the importance of the VLC system in terms of laser aiming accuracy when the target features an unpredictable 3D topology, as in real surgical scenarios. In these cases, it is impossible to guaranteed precision or be as accurate as needed for the application of interest with a static mapping from image space to target space. This can only be achieved by closing the laser control loop using, for example, computer vision as demonstrated by the VLC system (Fig. 8). Data from the application trials with real laser cutting (Fig. 9) corroborate the comments above. In this case, the results
presented in Tables 3 and 4 show the developed system is applicable to real surgical scenarios, enabling fast and accurate laser scanning motions even in the presence of smoke created during the laser cutting process. Furthermore, the data also shows the VLC system can effectively eliminate errors cause by imperfect calibration. This is clearly seen in Fig. 10. The figure shows, as expected, high-dependency between system accuracy and calibration quality for the OL case. However, when the VLC system is used, this dependency is eliminated. This indicates the VLC system is robust against calibration errors, validating the use of a simplified calibration method as described.
Conclusion In this paper, a new vision-based system for fast and accurate laser scanning control during robot-assisted phonomicrosurgery was introduced and evaluated. The system concept centers on merging the benefits of open-loop laser control (e.g., high-speed scanning capabilities) with the robustness and accuracy of closed-loop visual servoing systems. High-speed scanning is important in this application to guarantee the quality of laser cuts, minimizing carbonization and thermal damage to surrounding tissue during the delicate surgeries. Accuracy is important to precisely follow the surgeon’s commands, avoiding undesired tissue damage due to robotic system errors. The solution presented consists of adding a vision-based trajectory correction phase to a fast open-loop laser control system. The method follows a two-phase process: Visual servoing is initially used to realize one precise pass of the laser over the desired trajectory, during this phase waypoints are registered. Subsequently, the laser is controlled in openloop using the identified waypoints, effectively generating the required fast and accurate scanning motions. The development of this control system concept and its implementation and evaluation were all described in details. Laser positioning and trajectory following experiments performed with artificial targets and real tissue samples demonstrated the system was successful in accomplishing its objectives, eliminating open-loop aiming errors caused by system calibration limitations and by the unpredictable topology of real tissue. The obtained experimental results also demonstrated the new system is highly accurate, being able to reduce laser aiming errors to values that roughly correspond to pixel resolution on the real-time videos of the operating area. When compared with open-loop control in a realistic condition, i.e., in trajectory following trials involving real CO2 laser cutting on chicken tissue, the use of the new controller resulted in a 77 % reduction in the measured mean aiming errors. This corresponds to a reduction in RMS errors from 130 to 30µm,
showing the VLC system provides robust and accurate results even in the presence of smoke create by laser–tissue interactions. These same trials also demonstrate the VLC system can realize fast laser scanning with minimal deviations from desired trajectories, confirming the achievement of desired system characteristics in a realistic surgical condition. In this case, the average values for maximum absolute errors were reduced by 69 % (from 260 to 60 µm), which leads to the conclusion that the new system is also safer for real surgical procedures. In the future, additional trials and evaluations shall include the testion of standard phonomicrosurgery procedures on exvivo larynxes (i.e. swine larynxes), leading to subsequent animal trials and later clinical trials. A “real-time” VLC system will be developed for dynamic control and compensation for tissue motion. Furthermore, as the new VLC-enhanced robot-assisted surgical system has the potential to be applied to other types of laser microsurgeries (e.g., in dermatology, ophthalmology), future experimentations focusing on other surgical procedures are also envisioned. Acknowledgments The research has received funding from the European Union Seventh Framework Programme FP7/2007-2013— Challenge 2—Cognitive Systems, Interaction, Robotics—under grant agreement μRALP—no. 288233. The authors would like to thank Giorgio Peretti, Luca Guastini and Francesco Mora, ENT surgeons from University of Genoa, for precious discussions and information on laser microsurgeries. Conflict of interest Giulio Dagnino, Leonardo S. Mattos, and Darwin G. Caldwell declare that they have no conflict of interest. Ethical standard An approval by an ethics committee was not applicable. Animal tissue used on the trials was purchased from a supermarket. Informed consent Statement of informed consent was not applicable since the manuscript does not contain any patient data.
References 1. Zeitels SM (2001) Atlas of phonomicrosurgery and other endolaryngeal procedures for benign and malignant disease. Singular, San Diego 2. Giallo JF (2008) A medical robotic system for laser phonomicrosurgery, PhD Dissertation, North Carolina State University, Raleigh, NC, USA 3. Remacle M, Eckel HE (2010) Surgery of larynx and trachea. Springer, Berlin 4. Bless DM, Abbs JH (1983) Vocal fold physiology: contemporary research and clinical issues. College-Hill Press, San Diego 5. Garrett CG, Ossoff RH (2000) Phonomicrosurgery II: surgical techniques. Otolaryngol Clin N Am 33(5):1063–1070 6. McWhorter A, Hoffman H (2005) Transoral laser microsurgery for laryngeal malignancies. Curr Probl Cancer 29(4):180–189 7. Oswal V, Remacle M (2002) Principles and practice in otorhinolaryngology and head and neck surgery. Kugler Publications, The Hague 8. Chawla S, Carney AS (2009) Organ preservation surgery for laryngeal cancer. Head Neck Oncol 1:12. doi:10.1186/1758-3284-1-12
9. Lanfranco AR, Castellanos AE, Desai JP, Meyer WC (2004) Robotic surgery: a current perspective. Ann Surg 239(1):14–21 10. Berlinger NT (2006) Robotic surgery—squeezing into tight places. N Engl J Med 354:2009–2101 11. Mattos L, Dellepiane M, Caldwell D (2011) Next-generation micromanipulator for computer-assisted laser phonomicrosurgery. In: Proceedings of the 2011 international conference of the IEEE engineering in medicine and biology society, EMBC 2011 12. Mattos L, Dagnino G, Becattini G, Dellepiane M, Caldwell D (2011) A virtual scalpel system for computer-assisted laser microsurgery. In: Proceedings of the IEEE international conference on intelligent robots and systems, IROS 2011 13. Dagnino G, Mattos LS, Becattini G, Dellepiane M, Caldwell DG (2011) Comparative evaluation of user interfaces for robot-assisted laser phonomicrosurgery. In: Proceedings of the 2011 international conference of the IEEE engineering in medicine and biology society, EMBC 2011 14. Mattos LS, Caldwell DG (2012) Safe teleoperation based on flexible intraoperative planning for robot-assisted laser microsurgery. In: Proceedings of the 34th annual international conference of the IEEE engineering in medicine and biology society, EMBC 2012, San Diego, CA, USA 15. Krupa A, Doignon C, de Mathelin MF, Morel G, Leroy J, Soler L, Marescaux J (2003) Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing. IEEE Trans Robot Autom 19(5):842–853 16. Hynes P, Dodds G, Wilkinson A (2006) Uncalibrated visualservoing of a dual-arm robot for MIS suturing. In: The first IEEE/RAS-EMBS international conference on biomedical robotics and biomechatronics 17. Becker BC, Voros S, MacLachlan RA, Hager GD, Riviere CN (2009) Active guidance of a handheld micromanipulator using visual servoing. In: Proceedings of the IEEE international conference on robotics and automation, ICRA 2009, Kobe, Japan
18. Knoll A, Staub C, Osa T, Bauernschmitt R (2010) Autonomous high precision positioning of surgical instruments in robot-assisted minimally invasive surgery under visual guidance. In: Proceedings of the sixth international conference on autonomic and autonomous systems, ICAS, Cancun, Mexico 19. Staub C, Osa T, Knoll A, Bauernschmitt R (2010) Automation of tissue piercing using circular needles and vision guidance for computer aided laparoscopic surgery. In: Proceedings of the IEEE international conference on robotics and automation, ICRA 2010, Anchorage, Alaska, USA 20. Chaumette F, Hutchinson S (2006) Visual servo control, part I: basic approaches. IEEE Robot Autom Mag 13(4):82–90 21. Chaumette F, Hutchinson S (2007) Visual servo control, part II: advanced approaches. IEEE Robot Autom Mag 14(1):109–118 22. Ellekidle L-P, Favrholdt P, Paulin M, Petersen HG (2007) Robust control for high-speed visual servoing applications. Int J Adv Robot Syst 4(3):279–292 23. Mattos L, Caldwell D (2009) A fast and precise micropipette positioning system based on continuous camera-robot recalibration and visual servoing. In: Proceedings of IEEE conference on automation science and engineering, CASE 2009 24. Rosenfeld A, Kak A (1982) Digital picture processing, vol 2. Academic, New York 25. Dagnino G, Mattos LS, Caldwell DG (2012) New software tools for enhanced precision in robot-assisted laser phonomicrosurgery. In: Proceedings of the 34th annual international conference of the IEEE engineering in medicine and biology society, San Diego, CA, USA 26. Phillips CL, Nagle HT (1995) Digital control system analysis and design, 3rd edn. Prentice Hall, Englewood Cliffs 27. Choi Y, Chung WK (2008) PID trajectory tracking control for mechanical systems. Springer, Berlin 28. Kirk RE (2006) Statistics: an introduction. Wadsworth Publishing Co Inc, Belmont
