I would like to thank Zivthan Dubrovsky and Andy Levine for their project management and ... Robotic systems offer several advantages over manual tools, including ...... velocity proportional-derivative (PD) controller implemented in Arduino based ...... 3D printing with injection molding or CNC machining, obviating manual ...
Modular Robotic Systems for Interventional Endoscopy A dissertation presented by
Joshua B. Gafford to
Harvard John A. Paulson School of Engineering and Applied Sciences
in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the subject of Engineering Sciences
Harvard University Cambridge, Massachusetts March 2018
c 2018 Joshua B. Gafford
All rights reserved.
Dissertation Advisors: Professor Conor J. Walsh Professor Robert J. Wood
Author: Joshua B. Gafford
Modular Robotic Systems for Interventional Endoscopy
Abstract Nascent endoscopic therapeutic procedures, such as endoscopic submucosal dissection, enable unparalleled access to and removal of mid-size cancerous neoplasia from within the gastrointestinal tract. However, the remote locations of these lesions often require substantial dexterity which imparts appreciable cognitive loading on the clinician and opens up the possibility of adverse events such as intestinal perforation due to limited dexterity and a lack of sensory feedback. In this work, we introduce EndoMODRA (Endoscopic Module for On-Demand Robotic Assistance) which interfaces with commercially-available endoscopic tools and provides additional dexterity and feedback sensing using on-board actuators and sensors. Leveraging energy-dense actuation strategies and monolithic, printed-circuit-inspired manufacturing processes, we develop mm-scale actuation and sensing modalities that are fully contained within the distal module, obviating the need for a continuous mechanical transmission to a proximal actuation source. Closed-loop, position-controlled trajectory execution is demonstrated using on-board actuation and sensing, realizing the first instance where fully-distal loop closure is achieved in an endoscope-mounted robotic module with no proximal actuation or sensing component. System robustness and efficacy is demonstrated through ex vivo and in vivo tests on appropriate analogs. This research lays the groundwork for a new class of endoscopic robot modalities that bridges the gap between simplistic, low-cost add-on devices and sophisticated, stand-alone robotic systems.
iii
Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv Introduction
1
1
. .
1 3 5
. . . . . . . . . .
6 7 9 10 11 15 15 17 18 19
. . . . . . . . . . . .
22 22 23 25 26 27 28 32 36 38 39 41 42
2
Introduction 1.1 Gastrointestinal Endoscopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Frontiers of Endoscopy and Persisting Challenges . . . . . . . . . . . . . . . 1.1.2 The Rising Role of Advanced Instrumentation and Robotics in Interventional Endoscopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 State-of-the-Art in Advanced Endoscopic Devices and Systems . . . . . . . . . . . 1.2.1 Category 1: Mechanical Endoscopic Add-On Devices . . . . . . . . . . . . . 1.2.2 Category 2: Mechanical Multi-Articular Systems . . . . . . . . . . . . . . . 1.2.3 Category 3: Robotic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Other Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Opportunity for Innovation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Broader Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Clinical Parameterization 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Endoscopic Submucosal Dissection Overview . . . . 2.2.1 Endoscope Overview . . . . . . . . . . . . . . . 2.3 Ex Vivo ESD Workspace Analysis . . . . . . . . . . . . 2.3.1 Motion Tracking . . . . . . . . . . . . . . . . . 2.3.2 Data Post-Processing . . . . . . . . . . . . . . . 2.3.3 Results Analysis . . . . . . . . . . . . . . . . . 2.3.4 Motion Study Conclusions . . . . . . . . . . . 2.4 Endoscope/Tool Stiffness and Contact Force Analysis 2.4.1 Tool and Endoscope Stiffness . . . . . . . . . . 2.4.2 Electrosurgical Ablation Force . . . . . . . . . 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . iv
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
3
4
5
Distally-Integrated Actuation 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Actuator Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Actuator Performance Index . . . . . . . . . . . . . . . . . 3.2.2 Candidate Distal Actuator Modalities . . . . . . . . . . . . 3.3 SMA Modeling and Fabrication . . . . . . . . . . . . . . . . . . . . 3.3.1 A Brief Primer on Shape Memory Alloys . . . . . . . . . . 3.3.2 SMA Usage in Surgical Robotics . . . . . . . . . . . . . . . 3.3.3 Constitutive Modeling . . . . . . . . . . . . . . . . . . . . . 3.3.4 Fabrication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Experimental Validation . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Fixed Strain, Controlled Current (Thermodynamics) . . . 3.4.3 Variable Strain, Constant Temperature (Force vs. Stroke) . 3.4.4 Variable Strain, Variable Thermodynamics (Antagonism) 3.4.5 Forced Convection Cooling . . . . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
Distally-Integrated Sensing 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Sensor Manufacturing At Scale . . . . . . . . . . . . . . . . . . . . . 4.2.1 A Brief Introduction to Printed-Circuit MEMS . . . . . . . . 4.2.2 Using PCMEMS for Sensor Fabrication . . . . . . . . . . . . 4.2.3 Review of PCMEMS-Compatible Transduction Mechanisms 4.3 Endoscopic Multisensor Linkage Design and Modeling . . . . . . . 4.3.1 Linkage Kinematics Model . . . . . . . . . . . . . . . . . . . 4.3.2 Hall Effect Modeling . . . . . . . . . . . . . . . . . . . . . . . 4.4 A Force Sensor for Distal Force Feedback . . . . . . . . . . . . . . . 4.4.1 Optoelectronics Modeling . . . . . . . . . . . . . . . . . . . . 4.5 Multisensor Fabrication and Calibration . . . . . . . . . . . . . . . . 4.5.1 PCMEMS Fabrication . . . . . . . . . . . . . . . . . . . . . . 4.5.2 Electrical Implementation . . . . . . . . . . . . . . . . . . . . 4.5.3 Fold Pattern-Guided Assembly and Encapsulation . . . . . 4.5.4 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.5 Multisensor Demo Module . . . . . . . . . . . . . . . . . . . 4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
44 44 46 46 47 51 51 53 53 60 61 62 64 68 69 71 74
. . . . . . . . . . . . . . . . .
76 76 77 79 80 81 88 89 91 95 97 98 98 99 100 102 106 107
System Integration 108 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5.2 Distal Module Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
v
5.3 5.4 5.5
5.6
5.7 6
7
5.2.1 Mechanical Analysis . . . . . . . . . . . . . 5.2.2 Manufacturing and Assembly . . . . . . . Human Factors Design of Master Input Device . . Embedded System Overview . . . . . . . . . . . . 5.4.1 Controller Enclosure . . . . . . . . . . . . . Hydraulic System Design . . . . . . . . . . . . . . 5.5.1 Power Electronics . . . . . . . . . . . . . . . 5.5.2 Fluid Dynamics Analysis . . . . . . . . . . SMA Power Electronics and Control . . . . . . . . 5.6.1 Theoretical Justification for PWM Control 5.6.2 Power Electronics . . . . . . . . . . . . . . . 5.6.3 Performance Benchmarking . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . .
Closed-Loop Position Control 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . 6.2 PID/PWM SMA Control . . . . . . . . . . . . . . 6.3 Fuzzy PID-PWM Control with Gain Scheduling 6.3.1 Rule-Based Valve Control . . . . . . . . . 6.3.2 Fuzzification . . . . . . . . . . . . . . . . . 6.3.3 Defuzzification . . . . . . . . . . . . . . . 6.3.4 Gain Scheduling . . . . . . . . . . . . . . 6.3.5 Implementation . . . . . . . . . . . . . . . 6.3.6 Benchtop Results . . . . . . . . . . . . . . 6.4 Integrated Performance . . . . . . . . . . . . . . 6.4.1 Automated Trajectory Execution . . . . . 6.4.2 Teleoperation . . . . . . . . . . . . . . . . 6.4.3 Effect of Tool on Control Performance . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . Ex-Vivo and In-Vivo Testing 7.1 Introduction . . . . . . . . . . . . . . . . . . . . 7.2 Ex-Vivo Testing . . . . . . . . . . . . . . . . . . 7.2.1 Open Loop Control/Sensor Robustness 7.2.2 Closed-Loop Control/Feasibility Test . 7.3 In Vivo Tests . . . . . . . . . . . . . . . . . . . . 7.3.1 Animal Preparation . . . . . . . . . . . 7.3.2 Robustness Study (Open) . . . . . . . . 7.3.3 Feasibility Study (Closed) . . . . . . . . 7.4 Conclusion . . . . . . . . . . . . . . . . . . . . .
vi
. . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . .
111 112 113 115 117 118 119 119 121 122 123 127 131
. . . . . . . . . . . . . .
133 133 134 135 136 137 138 141 141 141 143 143 146 147 149
. . . . . . . . .
150 150 150 150 152 156 156 157 158 159
8
Conclusion 8.1 Insights and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Technical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2 Clinical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
160 161 162 166
References
170
Appendix A Custom Evaluation System Design A.1 Introduction . . . . . . . . . . . . . . . . . A.2 System Overview . . . . . . . . . . . . . . A.3 Mechanical Subsystem . . . . . . . . . . . A.3.1 Primary (Motor-Side) Drive . . . . A.3.2 Secondary (Load-Side) Drive . . . A.4 Electrical and Software . . . . . . . . . . . A.4.1 Software and Controls . . . . . . . A.5 System Performance Benchmarks . . . . . A.5.1 Force and Position Control . . . . A.5.2 Temperature and Current Control A.6 Soft Sensor Characterization . . . . . . . . A.7 Dynamic Calculations . . . . . . . . . . . A.7.1 Dynamic Bandwidth . . . . . . . . A.7.2 Oscillation Amplitude . . . . . . .
180 180 180 181 182 183 185 186 193 193 194 196 197 197 197
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Appendix B Disturbance Rejection via Analog Filtering and Machine Learning 200 B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 B.2 Rejection through Analog Signal Conditioning . . . . . . . . . . . . . . . . . . . . . . 201 B.3 Rejection through Machine-Learning-Based Regression . . . . . . . . . . . . . . . . . 204 B.3.1 Background of Machine Learning Techniques in Sensor Disturbance Rejection204 B.3.2 Experimental Multisensor Design . . . . . . . . . . . . . . . . . . . . . . . . . 205 B.3.3 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 B.3.4 A Modified Model For Kernel Methods . . . . . . . . . . . . . . . . . . . . . . 212 B.3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 B.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Appendix C PCMEMS Scalability, Robustness, and Biocompatibility C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.2 Process Economy . . . . . . . . . . . . . . . . . . . . . . . . . . . C.2.1 Applicability to Medical Devices . . . . . . . . . . . . . . C.2.2 Note on Sterilization and Usage . . . . . . . . . . . . . . C.3 PCMEMS Mechanism Failure Analysis . . . . . . . . . . . . . . C.3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . vii
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
222 222 222 225 225 226 226
C.3.2 Robustness Analysis . . . . . . C.3.3 Robustness Discussion . . . . . C.4 Persisting Challenges and Limitations C.4.1 Biocompatibility: . . . . . . . . C.4.2 Continuous Motion . . . . . . . C.4.3 Soft Encapsulation . . . . . . .
. . . . . .
Appendix D Control Implementation Details D.1 Introduction . . . . . . . . . . . . . . . . D.2 Fuzzy Rule Bases . . . . . . . . . . . . . D.3 Software Implementation . . . . . . . . D.3.1 Open-Loop Control . . . . . . .
. . . . . .
. . . .
viii
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
. . . .
. . . . . .
227 230 231 231 232 233
. . . .
234 234 234 235 237
List of Tables
1.1
Tabulated specifications of advanced endoscope-based mechanical and robotic devices. Note that we only consider degrees of freedom in addition to conventional endoscope degrees of freedom (axial translation, tip pitch, tip yaw, roll). Data compiled from (Patel et al., 2014; Yeung and Gourlay, 2012; Yeung and Chiu, 2016; Vitiello et al., 2013). (*: Data Not Available, § : Excluding insertable camera) . . . . . 14
2.1
Experimentally-derived functional requirements for an endoscope-mounted robotic system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1
Experimental protocol summary, where model colors correspond to the active controllers in Fig. 3.8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.1
Proprioceptive sensor specifications using linear regression with quadratic features where the green fill denotes satisfaction of functional requirements. (Note: resolution is peak-to-peak, hysteresis and linearity are worst-case) . . . . . . . . . . 105 Force sensor specifications using linear regression with linear features where the green fill denotes satisfaction of functional requirements. (Note: resolution is peak-to-peak) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.2
5.1 5.2 5.3 6.1
Approximate cost breakdown of single-use robotic module. (∗ : based on the average hourly wage of a medical device assembly technician) . . . . . . . . . . . . . . . . . 113 Embedded system specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 SMA power electronics specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Logical rule base for valve control (where (+) indicates the agonist actuator cooling valve is open, (-) indicates that the antagonist actuator cooling valve is open, and 0 indicates that both valves are closed). . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
A.1 Tabulated constraint analysis of secondary drive. . . . . . . . . . . . . . . . . . . . . 183 A.2 Comparison of motor requirement based on system desired specifications with manufacturer-specified performance metrics (Note: (*) denotes transmission-dependent specifications). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 A.3 Evaluation system specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
ix
B.1 Algorithm performance summary (Note: (*) indicates approximate full data run time based on downsampled data run time) . . . . . . . . . . . . . . . . . . . . . . . 218 C.1 PCMEMS material biocompatibility (actuator subsystems consider net biocompatibility of all constituent components) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 D.1 Fuzzy inference rules for agonist actuator (output sets are for K¯ p− , K¯ d− , and K¯ i− , respectively) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 D.2 Fuzzy inference rules for antagonist actuator (output sets are for K¯ p+ , K¯ d+ , and K¯ i+ , respectively). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
x
List of Figures
1.1
A history of robotic surgery in four pictures: (from left) Puma 560 used to place a brain biopsy (circa 1985), ROBODOCTM system for computer-guided hip replacement surgery (circa 1992), AESOPTM system (circa 1993), daVinciTM system (current). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 A history of endoscopy in three pictures: (left) Adolf Kussmaul’s ‘sword-swallowing’ technique for visualizing the stomach through a rigid metal tube (circa 1868), (middle) Basil Hirschowitz testing his flexible endoscope on an enthusiastic graduate student (circa 1960), (right) Olympus GIF-2TH180 endoscope (circa 2010). . . . . . . 1.3 Number of instances in which the following set of keywords show up in the title or abstract of articles archived in PubMed per year: (a) {Endoscopic Submucosal Dissection, Endoscopic Mucosal Resection, Peroral Endoscopic Myotomy}, (b) {Endoscopic Device, Robotic Resection, Endoscope Robot}. The dotted lines are extrapolations based on the data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 State-of-the-art for advanced endoscopy: (1) EndocuffTM (Arc Medical Design, Leeds, UK), (2) EndoringsTM (Aquilant Endoscopy, Hampshire, UK), (3) ThirdEyeTM panoramic camera module (Avantis Medical Systems, Inc., San Jose, CA), (4) OverstitchTM endoscopic suturing device (Apollo Endosurgery, Inc., San Diego, CA), (5) OTSCTM (Over-the-Scope Clipping) system (Ovesco Endoscopy AG, Tubingen, Germany), (6) CaptivatorTM for ligation-assisted EMR (Boston Scientific, Boston, MA), (7) AnubiscopeTM (Karl Storz, Tuttlingen, Germany), (8) CobraTM system (USGI Medical, San Clemente, CA), (9) EndoSamuraiTM (Olympus, Tokyo, Japan), (10) DDES (Direct-Drive Endoscopic System) (Boston Scientific, Boston, MA), (11) Incisionless Operating PlatformTM (USGI Medical, San Clemente, CA), (12) RScopeTM (Olympus, Japan), (13) IREP (Vanderbilt University, Knoxville, TN), (14) i-Snake (Imperial College, London, UK). (15) CYCLOPS (Imperial College, London, UK), (16) STIFF-Flop (EU Consortium) (17) RS-ALC (University of Twente, The Netherlands) (18) EOR (KIT, Japan) (19) EndoticsTM system (Endotics, Peccioli, Italy), (20) InvendoTM system (Invendo Medical, Kissing, Germany), (21) NeoGuideTM (Intuitive Surgical, Sunnyvale, CA), (22) MASTER (Endomaster, Singapore), (23) ViaCath (Hansen Medical, Mountain View, CA), (24) FlexTM system (Medrobotics, Raynham, MA). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xi
3
4
6
8
1.5 1.6
2.1
2.2 2.3
2.4 2.5 2.6 2.7 2.8
2.9
2.10
2.11
EndoMODRA: distally-mounted robotic add-on module for interventional endoscopy: (a) at-scale prototype, (b) view through endoscope and controller data. . . 16 Cooperative control methodology overview, where the endoscopist works in tandem with the distally-mounted robot (via hardware and software-based input devices) to articulate electrosurgical tools during endoscopic submucosal dissection. A ruggedized real-time control/drive unit consolidates control loop execution, data acquisition, power electronics and signal conditioning into a mobile suite-deployable system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Overview of endoscopic submucosal dissection (ESD) (from (Ryu and Chen, 2011)): (A) markings with electrosurgical knife around the edge of the lesion to establish a margin, (B) submucosal injection around the edge of the lesion, (C) incision with a needle-knife or hook-knife, (D) margin cutting with insulated tip (ITKnife) at the incision hole, (E) circumferential margin cutting, and (F) submucosal dissection with ITKnife beneath the muscularis mucosa. . . . . . . . . . . . . . . . . . . . . . . Overview of a standard endoscope: (a) technical terminology, (b) functional schematic of distal angulation (From (Waye et al., 2004)). . . . . . . . . . . . . . . . . Experimental procedure to quantify workspace and dexterity requirements in ESD: (left) image of the experimental setup for capturing motion data during ex-vivo ESD performed on an excised porcine stomach, (right) view through endoscope camera during the circumferential incision process. . . . . . . . . . . . . . . . . . . . . . . . . Illustration of successive rotations with respect to the current frame, mathematically described by Rxyz (φ, θ, ψ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Schematic of the experimental setup, showing relevant inertial and local frames and their respective orientations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pose data both pre- and post-processing for endoscope and tumor. . . . . . . . . . . Example workspace reconstruction during an incision task, wherein convex hulls are fitted to the trajectory data for a conservative workspace estimate. . . . . . . . . Workspace reconstructions from three distinct phases of ESD: (left) saline injection, (middle) axial incision using Olympus DualKnife forward-cutting monopolar cautery, and (right) circumferential dissection using Olympus ITKnife side-cutting monopolar cautery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example workspace reconstruction during a retroflexion event where the endoscope is bent back over 180 degrees on itself to access the distal side of the lesion. The numbers represent the temporal sequence of positions/orientations during this event. ESD motion analysis: (a) angular dexterity including retroflective event, (b) angular dexterity excluding retroflective event, (c) linear motion including retroflective event, and (d) task complexity including retroflective event. . . . . . . . . . . . . . . Relevant distal forces: (a) force (or torque) required to deflect flexible electrosurgical tools laterally, (b) contact force required to electrosurgically dissect tissue. . . . . . .
xii
24 25
27 30 31 34 35
36
37
38 39
2.12 Block diagram of cantilevered loading tests used to characterize flexural properties of demonstrative endoscope and endoscopy tools. . . . . . . . . . . . . . . . . . . . . 2.13 Stiffness analysis of flexible tools clamped 10mm proximal to distal end and subject to a cantilevered load at the distal end, where error bars indicate 1σ for n = 5: (a) Olympus DualKnife forward-cutting monopolar electrosurgical tool, (b) Olympus ITKnife side-cutting monopolar electrosurgical tool. . . . . . . . . . . . . . . . . . . . 2.14 Stiffness analysis of Olympus CF-100L endoscope, where shaded error bars indicate 1σ for n = 5: (a) Force vs. Displacement curves at various cantilevered distances l, (b) Stiffness vs. cantilever distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.15 Electrosurgical force analysis: (a) rendering of experimental setup where inset shows a close-up photograph of the tool-tissue interface, (b) Force vs. Displacement curves for different tissue speciments (Full-Thickness, Mucosa only, Muscularis only) where ‘x’ denotes electrosurgical pulses. . . . . . . . . . . . . . . . . . . . . . . 3.1
3.2 3.3 3.4 3.5
3.6
3.7 3.8
Effective actuator stress vs. characteristic stroke for various actuator modalities (where dotted lines denote isocontours of constant stroke work per unit volume). The red window highlights the modalities that satisfy our performance index for force and stroke. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qualitative thermomechanical loading diagram for a Nitinol-based shape memory alloy actuator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Schematic of a helical actuator and relevant geometric parameters. . . . . . . . . . . Schematic of antagonistic actuation and force balance. . . . . . . . . . . . . . . . . . (a) Comparison of cosine model with sigmoidal approximation for nonlinear optimization, (b) example results comparing nonlinear optimization results with the results obtained by running the discrete model forward in time from δ = 0 to δ = δ0 for 300 points, showing convergence in 26 iterations, and (c) comparison between optimized x after 26 iterations and solutions for ξ S and τ after running the discretized model forward from δ = 0 to δ = δ0 , where the x-axis plots the level of discretization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (a) Setup for winding and shape-setting helical SMA actuators, (b) actuator in different stages of fabrication process, where (left) shows the constrained actuator prior to annealing, and (right) shows the resulting actuator post-annealing. . . . . . Block diagram of real-time experimental platform for thermomechanical characterization of helical SMA actuators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Schematic of different control modes for thermomechanical characterization. Colored boxes correspond to controllers that are active during similarly-colored experimental protocols listed in Table 3.1 (i.e. the antagonism protocol requires both current and virtual spring controllers to be active simultaneously). . . . . . . . . . .
xiii
40
40
41
42
47 52 54 57
60
61 62
63
3.9
3.10
3.11
3.12
3.13
3.14
4.1
Thermodynamic characterization with fixed strain: (top) experimental and theoretical temperature plotted vs. applied current and time (error bars correspond to 1σ for n = 3, and the plane denotes the transition region), (bottom left) phase transition time vs. applied current, and (bottom right) steady-state temperature vs. applied current. Steady-state actuator temperature scales linearly with applied current as the analytical thermodynamic model suggests. . . . . . . . . . . . . . . . Thermomechanical characterization with fixed (100%) strain: (top) experimental and theoretical force plotted vs. applied current and time (error bars correspond to 1σ for n = 3, (bottom left) 50% rise time vs. applied current, and (bottom right) maximum blocking force vs. applied current. . . . . . . . . . . . . . . . . . . . . . . . Thermomechanical characterization with variable strain at constant temperature: (top) experimental and theoretical force plotted vs. temperature and strain (error bars correspond to 1σ for n = 3, (bottom left) residual strain (pseudoplastic deformation) vs. actuator temperature, and (bottom right) maximum force vs. actuator temperature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thermodynamic characterization with antagonistic strain dynamics at variable current: (top) experimental and theoretical displacement plotted vs. current (error bars correspond to 1σ for n = 3, (bottom left) 50% rise time vs. actuator current, and (bottom right) maximum stroke vs. actuator current. . . . . . . . . . . . . . . . . The effect of different cooling modalities (free convection in air, forced convection in air, forced convection in water) on actuator performance, where visual and infrared images show the actuator at four different states when pulling a 5N load, and the plots show representative actuator blocking force behavior. . . . . . . . . . . . . . . Performance summary: (a) experimental and theoretical force plotted vs. strain for both fully martensitic and fully austenitic states (shaded error bars correspond to 1σ for n = 3) indicating a maximum net force of 5N and stroke of 7mm, (b) blocking force vs. time for different cooling modalities. We observe nearly two orders-of-magnitude improvement in cooling time when using water as a force convection medium over free convection in air. . . . . . . . . . . . . . . . . . . . . . .
66
67
70
72
73
74
Printed-circuit MEMS fabrication of a single-degree-of-freedom flexural joint: (1) micromachining of single layers of material in a diode-pumped solid-state laser, (2) pin-aligned lamination via heat and pressure (ST=Steel, AD=FR0100 Acrylic Adhesive, KA=Kapton polyimide), (3) micromachining of composite laminate in DPSS laser to release part from alignment scaffold, (4) out-of-plane assembly wherein flexural material provides joint compliance while castellated features prohibit torsional bending. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
xiv
4.2
4.3
4.4
4.5 4.6
4.7
Different force and position transduction mechanisms compatible with printedcircuit MEMS fabrication: (a) metal-foil strain-gage-based transduction where the serpentine-style gage is machine from a high-resistivity material such as Constantan or Karma alloy, (b) light-intensity modulation where PCMEMS linkage kinematics or flexural mechanics dictate the transfer function between input force/position and output signal, (c) capacitance-based sensing where the dielectric material properties dictate the transfer function between input force/position and output signal, and (d) Hall effect sensing where the relative motion of a Hall element in the vicinity of a permanent magnet can be exploited to sense large articulations. . . . . . . . . . . Demonstrative PCMEMS mm-scale sensor implementations: (a) grip force-sensing micrograsper with integrated foil strain gage-based half-bridge (Gafford et al., 2016c), (b) three-axis force sensor with eight foil-based strain gages and a Maltese cross morphology (Gafford et al., 2014), (c) three-axis light-intensity modulationbased force sensor with four emitter-detector pairs and on-board temperature and environmental irradiance sensing for disturbance rejection (Gafford et al., 2016a), (d) capacitance-based force/displacement sensor with an integrated hydraulic actuator (Russo et al., 2016), (e) single-axis light-intensity modulation-based force sensor that ‘self-assembles’ via a pre-stretched spring (Gafford et al., 2016d), (f) early prototype of an SMA-actuated flexure-based articulating scaffold with four emitter-detector pairs for light-intensity-modulation-based proprioceptive sensing (Gafford et al., 2016b). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kinematic representation of proposed sensor linkage for proprioceptive angle sensing, where the detailed view (right) shows a qualitative representation of the magnetostatics finite element model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Representative output from finite element modeling of the flux density field around a Neodymium permanent magnet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FEM modeling of Hall effect-based proprioceptive sensor, where (top) shows the FEM simulation and kinematics, (middle) shows the component of the flux density field normal to the Hall effect sensor plane over the linkage stroke, and (bottom) is the resulting regression estimate for linear and quadratic feature spaces: (a) sensor placement results in non-monotonic field sensitivity over linkage stroke, (b) sensor placement results in monotonic, high sensitivity and nonlinear behavior over linkage stroke, and (c) sensor placement results in monotonic, low-sensitivity and more linear behavior over linkage stroke. . . . . . . . . . . . . . . . . . . . . . . . . . Hall element optimization surfaces (where the black line denotes the optimized configuration): (left) normal component of the magnetic field to the Hall element over entire stroke, (right) magnetic field misalignment φ over entire stroke. . . . . .
xv
81
83
89 92
93
94
4.8
4.9
4.10
4.11
4.12
4.13
4.14
4.15
Combined mechanical and optoelectronic model of differential light-intensity modulated force sensing methodology, where (left) shows the mechanical schematic and (right) shows qualitative irradiance and sensitivity dependencies on displacement and misalignment between emitter-detector pairs. . . . . . . . . . . . . . . . . . . . . Experimental setup to determine far-field behavior of infrared emitter/detector pairs: (a) experimental hardware where inset shows the array of opposing emitter/detector pairs, (b) collector current i PT vs. distance where the dotted black line shows the theoretical far-field cutoff. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Printed-circuit MEMS fabrication of the at-scale sensor linkage: (1) lamination of individually laser-machined material layers, (2) release cuts to free sensor linkage from alignment scaffold, (3) pick-and-placement of electronic components for reflow soldering, (4) tab-and-slot guided manual assembly. . . . . . . . . . . . . . . . . . . . Fabricated sensor: (a) combined proprioceptive (differential Hall effect), force (differential light-intensity modulation), and temperature (COTS IC) multisensor linkage, after component placement but prior to assembly (designed for implementation on a Fuji 450WR5 9.4mm diameter endoscope), (b) electrical schematic of on-board components, where qualitative plots show single-ended and differential sensing modalities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Origami-inspired assembly of force sensor employing differential light-intensity modulation, where insets show fold sequence: (a) shows E/D side 1 in the initial (flat) state, (b) shows E/D side 2 in the initial flat state, (c) shows a valley fold to bring E/D plane 1 perpendicular to that of the flexural beam, and mountain folds to assemble the enclosure, (d) shows a valley fold to bring E/D plane 2 perpendicular to that of the flexural beam, and (e) shows the linkage assembly, where linkage kinematics and tab/slot features guide E/D sides 1 and 2 into their final opposing configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (a) Test setup for calibrating to-scale PCMEMS angle sensor, where insets show the configuration at the extremes of travel, (b) snapshot of a calibration routine where a model is fit to the data using recursive least-squares estimation. . . . . . . . . . . . . Dynamic, cyclic and transient characterization data for the proprioceptive sensor: (a) sensor response to sudden input stimuli (where the FEM-predicted performance is shown for comparison), (b) snapshot of cyclic data around 200 cycles showing a good sensor estimate, (c) per-cycle RMSE compared to batch RMSE demonstrating negligible signal degradation after 200 cycles, (d) angle drift as a function of temperature (data and fit), (e) long-term drift (data and fit), and (f) power spectral density for a batch of four sensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Combined multisensor calibration results: (left, top to bottom) regression results [θ, θˆ(Φ HE )] vs. time, angle regression error vs. time, and angular regression linearity plot, (right, top to bottom) regression results [ F, Fˆ (XED )] vs. time, force regression error vs. time, and force regression linearity plot. . . . . . . . . . . . . . . xvi
96
97
99
100
101
103
104
106
4.16 Multisensor demonstration module where plots show real-time on-board sensor readings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.1
Illustrative overview of cooperative control scheme, highlighting systems-level hardware and software components and their relation to the distal module and the endoscopist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Principle of operation: (middle) shows the design of the distal module, highlighting the antagonistic actuation scheme and coupling to the proprioceptive sensor through the transparent deflector plate (a U.S. Penny provides scale), (left) shows the summarized experimental proprioceptive sensor characterization data from Chapter 4, and (right) shows the experimental antagonistic actuation data from Chapter 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Module attachment to endoscope: (left) the module is slid onto the distal end and the feeder channel of the deflection plate is aligned with the working channel of the endoscope, (middle) set screws can be (optionally) tightened for a more secure fit, and (right) the tool can be deployed into the deflection plate. . . . . . . . . . . . 5.4 Module articulation modeling: (a) moment balance about central pin joint with relevant geometric and mechanical variables, (b) overall angle of articulation measured with an IMU (left) and theoretical maximum lateral force generation measured over the reachable workspace (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Ergonomic master input device design process showing morphological and feature evolution as guided by clinical consultation and ex vivo testing. . . . . . . . . . . . . 5.6 Images showing integration and use of robotic module controller, where (left) shows conventional endoscope control, and (right) shows robot-assisted control. . . . . . . 5.7 Custom embedded system and power electronics design: (a) photograph of the embedded system enclosed in a custom electronics enclosure, (b) block diagram showing the system’s components, and (c) a photograph of the embedded system with the top shell removed, showing the internal components. . . . . . . . . . . . . . 5.8 Hydraulic system for providing forced water cooling to distal module: (a) hydraulic schematic, (b) electrical schematic (shown here providing control inputs to two hydraulic solenoid valves and one 12VDC submersible pump), and (c) hydraulics control relay board integrated into master controller. . . . . . . . . . . . . . . . . . . 5.9 Electrical schematic of dual-channel SMA power and signal conditioning electronics, where colored boxes denote different stages of the power electronics methodology. 5.10 Image showing the custom power electronics PCB and analog-to-PWM microcontroller board and their respective connections to auxiliary hardware. . . . . . . . . .
xvii
109
110
110
111 115 115
117
120 124 126
5.11 Two generations of SMA power electronics and custom laser-machined acrylic enclosures: (a) Generation 1 (low-power) with maximum 3.5A output simultaneously, PWM-only drive mode and no on-board filtering, (b) Generation 2 (high-power) with maximum 6A output simultaneously, PWM and analog drive mode, and on-board filtering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.12 SMA open-loop driver performance in analog input mode, where (top) is the analog input and (bottom) is the resulting current sensor output in open-loop control mode with a 10-Ω resistor as a dummy load: (bottom from left) unfiltered sensor readings, filtered via on-board active low-pass filters (manually tuned), filtered via software-based 60 Hz low-pass filters, filtered via both on-board and software filters.129 5.13 Closed-loop current control system identification: (a) chirp signal with peaks denoted, (b) experimental bode plot with a 2nd-order model fit, indicating a bandwidth of roughly 20 Hz. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 5.14 Closed-loop current control step response: (a) constant-load (10Ω power resistor) response for a 1A setpoint, (b) variable-load (SMA actuator) response for setpoints ranging from 1A to 5A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5.15 Thermal performance of combined Joule heating and hydraulic cooling, where inset images show visual and thermal infrared images of the module (right) corresponding to enumerated sections of the low-level controller data (left). . . . . . . . . . . . 131 6.1
6.2 6.3 6.4
6.5
6.6
Fuzzy-tuned PID-PWM control of a distally-mounted robotic module for endoscopy: (a) high-level controller overview, wherein reference position commands are sent to the controller either through hardware (teleoperation) or software (automation), (b) detail of fuzzy inference engine used to generate tuned gains Kθ , (c) high-level position PID controller, (d) low-level current PID-PWM controller. . . . . . . . . . . Triangular membership functions for fuzzification: (a) input variables e(t) and e˙(t) (for the agonist actuator), (b) output variables K p , Kd and Ki . . . . . . . . . . . . . . . Fuzzy control surfaces for agonist and antagonist PID gains: (from left) K p vs. e(t) and de(t)/dt, Kd vs. e(t) and de(t)/dt, and Ki vs. e(t) and de(t)/dt. . . . . . . . . . Simulink-implemented control architecture for FPID/PWM position control of antagonist SMA actuators with gain scheduling and a crisp valve control rule-base for forced convection cooling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Closed-Loop Controller Design: (a) experimental platform for off-board closed-loop controller design and verification, (b) GUI for tuning controllers and executing trajectories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bencthtop results (actuator-only module with reference sensor) for a FPID/PWM controller (shaded error bars show 1σ for n = 3 trials), where top shows desired vs. actual angle θ and bottom shows the angle error: (a) step response showing convergence in around 0.4 seconds, (b) stairstep pattern, (c) triangular profile with a 30 second period, and (d) 0.125 Hz sine wave. . . . . . . . . . . . . . . . . . . . . .
xviii
135 138 139
140
142
143
6.7
Comparison of the FPID/PWM controller and a traditional PID/PWM when tracking a fast (0.5 Hz) sinusoidal reference: (counter-clockwise from top left) reference and actual trajectories vs. time, angular error vs. time, and error integrated over time. . 6.8 Fully integrated distal module with on-board actuation, sensing, and closed-loop position control capabilities: (left) image of to-scale prototype with callouts to important features, (right) view through endoscope in different configurations. . . . 6.9 Automated trajectory execution of a FPID/PWM controller implemented on a fullyintegrated distal module with on-board sensing and actuation, where (a) shows a triangular (constant velocity) profile and (b) shows a sinusoidal profile (shaded error region denotes 1σ for n = 3): (from top) reference vs. actual trajectory, angular error, reference vs. actual SMA current, fluid valve states, agonist PID gains, and antagonist PID gains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10 System snapshot during closed-loop, rate-based teleoperation: (top left) embedded system, host computer, and master input device mounted on endoscope handle, (bottom left) desired vs. actual deflection angle and associated error, (right) close-up of distal module deflecting an Olympus ITKnife. . . . . . . . . . . . . . . . . . . . . 6.11 Teleoperation of a FPID/PWM controller implemented on a fully-integrated distal module with on-board sensing and actuation, where (a) shows a position control paradigm and (b) shows a rate control paradigm: (from top) reference vs. actual trajectory, angular error, reference vs. actual SMA current, fluid valve states, agonist PID gains, and antagonist PID gains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.12 Performance comparison with and without the tool in place: (left) controller response to step inputs varying from θd = 5 to θd = 45, (middle) 10%-90% rise time vs. setpoint angle θd , (right) steady-state error vs. setpoint angle θd . . . . . . . . . . 7.1 7.2
7.3
7.4
Schematic of open-loop controller employing the deterministic current and fluid logic profiles detailed in Appendix D, Fig. D.3. . . . . . . . . . . . . . . . . . . . . . . Ex Vivo testing of open-loop control and sensor/actuator robustness to clinical conditions: (a) experimental setup with an early version of the control electronics and the first version of the master input device, (b) image of sensorized distal module used during tests, (c) controller and sensor data during open-loop deflection of Olympus DualKnife. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Experimental setup for full ex-vivo ESD simulation to test closed-loop control capabilities, where inset shows a close-up view of the distal module affixed to the endoscope. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Controller data during robot-assisted ex-vivo ESD where images show endoscope view during the procedure: (a) open-loop teleoperation with an Olympus ITKnife, (b) closed-loop velocity-based teleoperation using Olympus DualKnife, (c) closedloop velocity-based teleoperation using Olympus ITKnife. . . . . . . . . . . . . . . .
xix
144
144
145
146
148
149
151
152
154
154
7.5 7.6 7.7
7.8
8.1
8.2
8.3
Results of ex vivo test: (left) lesion removed en bloc, (right) location from which the lesion originated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Procedure comparison between robot-assisted ESD and conventional ESD, where bars represent the task time normalized by the dissection lesion area. . . . . . . . . In vivo open ESD performed on anesthetized porcine model: (a) image of the experimental setup, (b) image of module performing lateral dissection, (c) image of stomach after successful dissection, (d) image of dissected lesion, and (e) system data during dissection where green boxes indicate approximate timing and duration of tissue contact and electrosurgery (qualitatively observed from videos of the procedure). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . In vivo closed ESD performed on anesthetized porcine model: (a) image of the experimental setup, (b) close-up of the view through the endoscopic camera, (c) image of dissected lesion where the margins are highlighted, and (d) exemplary system data during robot-assisted dissection via rate-based teleoperation. . . . . . .
155 155
157
158
Proposed visualization improvement: (left) schematic representation of visual occlusion for current design, (middle) visual field improvements due to singlejoint design, (right) unobstructed view (based on the data sheet for Fuji 600 series endoscopes) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Related work in tissue retraction using PCMEMS fabrication techniques: (a) a 3-DoF multi-articular arm based on pneumatic actuation and suction-based gripping (from (Russo et al., 2017)), and (b) a deployable retraction mechanism, where (1) shows the device encased in 23.81 mm outer diameter overtube (2) shows the device during deployment and overtube retraction, (3) shows the view from endoscope distal camera, with an endoscopic tool in the foreground, and (4) shows the deployed device retracting tissue (from (Becker et al., 2017)). . . . . . . . . . . . . . . . . . . . . 166 Related work in endoscope stabilization using PCMEMS fabrication techniques. In (a) we show the stabilization mechanism functional principle: (1) illustration of how endoscope flexibility hampers effective force transmission and tissue manipulation tasks, (2) Fm is the force exerted by a tool manipulating tissue and Fr is the reaction force on the endoscope, (3) the inherent flexibility of the endoscope causes the instrument to bend during the task, and (4) the proposed solution with an expandable device mounted around the endoscope. In (b) we show the fabricated stabilization mechanisms: (1) seven soft linear actuators are interconnected, enabling inflation with a single tube, (2) actuators integrated onto a sleeve mounted onto the endoscope, and (3) inflated actuators, expanding into the bracing structure, (4) five strain limiting structures are fabricated in a single batch with one channel for inflation, (5) stabilization mechanism positioned around the endoscope, and (6) inflated (from Ranzani et al. (2017)). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
xx
A.1 Overview of custom evaluation system with callouts to important features (configuration shown here is for evaluating SMA actuator performance). . . . . . . . . . . . 181 A.2 Sectional view of the primary drive subsystem, where the adjustable preload methodology places the dual radial ball bearings in a quasi back-to-back configuration.182 A.3 Constraint analysis of proposed secondary drive, illustrating the need for off-axis compliance in monolithic carriage design: (from left) free and constrained DoFs from the ballscrew alone, from the addition of a single guiderail, and from the addition of the second guiderail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 A.4 (left) Finite element analysis of monolithic carriage design under misalignment (off-axis) loading and operational (on-axis) loading (the mesh is refined to 0.001" element sizes on the flexural elements (roughly 1/6th of the flexure thickness), and 0.01" element size otherwise), (right) photograph of monolithic carriage integrated into the secondary drive. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 A.5 Schematic of custom evaluation system, configured for thermomechanical analysis of SMA actuators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 A.6 Graphical user interface for executing experimental protocols. . . . . . . . . . . . . . 187 A.7 Master controller block diagram of evaluation platform, showing the four different controllers embedded on the system’s real-time kernel. . . . . . . . . . . . . . . . . . 188 A.8 Simulink implementation of the master control system. . . . . . . . . . . . . . . . . . 189 A.9 Simulink implementation of high-level velocity controller for manual jogging and positioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 A.10 Simulink implementation of high-level position controller (PID control with velocity/acceleration/gravity feed-forward). . . . . . . . . . . . . . . . . . . . . . . . . . . 190 A.11 Simulink implementation of high-level force controller based on feedback from ATI Nano17 load cell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 A.12 Simulink implementation of virtual spring controller, where a custom MATLAB function computes desired spring position based on force feedback and desired bias spring parameters, which is subsequently fed into a PID position controller. . . 191 A.13 Simulink implementation of SMA current controller based on feedback from analog current sensors in SMA driver module. . . . . . . . . . . . . . . . . . . . . . . . . . . 192 A.14 Simulink implementation of SMA temperature control based on feedback from non-contact infrared temperature sensor. . . . . . . . . . . . . . . . . . . . . . . . . . 193 A.15 Position control example with a 5N dummy load: (a) sine wave (desired vs. actual stage location) and tracking error, (b) triangle wave (desired vs. actual stage location) and tracking error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 A.16 Force control example for a soft elastic load: (a) sine wave (desired vs. actual applied load) and tracking error, (b) triangle wave (desired vs. actual applied load) and tracking error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
xxi
A.17 System identification: (a) chirp waveform used in dynamic characterization, from which magnitude and phase information are extracted, (b) System dynamic analysis for position control of a 1mm sine wave, and the resulting model fit. Dynamic analysis shows a bandwidth of around 9 Hz for a 1mm peak-to-peak displacement. 195 A.18 SMA thermodynamics control. where shaded error bar denotes 1σ for n = 5: (a) current control results for maintaining steady current through SMA specimen, (b) temperature control results for maintaining steady temperature through SMA specimen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 A.19 Soft sensor characterization: (a) relative change-in-resistance of samples with varying pre-strain and laser treatment (strained until conductivity is lost), (b) a sensor’s performance as an electrical conductor, where the LED’s brightness shows qualitatively the level of electrical conductivity at various application strains. . . . . . . . . 196 B.1 Environmental disturbance in optoelectronic sensor systems: (a) PCMEMS Multisensor response to environmental disturbance with null force input: (top) environmental sensor readings (where green shading denotes a change in ambient light and red shading denotes a temperature change), (bottom) corresponding emitter/detector output changes, (b) operational schematic and graphical model demonstrating the complex inter-relationships between inputs, observations and outputs, illustrating how knowledge of s1 , ..., s4 alone is insufficient to accurately reconstruct the input force vector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.2 (left) Electrical architecture the source excitation and filtering of the phototransistor response, and (right) representative scope traces at various locations along the conditioning circuitry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.3 Theoretical and experimental frequency response at the output stage, where the pass band centers at 8 kHz as designed. . . . . . . . . . . . . . . . . . . . . . . . . . . B.4 Fabricated PCMEMS multisensor: (left) post-fabrication prior to assembly, still attached to assembly scaffold, (middle) assembled with components placed, (right) integrated into 3d-printed casing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5 Experimental setup, showing the encapsulated PCMEMS sensor mounted directly onto an ATI Nano16 reference load cell. . . . . . . . . . . . . . . . . . . . . . . . . . . B.6 Comparison of ground-truth force data with estimates generated via componentwise ordinary least-squares regression (Moore-Penrose pseudoinversion) using a linear combination of s1 , ..., s4 : (a) fˆ x,MP and fx , (b) fˆ y,MP and fy , (c) fˆ z,MP and fz . In general, we observe how the differential nature of Fx and Fy results in stable measurements, but the common-mode nature of Fz results in poor performance. . .
xxii
201
203 203
206 207
209
B.7 (a) Parametric sweep for ridge regression with a polynomial feature space transformation where the testing error is plotted as a function of basis dimension (overfittting occurs after order 2), (b) bar plot comparing component-wise test data RMSE for each feature space employed as discussed in Section 3.2, and (c) fˆ z,RR and fz where Φ has a quadratic basis. We observe significant force tracking performance over the Moore-Penrose solution with Xs data only. . . . . . . . . . . . . . . . . . . . B.8 Comparison of ground-truth force data with estimates generated via componentwise kernelized ridge regression with RBF kernel basis, where the data set has been downsampled for tractability (decimation = 100): (a) fˆ x,KRR and fx , (b) fˆ y,KRR and fy , (c) fˆ z,KRR and fz . We observe significant performance improvements in force tracking of Fz , as well as noticeable performance improvements in Fx and Fy . . . . . B.9 ǫ-SVR results: (a) fˆ z,SVR and fz using batch-based ǫ-SVR with RBF kernel on a downsampled data set (decimation = 100), (b) linear PEGASOS on raw data set (decimation = 1) using a linear kernel, and (c) kernelized PEGASOS on downsampled data set (decimation = 100) using RBF kernel. Dashed lines indicate RMSE obtained via batch ǫ-SVR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.10 Tracking performance of kernelized ridge regression (RBF kernel) vs. maximum sampling frequency (1/t predict ). Performance of other methods tested in this paper are provided for reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
211
214
217
220
C.1 PCMEMS fabrication process analysis: (top) 355nm Nd:YVO4 laser modeled as a Gaussian beam, and associated ablation rate dependence on depth of cut, (bottom) manufacturing timing diagram (hatched requires constant supervision/attention, solid requires no intervention) with qualitative graphs showing how the processing time scales with the number of layers in the laminate. . . . . . . . . . . . . . . . . . . 224 C.2 Experimental setup for robustness tests on PCMEMS assembly features, where insets show sample renderings and the geometric parameter of interest is circled (note that layer thickness is exaggerated for clarity): (from left) lap shear failure (loading plane parallel to adhesion plane), peel failure (loading plane perpendicular to adhesion plane), and castellated hinge torsion failure (planar torque about hinge).226 C.3 Experimental lap shear failure analysis where samples consist of two layers of steel with variable adhesion area: (a) failure shear force vs. lap area, (b) raw load/extension curves for all samples, (c) SEM image of failure plane. . . . . . . . . 227 C.4 Experimental lap shear failure analysis where samples consist of one layer of Kapton adhered between two layers of steel with variable adhesion area: (a) failure shear force vs. lap area, (b) raw load/extension curves for all samples, (c) SEM image of failure plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
xxiii
C.5 Experimental peeling failure analysis where samples consist of one layer of Kapton adhered between two layers of steel with variable adhesion area: (a) failure peel force vs. lap area, (b) raw load/extension curves for all samples, (c) SEM image of failure plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 C.6 Experimental torsional failure analysis where samples consist of one layer of Kapton adhered between two layers of steel with variable castellated hinge width, about which a torque is applied: (a) failure torque vs. hinge width, (b) raw load/extension curves for all samples, (c) SEM image of failure zone showing serrations in the Kapton layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 C.7 Experimental torsional failure analysis comparing flexural joints employing different thicknesses of Kapton flexural layer (25 µm and 50 µm) with a flexural joint consisting of 25 µm thick stainless steel as a flexural layer (shaded error bars denote standard deviation for n = 5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 D.1 Simulink implementation of EndoMODRA master controller . . . . . . . . . . . . . 236 D.2 Simulink implementation of EndoMODRA master controller . . . . . . . . . . . . . 236 D.3 Open-loop (sensorless) control paradigm: (top) control input, (middle) deterministic current profile, and (bottom) fluid cooling logic. . . . . . . . . . . . . . . . . . . . . . 237
xxiv
Acknowledgments I would like to begin by expressing my deepest gratitude to my advisers Professor Conor Walsh and Professor Robert Wood for giving me the opportunity to work on such an exciting project in a cutting-edge field. Professor Walsh taught me the importance of keeping the clinical application at the forefront of thought before getting bogged down in the technical details (seeing the forest from the trees), and his advising style granted me the liberty to carve out my own research path while offering helpful nudges and path corrections along the way. Professor Wood’s enthusiasm for my research and his involvement in getting animal protocols approved was instrumental especially in the later stages of my project. I would like to thank Professor Robert Howe for rounding out my defense committee and providing his expertise in surgical robotics during our infrequent yet deeply informative meetings. I would like to thank my clinical collaborators, Dr. Hiroyuki Aihara and Dr. Christopher Thompson, for offering both their time and their facilities at Brigham and Women’s Hospital to teach me about endoscopy, conduct experiments, engage in design reviews and define functional requirements. I would also like to thank Bob Platt and the staff at Pine Acres Rabbitry farm for helping with IACUC protocol approval and in vivo testing. This work would truly be incomplete without their contributions. I would also like to thank my colleagues, friends, and associates in the Harvard BioDesign, Microrobotics and Biorobotics labs who made my experience at Harvard unforgettable. There are far too many amazing people to name and the list that follows is by no means exhaustive. I would like to thank Dr. Sam Kesner for serving as my ‘ad-hoc adviser’ during the first few years of my Ph.D and for teaching me the value of divergent-convergent thinking. I would like to thank Dr. Tommaso Ranzani and Dr. Sheila Russo who were instrumental in helping me frame the direction of my thesis. I would like to thank Ye Ding and Jaehyun Bae for solidarity in many late night study sessions learning advanced machine learning and dynamics concepts. I would like to thank Peter York who approached our many brainstorming sessions with a vivacity that was truly contagious. I would like to thank Sebastien de Rivaz and Zhi Ern Teoh for the banter and and friendly foosball competition (even if they support Manchester United but I won’t hold it against them). I would like to thank Alperen Degirmenci for his willingness to collaborate despite tight
xxv
publication deadlines. I would like to thank Dr. James Weaver and Alex Meckes for their timely assistance in 3D printing my parts for jigs and prototypes. I would like to thank Frey Tesfaye and Marcia Mota for helping out with conference arrangements and making procurement a breeze. I would like to thank Zivthan Dubrovsky and Andy Levine for their project management and business development expertise. I would like to thank undergraduate researchers Shaan Ericksen and Katerina Soltero, and summer intern Cliff Bargar for developing hardware in support my work. I would like to thank the Defense Advanced Research Project Agency (DARPA) for providing the funding to get this research off of the ground through the Atoms to Products (A2P) initiative. I would like to acknowledge the Harvard John A. Paulson School of Engineering and Applied Sciences and the Wyss Institute for Biologically-Inspired Engineering for allowing me to use their facilities and instrumentation to generate high-quality results. I would like to thank the Harvard Office of Technology Development (OTD) for helping to protect intellectual property generated during this work. I would like to acknowledge the staff and facilities at Hemenway Gym for providing the venue for innumerable hours of iron-based stress relief. On a more personal note, I would like to thank my family and friends who supported me during this long, sometimes arduous, but always rewarding process. I’d like to thank the amazing Kristen Pluchino for her patience and support in the later stages of my thesis. Lastly and most importantly I’d like to thank my parents, John Gafford and Jodie Ball. Through your unconditional love and support, I learned how to think independently, work diligently, and keep my ambitions lofty yet grounded in reality. I attribute all of my successes, this thesis included, to you guys. I love you.
xxvi
To my friends and family for your love and support. Without you, none of this would have been possible.
xxvii
Chapter 1
Introduction Over the last several decades, surgical practice has seen a paradigm shift from high-morbidity, traumatic, open procedures to minimally-invasive surgical (MIS) techniques performed through a small number of strategically-placed, millimeter-sized incisions or through the body’s natural orifices. Such techniques promise improved outcomes by increasing patient safety through reduced intraoperative trauma, and associated reductions in cosmetic scarring and rehabilitation time. Further advancements in clinical practice and medical device technology have ushered in new classes of minimally-invasive procedures performed through natural orifices, as in natural orifice transluminal endoscopic surgery (NOTES) and laparoendoscopic single-site (LESS) surgery. While patient-side benefits are obvious, the paradigm shift towards minimal invasiveness has brought about new challenges on the clinician side due counterintuitive ergonomic principles, modifications to conventional surgical workflows, limited distal dexterity, reduced vision and a lack of haptic feedback (Vitiello et al., 2013). Computer and robotically-assisted surgery was developed in the late 1980’s/early 1990’s to overcome these limitations, and further, to enhance the interventional capabilities of surgeons and improve patient outcomes. Robotic systems offer several advantages over manual tools, including accurate knowledge regarding the end effector’s position in space for precision manipulation, the ability to compensate for or eliminate tremor and human fatigue, the ability enforce virtual constraints and fixtures to limit damage to adjacent healthy tissue, and the ability to reposition instruments quickly and execute complex trajectories with high accuracy. Computer-integrated systems might include pre-operative imaging and planning (based on single imaging modalities 1
or co-registration of multiple modalities), intraoperative registration of the patient to both pre- and intra-operative imaging, verification of surgical task completion, and quantitative post-operative analysis (Webster, 2007). Examples of computer-assisted surgical robotic platforms are shown in Fig. 1.1. Early explorations into robotic surgery consisted of retrofitting existing industrial robotic platforms (i.e. Puma 560) with specialized end-effectors to assist in placing brain biopsies for CT-guided stereotactic brain surgery (Kwoh et al., 1988). The ROBODOCTM system, designed to machine the femur for greater precision in hip replacement surgeries, was the first custom-designed robotic system to be approved by the United States Food and Drug Administration (FDA), however, the system failed to demonstrate substantial clinical benefit after a series of misuse cases in Germany led to legal issues that culminated in the removal of the system from the market (Nof, 2009). The Automated Endoscopic System for Optimal Positioning (AESOPTM ), developed by Computer Motion, Inc., was the first robot to be granted FDA approval for surgery in 1993 (Lanfranco et al., 2004). Six years later, the most well-known and widely-used robotic system, Intuitive Surgical’s daVinciTM , became the very first surgical robotic system to achieve FDA clearance for general laparoscopic procedures in 2000. The system implements a master-slave (teleoperative) control paradigm wherein the surgeon sits at a console and interacts with a master system to remotely operate a slave robot that has several highly-dexterous, rigid arms to perform a subset of laparoscopic surgeries. At the time of this writing, the daVinciTM system is the most successful commercial surgical robotic system with more than 3,800 units installed worldwide, and is widely considered to be the ‘gold standard’ for robolaparoscopic surgery. As robotic systems are being translated into clinical practice, the quest for further trauma reduction has pushed the frontiers of robotic surgery towards interventions performed through single incisions (LESS) or through natural orifices (NOTES) (Arulampalam et al., 2009). In this work, we consider an extreme class of minimally-invasive surgical robotics: endoscopic robots for intraluminal interventions within the gastrointestinal (GI) tract. While laparoscopic robots like the daVinciTM enjoy the benefits of rigid tools and easily-accessible anatomical workspaces, endoscopic robots must be flexible and highly-dexterous to access confined anatomy by navigating through the body’s natural lumens, while at the same time exhibiting distal stiffness to apply forces and interact with tissue. Despite substantial academic interest in robotic endoscopic devices, 2
Figure 1.1: A history of robotic surgery in four pictures: (from left) Puma 560 used to place a brain biopsy (circa 1985), ROBODOCTM system for computer-guided hip replacement surgery (circa 1992), AESOPTM system (circa 1993), daVinciTM system (current).
clinically-deployable GI robots are extremely limited. In the following sections, we present a brief history of GI endoscopy as well as a discussion of current and future trends. In addition, we analyze the landscape of the state-of-the-art in advanced interventional endoscopy.
1.1 Gastrointestinal Endoscopy Endoscopy (meaning ‘looking inside’) is the clinical technique of visualizing the inside of the body using a rigid or flexible instrument that is inserted into the body’s cavities or natural lumens. The term endoscopy itself is an umbrella term that comprises many sub-fields that are named depending on the anatomy that is being visualized (i.e. ‘colonoscopy’ refers to endoscopy of the colon, ‘oesophagoscopy’ refers to endoscopy of the esophagus, etc.). The very first GI endoscopy dates back to the 19th century and is credited to Adolf Kussmaul in 1868 who passed a rigid metal tube over a previously-inserted flexible obturator to visualize the stomach (gastroscopy) (Sivak, 2005). Despite lofty ambitions of accessing and visualizing confined anatomy inside the body in an atraumatic way, early pioneers faced difficulties associated with the gut’s tortuousness1 , in addition to the lack of viable illumination options for visualization of the gut. Ninety years later in 1958, Curtiss and Hirschowitz devised the first clinically-feasible endoscope that seemed to solve both of these problems through its flexible construction and the use of thin glass fibers which ran through the length of the instrument to transmit light to the distal end (Hirschowitz et al., 1958). 1 Kussmaul
himself circumnavigated this hindrance by cleverly engaging the services of a professional swordswallower ‘whose unusual talent allowed Kussmaul to understand more precisely how to navigate through the body’s most treacherous contours’ (Nezhat, 2011)
3
Figure 1.2: A history of endoscopy in three pictures: (left) Adolf Kussmaul’s ‘sword-swallowing’ technique for visualizing the stomach through a rigid metal tube (circa 1868), (middle) Basil Hirschowitz testing his flexible endoscope on an enthusiastic graduate student (circa 1960), (right) Olympus GIF-2TH180 endoscope (circa 2010).
This innovation is seen by many as the most significant development in the history of endoscopy which ushered in an explosion of technical innovations from the 1960’s to the 1990’s, including the introduction of digital imaging (through charge-coupled device (CCD) cameras) and computer interfacing for electronic recording and record-keeping, integrated irrigation and insufflation systems, controllable distal degrees-of-freedom, and working lumens for interchangeable flexible tool deployment. Despite the initial three-decade surge of innovation that began in the 1960’s, "technological development in terms of diagnostic and therapeutic endoscopy slowed substantially in the last decade by comparison to the earlier ‘golden era’" (Sivak, 2005). As can be seen in in Fig. 1.2, commercial endoscopes of today (right) are very similar, morphologically and fundamentally, to those that were being used fifty years ago (middle). Stagnation in endoscope innovation is rather conspicuous and can potentially be traced back to several sources, including (but not limited to) limitations in technical achievement, bureaucratic roadblocks, limited funding for endoscopebased research and under-representation of endoscope-related research within the National Institute of Health (NIH) (Sivak, 2005). However, while endoscope technology has stagnated, clinical ambitions continue to push the boundaries of therapy towards minimal invasiveness. To make these ambitions a reality, substantial advancements over current endoscope technology are paramount to usher in the future of advanced endoscopy.
4
1.1.1
Frontiers of Endoscopy and Persisting Challenges
A considerable motivator for technical innovation in endoscopy is a desire to bridge the gap between diagnosis and treatment. In diagnostic applications, the endoscope is a vehicle that provides access to and inspection of confined anatomy, whereas in interventional applications, the endoscope is both a vehicle for access and a tool for performing surgery. Most widespread applications of endoscopy are primarily diagnostic in nature, however the past few decades have seen the rise of endoscopy as a powerful minimally-invasive therapeutic tool for a small subset of intraluminal interventions. In diagnostic procedures, inadvertant intestinal wall perforation events are avoided at all costs as such complications are associated with high morbidity and mortality rates due to septic shock. Often times, expeditious perforation management entails surgical intervention which wholly negates the atraumatic benefits of endoscopy. However in the early 2010’s, pioneering practicioners demonstrated that, if the gut wall can be breached safely and intentionally, a multitude of new therapeutic possibilities are suddenly made available, including esophageal myotomy and cancerous tumor removal from the muscularis propria (Maydeo and Dhir, 2017; Inoue et al., 2010). This so called ‘third space’2 , or the potential space between the muscularis mucosa and the muscularis propria layer of the gut wall, can be exploited to separate the submucosa from the muscularis and perform any intervention in the muscularis propria layer. Nascent ‘third space’ procedures such as peroral endoscopic myotomy (POEM), endoscopic mucosal resection (EMR), and endoscopic submucosal dissection (ESD) are garnering steadilyincreasing interest in clinical and academic communities alike, as shown in Fig. 1.3 (a). However, a number of roadblocks stand in the way of ubiquitous adoption of these techniques.
Persisting Challenges While there is significant clinical and academic interest in performing complicated endoscopic procedures in the third space, the design of specialized instruments and systems is critical to ensure widespread clinical uptake of such procedures (few of which have recently passed 510(k) premarketing evaluation) (Wang, 2011). Intramural procedures such as ESD are in general very 2 "If
the lumen was historically the first and the peritoneal cavity the second, then the intramural space has come to represent the ‘third space’." (Khashab and Pasricha, 2013)
5
16000
2 ESD EMR POEM
14000
Endoscopic Device Robotic Resection Endoscopic Robot
12000
Number of Instances
Number of Instances
# 104
10000 8000 6000 4000
1.5
1
0.5
2000 0 1970
1980
1990
2000
2010
0 1970
2020
Year
1980
1990
2000
2010
2020
Year
(a)
(b)
Figure 1.3: Number of instances in which the following set of keywords show up in the title or abstract of articles archived in PubMed per year: (a) {Endoscopic Submucosal Dissection, Endoscopic Mucosal Resection, Peroral Endoscopic Myotomy}, (b) {Endoscopic Device, Robotic Resection, Endoscope Robot}. The dotted lines are extrapolations based on the data.
labor intensive and demand high levels of expertise and specialization, and anatomically difficult interventional locations can make the procedure challenging for even the most experienced endoscopist (Kim et al., 2011). The thickness of the gastric wall (no more than a few millimeters) makes third-space procedures a tightrope walk between successful (complication-free) intervention and intestinal perforation which can occur in up to 10% of cases (Lee et al., 2012; Thorlacius et al., 2013).
1.1.2
The Rising Role of Advanced Instrumentation and Robotics in Interventional Endoscopy
To answer the call for more sophisticated endoscopic tools and address some of the challenges associated with advanced endoscopic procedures such as those that take place in the submucosal space, many devices and systems have been developed both academically and commercially, and have received varying degrees of clinical testing and regulatory approval. Clinical interest in these advanced modalities is rapidly increasing as well, as shown in Fig. 1.3 (b). Advanced GI endoscopy devices and robots are primarily designed to satisfy at least one of the following five aims:
6
• Aim 1 (Simplifying Ergonomics): Designing mechanical or robotic systems to intuitively map from the user input space to the endoscope tip in task space, thereby alleviating the cognitive burden on the practicioner by offloading input-to-task space transformations to a computer processor. • Aim 2 (Improving Navigation and Locomotion): Designing highly dexterous, kinematically redundant (snake-like) or fully deployable (capsular) systems to seamlessly navigate through the body’s torturous luminal anatomy and deliver an end-effector to the site of intervention. • Aim 3 (Stiffness Tunability): Enabling active control of the endoscope’s stiffness to switch between low-stiffness/flexible (navigation) states and high-stiffness/rigid (interventional) states for transmitting forces distally. • Aim 4 (Enhancing Distal Dexterity): Designing tools and mechanisms that give the endoscopist additional degrees-of-freedom, multitasking capabilities and/or triangulation at the distal end to improve dexterity. • Aim 5 (Improving Distal Visualization and Sensory Feedback): Designing and implementing sensor systems to measure forces and displacements distally, or mechanisms or improved imaging technologies for enhancing visualization. With these primary aims in mind, in the following section we assess the landscape of advanced endoscopic technology to understand how the state-of-the-art maps to these aims, and identify where shortcomings exist in current solutions.
1.2 State-of-the-Art in Advanced Endoscopic Devices and Systems A snapshot of the current landscape of advanced endoscopic devices and systems for GI interventions is shown in Fig. 1.4. In this figure, we break the landscape up into three primary categories: (Category 1) simple mechanical or passive endoscopic add-on devices, (Category 2) mechanical multi-articular systems, and (Category 3) robotic systems. In addition, icons next to each technology indicate the degree of regulatory approval (FDA and/or CE, if any) as well as the primary aim(s) that is/are addressed by each device that were enumerated in the previous section. 7
Category 1: Mechanical/PassivemAdd-OnmDevices DiagnosticmDevices
A5
1
A5
2
A5
TherapeuticmxTask-SpecificVmDevices
A4
3
4
A4
5
6
Category 2: MechanicalmMulti-ArticularmSystems GeneralmEndoscopicmSystems
A3 A4
A3
7
8
A4
9
A4
13
A2
14
Task-Specific
A3 A4
A4
10
Category 3: RoboticmSystems AcademicmPrototypes
A4
11
A4
12
CommercialmSystems
A4
A2
15
19
A2
20
A2
21
A2 A3
16 Legend:
17 A1
A1
A1
A4
18
22
A2 A4
A4
23
FDAmApproval CEmMarkmofmApproval Ergonomics A2 Navigation A3 Stiffness A4 Dexterity/MT
24 A5
Sensing/Visualization
Figure 1.4: State-of-the-art for advanced endoscopy: (1) EndocuffTM (Arc Medical Design, Leeds, UK), (2) EndoringsTM (Aquilant Endoscopy, Hampshire, UK), (3) Third-EyeTM panoramic camera module (Avantis Medical Systems, Inc., San Jose, CA), (4) OverstitchTM endoscopic suturing device (Apollo Endosurgery, Inc., San Diego, CA), (5) OTSCTM (Over-the-Scope Clipping) system (Ovesco Endoscopy AG, Tubingen, Germany), (6) CaptivatorTM for ligation-assisted EMR (Boston Scientific, Boston, MA), (7) AnubiscopeTM (Karl Storz, Tuttlingen, Germany), (8) CobraTM system (USGI Medical, San Clemente, CA), (9) EndoSamuraiTM (Olympus, Tokyo, Japan), (10) DDES (Direct-Drive Endoscopic System) (Boston Scientific, Boston, MA), (11) Incisionless Operating PlatformTM (USGI Medical, San Clemente, CA), (12) R-ScopeTM (Olympus, Japan), (13) IREP (Vanderbilt University, Knoxville, TN), (14) i-Snake (Imperial College, London, UK). (15) CYCLOPS (Imperial College, London, UK), (16) STIFF-Flop (EU Consortium) (17) RS-ALC (University of Twente, The Netherlands) (18) EOR (KIT, Japan) (19) EndoticsTM system (Endotics, Peccioli, Italy), (20) InvendoTM system (Invendo Medical, Kissing, Germany), (21) NeoGuideTM (Intuitive Surgical, Sunnyvale, CA), (22) MASTER (Endomaster, Singapore), (23) ViaCath (Hansen Medical, Mountain View, CA), (24) FlexTM system (Medrobotics, Raynham, MA).
8
1.2.1
Category 1: Mechanical Endoscopic Add-On Devices
An add-on device is a distally-mounted, usually disposable device that can integrate with a standard endoscope and improve its function or add capabilities. Of the device categories identified in this thesis, mechanical add-on devices are the simplest in terms of functionality and implementation. Category 1 modalities are particularly attractive as they leverage the capabilities already present in conventional endoscopes (some degree of distal articulation, illumination, imaging, and working ports for tool interchangeability) while adding additional capabilities as-needed on a task- or procedure-specific bases. These devices are typically marketed as singleuse/disposable devices, thereby alleviating stringent on-site sterilization requirements3 , and can be diagnostic or therapeutic in nature. In the following sections we describe six clinically-approved add-on devices for advanced endoscopic procedures.
Diagnostic Add-On Devices Most add-on devices for diagnostics are passive devices employed distally to improve yield through enhanced visualization, either by mechanically ‘opening up’ collapsed lumens or improving imaging. Passive mechanical devices include the EndocuffTM (1) and EndoringsTM (2) devices which have soft features that expand collapsed lumens to create space for visualization. These devices passively deploy arms or expanding features when retracted to open up the lumens and expand the visual field. As the EndoCuffTM and EndoringsTM are inherently similar to commonly-used endoscopic overtubes and distal balloon attachments, predicate regulations were in place to facilitate approval through 510(k) by the FDA in 2012 and 2014, respectively (FDA, 2012, 2014b). The Third-EyeTM panoramic camera module (3) is an example of a sensorized add-on device, and features a tip-mounted, side viewing camera that provides an alternative view of the lumens for polyp visualization and removal. This device demonstrated substantial equivalency to the FDA-approved Third Eye RetroscopeTM by the same company (a camera that can be inserted through the endoscope working port and ‘bent back’ on itself to improve visualization of polyps during colonoscopy screenings), and as such, received expedited FDA approval through 510(k) in 3 Disposable
medical devices are typically sterilized before leaving the manufacturing site (as opposed to at the healthcare facility) using ethylene oxide gas, electron beam or gamma sterilization
9
2014 (FDA, 2014c).
Interventional Add-On Devices The Apollo OverstitchTM (4) ‘enables advanced endoscopic procedures by allowing physicians to place full-thickness sutures and secure the approximation of tissue through a dual-channel flexible endoscope’ and has received 510(k) approval through FDA and CE mark of approval in 2008 and 2012, respectively (USSEC, 2017). The OverstitchTM is perhaps one of the most mechanically complex clinically-approved add-on device on the market. Other interventional add-on devices include the Over-the-Scope Clip (OTSCTM ) (5) and the CaptivatorTM (6) for EMR which received 510(k) approval in 2010 and 2014, respectively (FDA, 2010, 2014a).
1.2.2
Category 2: Mechanical Multi-Articular Systems
Mechanical or direct-drive multi-articular systems have advanced features such as distal articulation and triangulation of tools, as in the AnubiscopeTM (7), CobraTM (8), EndoSamuraiTM (9) and DDESTM (10). Other systems provide more working ports to support multitasking, such as the Incisionless Operating Platform (IOP)TM (11). These systems are direct-drive in that the distal motion is generated purely mechanically by cables and pulleys routed to the distal end from a proximal input device with no motorized sources of actuation. These systems are all very similar morphologically, featuring articulating tool arms which expand outwards and back inwards for triangulation and have multiple distal degrees-of-freedom for bimanual dexterity. The AnubiscopeTM features a novel tulip shape that is closed during insertion (acting as a blunt trocar to prevent injury) and open during the procedure, allowing two opposing arms to emerge. Of the general endoscopic systems considered, only the AnubiscopeTM achieved any level of regulatory approval and was awarded the CE mark. The IOPTM and CobraTM systems also have the ability to stiffen their structures through shape-locking technology. In the case of the CobraTM system, the whole platform had to be removed from the body to exchange tools. In addition, the cable-driven controls led to imprecise dissection and the project was abandoned in 2012 (Patel et al., 2014). The EndoSamuraiTM has an unobstructed working channel through which standard endoscopic tools such as biopsy forceps (shown) can be inserted.
10
Criticisms of Category 2 general endoscopy technologies include unintuitive ergonomics, friction build-up in the tendon-sheath systems that results in time delays between proximal input motion to distal output motion, the need for multiple operators, poor visualization, and difficulties associated with exchanging tools (Patel et al., 2014). The R-ScopeTM (12) from Olympus is an example of a task-specific system that was designed to improve an endoscopist’s ability to provide vertical lifting/countraction and lateral dissection of tissue simultaneously during interventional endoscopy. The scope itself is a modified dualchannel scope where the instrument channels are movable to lift a grasping forceps and swing an electrosurgical knife (Yonezawa et al., 2006). Despite promising clinical results, the R-scope has yet to acquire regulatory approval. Of the Category 2 devices considered, only the IOPTM can be considered an ‘add-on’ platform as it can accommodate existing commercially-available endoscopes. All other systems feature custom-designed endoscopes and optics, reducing their modularity.
1.2.3
Category 3: Robotic Systems
Category 3 systems represent the most expensive, sophisticated and technically-complex endoscopic devices due to motorized operation and the need for drive electronics and computer processing/control. While academia is rich with prototype systems, there are also a number of robotic systems that have been commercialized and have even received some level of regulatory approval. In the remainder of this section we consider both academic and commercial systems.
Academic Prototype Systems The Insertable Robotic Effector Platform (IREP) was developed a Vanderbilt University for singleport access surgery, and although the highly-dexterous arms were capable of sub-millimetric precision in teleoperation, the low-resolution camera degraded performance and no substantial results have been reported since 2013 (Simaan et al., 2013). The i-Snake (14) from Imperial College London was a stand-alone 7-DoF system that had two working channels and was demonstrated in vivo on a porcine model (Shang et al., 2011). The project is currently inactive and there are no clear plans for further development. The CYCLOPS system (15), also from Imperial College
11
London, presented a unique implementation of parallel robotics to the field of robotic endoscopy by featuring a distally-mounted, bulb-like deployable peripheral structure with a network of cables running over the structure that actuate rigid metal overtubes inside the structure (Mylonas et al., 2014). While initial results are promising, there are a number of issues with regards to implementation for in vivo use, not the least of which includes the design of the peripheral structure in a way that is low profile and flexible upon insertion/retraction, and expansive and rigid during use. The STIFF-FLOP system (16) is a novel application of soft robotics to the field of interventional endoscopy, and the system uses pneumatic chambers to articulate. In addition, the system is able to stiffen through granular jamming to apply large forces distally. STIFF-FLOP demonstrated a number of advanced capabilities (high dexterity, granular jamming and shape/force sensing), however a path towards minimization is unclear as of this writing (Ranzani et al., 2016). The Robotic Steering and Automated Lumen Centralization (RS-ALC) (17) and Endoscopic Operation Robot (EOR) (18) systems are examples of motorized packages that interface with existing manual endoscope controllers, allowing a physician to use a more intuitive humanmachine interface (HMI) to control insertion and distal deflections robotically (Ruiter et al., 2012; Kume et al., 2015). The advantages of such systems are somewhat dubious. In an ex vivo caecum intubation task, the RS-ALC marginally improved the intubation time for novices, whereas expert endoscopists performed significantly worse with the robotic system than without (median intubation time of 781 seconds as opposed to 129 seconds without assistance) (Pullens et al., 2016). As for the EOR, currently there is not enough clinical data to comment on its potential benefits.
Commercial Robotic Systems Several robotic devices exist on the market that target improved navigation and interventional capability through powered autonomous locomotion, redundant kinematics, distal dexterity and variable stiffness. The InvendoscopeTM (20) from Invendo Medical is a single-use colonoscope primarily made up of soft materials that uses an electrohydraulic deflection mechanism to enable gastroenterologists to actively control the tip deflection. This system received FDA clearance for colonoscopies in 2016 (FDA, 2016). The EndoticsTM (19) system is another disposable robotic
12
colonoscope that emulates an inchworm to locomote through the colon. This is achieved by using vacuum suction to anchor the rear section to the lumens, actuating the middle section to expand axially, and then anchoring the distal section via vacuum suction while releasing the rear section. This process is repeated to locomote through the colon. The NeoGuideTM (21) (FDA approved in 2007 (FDA, 2007)) features 16 independently-controllable 2-DoF segments and a follow-the-leader control algorithm to navigate the colon and reduce pain due to lateral forces and looping during colonoscopy, and small-sample human trials have shown promise (Eickhoff et al., 2007). The Master and Slave Transendoluminal Robot (MASTER) (22) is a system that has recently been commercialized under the name EndomasterTM . The system has been in development for nearly ten years as of this writing, and early iterations featured instruments that ran along the outside of a standard dual-channel endoscope, and tendon sheaths which occupied the working channels (Phee et al., 2010b). The most recent iteration shown in Fig. 1.4, features two 7-DoF arms that pass through a modified endoscope (Phee et al., 2010a). The system has been successfully demonstrated in pre-clinical trials for NOTES procedures including liver resection and ESD, although critics of the system bemoan the cumbersome actuation package and the fact that it requires both an endoscopist and a robotic operator (and the implicit alteration of current clinical workflows) (Phee et al., 2010a; Yeung and Gourlay, 2012). Further, questions remain regarding its sterilizability. Hansen Medical’s ViaCathTM (23) consists of cable-actuated robotic arms that can integrate with conventional endoscopes. However, the arms are quite weak (only capable of generating 0.5N distally), thereby limiting the system’s interventional capabilities, and there is currently no published clinical data regarding its efficacy in GI procedures (Yeung and Chiu, 2016). The FlexTM robot (24) is a robotic endoscopy system that features a segmented, snake-like outer-mechanism which is steered by a clinician using a haptic input device in a follow-the-leader control methodology. An inner-mechanism is advanced into the outer mechanism which stiffens the overall structure. The distal end features a high-definition vision system and two working ports through which articulated instruments can be deployed to enhance distal dexterity. The system achieved 510(k) FDA clearance for transoral procedures in 2015 (where the NeoGuideTM was identified as one of the predicate systems) and colorectal procedures in 2017 (FDA, 2015).
13
14
15 17 (*) 35 N/A N/A 18 17 14-20 16 22 Variable
IREP
i-Snake CYCLOPS
STIFF-Flop
RS-ALC System
EOR System
InvendoscopeTM EndoticsTM
NeoguideTM
ViaCathTM
MASTERTM
FlexTM
ANUBIScopeTM Cobra EndoSamuraiTM DDESTM IOPTM R-ScopeTM
System
0
0
2F 2S
1
1
1 1
1-2
1-2
1
1 2
0
Interchangeable Ports§ 1 0 1 0 3 2
2S
0
0 0
0
0
0
0 0
2F
FixedF or Diameter SpecialS (mm) Tools 16 2S 17 2F 15-18 2S 16-22 2S 18 0 14.3 0
F
F
F
F
F F
I
I
I
I I
F
F I I I I F
Fixed (F) or Insertable (I) Camera
Y
Y
Y
Y
Y Y
Y
Y
Y
Y Y
Y
N N N N N N
Robotic 4E (per tool) IOP+5E per tool/camera 5E (per tool) 5E (per tool) Standard e-scope DoFs 2E (swing and lift) 20R (9 per arm, 2 for camera) 7R (articulated section) 5R DoF per distal overtube 3R per module (linear, pitch, yaw) Roboticized standard e-scope DoFs Roboticized standard e-scope DoFs R 1 (axial locomotion) 1R (axial locomotion) 32R (16 sections with 2 bending DoF each) 7R total for 2 tools (including grasping) 5R per tool (including grasping) R 3 (endoscope) and 1E wrist (per tool)
RoboticR or ExtraE DoFs (beyond conventional)
FDA, CE
-
FDA, CE
FDA
FDA, CE CE
-
-
-
-
-
CE CE FDA CE -
FDA/CE Approval
A,H
A,H
A
A,H
A,H A,H
-
H
-
A -
-
A,C A A,H A A,C,H A,H
Animal, Cadaver, Human Studies
Table 1.1: Tabulated specifications of advanced endoscope-based mechanical and robotic devices. Note that we only consider degrees of freedom in addition to conventional endoscope degrees of freedom (axial translation, tip pitch, tip yaw, roll). Data compiled from (Patel et al., 2014; Yeung and Gourlay, 2012; Yeung and Chiu, 2016; Vitiello et al., 2013). (*: Data Not Available, § : Excluding insertable camera)
1.2.4
Other Categories
There are several other categories that are not represented in Fig. 1.4 (and subsequently tabulated in Table 1.1) but still contribute to the rapidly-growing field of robotically-assisted endoscopic interventions. These include capsular robots (Beccani et al., 2015; Ciuti et al., 2016), reconfigurable serial (Petroni et al., 2013; Salerno et al., 2013) and parallel (Mahoney et al., 2016) robotic modules and active cannula/concentric-tube robotic systems (Webster et al., 2006; Dupont et al., 2010).
1.3 Opportunity for Innovation Category 1 (mechanical/passive add-on) devices are attractive due to their low cost, disposability, simplicity of operation, similarity to predicate devices for expedited regulatory review, minimal interruption of the current clinical workflow due to simple attachment and removal, and device transparency due to their low-profile and as-needed recruitment. However, their simplicity relegates them to task specificity, limiting their capabilities and candidate applications. At the other end of the spectrum, Category 2 (mechanical multi-articular) and Category 3 (robotic) systems offer improved distal dexterity and navigation capabilities, generalizing their applicability to a wider range of procedures. However, the addition of several degrees of freedom distally can introduce ergonomic issues and the need for multiple operators. The intimate coupling between the distal systems and off-board actuation packages substantially drives up system cost and complexity, and increases workflow interruption due to the necessary physical registration between the system and the patient. Friction build-up in tendon-sheath transmission systems can make teleoperation a challenge due to phase delays and unexpected behavior, and can reduce the distal force generation capabilities due to frictional losses. For multi-use systems, the distal integration of numerous heterogeneous mm-scale moving parts (as in IREP, MASTERTM and ViaCATHTM ) drives up the cost of manufacturing and assembly, and introduces concerns with respect to sterilization. In addition, in the event of an intraoperative complication or robot failure, it can be difficult and time-consuming to remove or ‘undock’ the robot from the patient. Especially when a patient’s health is on the line, every second counts. In this work, we attempt to bridge the gap between task-specific, low-cost, low-profile, disposable endoscopic add-on devices (Category 1) and robotic devices that offer enhanced distal 15
FujiWEG-450WR5 EndoscopeW Enclosed Actuators DeflectorWPlateWand FeederWTube Olympus ITKnife CoolingWPipe andWWiring
Force Sensor
10mm
Proprioceptive Sensing Linkage
(a)
(b)
Figure 1.5: EndoMODRA: distally-mounted robotic add-on module for interventional endoscopy: (a) at-scale prototype, (b) view through endoscope and controller data.
Feedback
Endoscope Feed
Patient
Endoscopist
System Data
Input Devices
Hardware
Distal Module
Software
RT Controller Signal Conditioning Master Controller Power Electronics, Hydraulics
Figure 1.6: Cooperative control methodology overview, where the endoscopist works in tandem with the distallymounted robot (via hardware and software-based input devices) to articulate electrosurgical tools during endoscopic submucosal dissection. A ruggedized real-time control/drive unit consolidates control loop execution, data acquisition, power electronics and signal conditioning into a mobile suite-deployable system.
dexterity (Category 3) to realize a new category of modular robotic add-on devices for roboticallyassisted interventional endoscopy. This work details the conception, fabrication and validation of EndoMODRA (Endoscopic Module for On-Demand Robotic Assistance), shown in Fig. 1.5, which consists of a disposable distal robotic module with integrated actuation and sensing that can be attached onto and removed from the distal end of conventional endoscopes in a similar vein as
16
Category 1 add-on devices. The distal module connects to a proximal control electronics unit via flexible electronic and fluid connections which provide control inputs for the distal module’s on-board actuators and feeds back information from the distal module’s on-board sensors to provide robotic assistance in a similar vein as Category 3 systems. The system engages with any tool passed through the endoscope working port to provide precision robotic control of the tool’s lateral motion, thereby facilitating and potentially automating otherwise technically challenging therapeutic maneuvers. The endoscopist interacts with the device in a cooperative fashion through both hardware and software inputs to control the distal articulation of the module or execute pre-determined trajectories (such as lateral sweeping) as shown in Fig. 1.6. Meanwhile, device transparency (limited visual field occlusion, compatibility with any tool passed through the working port, and and low profile such that the endoscope’s bending sections are not inhibited) ensures that the endoscopist maintains full control over all basic endoscope functions already available, allowing for seamless switching between standard/manual controls and robotic assistance on an as-needed basis. Parsimonious design and fabrication processes open up the possibility of single-use/disposability, greatly alleviating sterilization requirements.
1.4 Thesis Contributions The work presented herein contributes to the literature on endoscopic robots, cooperative control, and meso-scale manufacturing. Using novel smart composite manufacturing techniques and unconventional actuation and sensing schemes, we demonstrate the first modular robotic endoscope add-on capable of closed-loop control and automated trajectory execution with no off-board actuation or sensing component. The main contributions of this thesis are as follows: • Experimental clinical parameterization of ESD using motion tracking technology during several ex-vivo procedures to assess the approximate workspace and amount of distal dexterity required. • A complete and computationally-tractable analytical model of helically-wound shapememory alloy actuators and the experimental validation thereof.
17
• Methods for fabricating and experimentally characterizing high-force (∼10N maximum net force), high-stroke (∼80% working strain for 100% pre-strain) antagonistic SMA actuators using custom-developed fabrication and validation hardware. • Integrated hydraulics to realize practical actuation speeds (1-2 second rise and fall time) for teleoperation and trajectory execution. • The application of a novel smart composite manufacturing process (Printed-Circuit MEMS) in the fabrication of a multisensing system for the purpose of distal integration for closed-loop position control and force feedback in an endoscopic add-on robotic device. • The design and fabrication of a fully-integrated, moisture-resistant mobile embedded system and driver unit and real-time control architecture for endoscope suite-ready deployment. • First demonstration of closed-loop position control on a fully robotic endoscopic add-on device using actuators and sensors located exclusively at the endoscope tip. • Demonstration of system feasibility in in vitro, ex vivo and in vivo ESD experiments.
1.4.1
Broader Contributions
While the aforementioned contributions are specific to medical device development and surgical robotics, this work has also contributed to areas that are of interest to broader scientific communities. These contributions include: • The development of a toolbox of force/displacement sensor manufacturing capabilities of printed-circuit MEMS, including strain gage, light-intensity modulation, capacitance, and hall-effect modalities. • First demonstration of printed-circuit MEMS technology as a clinically-viable method for fabricating sensors and mechanisms for surgical robotics systems. • The conception of ‘distal, modular robotic add-on’ medical devices that combine simple, task-specific devices with robotic systems, meanwhile leveraging current capabilities of commercial endoscopes for navigation and gross positioning.
18
• Accurate, fast (∼Hz) position-based control of helical SMA-based actuation systems using high current transients and active fluid cooling. • A method for machine-learning-based disturbance rejection in multi-axis optoelectronic force sensors.
1.5 Thesis Outline This thesis presents the design, fabrication, and testing of a modular robotic system aimed at increasing clinician sensory awareness of and dexterity over tools passed through commercial endoscopes. A data-driven design methodology was implemented to benchmark system functional requirements based on clinically-derived force and workspace considerations. Novel mm-scale sensor and actuator methodologies were designed and fabricated using cost-effective smart composite manufacturing techniques, and integrated into a distal robotic module to enhance the therapeutic capabilities of conventional endoscopes. A ruggedized, endoscopy suite-ready controller/driver system was developed based on the power electronics and signal conditioning requirements of the integrated actuators and sensors. Closed-loop position control was demonstrated using only distally-located sensors and actuators, thereby realizing the first instance of a distally-mounted robotic assistant with closed-loop trajectory automation capabilities. The system was validated in vitro, ex vivo and in vivo to demonstrate robustness to clinically-realistic environments, as well as the potential to enable improved dissection speeds. Chapter 2 describes efforts to experimentally characterize endoscopic submucosal dissection for the purpose of defining system functional requirements. A methodology is presented for capturing high-fidelity motion data during ex vivo ESD procedures, and the experimental data is subsequently analyzed to define dexterity requirements for the proposed robotic system. In addition, contact forces during monopolar electrosurgery are measured in a benchtop setting using porcine stomach and a custom-designed metrology platform. Towards a complete system model, compliance of the endoscope and various endoscope-based flexible tools are measured and nonlinear models are fitted to the data. Chapter 3 presents the modeling, design, fabrication and testing of custom helical shapememory alloy (SMA) actuators for the purpose of generating the required force and stroke 19
to deflect flexible endoscopic tools laterally. A high-fidelity, discrete-time iterative model is presented for predicting actuator performance in both static and dynamic conditions. Actuators are fabricated, and a custom evaluation platform is designed and fabricated to validate actuator performance experimentally under different initial conditions. The fabricated actuators measure 2.5mm in diameter, are 10mm long and can provide more than 10N maximum net force and sufficient usable stroke in an antagonistic configuration to satisfy the requirements determined in Chapter 2. The form factor of these actuators is suitable for distal deployment in endoscopemounted robotic systems. Chapter 4 presents the design of a mm-scale, distal multisensing linkage capable of providing position and force feedback for loop closure and automated trajectory execution. Using a smart composite manufacturing process called printed-circuit MEMS (PCMEMS), the linkage is fabricated monolithically using high-tolerance laser machining and precision lamination processes, wherein origami-inspired fold patterns are leveraged to facilitate assembly. We investigate light-intensity modulation and hall-effect sensing as potential transduction mechanisms. The proprioceptive angle sensor has a range of ±45 degrees, a root-mean-squared error (RMSE) of as low as 1.5 degrees and resolution of less than 0.2 degrees. In addition, the sensor exhibits exemplary cyclic behavior (>500 cycles without noticeable performance degradation) and minimal drift (0.05 deg/hr). We also develop a differential force sensing methodology for haptic feedback purposes and present preliminary calibration results. Chapter 5 presents the design of the distal module and system-level hardware. Actuators and sensors are integrated into a low-profile distal module that mates with existing commerciallyavailable endoscopes, realizing a modular endoscope-mounted robot capable of providing fine control to otherwise fully-passive flexible tools. An ergonomically-designed master input control device was developed based on procedural observations and design iterations with clinician feedback. A hydraulic system was designed to enable water-based cooling with minimal hydraulic latency. We also present a custom power electronics implementation for providing high current (10 times) than similar SMA-based surgical robotic systems presented in literature. The closed-loop module was able to demonstrate automated trajectory execution for some deterministic waveforms (sinusoidal and triangular), in addition to both position- and rate-based teleoperation based on inputs from the master input controller. This lays the groundwork for automated robotic lateral incision in endoscope-based tissue dissection. Chapter 7 presents preliminary ex vivo and in vivo results using an excised porcine stomach and anesthetized live porcine model, respectively. Ex vivo results show that the integrated module can provide enough actuator force to deflect electrosurgical tools laterally to electrosurgically cut porcine tissue for successful circumferential incisions. In addition, the on-board sensors demonstrated robustness to clinically-relevant conditions and are impervious to the large potential fields generated by electrosurgical tools. Both open- and closed-loop velocity-based teleoperation control paradigms were employed in a complete robot-assisted ESD performed on an excised porcine stomach, demonstrating system efficacy. Ex vivo tests demonstrated that, with minimal learning and experience with the system, robot assistance can result in dissection speeds comparable to or marginally better than those achievable through unassisted ESD. In vivo tests on a live animal model, in collaboration with practicing endoscopists, demonstrated the integrated system’s efficacy as an interventional device. Chapter 8 summarizes the contributions of the work presented in this thesis. We also explore potential avenues for further development, including multi-DoF actuation and integrated countertraction. 21
Chapter 2
Clinical Parameterization† 2.1 Introduction Developing task-specific surgical systems requires a thorough understanding of the target procedure and associated force and workspace requirements. For endoscopic procedures where intervention sites are separated from points-of-entry via torturous anatomy, quantifying distal force and workspace requirements is particularly challenging due to the inherent difficulty associated with locating sensors and metrology interfaces distally. As of this writing, quantitative data regarding workspace considerations for endoscope-based therapies, and ESD in particular, are sparse in literature. Further, there has been little to no systematic attempt to quantify endoscope and tumor motion and orientation during ESD in a statistically meaningful way. In an effort to define system functional requirements based on real clinical data, we adopted an experimental approach to define system dexterity and force requirements. We collaborated with a veteran experimental endoscopist at Brigham and Women’s Hospital (Boston, MA) and used electromagnetic tracking technology to collect six-axis motion data of both the endoscope tip and a simulated tumor during three complete ESD procedures performed on an excised porcine stomach. Data were parsed into sub-tasks that characterize ESD, and post-processed in MATLAB to acquire tool and tumor position and pose information, as well as conservative estimates of the accessed workspace. An analysis was performed to quantify the amount of dexterity required to † Material
from this chapter was published in (Gafford et al., 2016b)
22
successfully complete ESD, the results of which guide the selection of functional requirements for the distal robotic module. In addition to dexterity requirements, custom hardware was developed to experimentally define force benchmarks based on endoscope and tool stiffness characteristics, as well as typical contact forces encountered during electrosurgical ablation.
2.2 Endoscopic Submucosal Dissection Overview Gastric cancer is the fourth most common cancer worldwide and the second leading cause of cancer death (Jemal et al., 1999). Gastric cancer has a poor prognosis (5-year survival rate is as low as 10%), most likely due to the fact that most patients present at an advanced stage (Dicken et al., 2005). Due to raised awareness regarding early treatment, as well as advancements in diagnostic imaging techniques (such as magnification narrow-band imaging, chromoendoscopy and confocal laser endomicroscopy), the proportion of early gastric cancer (EGC) diagnoses is increasing with time (Zhu et al., 2016). Early detection and treatment is essential for a favorable prognosis. Endoscopic submucosal dissection (ESD) is a well-established technique that allows for en bloc removal of epithelial lesions stemming from early-stage gastric cancer from within the gastrointestinal tract. Unlike endoscopic mucosal resection (EMR) which uses a snare to remove pedunculated or protruding lesions that are fairly easily accessible, in ESD the lesion lies beneath the surface, and the submucosa underneath the lesion is dissected using a suite of electrosurgical tools, allowing for en bloc removal of sessile and flat adenomas that cannot be snared using EMR (Thorlacius et al., 2013). The prognosis for successful ESD is very good, and the general consensus is that tumors resected using ESD do not typically need further surgical treatment (Shimomura et al., 2004). Disadvantages of ESD over EMR, as alluded to in Chapter 1, include technical/cognitive burdens as well as a high risk of intestinal perforation. While ESD techniques in eastern countries are well established and have yielded good results (84% en bloc resection rate with a perforation rate as low as 5%), multi-institutional ESD studies in western countries are sparse and have elucidated the need for advanced instrumentation and lower learning curves as indicated by lower en bloc resection rates (77%) and much higher perforation rates (18%) (Saito et al., 2007; Farhat et al., 2011). Perforation incidents are exacerbated by lesion size (as high as 30% for lesions larger than 30 mm in diameter) and location (higher perforation rates for more 23
Figure 2.1: Overview of endoscopic submucosal dissection (ESD) (from (Ryu and Chen, 2011)): (A) markings with electrosurgical knife around the edge of the lesion to establish a margin, (B) submucosal injection around the edge of the lesion, (C) incision with a needle-knife or hook-knife, (D) margin cutting with insulated tip (ITKnife) at the incision hole, (E) circumferential margin cutting, and (F) submucosal dissection with ITKnife beneath the muscularis mucosa.
distally-located lesions, such as in the duodenum for esophageal entry and the sigmoid for rectal entry). Fig. 2.1 illustrates the subtasks involved while performing ESD. In (A), the perimeter of the lesion is marked with an electrosurgical tool to establish cutting margins. In (B), a lifting agent (typically saline solution dyed blue) is injected underneath the lesion to lift the lesion off of the underlying mucosa. In (C), a forward-cutting electrosurgical tool such as an Olympus Dualknife is used to create an incision to grant entry to the submucosal space underneath the lesion. In (D) and (E), a side-cutting electrosurgical tool with a blunt ceramic tip (such as an Olympus ITKnife) is used to create a circumferential dissection around the lesion to separate it from the surrounding mucosa. In (F) the lesion has been successfully dissected en bloc and is ready for removal. The general consensus among ESD practitioners is that steps (C)-(E) are the most cognitively burdensome due to the need for high dexterity and scope flexion to perform the circumferential incision. Through our motion study, we hope to quantify the degree of dexterity required to corroborate this claim.
24
(a)
(b)
Figure 2.2: Overview of a standard endoscope: (a) technical terminology, (b) functional schematic of distal angulation (From (Waye et al., 2004)).
2.2.1
Endoscope Overview
A standard commercially-available endoscope is shown schematically in Fig. 2.2 (a) and consists of a flexible insertion tube with a tip that can be controlled in two degrees of freedom, a control interface that the endoscopist uses to control tip deflections, and a connection to the illumination source and video processor. At the proximal end, a chain-and-sprocket transmission system is
25
used to transmit rotations from the Up/Down (pitch) and Left/Right (yaw) input knobs to the distal tip bending section via cables that run the length of the flexible insertion tube, as shown in Fig. 2.2 (b). The input knobs also have locking features that maintain tension in the cables when engaged, allowing the endoscopist to lock the distal configuration. The control end also has various ports for suction, insufflation, irrigation, and tool insertion. Endoscopes can be purchased with different features (i.e. different diameters, multiple working ports, and stereoscopic imaging to name a few) and are typically designed with a particular procedure/application in mind. Through this work we encounter two endoscope models that can be used to perform ESD: (1) an Olympus CF-100L 13.3mm colonoscope with a single 3.2 mm working channel, and (2) Fuji 450WR5 9.4mm endoscope with a single 2.8 mm working channel.
2.3 Ex Vivo ESD Workspace Analysis The experimental setup for ESD motion analysis is shown in Fig. 2.3. All motion experiments were performed on an excised porcine stomach in a clinical laboratory at Brigham and Women’s Hospital, Boston, MA. The experimental endoscope is a 9.4mm diameter Fuji 450WR5 endoscope with a single 2.8mm instrument channel. Data collection was parsed into three distinct subtasks which characterize the lesion elevation and excision processes deemed to be the most complicated aspects of performing ESD. These are itemized below: (1) Mucosal Lift-Off: Using an endoscopic needle (InjectorForce Max, Olympus), fluid is injected into the submucosal space to lift the lesion off of the muscularis and provide countertraction for subsequent cutting tasks. This task is often repeated several times during the procedure as fluid leaks out or diffuses into surrounding anatomy. (2) Axial Incision: A forward-cutting electrosurgical knife (DualKnife, Olympus) is used to create an opening in the mucosa such that the submucosal space can be accessed for lift-off and excision. (3) Circumferential (Lateral) Incision/Dissection: A side-cutting electrosurgical knife (ITKnife, Olympus) is used to sweep under the tumor within the submucosal space until the lesion can be excised en bloc. 26
Endoscope Monitor
EM Probe Mucosa
Tumor Margins
Submucosa
EM Field Generator Porcine Stomach
Endoscope
Figure 2.3: Experimental procedure to quantify workspace and dexterity requirements in ESD: (left) image of the experimental setup for capturing motion data during ex-vivo ESD performed on an excised porcine stomach, (right) view through endoscope camera during the circumferential incision process.
2.3.1
Motion Tracking
Motion was tracked using Ascension Technology’s trakSTAR electromagnetic (EM) tracking system. The EM transmitter was placed outside of the stomach but close enough to ensure high signal quality. Two Model 90 sensor probes (0.9mm OD) were used for tracking both the endoscope tip and the simulated tumor simultaneously. Data were collected at a rate of 2 kHz. In addition to position and orientation data, sensor quality was logged for post-processing and filtering purposes. The endoscope probe was fastened to the tip of the endoscope using electrical tape and aligned to make the roll axis parallel to the endoscope axis. Prior to the procedure, the tumor probe was passed through the duodenal entry of the porcine stomach (so as not to interfere with endoscope which passes through the esophageal entry) and clipped into the simulated tumor using a surgical clip. The orientation of the tumor was deemed arbitrary, as we are only interested in the tumor’s position in euclidean space with respect to the endoscope tip. The experimental procedure is summarized as follows: (1) Tumor Marginalization: The endoscopist advances the endoscope into the insufflated porcine stomach and, using the Olympus DualKnife, ablates tissue superficially in a ring of about 2 cm in diameter to create a simulated tumor margin, as in Fig. 2.1 (A). The endoscope is retracted. 27
(2) Probe Placement: The tumor EM probe is manually passed through the duodenal entry of the porcine stomach (so as not to interfere with endoscope which passes through the esophageal entry). Meanwhile the endoscope is advanced through the esophageal entry to retrieve the probe and clip it into the simulated tumor using surgical clips. A hemostat clamps the duodenal orifice to the tumor probe leads to ensure an air-tight seal for insufflation purposes. The endoscope is retracted, and the endoscope probe is fastened to the outside of the endoscope using electrical tape. (3) Calibration: The endoscope, outfitted with the second EM probe, is advanced into the stomach via the esophageal entry and held stationary at this starting position. Data are collected for one second while the endoscope is stationary. This data will be used to define the endoscope origin frame { A0 }. (4) Perform Procedure: Data is collected continuously during each subtasks of the procedure. Data collection terminates at the end of each subtask, and the dataset is labeled with the procedure number, subtask identifier, and subtask number. The process repeats until the lesion is successfully excised en bloc.
2.3.2
Data Post-Processing
The raw six-axis data corresponding to endoscope and tumor position and orientation are given with respect to the inertial frame of the EM transmitter, which is arbitrarily defined in space. Therefore it is paramount to re-express this information in frames that are meaningful for subsequent analysis (for example, the local endoscope frame which is defined during the calibration step). Towards this aim, we lay out a fundamental mathematical foundation for performing coordinate transformations between relevant frames. A systematic way of doing so is to utilize transformation matrices. This section begins by defining a mathematical basis for transformation using homogeneous transformation matrices (HTM) from a general perspective.
Translations Along Axes We first describe the homogeneous transform for pure translations in Cartesian space, which is given by: 28
1 0 0 δx
0 1 0 δy T(δx , δy , δz ) = 0 0 1 δ z 0 0 0 1
(2.1)
where δx , δy and δz denote linear displacements in x, y, and z axes, respectively. Therefore, pre-multiplying a point p = ( x, y, z, 1) T in Cartesian space by this matrix will simply translate the point in Cartesian space to p’ = ( x + δx , y + δy , z + δz , 1) T . We will use this to transform the spatial location of the endoscope and tumor expressed in the inertial frame to respective local frames.
Rotations About Axes We describe all rotations using a right-handed Cartesian coordinate frame, wherein positive rotations are aptly defined by the ‘right-hand rule’ (i.e. if the thumb of the right hand points along the axis of rotation, a positive rotation is the direction in which the fingers naturally curl). The following represent the three basic rotation matrices which rotate vectors by angles φ, θ, ψ around axes x, y, and z, respectively:
1
0
0
0 cos φ − sin φ Rx (φ) = 0 sin φ cos φ 0 0 0 cos ψ − sin ψ 0 sin ψ cos ψ 0 Rz ( ψ ) = 0 1 0 0 0 0
0
0 , 0 1 0 0 0 1
cos θ
0 sin θ
0
0 1 0 0 Ry ( θ ) = , − sin θ 0 cos θ 0 0 0 0 1
(2.2)
Alternatively, these describe roll, pitch and yaw rotations. Successive rotations can be expressed by multiplying the rotation matrices in the desired sequence. Therefore, we define the following three-axis xyz (roll-pitch-yaw) rotation matrix:
29
z
z'
z'
φ
y' z'' φ
φ
θ
θ
y' z''
y'' ψ
ψ
y'
y
x
x
θ
ψ x'
x'
x''
Figure 2.4: Illustration of successive rotations with respect to the current frame, mathematically described by Rxyz (φ, θ, ψ).
Rxyz (φ, θ, ψ) = Rx (φ)Ry (θ )Rz (ψ) cos θ cos φ cos θ sin φ − sin θ sin ψ sin θ cos φ − cos ψ sin φ sin ψ sin θ sin φ + cos ψ cos φ cos θ sin ψ = cos ψ sin θ cos φ + sin ψ sin φ cos ψ sin θ sin φ − sin ψ cos φ cos θ cos φ 0 0 0
0
0 0 1 (2.3)
Note that we denote the sequence of rotations and their magnitudes in the subscripts and arguments of R, respectively (i.e. Rxyz (φ, θ, ψ) denotes a rotation about the x-axis by φ, followed by a rotation about the new y-axis by θ, followed by a rotation about the new z-axis by ψ). The three successive rotations defined by Rxyz (φ, θ, ψ) are schematically represented in Fig. 2.4. The Full Homogeneous Transform The homogeneous transformation matrix defining the full transformation (translational and rotational) from some inertial frame {α} to a local frame { β} is given by:
30
{A0}: Endoscope Origin Frame {A}: Endoscope Current Frame {B}: Tumor Current Frame {O}: Transmitter Inertial Frame
za φa Endoscope
{A0}
{A}
θa
ya
zo
ψa xa
δza
δya
φo δxa
δzb
δyb
ψo
θo
δxb
xo
{B}
Simulated Tumor
yo
{O} EM Transmitter
Figure 2.5: Schematic of the experimental setup, showing relevant inertial and local frames and their respective orientations.
{α}
H{ β} = H(φ, θ, ψ, δx , δy , δz ) = Rxyz (φ, θ, ψ)T(δx , δy , δz ) cos θ cos φ cos θ sin φ − sin θ δx sin ψ sin θ cos φ − cos ψ sin φ sin ψ sin θ sin φ + cos ψ cos φ cos θ sin ψ δy = cos ψ sin θ cos φ + sin ψ sin φ cos ψ sin θ sin φ − sin ψ cos φ cos θ cos φ δ z 0 0 0 1
(2.4)
where δx , δy and δz are the displacements separating the origins of the two frames. Therefore, if we have a point represented in an inertial frame {O} (given by p{O} ), we can represent this point in local frame { A} (p{ A} ) as follows: {O}
p{ A} = H{ A} p{O}
(2.5)
This fundamental transform will be used to transform endoscope/tumor positions/poses from the inertial (transmitter) frame to the local (calibration) frame.
31
2.3.3
Results Analysis
Consider the schematic shown in Fig. 2.5. An inertial frame {O} is defined at the EM transmitter which is placed somewhere outside the porcine stomach (but close enough to ensure sufficient signal quality). A local frame { A} is defined at the endoscope tip, where the roll axis x a runs along the axis of the endoscope, the yaw axis z a points vertically from the working channel through the endoscope camera, and the pitch axis y a completes the frame according to right-hand convention. Another local frame { B} is placed at the tumor, and the orientation is arbitrary as described previously. To express the global endoscope position and orientation with respect to the local endoscope frame as defined in Fig. 2.5, we define the following transformation matrix:
{ A}
H{O} = H(φa , θ a , ψa , δxa , δya , δza )
(2.6)
We wish to re-express this transformation with respect to the local endoscope calibration frame
{ A0 }. This frame is defined at the very beginning of the procedure and serves as the endoscope’s origin frame for all subsequent subtasks within the same procedure. This can be done as follows:
{ A} 0}
H{ A
1 = H− a,cal H( φa , θ a , ψa , δxa , δya , δza )
(2.7)
1 where H− a,cal is pre-multiplied such that all positions and orientations (expressed in the inertial
frame {O}) are expressed in { A0 }. Ha,cal is defined as:
{A }
Ha,cal = H{O}0
=
δxa,cal Rzy (ψa,cal , θ a,cal )
δya,cal δza,cal
0
0
0
1
(2.8)
where [ψa,cal , θ a,cal ] is the endoscope’s pose at the very beginning of the procedure, and [δxa,cal , δya,cal , δza,cal ] is the spatial position. As it was not possible to pass the EM probe through the endoscope working channel without interfering with the procedure, the probe is mounted along the outside of the endoscope and we assume that the tool has a fixed (time invariant) transform from the probe. We introduce a 32
simulated tool frame { A′ } that is offset 10 mm from the endoscope frame along the roll axis and 5mm along the negative pitch axis. The full transform between the endoscope starting frame and the current (time-varying) frame, based on data originally expressed in { A0 } is given by: { A′ } 0}
H a = H{ A
1 = H− a,cal H( φa , θ a , ψa , δxa , δya , δza )T(10, 0, −5)
(2.9)
The tumor position data is similarly expressed with respect to the endoscope’s starting position at the beginning of data collection via the following HTM:
{ B}
Hb = H{ A
0}
1 = H− a,cal H( φb , θb , ψb , δxb , δyb , δzb )
(2.10)
Signal Filtering Due to contamination from electrosurgical pulses and a subsequent reduction in EM signal quality (which manifests as sensor saturation due to the electromagnetic interference), the raw data goes through two stages of filtering. The first is an aggressive pre-filtering stage which applies a median filter (order = 50) and Savitsky-Golay filter (order = 50 and block size = 49) in series to smooth the data. The next filtering process steps through the raw data, assessing signal quality at each step. If the signal quality is below a certain threshold, a latch is initiated which is removed when the quality improves above the threshold, and all data in the latched region is replaced with the pre-filtered data. The resulting data preserves high-fidelity dynamics in areas where sensor quality is high, and smooths (interpolates) the dynamics where sensor quality is low. This filtering methodology can be seen in Fig. 2.6 which represents the raw position and orientation data for both the endoscope and tumor during an axial incision task. We observe how the aggressive pre-filter smooths over areas of high signal contamination, whereas the post-filter preserves the dynamics in between areas of high contamination. This data is subsequently stitched together to estimate the positions and orientations.
Data Visualization and Analysis All raw data were transformed into the local endoscope frame-of-origin { A0 } as described by Equation 2.7. The trajectory of both the tool tip and simulated tumor are plotted as point clouds,
33
Tumor Data
100
50
Azimuth [deg]
Azimuth [deg]
150
Endoscope Data
Raw Data Pre-Filter Post-Filter
50 0 -50
Raw Data Pre-Filter Post-Filter
0 -50 -100
-100 100
150
200
250
300
100
150
-40
Raw Data Pre-Filter Post-Filter
-60
-80
50
150
200
250
250
300
250
300
0
300
100
150
200
Time [s]
Raw Data Pre-Filter Post-Filter
0
Roll [deg]
Roll [deg]
300
Raw Data Pre-Filter Post-Filter
Time [s]
100
250
-50 100
150
200
Time [s]
Elevation [deg]
Elevation [deg]
Time [s]
50 0 -50 -100
Raw Data Pre-Filter Post-Filter
-50 -100 -150
100
150
200
250
300
100
Time [s]
150
200
Time [s]
Figure 2.6: Pose data both pre- and post-processing for endoscope and tumor.
convex hulls are fitted around these point clouds to generate a conservative visualization of the accessed workspace, and the volume of this hull is computed. Minimum and maximum position and pose information are recorded to identify the limits of travel in six axes, and this information is stored for each subtask analysis. Fig. 2.7 shows a sample visualization of the endoscope and tumor workspace hulls during an axial incision task. Trajectories are plotted in dashed lines where the darkness of the line corresponds to the passage of time. A red convex hull is fitted around the endoscope trajectory data, and a blue convex hull is fitted around the tumor trajectory data to provide conservative estimates of the total volumetric workspace accessed during the subtask. In this particular subtask, an incision using the DualKnife, we observe significant lateral motion of the endoscope with respect to the simulated tumor as the endoscopist creates access to the submucosal space. Fig. 2.8 shows representative workspace characterizations for the three procedural subtasks. We observe 34
Figure 2.7: Example workspace reconstruction during an incision task, wherein convex hulls are fitted to the trajectory data for a conservative workspace estimate.
that the injection and axial incision tasks entail primarily lateral motions proximal to the tumor, whereas the circumferential dissection task requires both axial and lateral motion. This illustrates a motion coupling as the endoscopist must retract the tool proximally while simultaneously sweeping the endoscope laterally to provide sufficient countertraction for excision. We experimentally captured an event that sufficiently illustrates the technical complexity of removing lesions endoscopically. Retroflexion occurs when the endoscope is bent back on itself to access the distal portion of a hard-to-reach lesion. While this technique is typically performed during diagnostic procedures to improve yields (i.e. visualizing polyps hidden behind intestinal folds), therapeutic interventions are seeing retroflexion being employed as a means to provide access to difficult-to-reach lesions for intervention which significantly drives up the cognitive load
35
Saline Injection
Endoscope Workspace
DualKnife
Endoscope Orientation
Endoscope Trajectory
ITKnife
Tumor Workspace
Tumor Orientation
Figure 2.8: Workspace reconstructions from three distinct phases of ESD: (left) saline injection, (middle) axial incision using Olympus DualKnife forward-cutting monopolar cautery, and (right) circumferential dissection using Olympus ITKnife side-cutting monopolar cautery.
on the endoscopist who must preserve the retroflected configuration (gross positioning) while also generating precise motion to dissect tissue (fine tip manipulation). This event is illustrated in Fig. 2.9, where we observe that the endoscope initially approaches the tumor from the left side in step (1), and through the course of the circumferential dissection, is bent back on itself nearly 180 degrees to access the right side of the tumor in step (4). Fig. 2.10 (a) shows the average angular range-of-motion during each subtask (error bars indicate standard deviation). We observe from this data that the lateral incision task requires the most angular dexterity on average, and also presents the most variability in terms of dexterity required as this is largely dependent on lesion origination within the GI tract which required retroflective maneuvers to ensure successful dissection. If we ignore the retroflective event and re-process the data (thereby only focusing on fine dissection maneuvers rather than coarse positioning maneuvers), the variability is substantially reduced, as shown in Fig. 2.10 (b). Fig. 2.10 (c) shows the linear range-of-motion (including retroflexion), and Fig. 2.10 (d) shows that subtask completion time increases with the volume swept out by the endoscope.
2.3.4
Motion Study Conclusions
Through this motion study, we were able to roughly quantify the amount of technical rigor involved in performing ESD in a clinically realistic environment. It was found that circumferential
36
Figure 2.9: Example workspace reconstruction during a retroflexion event where the endoscope is bent back over 180 degrees on itself to access the distal side of the lesion. The numbers represent the temporal sequence of positions/orientations during this event.
dissection of mid-size lesions from within the gastrointestinal tract requires 90.3±50.4 degrees of lateral motion, however this average and the wide variability is inflated by the singular retroflective event which, although necessary for successful dissection, does not correspond to the fine lateral maneuvers required to actually remove the lesion. If we ignore this singular event, the lateral angular motion requirement drops to 66.4±20.3 degrees. As such, we target a system that is capable of providing 90 degrees (or ±45 degrees) of active lateral motion.
37
160
160
j max(#) ! min(#)j [deg]
140 120
Injection (n=8) Axial Incision (n=3) Lateral Incision (n=4)
140
j max(#) ! min(#)j [deg]
Injection (n=8) Axial Incision (n=3) Lateral Incision (n=5)
120
100
100
80 60 40 20
80 60 40 20
0
0 Roll
Pitch
Yaw
Roll
Angular Degree-of-Freedom
Pitch
(a)
(b)
150
600 Injection (n=8) Axial Incision (n=3) Lateral Incision (n=5)
100
50
Injection Axial Incision Lateral Incision Fit
500
Accessed Volume [cm3 ]
j max(p) ! min(p)j [mm]
Yaw
Angular Degree-of-Freedom
400 300 200 100
0
0 x
y
z
0
100
200
300
400
500
600
700
Time of Subtask [s]
Linear Degree-of-Freedom
(c)
(d)
Figure 2.10: ESD motion analysis: (a) angular dexterity including retroflective event, (b) angular dexterity excluding retroflective event, (c) linear motion including retroflective event, and (d) task complexity including retroflective event.
2.4 Endoscope/Tool Stiffness and Contact Force Analysis In addition to satisfying anticipated workspace requirements, an endoscope-mounted robotic module must also provide enough force to (1) deflect flexible endoscopic tools laterally, and (2) ensure sufficient contact to achieve successful electrosurgical cutting, as shown schematically in Fig. 2.11 (a) and (b), respectively. In Fig. 2.11 (a) we are modeling the tool as a torsional spring with a variable stiffness k (θ ) which generates a moment about the pivot point that opposes the moment generated by the actuators for θ > 0. In Fig. 2.11 (b) the electrosurgical contact force Ft generates an additional moment about the pivot that resists actuator motion. These forces must 38
Fbias
Fbias k(θ) k(θ) Pivot Fact
Ft
(a)
Fact
(b)
Figure 2.11: Relevant distal forces: (a) force (or torque) required to deflect flexible electrosurgical tools laterally, (b) contact force required to electrosurgically dissect tissue.
be characterized experimentally to determine the required net force that the on-board actuators must generate. In addition, it is important to understand how forces applied at the distal end of the endoscope could affect the endoscope configuration due to inherent system flexibility, as this approach inherently assumes that the endoscope provides mechanical ground through which deflection forces are transmitted. As these effects are difficult to predict theoretically, we employ an experimental approach to define system force requirements.
2.4.1
Tool and Endoscope Stiffness
The proposed cooperative robotic strategy requires the distal module to engage with and deflect flexible tools laterally. As such, the stiffness properties of anticipated tools must be characterized to understand how much force must be generated. To characterize the flexural stiffnesses of a few common endoscopic electrosurgical tools, the evaluation system described in Appendix A was outfitted with an ATI Nano17 load cell and a 3D-printed force applicator, as shown schematically in Fig. 2.12. Two representative tools were clamped in a vice 10mm from the distal end (as advised by our clinical collaborators), and the force required to deflect the tool was measured. Force vs. displacement curves for these tools (Olympus DualKnife and Olympus ITKnife) are shown in Fig. 2.13 (a) and (b), respectively, where error bars correspond to the standard deviation across n = 5 tests. The tools exhibit viscoelastic characteristics with the nonlinear loading curve and substantial hysteresis upon unloading. For approximation purposes, we are only interested in the loading behavior which is well-modeled by a quadratic fit. If we extrapolate the data of the stiffest
39
ATI Nano17
Moving Stage
Secondary Transmission Primary Drive Drive xPCMTarget
Vice
Differential F
l
Tool UnderMTest
FVT Controller
Phasing
Copley Amplifier
DriveMSignal Experimental Protocol
Quadrature Decoder
Position PIDMController
MotionMControl Logic
Low0Pass 100MHz
File Storage
Master Controller
xPCMHost MatlabV SimulinkMxPC
GraphicalM UserMInterface
Figure 2.12: Block diagram of cantilevered loading tests used to characterize flexural properties of demonstrative endoscope and endoscopy tools.
2
2 F (3) = (-0.00082754)3 2 + (0.058038)3 + (-0.021313) R2 =0.99989
1.5
Force [N]
Force [N]
1.5
F (3) = (-0.0014561)3 2 + (0.099646)3 + (0.037929) R2 =0.99863
1
0.5
1
0.5 Data Model Fit
Data Model Fit
0
0 0
5
10
15
20
0
De.ection Angle 3 [deg]
5
10
15
20
De.ection Angle 3 [deg]
(a)
(b)
Figure 2.13: Stiffness analysis of flexible tools clamped 10mm proximal to distal end and subject to a cantilevered load at the distal end, where error bars indicate 1σ for n = 5: (a) Olympus DualKnife forward-cutting monopolar electrosurgical tool, (b) Olympus ITKnife side-cutting monopolar electrosurgical tool.
tool (the ITKnife) to 45 degrees given the maximum deflection requirement from the motion study, we will require about 1.6N of lateral force (corresponding to a linearized tool flexural stiffness of about 0.16 N/mm). Endoscope-mounted robotic systems effectively rely on the endoscope itself to provide adequate mechanical grounding for distal force transmission. Therefore, external loads from tissue contact forces must be transmitted to the local ground created by the endoscope. The experimental setup shown in Fig. 2.12 was used to analyze the flexural properties of an Olympus CF-100L colonoscope when under lateral tip loading with the cables locked. We observe from Fig. 2.14 (b) that the stiffness is very high near the tip, and decays to roughly 1 N/mm as we move proximally down the length. This shows that the endoscope stiffness is an order of magnitude higher than 40
14 25mm 30mm 35mm 40mm 45mm 50mm
Force [N]
4
12
Sti,ness [N/mm]
5
3
2
1
10 8 6 4 2
0 0
0.2
0.4
0.6
0.8
1
0 25
1.2
Displacement [mm]
30
35
40
45
50
Cantilever Distance [mm]
(a)
(b)
Figure 2.14: Stiffness analysis of Olympus CF-100L endoscope, where shaded error bars indicate 1σ for n = 5: (a) Force vs. Displacement curves at various cantilevered distances l, (b) Stiffness vs. cantilever distance.
the tool stiffness, indicating that we can reasonably approximate the endoscope as a mechanical ground through which tool deflection forces and tissue contact forces are transmitted.
2.4.2
Electrosurgical Ablation Force
The experimental setup for measuring electrosurgical ablation contact force Ft , shown as a point load on the tool in Fig. 2.11 (b), is illustrated in Fig. 2.15 (a). A linear stage features a DC motor coupled to a leadscrew via a worm gear drive, providing precision linear displacement with a velocity proportional-derivative (PD) controller implemented in Arduino based on feedback from a 500 counts-per-revolution optical encoder. Two Omega LCL-101 beam-based load cells couple a tool holder to the moving stage, and the experimental tool (Olympus DualKnife) is clamped into the tool holder. A porcine tissue specimen is clamped into a stationary clamp, and constant pretension force is achieved by a static weight hanging from a pulley. Three different tissue specimens were tested for the electrosurgical force required to perforate the tissue (a full-thickness porcine stomach specimen, a mucosa-only stomach specimen, and a muscularis-only stomach specimen), all excised from the same porcine stomach. The tool is advanced into the specimen at a rate of 1mm/s, and electrosurgical pulses are fired at regular intervals. The resulting contact force is recorded as a function of stage displacement. An Arduino
41
OlympusbDualKnife WormbDrive Encoder DCbMotor
500
LoadbCells
FT Mucosa Muscularis Pulse
ToolbHolder
Ft
Force [mN]
400
300
200
Breadboard
100
Tensioner TissuebSpecimen StationarybClamp
0 0
Leadscrew
1
2
3
4
5
6
7
8
Displacement [mm]
(a)
(b)
Figure 2.15: Electrosurgical force analysis: (a) rendering of experimental setup where inset shows a close-up photograph of the tool-tissue interface, (b) Force vs. Displacement curves for different tissue speciments (FullThickness, Mucosa only, Muscularis only) where ‘x’ denotes electrosurgical pulses.
Mega was used for motion control and data collection at a sample rate of roughly 2 kHz. We see from the results shown in Fig. 2.15 (b) that the maximum force required to provide sufficient contact for tissue ablation is about 400 mN, setting a lower-bound on the amount of net force our system must generate to successfully achieve ablation. This also shows how little force is necessary to fully perforate tissue, corroborating the fact that ESD is indeed quite prone to intestinal perforation. By combining the results of these analyses, we define a force requirement of 2N at full deflection (±45 degrees) to meet our functional requirement. Note that this is after we account for any transmission losses in the system due to geometry, friction or bias forces. This will serve as a basis for actuator design and development.
2.5 Conclusion The proposed system’s functional requirements are tabulated in Table 2.1. The deflection range and force requirements are derived from experimental motion and force analyses presented earlier in this section. The maximum length is endoscope-specific and is chosen such that the presence of the module does not interfere with the bending section or add stiffness to the endoscope’s manual degrees-of-freedom in the interest of device transparency. The diameter restriction was chosen
42
Table 2.1: Experimentally-derived functional requirements for an endoscope-mounted robotic system
Functional Requirement
Value
Unit
Actuated Deflection Range θ Deflection Sensing Range Deflection Sensing Resolution Lateral Force at Max Deflection F (θmax ) Force Sensing Range Force Sensing Resolution Maximum Length L Maximum Diameter D
±45 ±45 2 2 ±2 10 40 20
degrees degrees degrees N N mN mm mm
based on a review of prior systems presented in Chapter 1 in addition to discussions with our clinical collaborators. These requirements will inform actuator and sensor development detailed in Chapter 3 and 4.
43
Chapter 3
Distally-Integrated Actuation† 3.1 Introduction Robotic systems developed for flexible endoscopy require a means of transmitting forces distally to articulate and manipulate tools and tissue for interventional purposes. Given the limited working volume imposed by intraluminal constraints, it can be difficult to integrate local actuation systems close to the site of intervention inside the body. Referring back to Category 2 and Category 3 devices introduced in Chapter 1, most endoscope-based mechanical and robotic systems in literature circumvent this constraint by placing actuators outside of the body, and transmitting forces distally using tendon-sheath systems (TSS). This approach enables the use of arbitrarily large actuators or mechanical drives and transmission hardware (within reasonable constraints to minimize surgical arena occlusion). However, as discussed in Chapter 1, this approach substantially drives down the modularity and rapid deployability of the system as a therapeutic aid, thereby inherently affecting the clinical workflow in a potentially disruptive manner. Furthermore, TSS approaches often require specially-designed endoscope systems to accommodate the tendon sheaths. Of the robotic systems surveyed in Chapter 1 that aim to improve distal dexterity of endoscopic tools (Aim 4), there are three ways in which TSS transmissions are realized. The IREP system (Fig. 1.4 (13)) features a custom construction that accommodates tendon routing internally within the flexible section. The CYCLOPS, ViaCathTM † Material
from this chapter was published in (Gafford et al., 2016c)
44
and FlexTM systems (Fig. 1.4 (15), (23) and (24), respectively) feature special overtubes that run along the outside of the endoscope for routing the tendons and flexible tools to the distal end. The MASTERTM (Fig. 1.4 (22)) routes tendons through the working channels of the endoscope, prohibiting tool exchanges. These approaches drive down system modularity due to the presence of the tendons and their requisite mechanical engagement with proximal actuation sources. In addition, friction build-up within the sheaths themselves can adversely affect controllability due to mechanical latency. As we are targeting highly-modular, plug-and-play systems with minimal clinical workflow impedance, there is a fundamental aim to eliminate tendon-based transmissions by placing the actuators as close to the distal end as possible while satisfying the force and stroke requirements tabulated in Table 2.1. As such, with the goal of ‘moving everything to the tip,’ we consider alternative, energy-dense actuation strategies that are acquiescent to the rigorous form factor requirements incurred by fully-distal implementations. This chapter summarizes the design, fabrication, and validation of high force (1-10N), high strain (>100%) helical shape-memory alloy actuators for implementation in modular endoscopemounted robotic systems. The chapter begins with an assessment of potentially compatible actuator modalities through performance index comparisons to motivate the selection of an antagonistic SMA actuation scheme. We then derive a high-fidelity, computationally tractable thermomechanical model of helical SMA antagonists and simulate the following scenarios: (1) fixed-strain and variable thermodynamics (blocking force analysis), (2) externally-induced variable strain at steady thermodynamics (actuator force/stroke characterization), and (3) variable strain induced by variable thermodynamics given an elastic biasing force (full antagonism model). The fabrication process of custom helical actuators is described, and the chapter concludes with an extensive thermomechanical characterization of the optimized actuator configuration using custom characterization hardware to demonstrate the utility and generalizability of the model in predicting actuator performance both parametrically and dynamically. We also motivate the inclusion of forced convection fluid cooling to improve performance towards real-time (∼Hz) control.
45
3.2 Actuator Selection Actuator selection at the millimeter scale is a multivariate problem that is best approached by designing for a specific performance index of interest based on the requirements defined by the application. Defining the relevant performance index requires an understanding of the mechanical requirements on the application side in terms of force, displacement, stiffness, size, mass, response time (or operating frequency), power, efficiency and resolution.
3.2.1
Actuator Performance Index
To meet the functional requirements defined in Chapter 2, while being acquiescent to form factors required by intraluminal integration, we must be cognizant of the economies-of-scale for different actuation strategies. Based on previously-defined functional requirements summarized in Table 2.1, we would like an actuator with the following (order-of-magnitude) characteristics: • Characteristic Length L0 : 10 mm • Cross-Sectional Area A: 5 mm2 • Force F: 10 N (maximum) • Stroke ∆L: 10 mm (100% net strain) Secondary requirements include speed metrics (response time/bandwidth), and although dynamic requirements are not specifically defined in our performance index of interest, we would like an actuator to be as fast as possible for distal maneuvers, with a rise time of around 0.5-1 second (to approach near-Hz level bandwidth for control). Note that, because we assume constant electrical connection to an off-board controller which is plugged into the electrical grid, we are not concerned with power consumption and efficiency. With the additional objective of minimizing the occupied volume, the primary characteristics listed above can be combined into a performance index of stroke work per unit volume (i.e. PI = σǫ = ( FL0 )/(∆LA)). Given this, we would like to nominally design an actuator that achieves PI > 200 kJ/m−3 . A nice review of mechanical actuation strategies is given in (Huber
46
104
Design Goal
103 102 101 100 10-1 10-2 10-6
10-4
10-2
100
102
Figure 3.1: Effective actuator stress vs. characteristic stroke for various actuator modalities (where dotted lines denote isocontours of constant stroke work per unit volume). The red window highlights the modalities that satisfy our performance index for force and stroke.
et al., 1997). We summarize some of the salient points from this and other sources in the remainder of this section.
3.2.2
Candidate Distal Actuator Modalities
Fig. 3.1 shows the effective actuator stress versus characteristic stroke landscape for various compatible actuator modalities, where the dotted lines are isocontours of constant stroke work per unit volume. In general, high-force actuators lie towards the top of the diagram, and high-stroke actuators lie towards the right. The red window denotes actuator specifications that meet our functional requirements in terms of generated force and stroke (where the clipped corner denotes the performance index cutoff). In the following subsections we briefly describe candidate actuator technologies and explain why they may (or may not) be appropriate for the application of distal integration in modular therapeutic endoscopic systems.
47
Electromagnetic Actuation Traditional coil-based brushed and brushless DC motors are capable of meeting stroke requirements as they are capable of continuous motion. However, in terms of force generation, millimeterscale implementations perform an order-of-magnitude lower than what is necessary to deflect tools laterally for this application. Commercially-available solenoids have a maximum actuation stress of around 10 × 10−1 MPa with a stroke of about 40% of their characteristic length, and as such, do not provide the force or displacement required for our application (Huber et al., 1997). Force generation can be improved through gear reductions at the cost of speed, efficiency and increased system complexity. Given these considerations, in addition to concerns of electrical encapsulation and protecting moving parts to avoid jamming and locking in a heavily fluidic environment, electromagnetic approaches were not considered further.
Electric Actuation Electric actuators such as piezoelectric, ultrasonic, and dielectric elastomeric actuators require high voltages to actuate, thereby precluding their use inside the body due to the potential for electrical arcing and shorting through human tissue which poses obvious safety concerns. While piezoelectric materials are high-bandwidth (with the maximum operating frequency limited by the mechanical resonance of the actuator), they are are stroke-limited by the tolerable electric field, and as such, can require hundreds to thousands of volts to produce usable work outputs which has obvious negative implications on use inside the body. Despite the need for a substantial electric field, piezoelectric materials are in fact used in some interventional medical devices (such as harmonic scalpels) which place the actuators in an insulated handle outside the body, and the force/stroke from the actuator is transmitted to the distal end through a ceramic mechanical waveguide. This approach is obviously not applicable for flexible systems. Therefore, due to the high electric field requirement and limited stroke, piezoelectric approaches were not considered.
Fluid-based Actuation Hydraulic and pneumatic actuators leverage fluid pressure to drive a piston or expand an elastomeric chamber, thereby creating usable work. In conventional piston-based actuation
48
systems, strain is limited only by the design and mechanical construction of the actuator. Newer approaches in mm-scale hydraulic actuation utilize elastomeric materials that are mechanically programmed to actuate in designed directions due to strain-limiting materials, such that when the actuator is pressurized with the working fluid, the strain-limiting layer restricts motion in certain undesired directions (Russo et al., 2017). These modalities are strain-limited by the material properties of the elastomeric chamber. While such modalities can generate large forces and strokes per our target application, as shown in Fig. 3.1, they are comparatively slow. In addition, they require compressors, vacuum pumps, regulators and solenoid valves to control in closed loop which drives up the hardware requirements for implementation.
Magnetostrictive Actuation Magnetostriction refers to expansion/contraction of ferromagnetic materials under longitudinal magnetic stress. Although capable of large output forces, such actuators are strain-limited by magnetic saturation and typically exhibit magnetostrains up to about 2000 ppm (0.2%) which is several orders-of-magnitude lower than what our application requires (Huber et al., 1997).
Phase-Changing Actuation Shape-memory alloys (SMAs) and polymers (SMPs) undergo a temperature-induced phase change which increases the bulk material’s effective modulus when heated above or cooled below the material’s transition temperature. For SMAs, the shape memory effect arises from the introduction of plastic deformation that becomes elastic upon heating, allowing the alloy to return to its original shape. For SMPs, glass transition or melting of the crystalline phase affects the change in modulus. A fundamental difference in operation arises from the direction of the temperature-dependent modulus change: SMAs become stiffer at higher temperatures, whereas SMPs soften. Intrinsic SMP formulations exhibit high recoverable strain (up to 400%), low density, low cost, and low recovery stress (up to several MPa). SMPs can be treated to increase the recovery stress by several orders of magnitude (up to 130 MPa), however this substantially increases processing complexity and cost while reducing recovery strain (Xie, 2011). A limitation in recovery stress generation is a significant barrier towards adapting SMPs for mm-scale actuation schemes.
49
SMA actuators, on the other hand, have fairly low recoverable strain (less than 10%) and high recovery stress (up to several hundred MPa). However, SMAs can be shape-set to exhibit fairly high effective∗ recovery strain while still meeting effective recovery stress requirements for our application. SMA actuation presents several advantages over other candidate technologies, most notably the low-cost, simplicity, commercial availability, compact footprint and high energy density. Heating can be achieved by passing an electrical current through the material (requiring relatively inexpensive drive circuitry) which gets thermomechanically converted into a tensile strain or stress (depending on boundary conditions) for single-sided operation. For cyclic operation, a reset mechanism is needed such as an elastic biasing spring or another SMA actuator (an ‘antagonistic’ configuration which enables bidirectional motion). As SMA materials are extremely low impedance (with a resistivity of around 100 µΩ·cm, compared to a worst-case resistivity of ∼175-405 Ω·cm for human soft tissue, indicating a six order-of-magnitude difference), required voltages are low and there is negligible risk of electrical shock as the SMA itself will always be the lowest-impedance path to ground in the case of direct contact with human tissue (Faes et al., 1999). However, this low resistivity means substantial heat is generated during SMA operation, so thermal insulation or active cooling is necessary to prevent thermal damage to tissue. Due to the high energy density, inexpensive commercial availability, straightforward processing, performance tunability, and fairly simple power electronics requirements, we selected helical SMA actuators for distal integration as this modality most readily meets our performance index requirement. However, in the interest of actuator design, it can be deceitful to rely on model-based techniques alone to design the actuators and optimize their performance due to the complex thermomechanical phenomena that govern the behaviors of SMAs. As such, we implement an approach that combines phenomenological modeling and experimental validation to converge on and identify a configuration that meets functional requirements. ∗ For
effective strain, we normalize by the shape-set length ls which may differ from the length of the wire itself. For the case of a helically-wound actuator, ǫe f f = ∆ls /ls and ls = lw dw /πDs , where lw is the wire length, dw is the wire diameter, and Ds is the spring coil diameter.
50
3.3 SMA Modeling and Fabrication Towards actuator development and performance prediction, this section describes the derivation of a high-fidelity phenomenological model for coil-based SMA actuators. The model is used to inform the parametric design of actuators which are validated in an experimental setting.
3.3.1
A Brief Primer on Shape Memory Alloys
Shape Memory Alloys (SMAs) comprise a subset of unique phase-changing materials that can recover apparent permanent strains when heated above a certain transition temperature. This ‘shape memory’ effect can be exploited to create actuators that are heat-activated. Shape memory material exhibits two stable states, a high-temperature austenitic state (characterized by tight cubic packing symmetry, lending to high-stiffness pseudo-elastic behavior below a critical transition stress and strain is fully recovered), and a low-temperature martensitic state (characterized by packing asymmetry, lending to low-stiffness, nonlinear behavior with residual strain upon unloading). An exemplary loading curve for a helical SMA actuator comprised of a Nickel-Titanium alloy (Nitinol) is shown in Fig. 3.2. The loading phases are described in detail below: 1-2 Martensitic Detwinning (1-way SME, Shape Memory): Starting in the unstressed (σ0 = 0), unstrained (ǫ0 = 0) and low-temperature (LT) state (T < M f ), the actuator is mechanically strained until twinned (self-accomodated) martensite is converted into detwinned martensite. The loading curve shape is characterized by martensite detwinning start and finish stresses (σs and σ f , respectively). A relatively low stress is required to rearrange the atoms in the lattice during the detwinning process, as bonds are not broken. 2-3 Unloaded Detwinned Martensite (1-way SME): Mechanical loading is removed and, while some springback occurs, there is substantial residual strain as the martensitic structure has been deformed and finds a new stable state upon stress relief. 3-4 Austenitic (Reverse) Conversion (1-way SME): Heating the actuator above the austenitic start and finish temperatures (As and A f , respectively) relieves strain and causes the actuator to recall its original shape as the detwinned martensite is converted into austenite. Austenite only has one stable state (body-centered cubic) that exists at high temperature (T > A f ). 51
Stress σ Nickel Titanium
Detwinned Martensite Twinned Martensite
T 0, T0 = Tamb , ξ 0 = 1, ξ S0 > 0, ξ T0 = ξ 0 − ξ S0 ). At this point, the motion stage was locked in place to prevent any contraction of the specimen, thereby fixing the strain. A known current was applied at 200 mA intervals from 200 mA to 1.2 A for 60 seconds. The process was performed for n = 3 times at each current setting for a total of N = 18 tests. The resulting forces and temperatures were recorded at a sample rate of 2 kHz. The experimental conditions for fixed strain, variable temperature characterization are summarized below: (1)
• Initial Conditions: {τ (1) = τ0 , ∆(1) = δ0 , γ(1) = γ0 , T (1) = Tamb , ξ 0
(1)
= 1, ξ S
= ξ S0 ,
(1)
ξ T = 1 − ξ S0 } • Controller(s) Used: Current Controller • Controlled Input(s): i(n) = id is constant ∀n > 0 • Physical Constraint: ∆(n) = δ0
∀n > 0
• Degrees of Freedom: { F (n) , T (n) } • Parameters of Interest: { Tmax , Fmax , A f , As , τA f , τAs , τ50% } Results and Discussion Fig. 3.9 (top) shows the family of curves generated by this parametric study, where applied current is plotted on the x −axis, time on the y−axis, and temperature on the z−axis. The black line is the solution to Equation 3.8, the blue line is the experimental mean and the error bars correspond to the standard deviation for n = 3 trials. Fig. 3.9 (bottom left) plots the time to reach the austenite start and finish temperature (As and A f ), and Fig. 3.9 (bottom right) plots the steady-state temperature. We see that the model scales well both parametrically and temporally, allowing us to predict the thermodynamic behavior with a reasonable degree of fidelity. In analyzing the data further, we can draw important conclusions regarding the phase dynamics as they correspond to changes in actuator temperature. A high-level takeaway is that the rise time can be arbitrarily selected based on the temperature which is linear with the applied current, as expected from the steady-state form (dT/dt = 0) of Equation 3.8: 65
Thermodynamic Characterization, dw = 5007m, Ds = 2:5mm, N = 20 Model Experimental Max Force (mod) Max Force (exp) As Time (mod) As Time (exp) Af Time (mod) Af Time (exp)
Temperature [C]
150
100
50
1 0.8 0.6 0.4 0.2 0
Current [A]
10
0
Time [s]
35
140 As As Af Af
30 25
Time (mod) Time (exp) Time (mod) Time (exp)
120
Temperature [C]
Transition Time [s]
30
20
60
50
40
20 15 10
100
80
60 Model Experimental
5 0 0.2
0.4
0.6
0.8
1
40 0.2
1.2
Current [A]
0.4
0.6
0.8
1
1.2
Current [A]
Figure 3.9: Thermodynamic characterization with fixed strain: (top) experimental and theoretical temperature plotted vs. applied current and time (error bars correspond to 1σ for n = 3, and the plane denotes the transition region), (bottom left) phase transition time vs. applied current, and (bottom right) steady-state temperature vs. applied current. Steady-state actuator temperature scales linearly with applied current as the analytical thermodynamic model suggests.
T (t → ∞) =
IV + T∞ hc Ac
(3.27)
More specifically, the time spent in the phase transition region (As < T < A f ) is a direct function of the applied current, and in the interest of speed and bandwidth, it is preferable to 66
Thermodynamic Characterization, dw = 5007m, Ds = 2:5mm, N = 20
Force [N]
10
5
0 1
0.8
0.6
0.4
0.2
Applied Current [A] Model
Experimental
0
50% RT (mod)
10
0
30
20
60
50
40
Time [s]
50% RT (exp)
50
Maximum Force (mod)
Maximum Force (exp)
10 Model Experimental
9
Maximum Force [N]
50% Rise Time [s]
40
30
20
10
8
7
6 Model Experimental
0 0.2
0.4
0.6
0.8
1
5 0.2
1.2
Current [A]
0.4
0.6
0.8
1
1.2
Current [A]
Figure 3.10: Thermomechanical characterization with fixed (100%) strain: (top) experimental and theoretical force plotted vs. applied current and time (error bars correspond to 1σ for n = 3, (bottom left) 50% rise time vs. applied current, and (bottom right) maximum blocking force vs. applied current.
spend as little time in this region as possible. We also note that at low currents (200 mA) the actuator never leaves the transition region, reaching a steady-state temperature of about 50C. In Fig. 3.10 (top), we are plotting the blocking force as a function of applied current and time, where once again, the black line is the solution to Equation 3.15, the blue line is the experimental mean, and error bars correspond to standard deviation over n = 3 trials. Fig. 3.10 (bottom left) 67
plots the theoretical and experimental 50% rise time (or the amount of time it takes to reach a blocking force of 4.75N), while Fig. 3.10 (bottom right) plots the theoretical and experimental blocking force. While the force generation dynamics are not as accurately captured in the transition region, we observe diminishing returns after about 600-800 mA of applied current in terms of force generation.
3.4.3
Variable Strain, Constant Temperature (Force vs. Stroke)
The series of experimental tests described in this section serve to validate the thermally steady-state form of Equation 3.21 where temperature is held constant and strain is varied.
Experimental Setup The optimized actuator was clamped into the evaluation setup, initially in a strain/stress-free configuration at a constant temperature (generating the initial conditions τ0 = 0, γ0 = 0, ξ 0 ∈
{0, 1}, ξ S0 = 0, ξ T0 = ξ 0 , T0 > Tamb ). The temperature controller was turned on to a desired temperature setpoint Td . Once steady-state thermodynamics were reached, a triangular strain profile ∆(t) was applied with the evaluation system stage, ramping from 0 to δ0 and back to 0 with a ramp speed of 2 mm/sec for a total of n = 3 loading/unloading cycles at Td . Temperature Td was varied in 10C increments from 20C to 100C, and the loading/unloading experiments were repeated, generating N = 24 tests. The force, current, temperature, and displacement were captured at 2 kHz. The experimental conditions for variable strain, constant temperature characterization are summarized below. (1)
(1)
(1)
• Initial Conditions: {τ (1) = 0, ∆(1) = 0, γ(1) = 0, T (1) = Td , ξ 0 = ξ T0 , ξ S = 0, ξ T = ξ T0 } • Controller(s) Used: Temperature Controller, Position Controller • Controlled Input(s): ∆(n) = δ0 tri( f n), T (n) = Td is constant ∀n > 0 • Physical Constraint: Same as Controlled Input • Degrees of Freedom: { F (n) } 68
• Parameters of Interest:{ Fmax , ǫr } Results and Discussion Fig. 3.11 (top) shows both the loading and unloading behavior as a function of temperature and specimen strain. We observe the nonlinearity and hysteresis at low temperatures are adequately captured in both model (Equation 3.21) and experimental data, as martensitic detwinning leads to pseudo-plastic deformation above τcrit . At higher temperatures, the hysteresis and nonlinearity both decrease as the phase morphology is primarily austenitic, and the actuator approaches a linear spring in behavior. Fig. 3.11 (bottom left) shows the residual strain (i.e. pseudo-plastic deformation) as a function of temperature. This is found by observing where the force drops to zero on the unloading curve. This phenomenon occurs at lower temperatures because stress-induced detwinned martensite is stable at lower temperatures once the stress is removed. At higher temperatures, detwinned martensite becomes unstable, and as soon the applied stress becomes sub-critical, detwinned martensite converts to austenite which increases the effective shear modulus of the actuator. This occurs at about 60C which is consistent with the expected transition temperature based on manufacturer specifications (this is also the temperature required to generate the maximum output force, shown in Fig. 3.11 (bottom right)). It is also worth noting that, for the design stroke, the austenitic actuator never encounters the critical transformation stress τst , and linear-elastic behavior is observed over the entire stroke.
3.4.4
Variable Strain, Variable Thermodynamics (Antagonism)
The series of experimental tests described in this section serve to validate Equations 3.20 and 3.21 where both temperature and strain are varied, and the constraint equation in 3.22 is solved to simulate antagonism.
Experimental Setup To experimentally quantify antagonism, we command the evaluation stage to simulate a bias spring (with a spring constant equal to that of the detwinned martensitic spring constant) using
69
Thermomechanical Characterization, dw = 5007m, Ds = 2:5mm, N = 20
Force [N]
10
Model Experimental Residual Strain (mod) Residual Strain (exp) Maximum Force (mod) Maximum Force (exp)
5
0 100 80 60 40 20
Temperature [C]
2
0
12 Model Experimental
5
10
Maximum Force [N]
Residual Strain [mm]
6
Displacement [mm]
6
4 3 2
8
6
4
1 0 20
4
10
8
Model Experimental
40
60
80
2 20
100
Temperature [C]
40
60
80
100
Temperature [C]
Figure 3.11: Thermomechanical characterization with variable strain at constant temperature: (top) experimental and theoretical force plotted vs. temperature and strain (error bars correspond to 1σ for n = 3, (bottom left) residual strain (pseudoplastic deformation) vs. actuator temperature, and (bottom right) maximum force vs. actuator temperature.
the virtual spring controller shown in Fig. A.12. The experimental conditions are summarized as follows: (1)
• Initial Conditions: {τ (1) = τ0 , ∆(1) = δ0 , γ(1) = γ0 , T (1) = Tamb , ξ 0 (1)
ξ T = 1 − ξ S0 } 70
(1)
= 1, ξ S
= ξ S0 ,
• Controller(s) Used: Current Controller, Virtual Spring Controller • Controlled Input(s): i(n) = id is constant ∀n > 0 (n)
• Physical Constraint: Fa
(n)
= Fb
(antagonism force balance)
• Degrees of Freedom: { T (n) , ∆(n) } • Parameters of Interest:{ǫmax , τǫ,50% }
Results and Discussion As shown in Fig. 3.12 (top), the application of more current results in a faster dynamic response as expected. We also observe that the model fits the experimental behavior quite well at higher temperatures and, while less accurate at lower temperatures, still provides an accurate prediction of the steady-state stroke. The antagonistic actuators are able to generate ∼ 80% usable strain given 100% pre-strain. Fig. 3.12 (bottom left) depicts the rise time which, similarly to the blocking force results, decays exponentially as a function of applied current. In Fig. 3.12 (bottom right) we see that an application of about 0.4-0.6A allows the actuation scheme to achieve full stroke, meaning the agonist actuator is austenitic enough to overcome the biasing force of the martensitic antagonist. This agrees with the results of the blocking force test given in Fig. 3.10. As such, in any closed-loop control methodology, we can improve control efficiency by continuously running 0.4A through the actuators (even when not actuated) such that the temperature ‘hovers’ around the transition region, enabling a quicker response.
3.4.5
Forced Convection Cooling
While the actuation time is somewhat arbitrarily dependent on the applied current as seen in Fig. 3.10 (bottom left), SMA actuators are inherently rate-limited by their cooling properties (i.e. by the time it takes for T to drop below M f , signifying a full forward conversion to twinned martensite). Cooling dynamics are not as easily controllable, and many designers forego forced cooling entirely due to increased system complexity and depend on free convection in air to cool the actuators which severely limits achievable bandwidth. 71
Thermodynamic Characterization, dw = 5007m, Ds = 2:5mm, N = 20
Stroke [mm]
10
5
0 1
0.8
0.6
0.4
0.2
Applied Current [A] Model
Experimental
50% RT (mod)
0
10
0
Time [s]
50% RT (exp)
Maximum Stroke (mod)
Model Experimental
Maximum Stroke [mm]
16 14 12 10 8 6
8
7.5
7
6.5 Model Experimental
4 2 0.2
Maximum Stroke (exp)
8.5
18
50% Rise Time [s]
30
20
60
50
40
0.4
0.6
0.8
1
6 0.2
1.2
Current [A]
0.4
0.6
0.8
1
1.2
Current [A]
Figure 3.12: Thermodynamic characterization with antagonistic strain dynamics at variable current: (top) experimental and theoretical displacement plotted vs. current (error bars correspond to 1σ for n = 3, (bottom left) 50% rise time vs. actuator current, and (bottom right) maximum stroke vs. actuator current.
We can substantially improve actuator speed by forcing a working fluid over the actuator to dissipate heat through forced convection. The effects of different cooling modalities on actuator performance are illustrated in Fig. 3.13. A 5N load is hung from the midpoint of a 40-turn actuator (to simulate two 20-turn actuators in parallel), thereby creating pre-stress to convert some of the detwinned martensite into twinned martensite. A current is applied for two seconds to heat the 72
SwitchhCurrenthOn
SwitchhCurrenthOff
Coolinghto T100 dB. As such, the predicted effect is negligible. In addition, electrosurgical pulses often operate at 100’s of kHz, so contamination would likely be low-passed regardless.
91
Figure 4.5: Representative output from finite element modeling of the flux density field around a Neodymium permanent magnet.
increasing radial distance from the magnet, resulting in non-monotonicity which is undesirable for sensing systems as there is no longer a unique mapping from output signal to measurand. For this purpose, we exploit finite element modeling of the magnetostatic properties of permanent magnets to better understand how the motion of the Hall effect sensing element within the field will affect the output. We use the OctaveFEMM package which is an open-source MATLAB toolbox that employs variational formulations (based on minimizing energy) to solve the elliptic partial differential equation governing the magnetic vector potential:
∇×
1 ∇×A µ( B)
=J
B = ∇×A
(4.16)
(4.17)
where µ is the magnetic permeability, A is the vector potential, J is the current density, and B is the flux density. An example analysis is shown in Fig. 4.5. The numerical problem was formulated as a two-dimensional planar magnetostatics problem with Dirichlet boundary conditions at some radius R >> d, where d is the diameter of the magnet. The mesh was refined to 20 µm within the magnet, and 250 µm in free space. The flux density 92
(a)
(b)
(c)
Figure 4.6: FEM modeling of Hall effect-based proprioceptive sensor, where (top) shows the FEM simulation and kinematics, (middle) shows the component of the flux density field normal to the Hall effect sensor plane over the linkage stroke, and (bottom) is the resulting regression estimate for linear and quadratic feature spaces: (a) sensor placement results in non-monotonic field sensitivity over linkage stroke, (b) sensor placement results in monotonic, high sensitivity and nonlinear behavior over linkage stroke, and (c) sensor placement results in monotonic, low-sensitivity and more linear behavior over linkage stroke.
field was generated in FEMM and passed to MATLAB for post-processing and interfacing with the sensor linkage kinematics model. Only the normal component of the flux density field was assumed to contribute to the Hall voltage, as per Equation 4.5. Fig. 4.6 shows the FEM results for various locations of the Hall element with respect to the permanent magnet subject to the kinematics of the sensor linkage. We see that, in Fig. 4.6 (a), the sub-optimal sensor placement results in non-monotonic behavior due to the field misalignment.
93
Figure 4.7: Hall element optimization surfaces (where the black line denotes the optimized configuration): (left) normal component of the magnetic field to the Hall element over entire stroke, (right) magnetic field misalignment φ over entire stroke.
Fig. 4.6 (b) shows more desirable properties as the sensor output is both monotonic and highsensitivity. Fig. 4.6 (c) shows improved linearity at the cost of reduced sensitivity which decreases the signal-to-noise ratio.
Hall Element Location Optimization We observed how improper placement can result in reduced sensitivity or even non-monotonic behavior over the anticipated linkage stroke. This lends naturally to optimization, where the goal is to find a Hall element configuration that maximizes sensitivity, linearity and monotonicity. We formulate the following objective function Z:
maximize
Z=
Z θmax θmin
(B(c, θ ) · nˆ ) dθ
(4.18)
In other words, we want to maximize the integral of the magnetic flux density (normal to the plane of the Hall element) over the entire stroke of the sensor linkage. This objective function rewards high sensitivity and linearity while penalizing field misalignment (with respect to the Hall element normal vector) and subsequent non-monotonicity. This function was maximized in a brute-force manner by performing a parametric sweep over the FEM data for different values of c and choosing the value that maximizes Z. The curve families for both magnetic flux density B and field misalignment ψ are shown in Fig. 4.7. The black line denotes the sensor behavior at the optimized configuration.
94
4.4 A Force Sensor for Distal Force Feedback Force/tactile sensing is potentially useful in endoscopy for both diagnostic and interventional purposes, enabling tissue palpation and stiffness estimation, insertion force monitoring for perforation prevention, and haptic perception of the forces being applied to tissue during cutting/dissection. There is very little prior work in developing add-on modules to empower commercially-available endoscopes with force sensing capabilities. Examples from literature use the endoscope’s camera to detect deflections in an elastic structure to sense force through visual processing, and as such, visual field occlusion is a necessary construct of the transduction modality (Watanabe et al., 2014; Faragasso et al., 2014a,b). In addition, video processing algorithms can reduce system bandwidth. In this work, towards the aim of maximum transparency and minimum visual field occlusion, our desire is to develop a distally-mounted force sensor with a dedicated transduction modality that does not rely on visual processing of the endoscope’s camera feed. We mentioned previously that LIM is sub-optimal for large relative displacements where the optical transmission path can be difficult to mechanically encapsulate. However, in the context of a force sensor, the optical transmission path can be kept very small over the design strain for high signal-to-noise, and the low strain lends to small relative displacements which are easy to encapsulate at scale.
Mechanical Modeling A mechanical schematic of the proposed concept is shown in Fig. 4.8 (left). An infrared (IR) emitter is affixed to a beam-based flexural element which, when mechanically loaded, displaces the emitter with respect to two stationary IR phototransistors that are staggered to create a differential sensing methodology, wherein the signal rises for one detector and drops for the other detector due to the displacement of the emitter. To predict performance we require a joint model of the beam-bending mechanics and the optoelectronics phenomena. Assuming small deflection, we can use a first-order beam bending model to determine the emitter angle γ with respect to the horizontal plane:
γ=
F ( L − a )2 2EI 95
(4.19)
Mechanical Model F L-a a
Optoelectronics Model Irradiance Φ [mW/cm2] Φmax
ηLED(φ) detector 1
emitter
E,I
-φ 1
detector 2 g
φ Φ(d)
2
g
3
δx
d
β+ψ0 ψ0
c
1 δy
δy
Angular Sensitivity ηPT
c
δx
f
ψ0 f
γ
γ
ηPT(φ)
ψ0-γ
-φ
φ φ50%
Figure 4.8: Combined mechanical and optoelectronic model of differential light-intensity modulated force sensing methodology, where (left) shows the mechanical schematic and (right) shows qualitative irradiance and sensitivity dependencies on displacement and misalignment between emitter-detector pairs.
where F is the applied force, L is the beam length, a is distance from the end of the beam where the force is applied, E is the Young’s Modulus and I is the second moment of area. We can write the emitter’s displacement in x and y as follows:
δx = c sin (γ)
δy =
(4.20)
F (2L2 − 3L2 a + a3 ) 6EI
(4.21)
Ultimately, we are interested in the Euclidean separation between emitter and both detectors (d1 and d2 ) as well as angular misalignments φ1 and φ2 . Angular misalignments are given by the following:
φ1 = arctan
f + δy g + δx
+γ
(4.22)
φ2 = arctan
f − δy g + δx
−γ
(4.23)
96
PTlArray 500lCPRlEncoder
LEDlArray
8
Arduinolwith MotorlShield
6
4 Linearl Carriage
24:1lWorml GearlDrive
DClMotorlwith 100:1lGearhead
LED Array
2
Air Gap
Analogl Potentiometer
0 0
2
4
6
8
PT Array
(a)
(b)
Figure 4.9: Experimental setup to determine far-field behavior of infrared emitter/detector pairs: (a) experimental hardware where inset shows the array of opposing emitter/detector pairs, (b) collector current i PT vs. distance where the dotted black line shows the theoretical far-field cutoff.
The Euclidean separation is given by the following:
d1 =
q
( g + δx )2 + ( f + δy )2
(4.24)
d2 =
q
( g + δx )2 + ( f − δy )2
(4.25)
These linear and angular displacements will be plugged into the optoelectronics model presented in the next section to predict signal modulation due to an applied force.
4.4.1
Optoelectronics Modeling
The linear and angular displacements d• and φ• under load manifest as attenuation factors η in the optoelectronics model that are illustrated qualitatively in Fig. 4.8 (right). The angular attenuation can be extracted from the component-specific data sheets. The misalignment attenuation factor for the LED is adequately modeled by an inverse sigmoidal function with fit parameters α as given by the following:
η• LED =
α1 1 + exp (α2 − α3 φ• )
(4.26)
The misalignment attenuation factor for the PT is similarly described with fit parameters β:
97
η• PT =
β1 1 + exp ( β 2 − β 3 φ• )
(4.27)
We have shown experimentally that, assuming the emitter/detector pair is operating in the far field regime, the collector current of the phototransistor scales inversely with the square of the distance separating emitter from detector Gafford et al. (2016b). The experimental setup for determining the irradiance drop-off as a function of distance separating emitter/detector pairs is shown in Fig. 4.9 (a) where a custom linear stage was developed to displace an array of IR phototransistors from an opposing array of IR LEDs while measuring the current drained by the phototransistors. The resulting data in Fig. 4.9 (b) shows that the current does indeed fall off with the inverse square of the distance. Assuming the emitter of the phototransistor is in series with a gain resistor, we can solve for the generated voltage VPT :
VPT = R PT η• PT η• LED
γ1 Φmax + γ2 4πd2•
(4.28)
where R PT is the resistance value used to convert the collector current into a voltage, γ are curve fit parameters and Φ is the radiant intensity generated by the LED. This model can be used to predict the differential output voltage given an input load F.
4.5 Multisensor Fabrication and Calibration To fabricate the sensor linkage at the scale appropriate for distal implementation in endoscopy, we use PCMEMS. This section details the fabrication process.
4.5.1
PCMEMS Fabrication
The sensor laminate consists of eleven layers of material: two layers of 125 µm thick 304 stainless steel which form the mechanical substrate, one layer of 50 µm thick Kapton polyimide which forms the flexible layer that permits motion, one layer of 25 µm thick Kapton polyimide which forms an insulating layer and solder mask, two layers of 18 µm thick Kapton with an 18 µm thick Copper cladding which form separate ground and power/signal planes in a multi-layer flex cirtuit, and five layers of 25 µm FR0100 adhesive (duPont) to join all the layers together. The typical steps
98
Stainless Steel Kapton Adhesive Copper
1
2
E/D Pair 1 Assembly Slots
4
Temp Sensor
200 C Ambient Sensors
70 kg
3 E/D Pair 2
Assembly Tabs
2 hr
Figure 4.10: Printed-circuit MEMS fabrication of the at-scale sensor linkage: (1) lamination of individually lasermachined material layers, (2) release cuts to free sensor linkage from alignment scaffold, (3) pick-and-placement of electronic components for reflow soldering, (4) tab-and-slot guided manual assembly.
of this manufacturing process are more thoroughly described in Gafford et al. (2016c). Renderings of the fabrication process on a simpler (5-layer) laminate are shown in Fig. 4.10. The layers are individually machined in a diode-pumped solid-state (DPSS) laser and laminated together using heat and pressure (1). Release cuts are made in the DPSS laser (2) and the linkages are freed from their surrounding alignment scaffold. The top layer is a flexible circuit layer containing traces and land patterns for sensors and passive components which are soldered using reflow techniques (3). The sensor is then assembled as shown in (4) (where specifics are provided in Section 4.5.3). A detailed photograph of the fabricated sensor, post-release and populated with electrical components, is shown in Fig. 4.11 (a).
4.5.2
Electrical Implementation
An electrical schematic of the multisensor’s on-board electronics is shown in Fig. 4.11 (b). A 500 mA fuse provides protection against short circuiting, and a 5V Zener diode provides voltage regulation and protects against transient voltage spikes. A temperature sensor (MCP9700, Microchip Technologies) provides a ratiometric analog output for thermal compensation. Two Hall effect sensor ICs (A1326, Allegro Microsystems LLC) provide analog outputs corresponding to the incident magnetic field generated by two neodymium magnets in the linkage with a sensitivity of 2.5 mV/Gauss. The force sensor comprises a differential optoelectronic pair consisting of two IR LEDs and and two IR phototransistors, where a 220Ω resistor converts phototransistor collector current into an analog voltage output. Finally, programmable LEDs make it possible to provide visual feedback to the endoscopist. 99
Neodymium MagnetI1 p1b{O}
Hall-Effect SensorI1 p1a{O}
VHE1
L11
L12
Hall-Effect SensorI2 p2a{O}
VHE2
L21
Neodymium MagnetI2 p2b{O}
5mm
L22
5V GND
IRILEDIand PhototransistorI2
IRILEDIand PhototransistorI1
Vt Temperature Sensor
VLED
VFS1
Programmable LED
VFS2
(a) TemperatureF Compensation
FUSE3t5003mAi S1
+5V
Hall-Effect-BasedFProprioceptive Sensing
Light-IntensityFModulation-Based ForceFSensing 220
220 Hall-Effect
Temp3Sensor
Zener3t5Vi
Vin
220
220
Hall-Effect HE1
Temp
A1326LLHLT-T
A1326LLHLT-T
MCP9700T-E/TT
D1
Visual Feedback
HE2 U1
220
U2 F1
L1 220
F2 220 L2
VT
VHE
T
-θ
VHE
+θ
-θ
VLIM
+θ
L3
VLIM
F
F
(b)
Figure 4.11: Fabricated sensor: (a) combined proprioceptive (differential Hall effect), force (differential light-intensity modulation), and temperature (COTS IC) multisensor linkage, after component placement but prior to assembly (designed for implementation on a Fuji 450WR5 9.4mm diameter endoscope), (b) electrical schematic of on-board components, where qualitative plots show single-ended and differential sensing modalities.
4.5.3
Fold Pattern-Guided Assembly and Encapsulation
As the fabrication process is planar, we use fold patterns within the laminate structure to assemble the force sensor. This process is shown in Fig. 4.12. On the far left, we have the pre-assembled multisensor with components mounted and encapsulated in a silicone-based conformal coating, initially in the flat configuration as shown in insets (a) and (b). On one side (c), a series of mountain folds are made to enclose the emitter-detector air gap inside a box structure, and a valley fold brings the E/D plane perpendicular to the linkage plane. On the other side (d), a 100
(a)
(c)
Functional0Schematic
F
Detector01 Valley0Fold E/D0Plane
Flexural Beam
Mountain Fold Encapsulating0 Features
Emitter01
Emitter02
Emitter01,0Detector01
Detector02 2mm
2mm
F 10mm
(e)
(b)
(d)
Emitter02
Detector02
Valley0Fold E/D0Plane 2mm
2mm
2mm
Figure 4.12: Origami-inspired assembly of force sensor employing differential light-intensity modulation, where insets show fold sequence: (a) shows E/D side 1 in the initial (flat) state, (b) shows E/D side 2 in the initial flat state, (c) shows a valley fold to bring E/D plane 1 perpendicular to that of the flexural beam, and mountain folds to assemble the enclosure, (d) shows a valley fold to bring E/D plane 2 perpendicular to that of the flexural beam, and (e) shows the linkage assembly, where linkage kinematics and tab/slot features guide E/D sides 1 and 2 into their final opposing configuration.
single valley fold brings the E/D plane perpendicular to the linkage plane. Finally, as shown in inset (e), the linkage joints are folded out of plane, and the tabs on the top link are locked into corresponding slots, allowing the two emitter-detector pair planes to oppose each-other within the encapsulating structure. Note that the gaps within the encapsulating structure are small enough such that, in the case of full submersion in a fluid substance with properties similar to water, the surface tension will prevent the fluid from entering the enclosed air gap.
101
4.5.4
Calibration
Angle Sensor Calibration A jig was fabricated to enable calibration of the PCMEMS angle sensor linkage by coupling the kinematics of the sensor to a potentiometer which serves as the ground truth angle measurement. The experimental setup is shown in Fig. 4.13 (a), where insets show the extremes of travel. Although the differential configuration of the Hall effect sensors is quite effective at eliminating common-mode sources of noise, additional compensation can account for differences in the temperature coefficients between sensors. As such, the sensor integrates a ratiometric temperature sensor for thermal compensation.§ For convenience, we define feature space sets Ss = {s1 , s2 } ∈ R N ×2 where s1 , s1 are the emitter/detector output voltages, and N is the number of data points in the time series. Additionally, let Se = {st } ∈ R N ×1 , where st is the temperature sensor output voltage. We note that Ss ∩ Se = S ∈ R N ×3 (the union of the two sets comprises a complete data set) but Ss ∪ Se = ∅ (E/D data and environmental data are distinct and non-interfering). We study the following linear and quadratic feature space transformations: • Complete Data Linear Mapping S → X: We preserve a linear mapping for all sensor data, i.e. for so ∈ S, so 7→ a + bso = x. • Complete Data Quadratic Mapping S → Φ: We map all sensor data to a quadratic feature space: for so ∈ S, so 7→ a + bso + cso ⊙ so = φ (⊙ denotes element-wise multiplication). The proprioceptive sensor was calibrated in this setup by sweeping the input angle over the full range of motion and recording the multisensor and potentiometer outputs. The calibration results are shown in Fig. 4.13 (b). As expected, the quadratic feature space on the complete data set (Φ) offers the best estimate, as can be seen by observing the quality of fit of the magenta line which corresponds to θ (Φ HE ). The plot second-from-top shows that the combined analytical/finite element model very closely matches experimental performance. The bottom two plots demonstrate linearity of the different feature spaces, as well as some error statistics. § On a related note, the reader is encouraged to consult Appendix B where we discuss machine learning approaches to environmental disturbance rejection using a PCMEMS-manufactured three-axis force sensor as a case study.
102
-45 degrees
Pivot
Regression Results
3; 3^ [deg]
Potentiometer
50
^ HE ) 3(X ^ HE ) 3() 3
0 -50
Clamp
0
2
4
6
8
10
Hall Voltage [V]
Time [s] 3 HE1(e)
2
HE1(m)
HE2(e)
HE2(m)
1 0 0
2
4
6
8
10
Time [s] 50
3^ [deg]
+45 degrees
0
^ HE ) 3(X ^ HE ) 3() 3
-50 -50
0
50
3 [deg]
Counts
Sensor Linkage Strain Relief
200
0 -15
^ HE ) 3(X ^ HE ) 3()
-10
-5
0
5
10
15
Error [% FS]
(a)
(b)
Figure 4.13: (a) Test setup for calibrating to-scale PCMEMS angle sensor, where insets show the configuration at the extremes of travel, (b) snapshot of a calibration routine where a model is fit to the data using recursive least-squares estimation.
The proprioceptive sensor’s dynamic characteristics were measured as shown in Fig. 4.14 (a)-(c). Fig. 4.14 (a) shows the sensor’s electrical response to a sudden mechanical stimulus, and the sensor’s angle estimate actually leads that of the reference sensor due to damping and mechanical deadband within the reference potentiometer. However, it is clear that, given the rise time of both sensor and reference signals, the sensor can achieve bandwidths of more than 20 Hz which is an order of magnitude greater than the achievable positioning bandwidth of the SMA actuators. 4.14 (b) and (c) show the sensor outputs during the first 200 cycles of a repeated deflection test. 4.14 (c) shows the per-cycle RMSE compared to the batch RMSE, and the absence of any clear trend indicates that the sensor performance is unaffected through cyclic loading. Primary sources of fatigue include mechanical failure of the Kapton-based flexural hinges due to repeated loading, as well as cracking in the conductive traces routed across these hinges. As such, the hinge spacing was carefully selected such that the induced bend radius meets manufacturer specifications for long life. The sensor was also tested for its long-term, noise, and temperature-dependent characteristics, as shown in Fig. 4.14 (d)-(f). Fig. 4.14 (d) shows the induced error as a function of temperature
103
30
1.8
50 3 3^meas ()) 3^F EM ())
3 [deg]
3 [deg]
10 0
0
-10 3 3^ Peak
-20 -30 10.16
10.18
10.2
10.22
10.24
-50 80
10.26
80.4
80.6
Time [s]
(a)
(b) 0.2 Fit Data
80.8
Power/Frequency [dB/Hz]
Error [deg]
Error [deg] 6
8
0.05 0 -0.05 -0.1
10
12
Data Fit
T ! T1 [C]
(d)
50
100
150
200
150
250
Time [s]
(e)
300
350
400
1=f Noise
20 0
200
Sample Sample Sample Sample
1 2 3 4
-20 -40 -60 -80
Noise Floor -100
0
100
Fundamental
-0.2 4
50
40
Fit: e(t) = (1:3 # 10!5 )t + (44 # 10!5 )
-0.15 2
0
(c)
-0.6
-0.8
1.3
Number of Cycles
0.1
Fit: e("T ) = !0:05("T ) ! 0:02
1.4
81
0
-0.4
1.5
1.2
0.15
-0.2
1.6
1.1 80.2
Time [s]
0.2
0
Per-Cycle RMSE Batch RMSE
1.7
RMSE per Cycle
20
10-1
100
101
102
Frequency [Hz]
(f)
Figure 4.14: Dynamic, cyclic and transient characterization data for the proprioceptive sensor: (a) sensor response to sudden input stimuli (where the FEM-predicted performance is shown for comparison), (b) snapshot of cyclic data around 200 cycles showing a good sensor estimate, (c) per-cycle RMSE compared to batch RMSE demonstrating negligible signal degradation after 200 cycles, (d) angle drift as a function of temperature (data and fit), (e) long-term drift (data and fit), and (f) power spectral density for a batch of four sensors.
change, denoting a thermal coefficient of 0.05 deg/C. Long term drift was also tested, as shown in Fig. 4.14 (e), and the slew rate was found to be 0.05 deg/hr. Finally, the power spectral densities Pxx for four different samples in a batch are shown in Fig. 4.14 (f), demonstrating similar noise characteristics across different sensors. The fundamental (input calibration signal) can be seen at around f sig = 1 Hz (denoting a signal power of around 36 dB), and the noise falls off with increasing frequency (1/ f noise) until a noise floor is reached at about 300 Hz (Johnson noise). The total noise power was estimated by integrating the remainder of the spectral density through the noise floor (frequencies greater than 10 Hz and less than 5 kHz). The signal-to-noise ratio (SNR) was found to be around 60 dB. The angle sensor’s specifications are tabulated in Table 4.1 and compared to desired specifications as per previously-derived functional requirements. The fabricated prototype boasts high signal-to-noise, low error and good fatigue performance. The observed hysteresis is primarily 104
Table 4.1: Proprioceptive sensor specifications using linear regression with quadratic features where the green fill denotes satisfaction of functional requirements. (Note: resolution is peak-to-peak, hysteresis and linearity are worst-case)
Parameter
Design Goal
Angle Sensor
Units
Angular Range Angular Error (RMSE) Resolution Signal-to-Noise Hysteresis (max) Linearity (max) Bandwidth Cycles to Failure Temporal Drift Thermal Drift
±45 2.00 0.50 40 2.0 4.0 10 200 0.5 0.1
±45 1.53 0.19 61 3.3 5.2 >20 >500 0.05 0.05
deg deg deg dB %FS %FS Hz N/A deg/hr deg/C
due to the reference potentiometer which was empirically observed to have a slight deadband of 2-3 degrees, however the sensors themselves could have small hysteretic behavior due to slack in the Kapton flexural joints at the extremes of travel, as well as fractional resistance changes in the circuit traces over the articulation range that add a series resistance to the gain resistors used to amplify signals. As such, the tabulated hysteresis is conservative and the actual value is likely much lower. Regardless, this experimental analysis shows that the manufactured angle sensor meets most of the desired specifications.
Force Sensor Calibration The force sensor was calibrated by using a commercial load cell (Futek LSB302) to load the flexural beam shown in Fig. 4.12 (e) in approximately 100mN increments and performing ridge regression on the resulting phototransistor outputs using a linear feature space [v f 1 , v f 2 , vt ] = XED . An exemplary multisensor (angle, force) calibration is shown in Fig. 4.15, where the left column shows the angle sensor regression results (using a quadratic feature space) and the right column shows the force sensor regression results (using a linear feature space). We observe some deviation from the reference force due to viscous damping within the structure. However, the sensor is able to sense sub-Newton forces, as the resulting regression elucidates. The force sensor’s specifications are tabulated in Table 4.2. Note that, as the work is still preliminary, more work must be done to 105
Angle Calibration 3 ^ HE ) 3()
0
F F^ (XED )
1
F [N]
3 [deg]
Force Calibration 1.5
50
0.5
-50
0 0
2
4
6
8
10
5
10
4
20
25
20
25
0.2
2 0 -2 -4
0 -0.2
0
2
4
6
8
10
5
10
40 20 0 -20 -40
R2 : 0.997 RMSE: 1.53 deg Nonlinearity: 5.16% Resolution: 0.19 deg
-40
-30
-20
-10
0
15
Time [s]
F^ (XED ) [N]
Time [s]
^ H E ) [deg] 3()
15
Time [s]
Error [N]
Error [deg]
Time [s]
10
20
30
1 R2 : 0.986 RMSE: 35.9 mN Nonlinearity: 12.0% Resolution: 7.39 mN
0.5
40
0 0
0.2
0.4
3 [deg]
0.6
0.8
1
1.2
F [N]
Figure 4.15: Combined multisensor calibration results: (left, top to bottom) regression results [θ, θˆ(Φ HE )] vs. time, angle regression error vs. time, and angular regression linearity plot, (right, top to bottom) regression results [ F, Fˆ (XED )] vs. time, force regression error vs. time, and force regression linearity plot. Table 4.2: Force sensor specifications using linear regression with linear features where the green fill denotes satisfaction of functional requirements. (Note: resolution is peak-to-peak)
Parameter
Design Goal
Force Sensor
Units
Force Range Force Error (RMSE) Resolution Signal-to-Noise Linearity (max)
0-2 100 10 40 10
0-2 40 7.4 49 12
N mN mN dB %FS
characterize the force sensor’s transient and long-term characteristics, as we have done with the angle sensor.
4.5.5
Multisensor Demo Module
To demonstrate the multisensing capabilities of the distal robotic module, a peripheral sensor feedback module was custom manufactured using laser-cut acrylic. An Arduino Uno acquires raw sensor data with an analog resolution of 10 bits and multiplies by the appropriate calibration matrices. An LED ring shows the angular position of the distal module, a separate LED bar 106
Angle Feedback Temp Feedback
Force Feedback
Figure 4.16: Multisensor demonstration module where plots show real-time on-board sensor readings.
provides force feedback, and a four-digit seven-segment display provides temperature feedback. Images of the multisensor demo module, as well as plots of the real-time multisensor feedback, are shown in Fig. 4.16. The module can be deployed as a peripheral system for providing sensor feedback to the endoscopist during the proceure.
4.6 Conclusion In this chapter, we present a monolithically-fabricated multisensor that combines displacement, force and temperature sensing into a form factor that can be deployed distally on an endoscopemounted robotic module for sensory feedback. The chapter began with an overview of printedcircuit MEMS and a discussion on its applicability in the manufacturing of sensing systems for surgical robotics and medical devices. An angle sensor consisting of a differential pair of Neodymium magnets and Hall effect sensors transforms actuator motion into analog output voltages for real-time position feedback. A force sensor implements a differential infrared emitter/detector scheme and the principle of light intensity modulation to sense forces applied to the sensor structure. Printed-circuit MEMS was employed to fabricate the to-scale prototypes which were experimentally calibrated to determine performance specifications in terms of linearity, resolution, accuracy, bandwidth and fatigue. The manufactured sensors met or exceeded most of the functional requirements. While the force sensor work is preliminary, the angle sensor is well specified and will form the basis of closed-loop control work detailed in subsequent chapters.
107
Chapter 5
System Integration 5.1 Introduction Earlier sections of this thesis describe the design of mm-scale actuators (Chapter 3) and sensors (Chapter 4) on a component level with regards to the functional requirements derived in Chapter 2. This chapter covers the system-level design and fabrication of front- and back-end hardware to realize a complete, fully-deployable, endoscopy-suite-ready system for robotically-assisted ESD. System-level design includes the following: • Design and fabrication of the distal module itself with regards to sensor/actuator implementation considerations and clinical requirements • Ergonomic design of master input device for seamless, intuitive clinical integration • Design of power electronics systems for providing high-current digital control of on-board SMA actuators • Design of power electronics for driving pumps and solenoid valves for fluid cooling • Integration of power electronics with real-time embedded computation for high-bandwidth control loop implementation and data acquisition • Design of hardware and software for controlling the distal module both in open-loop (teleoperation) and closed-loop (trajectory execution)
108
Feedback
Endoscope Feed
Endoscopist
Input Devices
System Data
Patient
Hardware
Distal Module
Software
RT Controller Signal Conditioning Master Controller Power Electronics, Hydraulics
Figure 5.1: Illustrative overview of cooperative control scheme, highlighting systems-level hardware and software components and their relation to the distal module and the endoscopist.
Fig. 5.1 shows how all of these components integrate in the context of cooperative control to create a fully deployable cooperative system that works in tandem with the endoscopist to perform complex therapeutic procedures endoscopically.
5.2 Distal Module Design Towards distal integration, a tip-mounted module was carefully designed to establish the following functional requirements: 1. Creation of a loading path from the deflected tool to the endoscope body via the actuators through mechanical grounding to the endoscope itself 2. Creation of a mechanical ground serving as inertial frame for the sensor linkage kinematics 3. Coupling of the sensors to the actuators via mechanical engagement 4. Creation of an interface or manifold for fluidic cooling of the integrated actuators 5. Creation of mechanical and electrical connections for the SMA drive signals and sensor feedback signals 109
Neodymium Magnet
θact θest
0
PassiveCSMA CActuator
Hall5Effect Sensor FluidCFlow
Linkage Structure
10
-50 0
2
4
6
8
AntagonisticCActuatorCCharacterization
10
TimeC[s]
EndoMODRA
StrokeC[mm]
θact,θestC[deg]
ProprioceptiveCSensorCCharacterization 50
5
θestC[deg]
50 0
θact θest
0
-50 -50
1 0.8
RMSE:CxB)xCdeg SNR:Cx%5CdB Resolution:C7Bx7Cdeg 0
50
θactC[deg]
0.6 0.4 0.2
KevlarCPullwire
AppliedCCurrentC[A] ActiveCSMA CActuator
Model
Experimental
57pCRTC:mod1
0
0
57pCRTC:exp1
10
20
30
50
40
60
TimeC[s] MaximumCStrokeC:mod1
MaximumCStrokeC:exp1
Figure 5.2: Principle of operation: (middle) shows the design of the distal module, highlighting the antagonistic actuation scheme and coupling to the proprioceptive sensor through the transparent deflector plate (a U.S. Penny provides scale), (left) shows the summarized experimental proprioceptive sensor characterization data from Chapter 4, and (right) shows the experimental antagonistic actuation data from Chapter 3.
Commercial Endoscope Electrical and Fluid Connections
Wire Guide
Tighten Set-Screw
Tool Deployment and Deflection
Figure 5.3: Module attachment to endoscope: (left) the module is slid onto the distal end and the feeder channel of the deflection plate is aligned with the working channel of the endoscope, (middle) set screws can be (optionally) tightened for a more secure fit, and (right) the tool can be deployed into the deflection plate.
6. Creation of an ergonomic mounting interface to the endoscope A functional schematic of the fully-integrated distal module, designed to meet the aforementioned functional requirements, is shown in Fig. 5.2 (middle), where Fig. 5.2 (left) and Fig. 5.2 (right) summarize the actuator and sensor work discussed in Chapter 3 and 4, respectively. The module consists of two hollow chambers surrounding a central hollow bore, each with a 40-turn helical SMA actuator bent back on itself to realize two 20-turn actuators placed in parallel. Each chamber has an inlet for fluid flow which is routed to the module from proximal solenoid valves via 1m long, 2mm diameter flexible silicone tubing. Fluid leaves each chamber passively through holes and cutouts in the encapsulating shells. The inner bore is sized to ensure a snug press-fit onto a standard endoscope (with a diameter of up to 10.5 mm), and setscrews provide further engagement if necessary to prevent slipping. The distal attachment process is illustrated in Fig. 5.3. The module is 40mm long and 20mm in outer diameter.
110
20 Complimentary Kalman Filter
Fbias
9
Force [N]
50
δ
15
8 7
r kt(θ)
0
6
5
Force [N]
Articulation [deg]
θ
y-dimension [mm]
10
p
0 -5
5 4 3 2
-10
a
1
Fact
-15 0
δ
h
-20
-50 0
Fr
2
4
0
6
Time [s]
10
20
30
x-dimension [mm]
(a)
(b)
Figure 5.4: Module articulation modeling: (a) moment balance about central pin joint with relevant geometric and mechanical variables, (b) overall angle of articulation measured with an IMU (left) and theoretical maximum lateral force generation measured over the reachable workspace (right).
The actuators are coupled to each-other via Kevlar pullwire connections to a pin-jointed transparent deflector plate, realizing the antagonistic actuation scheme. As the active actuator contracts, the deflector plate pivots about a central pin joint, with the bias actuator providing the restoring force. The sensor linkage sits between the module body and the transparent deflector plate, with the ground link fastened to the distal module body, and the crank link fastened to the pin joint link, thereby coupling the crank link to the actuators and allowing the coupler and follower links to passively displace the emitters with respect to the detectors within the sensor linkage.
5.2.1
Mechanical Analysis
Referring to the mechanical schematic shown in Fig. 5.4 (a), we can compute the net force Fr by considering a simple moment balance about the pin joint, denoted p:
∑ M p,z = 0
(5.1)
= Fr (h + a) − Fact re
−µθ
+ Fbias re
−µθ
+ Ftool a
where r, a, and h are geometric parameters as shown in Fig. 5.4 (a), Fact is the instantaneous SMA agonist contraction force, Fbias is the instantaneous antagonist bias force, Ftool is the reaction force of the flexible tool, µ is the coefficient of friction between the Kevlar pullwire and the module body material, and θ is the angle of articulation. Note that we assume that the tendon actuation 111
force is perpendicular to the transparent deflection plate, but capstan losses are encountered upon angled entry into the fluid-filled chambers, thereby introducing the e−µθ term. Re-arranging, we have the net force left over for electrosurgical contact:
Fr =
i r h ( Fact − Fbias ) e−µθ − k t (θ )θa a+h
(5.2)
where we’ve plugged in k t as some linearly-variable tool stiffness as given by the quadratic approximation in Chapter 2:
k t ( θ ) = a k ( θ ) + bk
(5.3)
where ak and bk are curve fit parameters. The actuator and bias forces are functions of δ (and, accordingly, deflection angle θ) as given by the following:
Fact = Ka [δ0 − r sin(θ )]
(5.4)
Fact = Kb [δ0 − r sin(θ )] − β b (δ0 , θ )
(5.5)
where we repeat here the antagonistic force model previously derived in Chapter 3. Note that here we assume quasi-steady behavior in that the active actuator is fully austenitic and behaves like a high-stiffness linear spring with some pre-stretch δ0 . The bias model is more complicated as we must consider the conversion to detwinned martensite at high stresses as a function of instantaneous deflection θ and pre-stretch δ0 . Fig. 5.4 (b) shows the theoretical net force over the predicted workspace of the module. This is the force ‘left over’ to interact with tissue. Recall from Chapter 2, Fig. 2.15 that around 400 mN is necessary for electrosurgery.
5.2.2
Manufacturing and Assembly
As we target single-usage, an approximate cost breakdown of the distal module is given in Table 5.1. We observe that the constituent materials are fairly low-cost, and most of the manufacturing cost is
112
Table 5.1: Approximate cost breakdown of single-use robotic module. (∗ : based on the average hourly wage of a medical device assembly technician)
Part Number
Part Description
Quantity
Cost per EA
Cost Total
DM01 DM02 DM03 DM04 DM05 DM06 DM07 DM08 Various Various 1883T4 E1002S-1000-ND T1244-305-ND 609-3799-ND 209ME-ND 909GME-ND N/A N/A
3D-Printed Body 3D-Printed Shell 3D-Printed Relief Collar 3D-Printed Joint 3D-Printed Clamp Laser-Cut Acrylic Deflector SMA Actuator Sensor Laminate SM Electronics Fasteners Fluid Piping (1m) 22 AWG 2 Cond. Wire (1m) 30 AWG 4 Cond. Wire (1m) 4 Cond Screw Terminal 9-Pos Male DSUB Connector 9-Pos DSUB Backshell Sensor PCMEMS Module Assembly
1 2 1 2 3 1 2 1 1 1 2 2 2 1 1 1 5 hrs 5 hrs
$0.93 $0.18 $0.21 $0.05 $0.23 $0.14 $1.65 $8.00 $2.02 $3.51 $4.74/m $0.16/m $1.07/m $0.48 $0.47 $0.57 $15.00/hr∗ $15.00/hr∗
$0.93 $0.36 $0.21 $0.10 $0.69 $0.14 $3.30 $8.00 $2.02 $3.51 $9.48 $0.32 $2.14 $0.48 $0.47 $0.57 $75.00 $75.00
in manual assembly which could potentially be streamlined in a high-throughput setting.∗ Design modifications could substantially mitigate labor and drive down assembly time by replacing 3D printing with injection molding or CNC machining, obviating manual soldering through quick-disconnect electronic connections and pick-and-place automation of electronic component placement, snap-fit fastener-less assembly, and automated SMA winding and batch annealing.
5.3 Human Factors Design of Master Input Device Towards clinical integration, the user experience (i.e. how the clinician interacts with the system) is of utmost importance. As we desire seamless workflow integration, we must carefully consider system ergonomics to ensure a comfortable user experience, both physically and cognitively. If the clinician is not comfortable using a device, or if the device requires substantial time to set ∗ Labor
estimates are based on start-to-finish assembly of a single module with no task parallelization and, as such, are conservative.
113
up, the clinician will not want to use it regardless of its efficacy. Furthermore, poor ergonomics can lead to degenerative injuries from repetitive microtraumas to tendons, ligaments and joints. Repetitive injuries associated with endoscopy include high pinch force, repetitive hand activities, awkward postures, and contact stresses (Shergill et al., 2009). Therefore, human factors must be carefully considered to prevent unnecessary trauma to the patient and the clinician. Many advanced endoscopic systems presented in Chapter 1, particularly Category 2 systems, suffer from poor ergonomics in that multiple operators are required to perform procedures that are normally performed by a single operator. Ergonomics are especially important when considering add-on devices that interface with devices already in use, as poor ergonomics matching could increase the cognitive burden as the clinician must commit to memory multiple input mappings to produce desired output effects. In designing control interfaces for add-on interventional endoscopy, we identified two primary ergonomic barriers that could potentially prevent the system from experiencing clinical adoption. (1) The presence of the device hinders the clinician’s ability to do his/her job comfortably, effectively, or with familiarity (poor clinical workflow integration). (2) The device is well-integrated, but its use is so awkward or complicated, the clinician does not want to use it (good workflow integration, poor ergonomics matching). Through discussions with clinical collaborators and several design iterations, in addition to procedural observations, we developed a master input controller that interfaces seamlessly with commercial endoscopes. The design evolution is shown in Fig. 5.5, where red and green arrows show the preservation or termination of certain features between designs. The final controller design mounts onto the working port inlet of the endoscope through positive mechanical engagement, thumbscrews and an adjustable Velcro strap. The form factor of the device ensures that it does not interfere with standard endoscope controls, providing uninhibited access to the manual control knobs and insufflation/irrigation controls. The robotic control joystick is integrated with tools passed through the working port, allowing the endoscopist to use his or her left pinky or ring finger to control robotic deflection, as shown in Fig. 5.6. This leaves the right hand free to insert/retract/roll the endoscope, or insert/retract tools through the working port. A
114
ManualbPositioning Knobs
Insufflationband Irrigation Control Working Channel
StandardbFuji Endoscope
ModebSelection andbFeedback
Version 1 ex-vivobj EndoscopeJMounted Binaryb1ForwardIReverse2bControl Nobmodebselect OnebprogrammablebpushJbutton BiJmanualImultiJuserboperation
Version 2 ex-vivo 2 HandleJMounted AnalogbControlIAutoJExecute ModebSelect ThreebprogrammablebpushJbuttons BiJmanualboperation
Robot Positioning Joystick
Robot Controller
Version 3 HandleJMounted AnalogbControlIAutoJExecute ModebSelect ThreebprogrammablebpushJbuttons Unimanualboperation
Figure 5.5: Ergonomic master input device design process showing morphological and feature evolution as guided by clinical consultation and ex vivo testing.
Figure 5.6: Images showing integration and use of robotic module controller, where (left) shows conventional endoscope control, and (right) shows robot-assisted control.
toggle switch on the controller allows the user to switch between control modes, and a series of tactile switches on the reverse side allow the execution of different control trajectories.
5.4 Embedded System Overview Making the leap from laboratory benchtop to clinic for ex-vivo and in-vivo testing requires a portable control electronics system capable of providing the necessary current to power the SMA actuators, driving solenoid valves for active cooling, processing on- and off-board sensor data, and running control loops in real-time at high bandwidth. Towards this aim, a custom controller and driver 115
system was developed and manufactured that combines all of the aforementioned functions while adopting a ruggedized, mobile form factor, as shown in Fig. 5.7 (a). The integrated controller combines a high-current SMA driver with custom-designed power and signal conditioning electronics, a relay-based fluid logic driver and pump power module, on-board sensor signal conditioning, high-resolution data acquisition, a dedicated 12V, 30A power supply and a singleboard computer for open and closed-loop control execution. A block diagram of the proposed controller structure is shown in Fig. 5.7 (b). The system is powered by a 12VDC, 30A power supply which plugs directly into a 120VAC power outlet. A PC104 stack consisting of an Aurora single-board computer (SBC), VGA Module, dedicated stack power supply, and MM-32DX-AT DAQ module (all from Diamond Systems, Inc.) implements real-time control and data acquisition using MATLAB/Simulink xPC real-time operating system (RTOS). The stack provides control outputs to both the integrated SMA driver and hydraulics driver modules, and receives sensor feedback from the SMA current/self-sensors as well as the distal module’s on-board proprioceptive sensors. The embedded computer runs off of a dedicated power supply (Jupiter-MM-LP, Diamond Systems) to decouple from the master 12V, 30A power supply which provides high currents to the on-board SMA and hydraulics drive electronics. An integrated VGA monitor provides real-time visualization of the target control parameters and sensor feedback signals. An Arduino-controlled thermal regulation system monitors two temperature sensors placed within the SMA and hydraulics drive electronics and controls the speed of two cooling fans accordingly to prevent overheating. This information (system temperature and fan speed) is displayed on a small LCD display. A front panel-mounted emergency stop cuts power to the peripheral (high-current) modules (SMA driver and hydraulics driver) without cutting power to the embedded controller, which can only be turned off via the rear panel-mounted master power switch. The host machine is any computer running MATLAB/Simulink xPC real-time environment (in this case, a Dell Precision M4800 laptop running MATLAB/Simulink 2015a) connected to the target via TCP/IP ethernet protocol. Embedded system specifications are tabulated in Table 5.1.
116
Ethernet,,Power (Rear)
Valve,Outputs (Rear)
Real-Time Monitor Auxilliary E-Stop Thermal SMA,Out Controller Sensor,In
Filtered Fan,Inlet Control,In
(a) FillHTankFH SolenoidHValves
HostHComputer withHMATLABHxPC
Ethernet Connector
120 VAC
Integrated Controller/Driver
Analog/PWM
AnalogHBreakout
DedicatedHPS VGAHModule AuroraHSBC
DigitalHBreakout
xPCHTarget monitorHwithH sensorLcontroller feedback
Arduino Thermal Regulation
TempH2
FanH2
Hydraulics ElectronicsLRelays TempHI FanHSpeedH Controller
FanHI
PCI54HStack DAQHModule
Valve Outpts
Fan 2 PWM
SMAHDriveL CurrentLSelf Current/Self Sensing SensorHElectronics
12V,030A Power0Supply SMA0 Driver
Fan 1 PWM
12 VDC
5 VDC
VGA
Ethernet
12 VDC
I2VHRegulated PowerHSupply
PC1040SBC Stack
LCD Controller Inputs
Sensor Inputs
SMAHOutputs
Real-Time Monitor
FluidHLines I2VHPowerHLines 5VHPowerHLines DataHLines HMI
Relay0 Board
Thermal Controller
Distal Module
(b)
(c)
Figure 5.7: Custom embedded system and power electronics design: (a) photograph of the embedded system enclosed in a custom electronics enclosure, (b) block diagram showing the system’s components, and (c) a photograph of the embedded system with the top shell removed, showing the internal components.
5.4.1
Controller Enclosure
The controller enclosure was designed to be moisture-proof and ruggedized prior to ex- and in-vivo experiments where potential interaction with fluids and detritus is a concern. The enclosure was designed from 0.078" powder-coated galvannealed steel sheet metal and manufactured professionally by Protocase. Captive nuts enable assembly and self-clinching threaded fasteners 117
Table 5.2: Embedded system specifications
Feature
Value
Unit
Notes
Supply Line Voltage Boosted Voltage (SMA Driver) Dedicated PC104 PS Voltage Processor Speed Random Access Memory Max (Continuous) Current Draw Max (Continuous) Power Draw No. Analog Inputs No. Analog Outputs No. SMA Drive Channels SMA Drive PWM Frequency No. Fluid Valve Drive Channels Controller Bandwidth
120 12-24 12 1.6 2.0 14 168 12 4 2 50-800 8 5000
VAC VDC VDC GHz GB A W N/A N/A N/A Hz N/A Hz
N/A Fixed 25 Watt Intel Atom Z-Series RSODIMM Fused 12VDC 16 bit resolution 16 bit resolution PWM out SW or HW Programmable Optoisolated, Relayed Maximum
provide mounting surfaces for components. All fasteners and viewports are gasketed for moisture sealing, and fan inlets/outlets are filtered and louvered for additional moisture protection. A high-current 4-terminal barrier block provides an electrical connection to the distal module’s SMA actuators, and a 9-pin D-Sub connector provide electrical connections to the on-board sensor and the control input. Fig. 5.7 (c) shows the fully-populated controller with the top shell removed to show each of the controller subsystems.
5.5 Hydraulic System Design In Chapter 3 we motivated the inclusion of forced fluidic cooling to achieve practical actuation speeds for control. We demonstrated that forcing a fluid medium over the actuators can improve cooling speed by two orders of magnitude over air cooling. In this section we discuss the design, analysis and implementation of the hydraulic cooling system to deliver fluid to the actuator chambers of the distal module.
118
5.5.1
Power Electronics
To control forced convection cooling, we toggle the on/off state of hydraulic solenoid valves that are placed in series with the submersible pump and selectively deliver fluid to the distal module’s actuator chambers, as shown schematically in Fig. 5.8 (a). The power electronics used to digitally control the on/off states of pumps and solenoid valves for fluid cooling are shown in Fig. 5.8 (b) and (c). A logic-level input drives an optically-isolated flyback-protected relay which connects the high side of the valves and pumps either to 12VDC or ground depending on relay state. As the relay board is configured such that a low-level signal turns the relay ‘on’, we add an extra pull-up resistor to the digital input to force floating signals to 5VDC (such that the pumps/valves are in a ‘normally off’ state when the control module is powered on). Each relay is also wired to an RGB LED strip, where the strip is configured to shine red when the relay is closed, and blue when the relay is open. This gives a visual representation of the valve states through the EndoMODRA panel cutout on the front face of the metal enclosure. The system has eight relay outputs to control more valves and pumps in support of future work in hydraulically-actuated retraction and stabilization structures, discussed more thoroughly in Chapter 8.
5.5.2
Fluid Dynamics Analysis
In designing the hydraulics system and selecting the appropriate pump size, we must account for major losses due to friction within the pipes, as well as minor losses due to fittings, valves, and diameter reductions. Assuming incompressible flow, the steady-state form of the Bernoulli equation between points (A) and (B) in Fig. 5.8 (a) is given by: v2 v2 pA p + A + z A = B + B + zB + hL − h p γ 2g γ 2g
(5.6)
where vα is the fluid velocity at point α, pα is the fluid pressure at point α, γ is the specific weight, g = 9.81m/s2 is the gravitational constant, zα is the elevation at point α, h L is the loss and h P is the pump head. We simplify this by noting that p B = p A = p ATM , and we assume v A = 0. As such, we can express the required pump head as follows:
hp =
v23 + le + h L 2g 119
(5.7)
+ 5V
+ 12V
R1
Distal Module
20 0 RY1
U1
V1
B
RY1 +
Q, PATM
CH1
-
RY2
vKexitR + 5V
d3
RY3
+ 12V
R3
le
RY4
20 0
3 R4 2k
FillDTank
LuerDFittings DvK4R
SubmersibleDPump v215W,D1560DGPHR
Q, PATM
V2
RY5 +
CH2
-
ine lL
o ntr
Reduction vK1R 2
A
RY2
U2
+ 5V
Co
+ 12V
d1 PP
RY6
RY7
V2 R5
1
Light Bar for Relay Status Visualization
R2 2k
lm
RY8
20 0
d2 Solenoid ValvesDvK3R
R6 2k
RY3
U3
P1
V1
vKentR
+
CH3
-
T-JunctionDvK2R
(a)
(b)
(c)
Figure 5.8: Hydraulic system for providing forced water cooling to distal module: (a) hydraulic schematic, (b) electrical schematic (shown here providing control inputs to two hydraulic solenoid valves and one 12VDC submersible pump), and (c) hydraulics control relay board integrated into master controller.
We also note that only one chamber is ever being flushed at a time. As such, we can block one valve and only consider flow through the other valve. Incompressible flow states that flow rate through the system is constant:
Q = v1
πd21 πd2 πd2 = v2 2 = v3 3 4 4 4
(5.8)
where Q is the flow rate and di is the pipe diameter of section i. To solve for the flow rate required, we need to approximate the desired flow rate at the inlet of the distal module. Assuming the module is held vertically at the full extent of the endoscope’s length (worst-case), we want the fluid jet to have enough velocity to adequately wet the SMA chamber, with a safety factor to account for p modeling inaccuracies. Given this, we are designing for an inlet velocity of v3 = 4glm (in other
words, the jet of water entering the module should reach a maximum height of two times the module length lm to ensure adequate wetting of the actuator chamber). The desired volumetric flow rate is as follows:
120
p
Q=
4glm
πd23 4
(5.9)
We sum major head losses h f due to friction within the pipes and minor head losses hm due to fittings, valves and nozzles: h L = h f + hm
=∑ 3
=
v2 f lv2 + ∑ KL 2dg 2g li v2
1
∑ fi 2diig + 2g
i =1
(5.10)
Kentry v21 + K1 v22 + K2 v22 + K3 v22 + K4 v23 + Kexit v23
where K are the loss coefficients and f i is the Darcy friction factor which differs between laminar and turbulent flow and can be approximated via the following:
fi =
64
if Rei ≤ 2300
Rei
0.316Re−1/4 i
(5.11)
if Rei > 2300
where Rei = (ρvi di )/µ is local the Reynolds number of the flow through pipe section i, ρ is the density and µ is the viscosity. Combining equations and solving for h p , we require a pump that can generate at least 3.5m of head. As such, we selected the EcoPlus 1584 submersible pump which is capable of generating 3.8m of head.
5.6 SMA Power Electronics and Control To control each SMA digitally, we need a means of using logic-level, low-current (10’s of milliamps) analog voltage signals from the PC104 DAQ module to control relatively high (several amps) currents through the SMAs. Ideally we would like an analog implementation of a perfectly linear transconductance amplifier (where output current is a linear function of input voltage), and while these exist in monolithic form, the output is dependent on load resistance which is potentially variable depending on the electrical resistance properties of the SMA actuators which change as a function of temperature. A current mirror is another possibility, however the linear mirroring capabilities are highly dependent on transistor matching and load resistance as well. 121
Analog design of voltage-controlled current sources that are insensitive to thermal gradients and load resistances is a challenging problem, especially when substantial output currents are required. Therefore, due to the anticipated variable load resistance of the SMA actuators and the requirement for high (>4A) transient current capabilities, we seek digital control by leveraging the slow thermodynamics of the actuators themselves and using MOSFET-based current amplification and pulse-width modulation (PWM) where a low-level current control loop is closed via a microcontroller. The actuators themselves effectively act as thermomechanical low-pass filters, and high-frequency (>100 Hz) PWM current inputs get averaged in the thermodynamic conversion via Joule heating, thereby imitating an analog current input. This amounts to using a constant voltage source and switching the voltage on and off from this source to the SMAs at a high frequency (relative to the SMA thermomechanical bandwidth) while using a known resistor in series to limit the current. Modulating the time the switch is ‘on’ vs. ‘off’ in a single cycle (i.e. changing the duty cycle) will modulate the average current across the SMA actuators.
5.6.1
Theoretical Justification for PWM Control
As described more thoroughly in Chapter 3, the phase volume fraction ξ (and, accordingly, the force/displacement characteristics) of an SMA actuator is a direct function of the actuator’s temperature. If we consider the simplified thermodynamics of the SMA actuator under Joule heating, we define ∆ T (t) as the instantaneous SMA temperature with respect to ambient temperature T∞ (which is assumed to be constant) as below:
∆ T (t) = T (t) − T∞
(5.12)
The associated derivative is simply the following:
∆˙ T (t) = T˙ (t)
(5.13)
The thermodynamics in the time domain are given by:
mC p ∆˙ T (t) = i (t)V − hc Ac ∆ T (t)
(5.14)
where m is the actuator mass, C p is the specific heat, i (t) is the applied current, V is the voltage 122
across the actuator, hc is the convective heat transfer coefficient, and Ac is the surface area over which convection takes place. We can take the Laplace transform to represent the thermodynamics in the frequency domain with respect to an imaginary frequency variable s = jω:
mC p ∆ T (s)s = i (s)V − hc Ac ∆ T (s)
(5.15)
Re-arranging, we have the transfer function relating input current i to SMA temperature ∆ T in the frequency domain: ∆T (s) V = i(s) mC p s + hc Ac
(5.16)
We observe that this transfer function has the same form as that of a first-order low-pass filter with a cutoff frequency equivalent to:
f c,SMA =
hc Ac 2πmC p
(5.17)
For practical values of hc , Ac , m and C p , this cutoff frequency sits somewhere between 0.1-1 Hz which is more than two orders-of-magnitude lower than the proposed PWM drive frequency. As such, PWM is a simple and appropriate drive scheme to achieve quasi-analog temperature control of SMA actuators.
5.6.2
Power Electronics
Custom power/sensing electronics, shown schematically in Fig. 5.9, were designed and fabricated to enable digital control of the average current in two actuators simultaneously via PWM. As the embedded SBC stack runs off of a 12V supply, an off-board boost converter boosts the SMA driver supply voltage to 24V which, in series with a 2Ω power resistor Rr , the SMA actuators RSMA , and lead resistance Rl (resulting in a total of 3Ω-4Ω resistance), can sink more than 6A across each SMA simultaneously for rapid heat-up and improved control. The SMA driver has two on-board drive options: PWM and analog. In PWM drive mode, the PWM signal must be generated by the master controller (in this case, the SBC stack and multifunction DAQ) which subsequently drives the gates of the N-channel power MOSFETs through optical coupling. This approach is not ideal, as the PWM signal must be created point123
a
tk
a
t o
t o W
ADfoaN
ICo ADlo N
f
Rso
n
n
m
WVS
OFF
s *VS *
OFF
Rto t k
REF WV
o
m
oo
Rtf
Rtt o k
Rtl t k
m
SpiceOrder o
Da tNntnlDOsmWa
SpiceOrder o
tk
a
Rst
Rtm t k
tk
Ra
SpiceOrder t
s o
t o
tuF
m n
XlWn SelfWSenseVo
t_mk
t V
Do
t_mk
Roa
oo
Rta
t V
Dt
tuF
Rol
Ca
Cf SMAVtVIn
TLPols OptoVIsolation
XlWs SelfWSenseVt
t f St
s
o
Rs
Dn tNntnlDOsmWa XaWo Midpoint PickoffVo JPs
f
l RG
SpiceOrder t
St
oo
oo
WV
f
Qt
Rtn
Ro
XoWo
o
JPo
ICm
t RG
ADfoaN
REF
Rts XmWo
tk
tk Rf
oo Rl
m
Rt n
XmWt
XoWt
OKt t
XaWt Midpoint PickoffVt
SMAVoVIn
_tuF oo
t RG
l RG
s o t
Ct
*V
_tuF ICn
n
JPt
*V
Cl
s
oo Rt o k
o
VO GND
s
Co
s
LEDt
VI
LEDs
XtWt
R9
Cs
t uF t uF
ICt LMsn HW m t
XtWo
XfWt onVVIn FromVBoostXfWo
XnWo toVVOut XnWtToVBoost
Fo
X9Wt ToVVoltmeter X9Wo
LEDo
XsWo FanVOut XsWt
s k Rs
s k Ro9
OKo n s
l a
IP*
VCC VOUT
l a
BW_SEL GND IPW
f m
BW_SEL GND IPW
f m
CSt XttWt
AMPt
Cm
o
XlWt CurrentVSenseVt
m
XlWo CurrentVSenseVo
t
a f
MCPf oP
tuF
So
f So
Rom
CSo
s
ICaA
n
t
s k Ros
o
MCPf oP
Cn
AMPo
tuF
LEDm
LEDn
Ro
oo tk
Roo
oo
Rt9
Low-Current))Isolated Drive High)Current)Drive and)Sensing Self-Sensing Signal)Conditioning Current)Sense Signal)Conditioning Voltage Regulation Boost-Conversion)and Peripherals)(Off-Board)
Qn BSSt t
Qs BSSt t
tk
ICaB
m
s k Ron
t_mk
XttWo
Xt Wo
Rot
Rof ICs
VCC VOUT
IP*
s9o
tk
Rm
tk
Rn
ICf
l
t9o
toV
Df
toV
Dm
mV
mV
D9
Xt Wt
t9o
t_mk
TLPols OptoVIsolation
s9o
o
Dl
s
Qo
n
t
Figure 5.9: Electrical schematic of dual-channel SMA power and signal conditioning electronics, where colored boxes denote different stages of the power electronics methodology.
to-point, and there is a direct performance trade-off between PWM frequency and duty cycle resolution (i.e., for a 2 kHz master controller sample rate, a 100Hz PWM signal is characterized by 20 discrete points, affording 5% duty cycle resolution, whereas a 200Hz PWM signal affords only 10% duty cycle resolution). In analog drive mode, an Arduino-based accessory board (shown on the left in Fig. 5.10) takes in an analog signal varying from 0 to 5V from the multifunction DAQ and locally converts this signal to a 800 Hz PWM output with a duty cycle that varies between 0% and 100% proportionally with 8-bit resolution (i.e. 0.4% duty cycle resolution) independent of the master controller sample rate. This locally-generated PWM signal is then fed into the driver board directly, thereby offloading the burden of generating a digital PWM signal from the master embedded controller. In addition, it should be noted that using a much higher PWM frequency means we can use a significantly less aggressive low pass filter, making the system much more responsive to dynamic inputs. The effective (average) current passed through the SMA is given by: DC i¯ = 100
Vboost RSMA + Rr + Rl
(5.18)
where DC is the commanded duty cycle in percent, and Vboost is the voltage generated by the
124
boost converter (24V). Therefore, by varying the duty cycle, we can digitally vary the (average) current to each SMA to any arbitrary value between 0 and 6A. This forms the basis of a lower-level current loop based on current sensor feedback that will be used in both open-loop control and closed-loop position control. As the actuator’s electrical resistance changes as a function of phase, precision current control cannot be performed open-loop, and as such, we require current feedback to ensure consistent average current delivery. A 5V regulator provides power to two Hall effect-based ratiometric current sensors which measure the current through each SMA. An alternative approach to sense current is to measure the voltage drop across a shunt resistor. However, as our application calls for power delivery on the order of 60 W, heat dissipation is a concern and any shunt resistor must be fairly large. As this signal is a square wave with a fixed amplitude and varying duty cycle, the board features user-tunable low-pass filters to convert the square wave into an average signal as given by Equation 5.18. Future iterations of power electronics could use low-level interrupts at the microcontroller level to measure the pulse width of the current sensors and, accordingly, the duty cycle, thereby staying in the digital domain for a more accurate reading. Although self-sensing was not explored in this work, the electronics were designed to enable phase-based feedback by measuring the electrical resistance of the actuators. The midpoint of each SMA and power resistor are fed into an instrumentation amplifier and referenced to an adjustable DC offset, forming a Wheatstone bridge. This approach was chosen over other resistance measuring approaches in literature which pick off the voltage on either side of the SMA actuator and send it to a differential amplifier, as such an approach would produce a large DC offset and limit sensitivity. In a Wheatstone bridge morphology, as the resistance of the actuator changes due to phase transformation, the bridge balance is upset, generating a differential voltage offset from zero that is amplified to be processed by a data acquisition unit. This signal is essentially a square wave with a duty cycle equal to that of the PWM wave driving each SMA, where the amplitude varies as a function of SMA resistance. As this signal is not ideal for processing, there are a few methods to convert it into a meaningful DC-level signal for control purposes. The first is to digitally pick off the peak of the signal by software triggering data acquisition to the drive PWM signal. Another approach is to use analog components to 125
Vboost
24VvBoostv Converter
BoostvV
oltage vMeter
SMAv1
SMAv2
Power Resistors
Sigv1 ControlvInv FromvPC104 Sigv2
Envelope FiltervTuning
PowervSupply 12V GND Input
Low-Pass FiltervTuning
SensorvOut TovPC104
CoolingvFans
SMAv1vAmmeter
SMAv2vAmmeter
Figure 5.10: Image showing the custom power electronics PCB and analog-to-PWM microcontroller board and their respective connections to auxiliary hardware.
detect the signal envelope and pass this directly into the controller. Towards this end, the board features user-tunable analog envelope detectors for picking off the amplitude of the self-sensing signals for processing in the master controller. Potentiometers allow the user to select a cutoff frequency from 5Hz to 100Hz. On-board analog filters and envelope detectors can be enabled or disabled by a switch, allowing for software-based filtering if necessary. Although the functionality is there, self-sensing of SMA resistance is beyond the scope of this thesis and is a subject for further investigation towards the implementation of a low-level power limiting controller that can prevent the actuators from overheating. An integrated fuse limits the current to the board so as not to damage the boost converter. In addition, several Zener diodes are implemented as voltage clamps to (1) clamp the voltage input to the optoisolators at 5V to prevent over-current to the LED, (2) clamp the gate voltage to each N-channel power MOSFET at 12V (half the boost supply), (3) clamp the sensor voltage outputs at acceptable levels (5V for current sensors and 10V for self-sensors) so as to not damage the ADC inputs of PC-104 multifunction DAQ card, and (4) protect against transient voltage spikes.
126
OutputMAmmeters SMA Out
BuckMConverter OffsetMAdjust Knobs
Boost-Converter OutputVoltage Ammeters
Cooling-Fans (Rear)
Cooling Fans
Adjustment Power-In Slider Variable PowerMResistor PWM/Current/ SelfMSenseMD-Sub
Current Sensors
PowerMIn
SMA-Out(Rear)
Inst.MAmp GainMAdjust PWM/Current/ Self-Sense-D-Sub
(a)
(b)
Figure 5.11: Two generations of SMA power electronics and custom laser-machined acrylic enclosures: (a) Generation 1 (low-power) with maximum 3.5A output simultaneously, PWM-only drive mode and no on-board filtering, (b) Generation 2 (high-power) with maximum 6A output simultaneously, PWM and analog drive mode, and on-board filtering.
In an earlier iteration of the power electronics module, shown in Fig. 5.11 (a), components were wired and soldered by hand using jumper wire and protoboards. The early version was PWM-only and had a max output current of about 3.5-4A per actuator (7-8A simultaneously). The power electronics were redesigned, including the addition of the Arduino-controlled analog-toPWM conversion board, and PCBs were fabricated professionally by OSH Park. Fig. 5.10 shows the fully-assembled boards as well as their connections to peripheral components. The power electronics are enclosed within a fan-cooled acrylic enclosure with one digital voltmeter showing the voltage output of the boost converter, and two digital ammeters showing the average current through each SMA, as shown in Fig. 5.11 (b). Input power connections can be made via banana plug input or a screw terminal block. A 9-pin D-SUB connector provides an interface for control input and sensor output signals.
5.6.3
Performance Benchmarking
SMA driver specifications are summarized in Table 5.3. This section describes driver performance when implemented as a low-level current controller.
127
Table 5.3: SMA power electronics specifications
Feature
Value
Unit
Notes
No. SMA Output Channels Supply Voltage Boosted Voltage (Optional) Absolute Max Current Input Absolute Max Transient Power Draw Design Max Transient Output Current Per Channel On-Board PWM Frequency On-Board Analog-to-PWM Resolution Current Sensor Low-Pass Filter Range Self-Sensor Envelope Filter Range Current Sensor Sensitivity Current Sensor Zero-Offset Current Sensor Resolution (filtered) Current Sensor 1 X-talk Current Sensor 2 X-talk
2 12 24 20 200 6 800 8 5-80 5-80 480 0.53 10 0
K− =
,
if e(t) ≤ 0
0
K−
if e(t) ≥ 0
(6.5)
if e(t) < 0
Note that, in some instances, co-contraction might be desirable to increase the overall stiffness of the system. This is something that must be determined empirically. The resulting control surfaces are shown in Fig. 6.3. We note that they are symmetric about e˙(t) = 0. As suggested in Kha and Ahn (2006), we limit the universe of discourse for each output variable {K¯ p , K¯ d , K¯ i } ∈ range(0, 1) to obtain feasible rule bases with high efficiency. We convert these outputs into usable gains by 139
Discretized Fuzzy PID Controller (Agonist)
theta in Scheduled Gains Gains
Proportional Gain Scheduler Agonist Fuzzy Inference Fuzzy
butter
2 pos_d
K (z-1) Ts z
Velocity Smoothing
Kd*de(t)
de(t)
e(t)
K Ts (z+1) 2(z-1)
int e(t)
Discrete-Time Integrator
1 i_d
Kp*e(t)
Discrete Derivative
Saturation2
Dead Zone
Ki*int e(t)
Agonist Gain Select
3 error butter Velocity Smoothing1
e
v1
de
v2
4 valve out
Flushing Rule Base
5 Gains
Discretized Fuzzy PID Controller (Antagonist) butter
K (z-1) Ts z
Velocity Smoothing2
de(t)
Kd*de(t)
Discrete Derivative1 e(t) Kp*e(t)
1 pos_a
K Ts (z+1) 2(z-1)
int e(t)
Discrete-Time Integrator1
Fuzzy
Saturation1
Dead Zone1
2 i_d2
Ki*int e(t)
Antagonist Gain Select
Gains Scheduled Gains theta in
Antagonist Fuzzy Inference
Proportional Gain Scheduler1
Figure 6.4: Simulink-implemented control architecture for FPID/PWM position control of antagonist SMA actuators with gain scheduling and a crisp valve control rule-base for forced convection cooling.
scaling them according to the following: K p = K¯ p (K p,max − K p,min ) + K p,min Kd = K¯ d (Kd,max − Kd,min ) + Kd,min
(6.6)
Ki = K¯ i (Ki,max − Ki,min ) + Ki,min where K·,max , K·,min define the gain range. For this application, we have K p ∈ (0, 2), Ki ∈ (0, 1), and Kd ∈ (0, 0.01).
140
6.3.4
Gain Scheduling
Referring back to Equation 5.2 in Chapter 5, the net force is maximized when θ act = 0 due to the geometry of the system (namely, capstan losses as the system articulates), and falls off with e−µθact for increasing magnitudes of θ. As such, less current needs to be applied to the agonist actuator near θ act = 0 as this is the area where the actuator has the most mechanical advantage. To compensate for this nonlinearity, we implement gain scheduling to effectively modulate the proportional gain based on the value of θ act as per the following equation:
K p,gs = K p e−µ(θmax −θact )
(6.7)
where θmax = 45 is the maximum single-sided deflection of the articulating system, and µ = 0.5.
6.3.5
Implementation
The fuzzy inference engine was designed using MATLAB’s Fuzzy Logic Designer toolbox using a Mamdani inference model. To speed up real-time execution and enable high sampling rates, the control surfaces were generated offline using the previously-described rule bases, inference, fuzzification and defuzzification processes, and exported as 2D matrices to be stored on the real-time target to be quickly accessed. Given some combination of inputs {e(t), e˙(t)}, each control surface was linearly interpolated at each time step to generate the variable gain output. The Simulink implementation of the FPID/PWM controller is shown in Fig. 6.4. The Agonist/Antagonist Fuzzy Inference blocks perform control surface interpolation depending on the sign and magnitude of control inputs {e(t), e˙(t)} to output the scaled fuzzy gains {K p , Kd , Ki }. Note that, for the antagonist actuator, we invert the sign of the error term such that positive currents are generated for negative errors. The rule-based valve controller is encoded as a custom MATLAB function within the controller. The outputs are once again clipped to prevent negative currents and overcurrents, and sent to the current PID/PWM controller for each SMA.
6.3.6
Benchtop Results
An actuator-only module was attached to the experimental setup shown in Fig. 6.5 (a), where control parameters are tuned heuristically using the GUI shown in 6.5 (b). Fig. 6.6 shows the 141
Mechanical Breadboard
DistalhModulehBody EndoscopehDummy
ATIhNano17
FluidhCatch SerieshSpring SyncromeshhPulley andhCable
Potentiometer
(a)
(b)
Figure 6.5: Closed-Loop Controller Design: (a) experimental platform for off-board closed-loop controller design and verification, (b) GUI for tuning controllers and executing trajectories.
results of the fuzzy controller for various reference trajectories, both smooth and discontinuous. We observe in Fig. 6.6 (a) that the controller is quick to converge to steady-state with a rise time of about 0.4 seconds (implying a bandwidth of 0.88 Hz), exhibits a maximum speed of 75 degrees/second, has a steady-state positioning error of less than 0.06 degrees (which is well within the error limits of the on-board sensor), and there is minimal overshoot for large (>30 degrees) reference inputs. In addition, the width of the shaded error region (standard deviation over three tests) indicates that control is very repeatable. We also observe the controller’s ability to track rate-based profiles, as shown in Fig. 6.6 (c), and smooth trajectories, as shown in Fig. 6.6 (d). We observe the benefits of a FPID/PWM controller over a traditional PID/PWM controller in Fig. 6.7. The reference input is a 0.5 Hz, 10 degree sine wave. The plot on the right shows the absolute value of the positioning error integrated over time, and we observe how error accumulates in the PID controller faster than that of the FPID controller. This result corroborates the hypothesis that adaptive tuning of the gains in a standard PID controller can serve to better compensate for nonlinearities present in an antagonistic SMA control scheme.
142
40
40 Actual
3 [deg]
3 [deg]
Desired
20
0
0 -20
1
2
3
4
5
0
5
10
0
5
10
0
-20
20
25
30
15
20
25
30
0
-20 0
1
2
3
4
5
Time [s]
Time [s]
(b)
40 Desired Actual
20 0 -20
Angle 3 [deg]
(a) Angle 3 [deg]
15
20
3d ! 3a [deg]
20
-40
40 20 0 Desired Actual
-20 -40
10
20
30
40
50
Angle Error [deg]
0
Angle Error [deg]
Actual
-40 0
3d ! 3a [deg]
Desired
20
5
0
-5 0
10
20
30
40
50
Time [s]
0
10
20
0
10
20
30
40
50
30
40
50
5
0
-5
Time [s]
(c)
(d)
Figure 6.6: Bencthtop results (actuator-only module with reference sensor) for a FPID/PWM controller (shaded error bars show 1σ for n = 3 trials), where top shows desired vs. actual angle θ and bottom shows the angle error: (a) step response showing convergence in around 0.4 seconds, (b) stairstep pattern, (c) triangular profile with a 30 second period, and (d) 0.125 Hz sine wave.
6.4 Integrated Performance The optimized FPID/PWM controller was tested on a distal module where the loop is closed using on-board proprioceptive sensing and SMA actuation, as shown in Fig. 6.8.
6.4.1
Automated Trajectory Execution
Example automated trajectories are shown in Fig. 6.9 (a) and (b). In Fig. 6.9 (a), we study the controller’s ability to track slowly-varying inputs by providing a triangular input profile with a period of 20 seconds. The controller is able to keep the error within ±3 degrees over the entire 143
Desired PID Fuzzy
0
PID Fuzzy
-10 5
10
15
20
25
PID Fuzzy
5 0
15
10
5
Z
Angle Error [deg]
0
j3d (t) ! 3a (t)jdt [deg*s]
Angle 3 [deg]
20
10
-5 0 0
5
10
15
20
Time [s]
25
0
5
10
15
20
25
Time [s]
Figure 6.7: Comparison of the FPID/PWM controller and a traditional PID/PWM when tracking a fast (0.5 Hz) sinusoidal reference: (counter-clockwise from top left) reference and actual trajectories vs. time, angular error vs. time, and error integrated over time. FujiWEG-450WR5 EndoscopeW Enclosed Actuators DeflectorWPlateWand FeederWTube Olympus ITKnife CoolingWPipe andWWiring
Force Sensor
10mm
Proprioceptive Sensing Linkage
Figure 6.8: Fully integrated distal module with on-board actuation, sensing, and closed-loop position control capabilities: (left) image of to-scale prototype with callouts to important features, (right) view through endoscope in different configurations.
trajectory. In Fig. 6.9 (b), we provide a faster, continuously-differentiable 0.2 Hz sinusoidal input profile, and observe that the closed-loop system is able to track this trajectory as well. We observe error spikes at locations where the input trajectory derivative changes sign, and this is due to the transition of the current active actuator (under continuous current control) to bias (hysteretic fluid
144
15
20
25
30
35
3d ! 3a [deg]
-10 5
10
15
20
25
30
35
40
4 Cd+ Ca+ Cd! Ca!
2 0 0
5
10
15
20
25
30
35
40
1 Valve (+) Valve (-)
0.5 0 5
10
15
20
25
30
35
7 p+ K 7 d+ K 7 i+ K
0.5
5
10
15
20
25
30
35
40
0
5
10
15
20
25
30
35
40
-10 4 Cd+ Ca+ Cd! Ca!
2 0 0
5
10
15
20
25
30
35
40
1 Valve (+) Valve (-)
0.5 0 0
0
5
10
15
20
25
30
35
40
1 7 p+ K 7 d+ K 7 i+ K
0.5 0
0
5
10
15
20
25
30
35
40
0
5
10
15
20
25
30
35
40
1 7 p! K 7 d! K 7 i! K
0.5
(-) Gains
1
(-) Gains
0
0
40
1
Desired Actual
10
SMA Current [A]
0
40 20 0 -20 -40
40
Valve States
Valve States
10
0
0
(+) Gains
5
10
SMA Current [A]
3d ! 3a [deg]
0
3 [deg]
Desired Actual
(+) Gains
3 [deg]
40 20 0 -20 -40
0
7 p! K 7 d! K 7 i! K
0.5 0
0
5
10
15
20
25
30
35
40
0
Time [s]
5
10
15
20
25
30
35
40
Time [s]
(a)
(b)
Figure 6.9: Automated trajectory execution of a FPID/PWM controller implemented on a fully-integrated distal module with on-board sensing and actuation, where (a) shows a triangular (constant velocity) profile and (b) shows a sinusoidal profile (shaded error region denotes 1σ for n = 3): (from top) reference vs. actual trajectory, angular error, reference vs. actual SMA current, fluid valve states, agonist PID gains, and antagonist PID gains.
cooling control). It should be noted that, as discussed in Chapter 3, active cooling based on the logic-based fluid cooling controller is absolutely instrumental in allowing the system to track trajectories with this speed. Without fluid cooling, the actuators would require nearly 100 seconds of cool-down time, thereby substantially limiting achievable bandwidth and making real-time control impossible. By correlating the angular error with the agonist and antagonist gains (most easily seen in
145
Figure 6.10: System snapshot during closed-loop, rate-based teleoperation: (top left) embedded system, host computer, and master input device mounted on endoscope handle, (bottom left) desired vs. actual deflection angle and associated error, (right) close-up of distal module deflecting an Olympus ITKnife.
Fig. 6.9 (b)), we see the effects of gain tuning and co-contraction avoidance. We observe that the agonist gains are active only at positive errors, and vice-versa for the antagonist gains. We also see that proportional gain is highest for high errors, and integral and derivative gains are maximized at low error and high derivative errors.
6.4.2
Teleoperation
Under teleoperation, the endoscopist maintains full control over the robot’s configuration via the master input device presented in Chapter 5, Fig. 5.6. Teleoperation can be implemented in two forms: a position-based controller and a rate-based controller. Under position control, the robotic deflection directly tracks the position of the joystick on the master input device. This is implemented as follows:
[n]
[n]
θd = η p (Vj
− Vo f f )
(6.8)
where n is the current time step, Vj is the joystick potentiometer voltage, Vo f f is the zero-offset 146
voltage, and η p is a scaling factor to convert from voltage to degrees. Under rate-based control, the position of the joystick dictates the angular velocity of the lateral deflection. This is implemented as follows:
[n]
[ n −1]
θd = θd
[n]
+ ηr (Vj
− Vo f f )∆ts
(6.9)
where ηv is a scaling factor to convert from voltage to degrees-per-second, and ∆ts is the sample period. Both controllers have their potential benefits. Under position control, it is possible to perform very fine, precision maneuvers in real time, and the robot will always return to θ = 0 when the control command is removed. Under rate control, the robot holds its position when the control command is removed, which might be beneficial during lateral dissection when the endoscopist wants to incrementally advance the robot and perform electrosurgery. A system snap-shot during rate-based teleoperation is shown in Fig. 6.10. In addition, data for both position- and rate-based teleoperation modes are shown in Fig. 6.11 (a) and (b), respectively. We observe the ability of the module to track fast, dynamic input trajectories. We also see that, using a rate-based controller, it is easier to generate smooth trajectories over a wide range, whereas the position controller is better at generating precise, incremental motions.
6.4.3
Effect of Tool on Control Performance
When the module is actively deflecting a tool, an elastic load is placed in parallel with the stiffness of the bias actuator. In order to actively deflect the tool, the actuators must generate a net positive force when a tool is present, i.e. Fact − Fbias > Ftool , where Ftool = f (θ ) is not necessarily a linear function, as we saw in Chapter 2. As such, control performance is affected due to the tool’s presence, as the elastic contribution from the tool essentially appears as a disturbance to the system. To study the effect of the tool on the control performance, we performed a parametric sweep where step inputs of variable magnitude were applied to a module both with and without the tool in place. These results are presented in Fig. 6.12. The tool under consideration is an Olympus DualKnife which is very commonly used during ESD and other endoscopic interventions. We
147
30
3d ! 3a [deg]
-10 10
20
30
40
4 Cd+ Ca+ Cd! Ca!
2 0 0
10
20
30
40
1 Valve (+) Valve (-)
0.5 0 10
20
30
10
20
30
40
0
10
20
30
40
-10 4 Cd+ Ca+ Cd! Ca!
2 0 0
7 p+ K 7 d+ K 7 i+ K
10
20
30
40
1 Valve (+) Valve (-)
0.5 0 0
1
0
10
20
30
40
1 7 p+ K 7 d+ K 7 i+ K
0.5 0
0
10
20
30
40
50
0
1
10
20
30
40
50
1 7 p! K 7 d! K 7 i! K
0.5
(-) Gains
(-) Gains
0
0
40
0.5
Desired Actual
10
SMA Current [A]
0
40 20 0 -20 -40
40
Valve States
Valve States
20
0
0
(+) Gains
10
10
SMA Current [A]
3d ! 3a [deg]
0
3 [deg]
Desired Actual
(+) Gains
3 [deg]
40 20 0 -20 -40
0
7 p! K 7 d! K 7 i! K
0.5 0
0
10
20
30
40
50
0
Time [s]
10
20
30
40
50
Time [s]
(a)
(b)
Figure 6.11: Teleoperation of a FPID/PWM controller implemented on a fully-integrated distal module with on-board sensing and actuation, where (a) shows a position control paradigm and (b) shows a rate control paradigm: (from top) reference vs. actual trajectory, angular error, reference vs. actual SMA current, fluid valve states, agonist PID gains, and antagonist PID gains.
observe from Fig. 6.12 (middle) that the tool’s presence has very little impact on the controller’s transient dynamics for deflections of less than 35 degrees, after which, the tool’s stiffness starts to slow down the response (albeit marginally). We also observe that, for deflections less than 30 degrees, we approach 1 Hz bandwidth (where we use the first-order approximation f BW = 0.35/t10%−90% ). Another notable observation, apparent in 6.12 (left), is that the tool adds a damping component that reduces overshoot especially for smaller deflections with negligible effect on
148
50
1
1
30
20
3d 3act (no tool) 3act (tool)
10
0
0.8 0.5 Hz
0.6
0.4
1 Hz
0.2
1
2
3
Time [s]
0.6 0.4 0.2 0 -0.2 -0.4
0 0
Without Tool With Tool
0.8
Steady-State Error [deg]
3 [deg]
40
10%-90% Rise Time [s]
Without Tool With Tool
-0.6 10
20
30
40
Setpoint Angle 3d [deg]
10
20
30
40
Setpoint Angle 3d [deg]
Figure 6.12: Performance comparison with and without the tool in place: (left) controller response to step inputs varying from θd = 5 to θd = 45, (middle) 10%-90% rise time vs. setpoint angle θd , (right) steady-state error vs. setpoint angle θd .
transient dynamics, so performance actually improves due to the tool’s damping contribution. We observe in Fig. 6.12 (right) that the tool does not have any notable or systematic effect on steady-state error of the system.
6.5 Conclusion In this chapter, we designed a fuzzy-tuned PID/PWM (FPID/PWM) controller for precise position control of the distal module based on feedback from on-board proprioceptive sensors. The proposed closed-loop system performs significantly faster than similar coil-based SMA-actuated surgical systems presented in literature (100 times faster than passively-cooled systems (Ho and Desai, 2013) and nearly 10 times faster than acively-cooled systems (Cheng et al., 2018)), achieving 1Hz positioning bandwidth is possible for setpoint amplitudes of less than 30 degrees. The controllers developed in this chapter form the basis for teleoperation and automated trajectory execution during lateral dissection.
149
Chapter 7
Ex-Vivo and In-Vivo Testing 7.1 Introduction While developing hardware to work on the benchtop answers fundamental questions regarding feasibility, demonstrating device efficacy in realistic clinical conditions where blood, saline, mucus and organic material are present is a substantial challenge. In this chapter, we demonstrate that the add-on robotic system can operate both open- and closed-loop in realistic ex- and in-vivo analogs. Through collaboration with endoscopists at Brigham and Women’s hospital, we performed a series of ex vivo tests using an excised porcine stomach to evaluate module performance (sensor/actuator robustness, closed-loop position control) in a clinically realistic environment. Overall it was demonstrated that robot assistance could lead to potentially higher dissection speeds when compared to conventional ESD. We also performed a series of IACUC-approved in vivo tests on an anesthetized pig to demonstrate system efficacy on living tissue.
7.2 Ex-Vivo Testing 7.2.1
Open Loop Control/Sensor Robustness
To test actuator and sensor robustness in open-loop, we use the controller shown schematically in Fig. 7.1 which generates determinstic SMA current profiles based on user input (as detailed in Appendix D, Fig. D.3) and closes the loop around the low-level current controller described 150
Embedded Controller
+Current Profile
Control Logic
Ki Kd
Low-Level Current Controller Analogto-PWM
+ ++
Peripheral Drivers SMA Driver
High Current iSMA1,iSMA2
Kp Current Sensors
5Hz
Input Device
Hydraulics Driver Hydraulics Profile
Data Acquisition 50Hz
On-Board Sensor s1,s2,st,se1,se2
Distal Module
Figure 7.1: Schematic of open-loop controller employing the deterministic current and fluid logic profiles detailed in Appendix D, Fig. D.3.
in Section A.4.1 and shown in Fig. A.13. Low pass filters are implemented in software to filter out high frequency noise in both the current sensor and on-board proprioceptive sensor measurements. An early version of the master input device (Version 1 in Fig. 5.5) dictates the direction of articulation. System data (desired vs. actual current profile, SMA electrical resistance, fluid logic states, and on-board sensor data) is acquired at a rate of 1 kHz.
Ex-Vivo Test with Porcine Stomach Collaborating with endoscopists at Brigham and Women’s hospital (Boston, MA), we used the robotic module to perform a simulated tissue dissection task on an excised porcine stomach. The experimental setup is shown in Fig. 7.2 (a). A photograph of the sensorized distal module is shown in Fig. 7.2 (b). A simulated tumor was marked on the porcine stomach, and saline fluid was injected into the submucosal space to lift the tumor off of the muscularis. The module was then used to deflect an Olympus DualKnife electrosurgical tool to create a circumferential incision around the simulated tumor margin, and system data were recorded (as shown in Fig. 7.2 (c)). We see a clear correlation between the applied current and the angle of articulation, indicating that the sensor is capable of providing real-time proprioceptive angle feedback. The on-board temperature sensor, in addition to providing disturbance rejection for the proprioceptive sensor, is also useful for measuring the temperature of the system as a whole. As such, a higher-level controller can use this data to monitor the system temperature in the event that the SMAs heat up the entire module near the pain threshold, thereby triggering fluid flushing to cool the system
151
Current-[A]
Host-PC-with-GUI
4 Cd+ Ca+ CdCa-
2 0 0
Cooling-Pump
Input-Device Distal-Module
Fluid-Logic
Target-Controller
5
10
15
20
1 Valve-wIG Valve-w-G
0.5 0
-Stomach Solenoid-Valves
Hall-Voltage-[V]
0
Fluid/Electrical Lines Strain-Relief-Collet
Actuator Sensor-Linkage
20 HE1 HE2
0
5
10
15
20
0
5
10
15
20
0
5
10
15
20
50 0 -50 40
0.05
30 20
FS1 FS2
0 -0.05 0
Deflecting-Plate
15
0
Temperature-[C]
Stomach
10
0.2
E/D-Voltage-[V]
Angle-θ-[deg]
(a)
5
0.4
5
(b)
10 Time-[s]
15
20
(c)
Figure 7.2: Ex Vivo testing of open-loop control and sensor/actuator robustness to clinical conditions: (a) experimental setup with an early version of the control electronics and the first version of the master input device, (b) image of sensorized distal module used during tests, (c) controller and sensor data during open-loop deflection of Olympus DualKnife.
down back to an acceptable temperature. It is also important to note that electrosurgical pulses do not introduce any noise or contamination to the sensor readings.
7.2.2
Closed-Loop Control/Feasibility Test
With the on-board sensors and actuators validated, we simulated a full ESD procedure on an excised porcine stomach under closed-loop control based on on-board sensor data and the Fuzzy-
152
tuned PID/PWM controller detailed in Chapter 6. The experimental setup is shown in Fig 7.3. The procedure was performed at Brigham and Women’s hospital (Boston, MA) using a Fuji 450WR5 9.4mm endoscope. The module is affixed to the endoscope through a press-fit, and two set-screws are tightened to provide further mechanical engagement for stability. A hole is cut into the sidewall of the stomach, and the roboticized endoscope is inserted into the hole. The tissue around the endoscope is clamped such that proper insufflation can be achieved. The simulated lesion margins were marked in the antrum chamber of the stomach. The endoscope’s live feed was captured by a StarTech frame grabber, and system data was captured in two-minute intervals at a rate of 500 Hz. In this series of tests, three different control modes were evaluated: • Passive: Robotic control is turned off and the endoscope is used as though the module is not there. • Open-Loop Teleoperation: Robotic module is controlled using deterministic current profiles and valve state profiles (shown in Appendix D, Fig. D.3), and on-board proprioceptive sensor data is ignored. • Closed-Loop Teleoperation: On-board proprioceptive sensor data closes a rate-based position controller given in Equation 6.9. Three tools were used throughout the procedure. An Olympus InjectorForce Max was used to inject saline underneath the simulated lesion to provide marginal countertraction for cutting. The robot was in passive mode for this aspect of the procedure. Once fluid injection was complete, an Olympus DualKnife (forward-cutting monopolar) and ITKnife (side-cutting blunt-tip monopolar) electrosurgical tools were used to create an incision to access the submucosal space and to dissect the tumor from the submucosal space, respectively. Both open-loop teleoperation and closed-loop teleoperation were used to deflect these tools laterally to facilitate dissection. Example controller data for each of these control modes, along with snapshots of the endoscopic field-of-view during the procedure, are shown in Fig. 7.4. There are several key takeaways from this figure. The first is the ability of the on-board sensor to provide stable, accurate position feedback in the presence of physical debris as well as any EM noise generated by the electrosurgical pulses. The second is the ability of the open-loop controller to deflect the tool in the absence of closed-loop 153
HosthMonitorhGUI
ElectrosurgicalhGenerator
EndoscopehMonitor
Controller TargethSystem EndoscopyhTower PighStomach SubmersiblehPump FujihEGh450WR5h
SolenoidhValves DistalhModule
Figure 7.3: Experimental setup for full ex-vivo ESD simulation to test closed-loop control capabilities, where inset shows a close-up view of the distal module affixed to the endoscope.
40 20 0 -20 -40 55
40 20 0 -20 -40 30
40 20 0 -20 -40 60
65
70
75
80
85
5
5
4
0
0
2
-5 55 4
0 30
40
50
60
70
50
60
70
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
-5 60
65
70
75
80
85 4
2 40
0
80
2
80 0 55
1
0 60
65
70
75
80
85
1
1
0.5
0.5
0.5
0 30
40
50
60
(a)
70
80
0 55
0 60
65
70
(b)
75
80
85
(c)
Figure 7.4: Controller data during robot-assisted ex-vivo ESD where images show endoscope view during the procedure: (a) open-loop teleoperation with an Olympus ITKnife, (b) closed-loop velocity-based teleoperation using Olympus DualKnife, (c) closed-loop velocity-based teleoperation using Olympus ITKnife.
154
Figure 7.5: Results of ex vivo test: (left) lesion removed en bloc, (right) location from which the lesion originated.
Injection Incision Dissection
RobotAssisted
No RA 1
No RA 2
No RA 3
0
1
2
3
4
5
6
7
Procedure Time Normalized by Lesion Size [s/mm2]
Figure 7.6: Procedure comparison between robot-assisted ESD and conventional ESD, where bars represent the task time normalized by the dissection lesion area.
sensor feedback. This implies that, if for some reason the sensor loses power or malfunctions, the operator can always revert to open-loop mode and maintain control over tool deflections. Finally, (b) and (c) show the module deflecting the Olympus DualKnife and ITKnife, respectively, which have different flexural properties as seen in Fig. 2.13, demonstrating the ability of the module to accommodate various commonly-used tools. Through robot assistance, the endoscopist was able to remove the lesion en bloc with no incidence of perforation. The excised lesion is shown in Fig. 7.5. This test demonstrated the efficacy of the module as an interventional device and validated robustness of the sensors, actuators, and closed-loop control scheme to clinically-realistic environmental conditions. 155
In Fig. 7.6 we compare the robot-assisted (RA) ESD to three ESDs performed conventionally by the same user. Note that the bar graphs represent task time normalized by the dissected lesion area, or the ‘inverse procedure speed’. The dissected lesions are shown for each procedure (where the US penny provides scale). Overall we observe that robot-assisted ESD can match or even exceed conventional ESD in terms of minimizing task time (when lesion size is considered). The associated circumferential dissection rate (lesion area divided by dissection time) for robot-assisted ESD was 41.1 mm2 /min, compared to an average unassisted dissection rate of 37.0±21.0 mm2 /min for ex vivo ESDs performed conventionally by the same endoscopist. Overall procedure speed (lesion size divided by total time of procedure) was 16.5 mm2 /min for RA and 14.7 mm2 /min for conventional. In general, an average procedure speed of around 15 mm2 /min is a benchmark for expert proficiency in ESD (Oyama et al., 2015). A retrospective study of 77 conventional ESDs reported procedure speeds of 14.0±13.0 mm2 /min (He et al., 2016). Given these results, we observe that robot-assisted ESD could potentially lead to comparable or faster dissection rates due to the additional distal degree-of-freedom. We observe that, in robot-assisted ESD, incision requires more time proportional to other tasks, potentially as a result of reduced visualization due to the module’s presence which drives up incision time. It is possible that more experience with the module could further improve performance. Improved visualization is a subject of future work, as discussed in Chapter 8.
7.3 In Vivo Tests Live animal studies were performed under the Institutional Animal Care and Use Committee (IACUC) approved protocol number 17-04, entitled ‘Robotic Module for Endoscopic Submucosal Dissection’, at Pine Acres Rabbitry Farm in Norton, MA. A series of two studies were performed on 50 kg female Yorkshire pigs.
7.3.1
Animal Preparation
Animals were delivered to the facility 10 days prior to each procedure and were fed a liquid diet 72 hours prior. On the day of the study, the surgical team performed parenteral induction using Telazol 4.4-4.6 mg/kg, Xylazine 1.1-2.2 mg/kg, Atropine 0.05 mg/kg (Telazol and Xylazine were 156
Angle [deg]
40 20 0 -20 -40
θref θact
Error [deg]
20
40
60
80
100
120
140
5 Error
0 -5 20
(b) Current [A]
(a)
40
60
80
100
120
140
4 CdCaCd+ Ca+
2 0
Fluid Logic
20
40
60
80
100
1
120
140
Valve (-) Valve (+)
0.5 0 20
40
60
80
100
120
140
Time [s]
(c)
(d)
(e)
Figure 7.7: In vivo open ESD performed on anesthetized porcine model: (a) image of the experimental setup, (b) image of module performing lateral dissection, (c) image of stomach after successful dissection, (d) image of dissected lesion, and (e) system data during dissection where green boxes indicate approximate timing and duration of tissue contact and electrosurgery (qualitatively observed from videos of the procedure).
injected to promote significant pain relief, immobility, freedom from reflex movement and muscle relaxation. Atropine was used to reduce oral secretions and to protect against reflex cardiac dysrhythmias), intubation (spray Lidocaine to avoid laryngospasm), maintenance using isoflurane (1-4%) and placement of marginal ear vein catheter. The pigs were placed on an operating table and were in left lateral position under monitored anesthesia care, as shown in 7.7 (a).
7.3.2
Robustness Study (Open)
The first in vivo study was performed to test device robustness during a live tissue dissection experiment, in the presence of blood. The porcine specimen was opened up and the stomach was pulled outside of the body, as shown in 7.7 (b), and device was used to deflect an electrosurgical tool laterally to dissect tissue to simulate ESD in an open environment. The endoscopist was instructed to hold the module body stationary, relying only on the module’s integrated actuation to deflect the tool laterally for tissue dissection. The experimental results demonstrated that the module provided enough force to deflect the surgical knife and successfully dissect tissue, en bloc, as shown in 7.7 (d). In addition, the sensor and control strategy was robust to the presence of
157
Angle)[deg]
40 20 0 -20 -40
θref θact
Error)[deg]
0
20
40
60
80
100
5 Error
0 -5 0
Current)[A]
(a) Superficial)Lesion Margin
20
40
60
80
100
4 CdCaCd+ Ca+
2 0
Fluid)Logic
0
20
40
60
80
1
100 Valve)(-) Valve)(+)
0.5 0 0
20
40
60
80
100
Time)[s]
(b)
(c)
(d)
Figure 7.8: In vivo closed ESD performed on anesthetized porcine model: (a) image of the experimental setup, (b) close-up of the view through the endoscopic camera, (c) image of dissected lesion where the margins are highlighted, and (d) exemplary system data during robot-assisted dissection via rate-based teleoperation.
external forces and substantial fluid/debris, as shown in 7.7 (e) where the green bars indicate the timing and approximate duration of electrosurgical dissection as observed visually. Dissection instances are associated with high errors that are quickly overcome when the contact force is removed (by observing that, in the second plot in Fig. 7.7 (e), the error spikes typically align with the green bars which denote tissue contact). As the position error is proportional to the contact (disturbance) force Ft given the PID control methodology, this means that the module is able to provide positive lateral contact force to facilitate electrosurgical dissection.
7.3.3
Feasibility Study (Closed)
A full robot-assisted closed ESD was performed on an anesthetized porcine specimen. The results are summarized in Fig. 7.8, where (a) shows the experimental setup, (b) shows a close-up of the endoscope monitor, (c) shows the dissected lesion, and (d) shows controller data during rate-based teleoperation. In (a) we can see the placement of the embedded system and host computer in the foreground, sufficiently far away from the surgical arena. In (c) we observe that the resected lesion contains all of the superficial markings within its boundaries, which is an ideal outcome. The entire procedure lasted roughly 70 minutes, corresponding to a procedure speed of around 6.7 158
mm2 /s. The reduction in speed is likely due to visualization issues, described more thoroughly in Chapter 8, which necessitated several rounds of water flushing to clear away the substantial amounts of mucus, stomach acid and blood. The boundaries of the resected lesion are smooth and there is no evidence of burning/charring which can be symptomatic of electrosurgical overuse. Given these results, while feasibility has been demonstrated, improving visualization is a notable next step to replicate procedure speeds observed during ex vivo testing.
7.4 Conclusion The results presented in this chapter demonstrate the first ex and in-vivo studies of robot-assisted interventional endoscopic techniques using a tip-mounted robotic module where all actuation and sensing is fully integrated at the endoscope tip. We demonstrate the important result of system feasibility and robustness in clinically realistic conditions. The sensors and actuators operated reliably in the presence of significant fluid and debris occlusion, and both open- and closed-loop controllers enabled the endoscopist to deflect tools laterally in real-time, meanwhile decoupling tool motion from endoscope motion. While initial results are extremely promising and suggest the potential for improved performance and faster dissection speeds through robotic assistance, the ex and in-vivo tests presented in this thesis contribute only a few data points and the reader should interpret the results with this understanding. Future work will entail a systematic, multi-user in vivo animal trial to assess any associated reductions in learning curve and cognitive burden using NASA task load indices. Such a study is a necessary pre-cursor towards clinical validation.
159
Chapter 8
Conclusion This thesis investigates the utility and efficacy of distally-mounted, fully deployable robotic modules for on-demand robotic assistance in complex therapeutic endoscopic procedures. Chapter 2 describes efforts to experimentally characterize endoscopic submucosal dissection for the purpose of defining system functional requirements. Chapter 3 presents the modeling, design, fabrication and testing of custom helical shape-memory alloy (SMA) actuators for the purpose of generating the required force and stroke to deflect flexible endoscopic tools laterally. Chapter 4 presents the design of a mm-scale, distal multisensing linkage capable of providing position and force feedback for loop closure and automated trajectory execution. Chapter 5 presents the design of the distal module and system-level hardware. Chapter 6 demonstrates the first instance closed-loop position control of a roboendoscopic module using sensors and actuators integrated distally within the module. Chapter 7 presents preliminary ex vivo and in vivo results using an excised porcine stomach and anesthetized live porcine model, respectively. This work introduces several key topics that contribute to existing literature in endoscopic robotic surgery and are of general interest to the surgical robotics community. Specific contributions include: • A rigorous experimental workspace characterization of endoscopic submucosal dissection on an ex vivo pig model, the results of which can provide dexterity benchmarks for researchers aiming to develop devices and systems to facilitate ESD. • A method for heuristic design and optimization of optimally-packed, helically-wound SMA 160
actuators, which includes (1) solving the discrete form of the shear-dominated constitutive equation, (2) performing custom fabrication with parametric flexibility, and (3) performing a complete thermomechanical characterization using a custom-developed evaluation and characterization platform. • An analysis of printed-circuit MEMS as a viable fabrication method for manufacturing sensor systems for distal placement in surgical robotic systems. • Design and conception of a distally-mounted module that combines on-board actuation and sensing to give on-demand robotic capabilities to otherwise passive commercially available endoscopic tools. • Ergonomic design of systems-level hardware with a specific focus on seamless clinical workflow integration. • Development of a hybrid position controller that combines fuzzy-tuned PID/PWM control of antagonistic SMA actuators (based on on-board sensor feedback) with hysteresis control of forced convection fluid cooling to speed up actuation. • Demonstration of on-demand robotic assistance in both ex vivo and in vivo endoscopic therapeutic procedures.
8.1 Insights and Future Work The work covered in this thesis presents a first step towards the development of low-cost, disposable robotic modules that can augment the capabilities of existing commercially-available endoscopic devices. Ultimately it was demonstrated through ex- and in-vivo testing that the system is robust to clinically-realistic conditions and is potentially useful as an interventional device. While the results are very promising, there are a number of avenues to be explored that can improve the reliability and efficacy of systems developed under this ‘robotic add-on’ paradigm. We consider both technical improvements and clinical/workflow improvements in the remainder of the chapter.
161
8.1.1
Technical Considerations
Multiple-DoF Robotic Control Single-degree-of-freedom lateral control is sufficient in many cases as the endoscopist can simply roll the endoscope to transition between pitch and yaw control (with respect to the inertial frame). However, in some instances, true pitch control (with respect to the endoscope frame) is desirable especially if the lesion is in a particularly difficult orientation and the scope requires substantial coarse positioning or retroflexion. True pitch/yaw control could also further reduce the cognitive burden and potentially obviate the need to roll the endoscope during dissection. The requirement for an unobstructed bore rules out the use of a central flexible backbone and peripherally-routed tendons which is a common implementation of multi-DoF continuum robots. As such, this application requires discrete, serial linkages for multi-DoF control, and the number of actuators Nact required scales linearly with the number of degrees-of-freedom NDoF (Nact = 2NDoF due to the antagonistic actuation scheme), as does the number of cooling pipes Npipes and solenoid valves Nvalves (Npipes = Nvalves = NDoF ). For position-sensing only, the number of electrical connections Ne also scales with additional degrees-of-freedom (Ne = 3 + 2NDoF ) (assuming shared power, shared ground, and a single temperature sensor for thermal compensation). A theoretical upper-bound on the number of degrees of freedom as determined by the maximum number of actuators that can be placed around the endoscope is NDoF,max = ⌊(π (dm,i + tm + d act )/(4d act + ǫ)⌋, where dm,i is the module inner diameter, tm is the module wall thickness, d act is the actuator diameter, and ǫ is a spacing parameter (the actuators cannot physically touch each-other due to electrical shorting). For the system proposed here, where dm,i =11mm, tm =1mm, d act =2.5mm, and ǫ=2mm, the theoretical maximum is NDoF,max = 3, meaning we could integrate three antagonistic pairs (six actuators) before there is no more physical space left to accommodate additional actuators. Note that, due to the requirement for serial linkages instead of a continuum structure, more degrees-of-freedom results in further recession of the endoscope camera from the deflector plate which could affect visual acuity due to increasing distance from the surgical field and the placement of more mass within the field-of-view. As system complexity scales with the number of degrees-of-freedom, and further analysis is necessary to realize the optimal dexterity requirement to avoid diminishing returns in terms of performance. 162
Figure 8.1: Proposed visualization improvement: (left) schematic representation of visual occlusion for current design, (middle) visual field improvements due to single-joint design, (right) unobstructed view (based on the data sheet for Fuji 600 series endoscopes)
Visualization It was observed during in vivo tests that, if sufficient mucus, blood and stomach acid is present, the fluids can ‘drape’ across the deflector plate and adversely affect visual acuity. As the deflector plate is circular to connect the pin joints both above and below the endoscope camera, it can have a ‘bubble wand’ effect where thin films of fluid stretch across the inner-diameter to create a membrane that occludes the visual field. This could be mitigated by removing the pin joint above the camera, relying on a single pin joint below the working channel to couple the actuators and deflect the tool, enabling the use of a much lower-profile deflector plate, and placing most of the visible mass below the working channel which is already obscured by the tool’s presence. A potential design improvement is shown schematically in Fig. 8.1. The reduced deflector plate and feeder tube are below the camera, and the removal of the top pin joint ensures that any fluid accumulation will occur in the bottom half of the visual field which is already obscured by the tool’s presence. It should be noted that the Kevlar tendon attachment points could be placed closer to the feeder tube for further occlusion reduction,at the cost of a reduced deflection force margin as this shortens the moment arm between the actuators and the tool being deflected.
Force Feedback In this work we developed the capability to sense forces distally using light-intensity modulation, as described in Chapter 4. This enables stand-alone force sensing for monitoring insertion forces and enabling tissue palpation endoscopically, and while the fundamental technology developed and detailed in Chapter 4 demonstrates the concept, force sensing was not validated in ex- and in-
163
vivo studies, and no closed-loop control efforts were attempted. In future iterations, force sensing could be directly coupled to the tools passed through the deflection mechanism to provide realtime haptic feedback and force limiting for perforation prevention. For example, the module could be configured with a low-level position controller and a high-level force controller that overrides the position controller when a certain contact force threshold is exceeded. A limitation of the current force sensor design is that only lateral forces can be detected, which are dominant during dissection. In incision tasks where forces are primarily axial, a multi-axis sensing methodology must be designed with negligible cross-talk. Depending on space limitations it may not be possible to integrate differential axial force sensing, in which case, techniques presented in Appendix B could be used for disturbance rejection.
Sensor Exploration A broader contribution of this work is the demonstration of several different sensing modalities that can be realized via PCMEMS towards position and force feedback in surgical robotics applications. Of the modalities tested (foil-based strain gage, light-intensity modulation, capacitance, and Hall effect sensing), differential Hall effect sensing was the most robust in terms of high sensitivity, negligible thermal and temporal drift, and resolution (assuming the system is properly designed for high-sensitivity, monotonic behavior as demonstrated in Fig. 4.6). The combination of precise meso-scale kinematics and Hall effect-based sensing is a powerful approach for proprioceptive sensing, with potential applications in shape sensing and reconstruction for continuum robotic systems.
Actuator Temperature Estimation A notable drawback of the control approach developed in this dissertation is the unobservable nature of actuator temperature. We found in Chapter 2 that the austenitic transition temperature is around 60C, at which point, any further heating is superfluous, inefficient, and potentially destructive. Measuring actuator temperature either directly (via thermocouples) or indirectly (by measuring actuator resistance) would allow us to have an outer temperature regulation loop that monitors temperature to prevent overheating and improve system efficiency by keeping the
164
temperature near the conversion temperature. The power electronics developed in Chapter 5 were designed to sense actuator resistance through a Wheatstone bridge methodology (shown highlighted in green in Fig. 5.9), and several prior works demonstrate the ability to estimate temperature or strain through actuator resistance (Wang et al., 2012). However, the correlation between resistance and temperature is non-monotonic, nonlinear and highly hysteretic, and all of these behaviors must be adequately captured in the thermoelectric characterization for accurate temperature regulation. An alternative approach is to use a model-based approach to predict temperature in realtime, which would require no further sensing modalities. We showed in Chapter 3, specifically Fig. 3.9, that we can predict temperature with a fairly high degree of fidelity by knowing the history of the electrical current passed through the actuator and the heat transfer characteristics between the actuator and the environment. However, the model assumes a constant heat transfer coefficient, and for our application, the transition between air and water cooling must be captured for model-based thermal regulation.
Improved Hydraulics It was observed experimentally that improper wetting of the actuators during the cooling phase could adversely affect performance and potentially lead to overheating. This can be seen in the infrared image corresponding to (3) in Fig. 5.15, where only a section of the actuator is being cooled∗ . This is due to the use of small-diameter, long tubes which deliver the fluid cooling medium from the submersible pump to the distal module. Frictional losses along the tube substantially reduce the flow rate of the fluid to the point where, once the fluid reaches the module, the flow is very laminar. This can result in a ‘jet’ of fluid which only cools one portion of the actuator. In this work, localized cooling was circumvented by pinching the inlet tube to spread the flow laterally, and aiming the tube such that the flow converged with the inner surface of the actuator chamber to wet the surface to spread the flow. More work could be spent on the design of the module flow inlet to induce turbulence (or, at the very least, divergent flow) to ensure adequate actuator cooling. ∗ This image is somewhat misleading, as the presence of the actuator chamber encapsulating shell (which was removed for clarity) actually improves the wetting performance.
165
1
2
3
4
(a)
(b)
Figure 8.2: Related work in tissue retraction using PCMEMS fabrication techniques: (a) a 3-DoF multi-articular arm based on pneumatic actuation and suction-based gripping (from (Russo et al., 2017)), and (b) a deployable retraction mechanism, where (1) shows the device encased in 23.81 mm outer diameter overtube (2) shows the device during deployment and overtube retraction, (3) shows the view from endoscope distal camera, with an endoscopic tool in the foreground, and (4) shows the deployed device retracting tissue (from (Becker et al., 2017)).
8.1.2
Clinical Considerations
Retraction-Enabled ESD The work presented in this thesis satisfies a fundamental need for precision lateral control of electrosurgical tools during circumferential tumor dissection. Marginal countertraction is provided through saline injection into the submucosal space which lifts the tumor off of the muscularis. However, saline injection is often not sufficient for retraction due to fluid leakage which causes the retraction tension to decay over time. In addition, retraction through injection is not directional or controllable. Numerous technique- and device-based retraction methods have been developed, including the clip-with-line method, percutaneous traction method, sinker-assisted method, magnetic anchor method , external forceps method, internal-traction method, double-channel-scope method, outerroute method, and double-scope method (Imaeda et al., 2014). Each of these techniques has its own advantages and disadvantages, however there is no universally-accepted retraction solution. In related work, PCMEMS has been used to develop retraction modalities both rigidly mounted to the endoscope (Russo et al., 2017), and decoupled from the endoscope (Becker et al., 2017). As robot assistance begins to further penetrate the space of therapeutic endoscopy, traction-enabled ESD will become more ubiquitous due to advances in technology.
166
(1)
(2)
(3)
(3)
(1)
(2)
(3)
(4)
(5)
(6)
(a)
(b)
Figure 8.3: Related work in endoscope stabilization using PCMEMS fabrication techniques. In (a) we show the stabilization mechanism functional principle: (1) illustration of how endoscope flexibility hampers effective force transmission and tissue manipulation tasks, (2) Fm is the force exerted by a tool manipulating tissue and Fr is the reaction force on the endoscope, (3) the inherent flexibility of the endoscope causes the instrument to bend during the task, and (4) the proposed solution with an expandable device mounted around the endoscope. In (b) we show the fabricated stabilization mechanisms: (1) seven soft linear actuators are interconnected, enabling inflation with a single tube, (2) actuators integrated onto a sleeve mounted onto the endoscope, and (3) inflated actuators, expanding into the bracing structure, (4) five strain limiting structures are fabricated in a single batch with one channel for inflation, (5) stabilization mechanism positioned around the endoscope, and (6) inflated (from Ranzani et al. (2017)).
Scope Stabilization In Chapter 2 we assessed the inherent stiffness of endoscopes, and demonstrated that substantial forces applied at the distal end could deflect the scope, limiting distal force transmission. While this is unlikely to happen during standard ESD due to forces commonly exerted during electrosurgery (300-400mN), forces required for adequate tissue countertraction could exceed 2N depending on the countertraction approach, and as such, ‘scope sag’ is a potential concern, as shown in Fig. 8.3 (a). In related work (Ranzani et al., 2017), PCMEMS has been used to develop expanding scope stabilization mechanisms that combine soft, air- or water-filled bladders with strain limiting structures (shown in Fig. 8.3 (b)) to enable the scope to expand against the intraluminal wall, thereby using the anatomy as a local mechanical ground for force transmission and increasing the stiffness of the combined system. Similar structures could be built into the distal module and, as the expansion structures only require one inflation tube, the same hydraulics power electronics and hardware described in Chapter 5 could be used to control inflation and deflation.
167
Data Collection for Surgical Skill Assessment Increasing interest in performing complex interventional endoscopic procedures naturally leads to a focus on new methods of training and educating novice practicioners. Current approaches to endoscopic surgical skill assessment are subjective, where an expert judging committee reviews videos of procedures and scores them based on a list of criteria (Kimura et al., 2010). While other MIS procedures (laparoscopy, microsurgery) are seeing innovation towards objective skill assessment via instrument motion/speed/acceleration tracking, endoscopy lags in this regard due to difficulties associated with sensor integration. Sensorizing endoscopes via add-on modules could provide real-time position, orientation, or force data which, combined with machinelearning-based classification, could provide an objective assessment of skill in advanced endoscopic interventions.
Assessment through Multi-Procedure/Multi-User Study The animal results presented in this thesis demonstrate module robustness to clinical conditions, and preliminary data suggests that dissection speeds associated with robot-assisted ESD (with minimal prior experience with the system) could match or exceed those associated with conventional ESD. These results are promising, but a thorough clinical validation requires more in vivo data points to fully characterize robot-assisted performance compared to conventional ESD. In performing multi-user and multi-procedure studies, performance metrics of interest include incision and dissection speed normalized by lesion circumference/area (higher speeds are indicative of an ‘easier’ procedure) (Iacopini et al., 2017), en bloc resection rate, and complication rate (bleeding or perforation). Given these metrics, a rigorous validation could include the following three testing protocols: • Cognitive Reduction: Comparing conventional vs. robot-assisted ESD with endoscopists with a wide range of experience (novice to expert) will qualify how prior experience affects the uptake of the robot-assisted approach and allow for qualitative assessment of associated reductions in cognitive burden through NASA task load indices. Procedures should be performed in vivo on anesthetized porcine models, in roughly the same location of the stomach so that intra-procedure difficulty is roughly uniform. The previous performance 168
metrics will be tabulated and compared to years of experience to (1) analyze the effects of expertise on system usage, and (2) determine any associated cognitive reduction due to robot assistance. • Learning Curve: With a single user, robot-assisted ESD will be repeated numerous (n ≥ 10) times on an anesthetized pig model. The lesion location should be kept consistent for uniform intra-procedure difficulty. Assessing previously-defined performance metrics will define any experience-based learning and adaptation associated with the robotic system compared to conventional ESD. • Sensitivity to Difficulty: Varying intraoperative difficulty predictors such as lesion size (larger lesions are typically more difficult to dissect), lesion location and orientation versus gravity for countertraction will assess the system’s sensitivity to different intraoperative conditions.
Other Clinical Applications The concept of add-on robotic modules could potentially extend beyond ESD and, further, endoscopy in general. Some examples of candidate endoscopic procedures which would benefit from advanced distal dexterity include per-oral endoscopic myotomy (POEM), Endoscopic Mucosal Resection (EMR) and Endoscopic Retrograde Cholangiopancreatography (ERCP), to name a few. As clinical ambitions continue to enable more advanced interventional endoscopic procedures, the list will surely grow.
169
References Aeby, N. and Leo, G. (2017). Triaxial fiber optic force sensing catheter and method of use. US Patent 9,597,036. Ahn, K. K. and Nguyen, B. K. (2006). Position Control of Shape Memory Alloy Actuators Using Self Tuning Fuzzy PID Controller. International Journal of Control, Automation, and Systems, 4 (6), 756–762. An, S.-m., Ryu, J., Cho, M. and Cho, K.-j. (2012). Engineering design framework for a shape memory alloy coil spring actuator using a static two-state model. Smart Materials and Structures, 21, 1–16. Arulampalam, T., Patterson-Brown, S., Morris, A. J. and Parker, M. C. (2009). Natural orifice transluminal endoscopic surgery. Annals of the Royal College of Surgeons of England, 91 (6), 456–459. Atalay, O., Atalay, A., Gafford, J. and Walsh, C. (2017a). A Highly Sensitive Capacitive-Based Soft Pressure Sensor Based on a Conductive Fabric and a Microporous Dielectric Layer. Advanced Materials Technologies, 1700237, 1700237. —, —, —, Wang, H., Wood, R. and Walsh, C. (2017b). A Highly Stretchable Capacitive-Based Strain Sensor Based on Metal Deposition and Laser Rastering. Advanced Materials Technologies, 1700081, 1700081. Baisch, A. T., Ozcan, O., Goldberg, B., Ithier, D. and Wood, R. J. (2014). High speed locomotion for a quadrupedal microrobot. International Journal of Robotics Research, 33 (8), 1063–1082. Beccani, M., Tunc, H., Taddese, A., Susilo, E., Volgyesi, P., Ledeczi, A. and Valdastri, P. (2015). Systematic design of medical capsule robots. IEEE Design and Test, 32 (5), 98–108. Becker, S., Ranzani, T., Russo, S. and Wood, R. J. (2017). Pop-up tissue retraction mechanism for endoscopic surgery. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (December), 920–927. Brinson, L. (1993). One-Dimensional Constitutive Behavior of Shape Memory Alloys: Thermomechanical Derivation with Non-Constant Material Functions and Redefined Martensite Internal Variable. Journal of Intelligent Material Systems and Structures, 4, 229–242. Cao, H., Yu, Y. and Ge, Y. (2009). A research of multi-axis force sensor static decoupling method based on neural network. Automation and Logistics, 2009. ICAL ’09. IEEE International Conference on, pp. 875–879.
170
Chapelle, O. (2007). Training a support vector machine in the primal. Neural Comput., 19 (5), 1155–1178. Chatterjee, A., Munshi, S., Dutta, M. and Rakshit, A. (2000). An artificial neural linearizer for capacitive humidity sensor. Instrumentation and Measurement Technology Conference, 2000. IMTC 2000. Proceedings of the 17th IEEE, 1, 313–317 vol.1. Cheng, S. S., Kim, Y. and Desai, J. P. (2018). Towards Real-Time SMA Control for a Neurosurgical Robot: MINIR-II, Cham: Springer International Publishing, pp. 187–200. Ciuti, G., Caliò, R., Camboni, D., Neri, L., Bianchi, F., Arezzo, A., Koulaouzidis, A., Schostek, S., Stoyanov, D., Oddo, C. M., Magnani, B., Menciassi, A., Morino, M., Schurr, M. O. and Dario, P. (2016). Frontiers of robotic endoscopic capsules: a review. Journal of Micro-Bio Robotics, 11 (1-4), 1–18. Claesen, M., Smet, F. D., Suykens, J. A. K. and Moor, B. D. (2014). Fast prediction with SVM models containing RBF kernels. CoRR, abs/1403.0736. Dicken, B. J., Bigam, D. L., Cass, C., Mackey, J. R., Joy, A. A. and Hamilton, S. M. (2005). Gastric Adenocarcinoma Review and Considerations for Future Directions. Ann Surg, 241 (1), 27–39. Dupont, P. E., Lock, J., Itkowitz, B. and Butler, E. (2010). Design and control of concentric-tube robots. IEEE Transactions on Robotics, 26 (2), 209–225. Eickhoff, A., Jakobs, R. and Kamal, A. (2007). Computer-assisted colonoscopy (the NeoGuide Endoscopy System): results of the first human clinical trial ("PACE study"). Am J Gastroenterol, 102, 261–266. Faes, T. J. C., van der Meij, H. A., de Munck, J. C. and Heethaar, R. M. (1999). The electric resistivity of human tissues (100 hz-10 mhz): a meta-analysis of review studies. Physiological Measurement, 20 (4), R1. Faragasso, A., Bimbo, J., Noh, Y., Sareh, S., Liu, H., Nanayakkarra, T., Wurdemann, H. A. and Althoefer, K. (2014a). Novel Uniaxial Force Sensor based on Visual Information for Minimally Invasive Surgery. IEEE International Conference on Robotics and Automation (ICRA), pp. 1405–1410. —, Stilli, A., Bimbo, J., Noh, Y., Liu, H., Nanayakkara, T., Dasgupta, P., Wurdemann, H. A. and Althoefer, K. (2014b). Endoscopic add-on stiffness probe for real-time soft surface characterisation in MIS. Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference, 2014, 6517–6520. Farhat, S., Chaussade, S., Ponchon, T., Coumaros, D., Charachon, a., Barrioz, T., Koch, S., Houcke, P., Cellier, C., Heresbach, D., Lepilliez, V., Napoleon, B., Bauret, P., Coron, E., Le Rhun, M., Bichard, P., Vaillant, E., Calazel, a., Bensoussan, E., Bellon, S., Mangialavori, L., Robin, F. and Prat, F. (2011). Endoscopic submucosal dissection in a European setting. A multi-institutional report of a technique in development. Endoscopy, 43 (8), 664–70. FDA (2007). NeoGuide 510(k) Approval (No. K070622). — (2010). Ovesco OTSC 510(k) Approval (No. K101428). — (2012). Arc Endocuff 510(k) Approval (No. K122565). 171
— (2014a). Boston Scientific Captivator 510(k) Approval (No. K140726). — (2014b). EndoAid Endorings 510(k) Approval (No. K133359). — (2014c). Third-Eye Panoramic 510(k) Approval (No. K140595). — (2015). Medrobotics Flex 510(k) Approval (No. K150776). — (2016). Invendoscope 510(k) Approval (No. K161355). Tech. rep. Feng, Y., Rabbath, C. A., Hong, H., Su, C. Y. and Lin, W. (2010). Inverse hysteresis control for shape memory alloy micro-actuators based flap positioning system. Proceedings of the IEEE Conference on Decision and Control, pp. 3662–3667. Gafford, J., Aihara, H., Thompson, C., Wood, R. and Walsh, C. (2018a). Distal Proprioceptive Sensor for Motion Feedback in Endoscope-Based Modular Robotic Systems. IEEE Robotics and Automation Letters, 3 (1), 171–178. —, —, —, — and — (2018b). Fuzzy-Based Feedback Control of a Tip-Mounted Module for Robot-Assisted Endoscopy. IEEE Robotics and Automation Letters (submitted). —, Doshi-Velez, F., Wood, R. and Walsh, C. (2016a). Machine learning approaches to environmental disturbance rejection in multi-axis optoelectronic force sensors. Sensors and Actuators A: Physical, 248, 78–87. —, Kesner, S., Degirmenci, A., Wood, R., Howe, . and Walsh, C. (2014). A monolithic approach to fabricating low-cost, millimeter-scale multi-axis force sensors for minimally-invasive surgery. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pp. 1419–1425. —, —, Wood, R. and Walsh, C. (2013). Microsurgical devices by pop-up book mems. In Proceedings of the 2013 ASME International Design Engineering Technical Conference and Computers and Information in Engineering Conference. —, Ranzani, T., Russo, S., Aihara, H., Thompson, C., Wood, R. and Walsh, C. (2016b). Snap-on robotic wrist module for enhanced dexterity in endoscopic surgery. 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 4398–4405. —, —, —, Degirmenci, A., Kesner, S., Howe, R. D., Wood, R. J. and Walsh, C. (2016c). Towards Medical Devices with Integrated Mechanisms, Sensors and Actuators via Printed-Circuit MEMS. Journal of Medical Devices, 11 (March). —, Wood, R. and Walsh, C. (2016d). Self-Assembling, Low-Cost, and Modular mm-Scale Force Sensor. IEEE Sensors Journal, 16 (1), 69–76. Guo, J.-r., He, Y.-g. and Liu, C.-q. (2011). Nonlinear correction of photoelectric displacement sensor based on least square support vector machine. Journal of Central South University of Technology, 18 (5), 1614–1618. Hammond, F. L., Smith, M. J. and Wood, R. J. (2014). Estimating surgical needle deflection with printed strain gauges. Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference, 2014, 6931–6936.
172
He, Y., Wang, X., Du, Y., Yee-Wa Yung, V., Jin, P. and Sheng, J. (2016). Predictive factors for technically difficult endoscopic submucosal dissection in large colorectal tumors. Turkish Journal of Gastroenterology, 27 (6), 541–546. Hirschowitz, B., Curtiss, L., Peters, C. and Pollard, H. (1958). Demonstration of a New Gastroscope, the ‘Fiberscope’. Gastroenterology, 35, 50–53. Ho, M. and Desai, J. (2013). Modeling , Characterization and Control of Antagonistic SMA Springs for use in a Neurosurgical Robot. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, Karlsruhe, Germany, pp. 2503–2508. —, McMillan, A. B., Simard, J. M., Gullapalli, R. and Desai, J. P. (2012). Toward a mesoscale SMA-actuated MRI-compatible neurosurgical robot. IEEE Transactions on Robotics, 28 (1), 213–222. Hu, J., Chang, C.-Y., Tardella, N., Pratt, J. and English, J. (2006). Effectiveness of haptic feedback in open surgery simulation and training systems. Studies in health technology and informatics, 119, 213—218. Huber, J., Fleck, N. and Ashby, M. (1997). The selection of mechanical actuators based on performance indices. Proc. R. Soc. Lond. A, 453, 2185–2205. Iacopini, F., Saito, Y., Bella, A., Gotoda, T., Rigato, P., Elisei, W., Montagnese, F., Iacopini, G. and Costamagna, G. (2017). Colorectal endoscopic submucosal dissection: predictors and neoplasm-related gradients of difficulty. Endoscopy International Open, 05 (09), E839–E846. Ianagui, A. and Tannuri, E. A. (2015). A sliding mode torque and position controller for an antagonistic SMA actuator. Mechatronics, 30, 126–139. Imaeda, H., Hosoe, N., Kashiwagi, K., Ohmori, T., Yahagi, N., Kanai, T. and Ogata, H. (2014). Advanced endoscopic submucosal dissection with traction. World journal of gastrointestinal endoscopy, 6 (7), 286–95. Inoue, H., Minami, H., Kobayashi, Y., Sato, Y., Kaga, M., Suzuki, M., Satodate, H., Odaka, N., Itoh, H. and Kudo, S. (2010). Peroral Endoscopic Myotomy (POEM) for Esophageal Achalasia. Endoscopy, 42, 265–271. Jemal, A., Bray, F. and Ferlay, J. (1999). Global Cancer Statistics: 2011. CA Cancer J Clin, 49 (2), 1,33–64. Kazanzides, P., Zuhars, J., Mittelstadt, B. and Taylor, R. H. (1992). Force Sensing and Control for a Surgical Robot.pdf. Kha, N. and Ahn, K. (2006). Position Control of Shape Memory Alloy Actuators by Using Self Tuning Fuzzy PID Controller. IEEE Conference on Industrial Electronics and Applications, pp. 1–5. Khan, F., Roesthuis, R. J. and Misra, S. (2017). Force Sensing in Continuum Manipulators using Fiber Bragg Grating Sensors. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) September 24–28, 2017, Vancouver, BC, Canada, pp. 2531–2536. Khashab, M. and Pasricha, P. (2013). Conquering the third space: challenges and opportunities for diagnostic and therapeutic endoscopy. Gastrointestinal Endoscopy, 77 (1), 146–148.
173
Kim, K. O., Kim, S. J., Kim, T. H. and Park, J. J. (2011). Do you have what it takes for challenging endoscopic submucosal dissection cases? World Journal of Gastroenterology, 17 (31), 3580–3584. Kim, U., Lee, D. H., Yoon, W. J., Hannaford, B. and Choi, H. R. (2015). Force Sensor Integrated Surgical Forceps for Minimally Invasive Robotic Surgery. IEEE Transactions on Robotics, 31 (5), 1214–1224. Kimura, T., Mori, T., Konishi, F. and Kitajima, M. (2010). Endoscopic surgical skill qualification system in Japan: Five years of experience in the gastrointestinal field. Asian Journal of Endoscopic Surgery, 3 (2), 66–70. Kuebler, B., Seibold, U. and Hirzinger, G. (2005). Development of actuated and sensor integrated forceps for minimally invasive robotic surger. The international journal of medical robotics + computer assisted surgery : MRCAS, 1 (3), 96–107. Kume, K., Kuroki, T. and Shingai, M. (2015). Development of a Novel Endoscopic Manipulation System: The Endoscopic Operation Robot ver. 2. Hepato-gastroenterology, 62 (140), 843–845. Kwoh, Y. S., Hou, J., Jonckheere, E. A. and Hayati, S. (1988). A robot with improved absolute positioning accuracy for CT guided stereotactic brain surgery. IEEE Transactions on Biomedical Engineering, 35 (2), 153–160. Lanfranco, A. R., Castellanos, A. E., Desai, J. P. and Meyers, W. C. (2004). Robotic Surgery. Annals of Surgery, 239 (1), 14–21. Lazeroms, M., Villavicencio, G., Jongkind, W. and Honderd, G. (1996). Optical fibre force sensor for minimal-invasive-surgery grasping instruments. In Engineering in Medicine and Biology Society, 1996. Bridging Disciplines for Biomedicine. Proceedings of the 18th Annual International Conference of the IEEE, vol. 1, pp. 234–235 vol.1. Lee, E. J., Lee, J. B., Choi, Y. S., Lee, S. H., Lee, D. H., Kim, D. S. and Youk, E. G. (2012). Clinical risk factors for perforation during endoscopic submucosal dissection (ESD) for large-sized, nonpedunculated colorectal tumors. Surgical Endoscopy and Other Interventional Techniques, 26 (6), 1587–1594. Lei, J., Qiu, L., Liu, M., Song, Q. and Ge, Y. (2006). Application of neural network to nonlinear static decoupling of robot wrist force sensor. Intelligent Control and Automation, 2006. WCICA 2006. The Sixth World Congress on, 2, 5282–5285. Liang, C. and Rogers, C. A. (1992). A multi-dimensional constitutive model for shape memory alloys. Journal of Engineering Mathematics, 26 (3), 429–443. Ma, K. Y., Felton, S. M. and Wood, R. J. (2012). Design, fabrication, and modeling of the split actuator microrobotic bee. IEEE International Conference on Intelligent Robots and Systems, pp. 1133–1140. Maffiodo, D., Raparelli, T. and Belforte, G. (2010). Fuzzy Logic Position Control of a Shape Memory Alloy Wire. Vodenje Sistema, pp. 136–145. Mahfuzur Rahman, S. M., Ahn, K. K. and Yoon, J. I. I. (2008). Experimental study of the tracking control of a SMA actuated manipulator. ICSMA 2008 - International Conference on Smart Manufacturing Application, pp. 152–156. 174
Mahoney, A. W., Anderson, P. L., Swaney, P. J., Maldonado, F. and Webster, R. J. (2016). Reconfigurable parallel continuum robots for incisionless surgery. IEEE International Conference on Intelligent Robots and Systems, 2016-November, 4330–4336. Malka, R., Desbiens, A. L., Chen, Y. and Wood, R. J. (2014). Principles of microscale flexure hinge design for enhanced endurance. In IEEE International Conference on Intelligent Robots and Systems, Iros, pp. 2879–2885. Maydeo, A. and Dhir, V. (2017). Third-space endoscopy: stretching the limits. Gastrointestinal Endoscopy, 85 (4), 728–729. Mylonas, G. P., Vitiello, V., Cundy, T. P., Darzi, A. and Yang, G.-z. (2014). CYCLOPS : A Versatile Robotic Tool for Bimanual Single-Access and Natural-Orifice Endoscopic Surgery. IEEE International Conference on Robotics and Automation, pp. 2436–2442. Nezhat, C. (2011). Nezhat’s History of Endoscopy: A Historical Analysis of Endoscopy’s Ascension Since Antiquity. Endo-Press. Nguyen-Tuong, D., Seeger, M. and Peters, J. (2009). Model learning with local gaussian process regression. Advanced Robotics, 23 (15), 2015–2034. Nof, S. (2009). Springer Handbook of Automation, vol. 35. Springer-Verlag Berlin Heidelberg. Noh, Y., Sareh, S., Wurdemann, H., Liu, H., Back, J., Housden, J., Rhode, K. and Althoefer, K. (2015). Three-axis fiber-optic body force sensor for flexible manipulators. Sensors Journal, IEEE, PP (99), 1–1. Oyama, T., Yahagi, N., Ponchon, T., Kiesslich, T. and Berr, F. (2015). How to establish endoscopic submucosal dissection in Western countries. World Journal of Gastroenterology, 21 (40), 11209–11220. Pandya, H. J., Sheng, J. and Desai, J. P. (2017). Towards a tri-axial flexible force sensor for catheter contact force measurement. Proceedings of IEEE Sensors, 26 (1), 264–272. Pappafotis, N., Bejgerowski, W., Gullapalli, R., Simard, J. M., Gupta, S. K. and Desai, J. P. (2009). Towards Design And Fabrication Of A Miniature Mri-Compatible Robot For Applications In Neurosurgery. Detc 2008: 32nd Annual Mechanisms And Robotics Conference, Vol. 2, Pts A & B, (August), 747–754. Park, W. T., Kotlanka, R. K., Lou, L., Hamidullah, M. and Lee, C. (2013). MEMS tri-axial force sensor with an integrated mechanical stopper for guidewire applications. Microsystem Technologies, 19 (7), 1005–1015. Passino, K. and Yurkovich, S. (1998). Fuzzy Control: The Basics. In Fuzzy Control, Addison Wesley Longman, Inc., pp. 24–25. Patel, N., Seneci, C., Yang, G.-Z., Darzi, A. and Teare, J. (2014). Flexible platforms for natural orifice transluminal and endoluminal surgery. Endoscopy International Open, 2 (2), E117–23. Penrose, R. (1954). A generalized inverse for matrices. Mathematical Proceedings of the Cambridge Philosophical Society, 51 (3), 406–413.
175
Petroni, G., Niccolini, M., Menciassi, A., Dario, P. and Cuschieri, A. (2013). A novel intracorporeal assembling robotic system for single-port laparoscopic surgery. Surgical Endoscopy and Other Interventional Techniques, 27 (2), 665–670. Phee, S. J., Ho, K. Y., Lomanto, D., Low, S. C., Huynh, V. A., Kencana, A. P., Yang, K., Sun, Z. L. and Chung, S. C. S. (2010a). Natural orifice transgastric endoscopic wedge hepatic resection in an experimental model using an intuitively controlled master and slave transluminal endoscopic robot (MASTER). Surgical Endoscopy and Other Interventional Techniques, 24 (9), 2293–2298. —, Kencana, A. P., Huynh, V. A., Sun, Z. L., Low, S. C., Yang, K., Lomanto, D. and Ho, K. Y. (2010b). Design of a master and slave transluminal endoscopic robot for natural orifice transluminal endoscopic surgery. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, 224 (7), 1495–1503. Polygerinos, P., Seneviratne, L. D., Razavi, R., Schaeffter, T. and Althoefer, K. (2013). Triaxial catheter-tip force sensor for MRI-guided cardiac procedures. IEEE/ASME Transactions on Mechatronics, 18 (1), 386–396. Puangmali, P., Liu, H., Seneviratne, L., Dasgupta, P. and Althoefer, K. (2012). Miniature 3-axis distal force sensor for minimally invasive surgical palpation. Mechatronics, IEEE/ASME Transactions on, 17 (4), 646–656. Pullens, H. J., Van Der Stap, N., Rozeboom, E. D., Schwartz, M. P., Van Der Heijden, F., Van Oijen, M. G., Siersema, P. D. and Broeders, I. A. (2016). Colonoscopy with robotic steering and automated lumen centralization: A feasibility study in a colon model. Endoscopy, 48 (3), 286–290. Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Ranzani, T., Cianchetti, M., Gerboni, G., Falco, I. D. and Menciassi, A. (2016). A Soft Modular Manipulator for Minimally Invasive Surgery: Design and Characterization of a Single Module. IEEE Transactions on Robotics, 32 (1), 187–200. —, Russo, S., Schwab, F., Walsh, C. J. and Wood, R. J. (2017). Deployable stabilization mechanisms for endoscopic procedures. Proceedings - IEEE International Conference on Robotics and Automation, pp. 1125–1131. Roesthuis, R. J., Kemp, M., van den Dobbelsteen, J. J. and Misra, S. (2014). Three-dimensional needle shape reconstruction using an array of fiber bragg grating sensors. IEEE/ASME Transactions on Mechatronics, 19 (4), 1115–1126. Ruiter, J., Rozeboom, E., Van Der Voort, M., Bonnema, M. and Broeders, I. (2012). Design and evaluation of robotic steering of a flexible endoscope. Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 761–767. Russo, S., Ranzani, T., Gafford, J., Walsh, C. J. and Wood, R. J. (2016). Soft pop-up mechanisms for micro surgical tools: Design and characterization of compliant millimeter-scale articulated structures. 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 750–757. —, —, Walsh, C. J. and Wood, R. J. (2017). An Additive Millimeter-Scale Fabrication Method for Soft Biocompatible Actuators and Sensors. Advanced Materials Technologies, 1700135, 1700135. Ryu, C. and Chen, Y. (2011). Endoscopic Therapy for Gastric Neoplasms, Elsevier Health Sciences. 176
Ryu, S. C. and Dupont, P. E. (2014). Fbg-based shape sensing tubes for continuum robots. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pp. 3531–3537. Saito, Y., Uraoka, T., Matsuda, T., Emura, F., Ikehara, H., Mashimo, Y., Kikuchi, T., Fu, K. I., Sano, Y. and Saito, D. (2007). Endoscopic treatment of large superficial colorectal tumors: a case series of 200 endoscopic submucosal dissections (with video){A figure is presented}. Gastrointestinal Endoscopy, 66 (5), 966–973. Salerno, M., Tognarelli, S., Quaglia, C., Dario, P. and Menciassi, A. (2013). Anchoring frame for intra-abdominal surgery. The International Journal of Robotics Research, 32 (3), 360–370. Saunders, C., Gammerman, A. and Vovk, V. (1998). Ridge regression learning algorithm in dual variables. Proceedings of the Fifteenth International Conference on Machine Learning, pp. 515–521. Schwalb, W., Shirinzadeh, B. and Smith, J. (2017). A force-sensing surgical tool with a proximally located force/torque sensor. International Journal of Medical Robotics and Computer Assisted Surgery, 13 (1). Shalev-Shwartz, S., Singer, Y., Srebro, N. and Cotter, A. (2011). Pegasos: primal estimated sub-gradient solver for svm. Mathematical Programming, 127 (1), 3–30. Shang, J., Noonan, D. P., Payne, C., Clark, J., Sodergren, M. H., Darzi, A. and Yang, G. Z. (2011). An articulated universal joint based flexible access robot for minimally invasive surgery. Proceedings - IEEE International Conference on Robotics and Automation, 1, 1147–1152. Shergill, A. K., McQuaid, K. R. and Rempel, D. (2009). Ergonomics and GI endoscopy. Gastrointestinal Endoscopy, 70 (1), 145–153. Shi, C., Luo, X., Qi, P., Li, T., Song, S., Najdovski, Z., Fukuda, T. and Ren, H. (2017). Shape sensing techniques for continuum robots in minimally invasive surgery: A survey. IEEE Transactions on Biomedical Engineering, 64 (8), 1665–1678. Shi, Z. Y., Liu, D. and Wang, T. M. (2014). A shape memory alloy-actuated surgical instrument with compact volume. Int J Med Robotics Comput Assist Surg, 2000 (October 2013), 474–481. Shimomura, T., Ishiguro, S., Konishi, H., Wakabayashi, N., Mitsufuji, S., Kasugai, T., Manou, M. and Kodama, T. (2004). New indication for endoscopic treatment of colorectal carcinoma with submucosal invasion. Journal of gastroenterology and hepatology, 19 (1), 48–55. Simaan, N., Bajo, A., Reiter, A., Wang, L., Allen, P. and Fowler, D. (2013). Lessons learned using the insertable robotic effector platform (IREP) for single port access surgery. Journal of Robotic Surgery, 7 (3), 235–240. Sivak, M. V. (2005). Gastrointestinal endoscopy: past and future. Gut, 55 (8), 1061–1064. Smola, A. and Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14 (3), 199–222. Sreetharan, P. S., Whitney, J. P., Strauss, M. D. and Wood, R. J. (2012). Monolithic fabrication of millimeter-scale machines. Journal of Micromechanics and Microengineering, 22 (5), 055027. Szewczyk, J., de Sars, V., Bidaud, P. and Dumont, G. (2001). An active tubular polyarticulated micro-system for flexible endoscope, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 179–188. 177
Tabrizi, V. and Moallem, M. (2007). Nonlinear Position Control of Antagonistic Shape Memory Alloy Actuators. 2007 American Control Conference, (1), 88–93. Tai, N. T. and Ahn, K. K. (2010). A RBF neural network sliding mode controller for SMA actuator. International Journal of Control, Automation and Systems, 8 (6), 1296–1305. Thorlacius, H., Uedo, N. and Toth, E. (2013). Implementation of Endoscopic Submucosal Dissection for Early Colorectal Neoplasms in Sweden. Gastroenterology Research and Practice, 2013. Toi, Y., Lee, J.-b. and Taya, M. (2004). Finite element analysis of superelastic , large deformation behavior of shape memory alloy helical springs. Computers and Structures, 82, 1685–1693. Trong Tai, N. T. and Ahn, K. K. (2010). Apply Adaptive Fuzzy Sliding Mode Control to SMA Actuator. International Conferance on Control, Automation and System, (ICROS), 433–437. USSEC (2017). Apollo Endosurgery, Inc. Annual Report. Tech. rep. Valdastri, P., Harada, K., Menciassi, A., Beccai, L., Stefanini, C., Fujie, M. and Dario, P. (2006). Integration of a miniaturised triaxial force sensor in a minimally invasive surgical tool. IEEE Transactions on Biomedical Engineering, 53 (11), 2397–2400. Vijayakumar, S. and Schaal, S. (2000). Locally weighted projection regression: An o(n) algorithm for incremental real time learning in high dimensional spaces. Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), 1, 288–293. Vitiello, V., Lee, S., Cundy, T. and Yang, G. (2013). Emerging Robotic Platforms for Minimally Invasive Surgery. IEEE Reviews in Biomedical Engineering, 6, 111–126. Wang, A. Y. (2011). The international emergence of endoscopic submucosal dissection for early gastric cancer. Gastrointestinal Endoscopy, 73 (5), 928–931. Wang, T. M., Shi, Z. Y., Liu, D., Ma, C. and Zhang, Z. H. (2012). An accurately controlled antagonistic shape memory alloy actuator with self-sensing. Sensors, 12 (6), 7682–7700. Wang, X. (2008). Non-linearity estimation and temperature compensation of capacitor pressure sensors using least square support vector regression. Knowledge Acquisition and Modeling Workshop, 2008. KAM Workshop 2008. IEEE International Symposium on, pp. 1161–1164. Watanabe, T., Iwai, T., Fujihira, Y., Wakako, L., Kagawa, H. and Yoneyama, T. (2014). Force sensor attachable to thin fiberscopes/endoscopes utilizing high elasticity fabric. Sensors (Switzerland), 14 (3), 5207–5220. Waye, D., Rex, D., Williams, C. and Dumont, G. (2004). Oxford, United Kingdom, pages=: Blackwell Publishing. Webster, R., Okamura, A. and Cowan, N. (2006). Toward Active Cannulas: Miniature Snake-Like Surgical Robots. 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2857–2863. Webster, R. J. (2007). Design and Mechanics of Continuum Robots for Surgery. Ph.D. thesis, Johns Hopkins University. Xie, T. (2011). Recent advances in polymer shape memory. Polymer, 52 (22), 4985–5000. 178
Yao, Z., Wang, F., Wang, W. and Qin, Y. (2010). Neural-network-based six-axis force/torque robot sensor calibration. Electrical and Control Engineering (ICECE), 2010 International Conference on, pp. 1336–1338. Yeung, B. P. M. and Chiu, P. W. Y. (2016). Application of robotics in gastrointestinal endoscopy: A review. World Journal of Gastroenterology, 22 (5), 1811–1825. — and Gourlay, T. (2012). A technical review of flexible endoscopic multitasking platforms. International Journal of Surgery, 10 (7), 345–354. Yonezawa, J., Kaise, M., Simiyama, K., Goda, K., Arakawa, H. and Tajiri, H. (2006). A novel double-channel therapeutic endoscope, r-scope, facilitates endoscopic submucosal dissection of superficial gastric neoplasms. Endoscopy, 38 (10), 1011–1015. Zakerzadeh, M. R. and Sayyaadi, H. (2013). Precise position control of shape memory alloy actuator using inverse hysteresis model and model reference adaptive control system. Mechatronics, 23 (8), 1150 – 1162. Zhao, B. and Nelson, C. A. (2015). Sensorless Force Sensing for Minimally Invasive Surgery. Journal of Medical Devices, 9 (4), 041012. Zhu, L., Qin, J., Wang, J., Guo, T., Wang, Z. and Yang, J. (2016). Early Gastric Cancer: Current Advances of Endoscopic Diagnosis and Treatment. Gastroenterology Research and Practice, 2016 (1).
179
Appendix A
Custom Evaluation System Design A.1
Introduction
A full thermomechanical characterization of SMA actuators requires a means of measuring and controlling the stress, strain, temperature, and current applied to the actuator, sometimes simultaneously. While commercial tensile testing systems and dynamic mechanical analysis (DMA) machines can control the stress and strain, tightly integrating temperature and current control into proprietary commercial control software, and syncing subsequent data acquisition, is not straightforward. This appendix describes the detailed design and implementation of a custom mechanical characterization platform and associated control implementation for systematically analyzing, characterizing, validating and simulating mechanisms, sensors and actuators built using PrintedCircuit MEMS and similar mesoscale manufacturing techniques. Although used extensively for thermomechanical characterization of custom SMA actuators as described in Chapter 3, the open-architecture design has allowed the system to support soft sensor characterization and cyclic/long-term testing of fabric-based sensors (Atalay et al., 2017b,a).
A.2
System Overview
To support research in meso- and milli-scale manufacturing of sensors, actuators and mechanisms for surgical robotics applications, we have developed custom hardware for thermomechanical
180
Linear-Encoder-with 7-micron-resolution
es ch n i 9-
Timing-Belt-with 3:7-Reduction
Precision-Ballscrewwith-3mm-Lead
Dual-Radial Bearing-Block
ATI-Nano77
72-inches
Electromechanical Power2Off-Brake
Linear-Carriage
SMA-Specimen
Helical-Coupling
Maxon-motorb74:7-Gearheadb 5NN-CPR-Encoder
Mechanical Breadboard
5-i nch
es
Figure A.1: Overview of custom evaluation system with callouts to important features (configuration shown here is for evaluating SMA actuator performance).
characterization and performance validation. The system, shown in Fig. A.1, features a single degree-of-freedom moving metrology carriage that can be precisely positioned in relation to a specimen holder. The metrology carriage features linear optical quadrature encoding to resolve its position in space with 0.5 µm resolution. An ATI Nano17 6-axis load cell is used to apply or measure forces and close a force control loop for quasi-steady and dynamic mechanical characterization. The system is designed to execute known position/force-based trajectories while acquiring data and writing to a file for post-processing. The system occupies a space of 12"x9"x5", making it extremely modular and portable. The system can be mounted vertically as shown for characterization efforts. It can also be mounted on its side and back depending on the needs of the experimental protocol.
A.3
Mechanical Subsystem
The mechanical system consists of two subsystems, a primary drive and a secondary drive. The function of the primary drive is to generate and transmit rotary motion to the secondary drive, 181
Figure A.2: Sectional view of the primary drive subsystem, where the adjustable preload methodology places the dual radial ball bearings in a quasi back-to-back configuration.
which converts this rotary motion into linear motion.
A.3.1 Primary (Motor-Side) Drive The primary drive features a Maxon ECMax 30 brushless DC motor (40W, P/N272768) with a 14:1 gearhead reduction and a 1000 count-per-revolution optical encoder. The motor is coupled to the driveshaft through a helical coupling, and the driveshaft is rigidly attached to an electromechanical power-off brake to hold the load in the case of a sudden power loss. When power is provided to the system, the brake disengages and the load is held by the holding torque of the motor. The brake also engages when (1) the secondary side linear stage exceeds positive and negative travel limits, or (2) the applied load exceeds the safe load limits of the load cell attached to the stage on the secondary side as defined in software. A custom-designed bearing block (with bearing surface tolerances machined according to ISO standard) implements a dual radial bearing system in a quasi-back-to-back configuration made possible by a spring-loaded adjustable preload as shown in Fig. A.2 to increase the effective moment arm for radial load resistance (primarily due to timing belt pre-tension), and also to provide some thrust load resistance (although not anticipated during normal operation). The driveshaft is coupled to the secondary drive via a timing belt setup with a 1:2.5 reduction and an adjustable belt tensioner. 182
Figure A.3: Constraint analysis of proposed secondary drive, illustrating the need for off-axis compliance in monolithic carriage design: (from left) free and constrained DoFs from the ballscrew alone, from the addition of a single guiderail, and from the addition of the second guiderail. Table A.1: Tabulated constraint analysis of secondary drive.
Element Ballscrew First Guiderail Second Guiderail
New Constraints x, y, θ x , θy θz none
Free DoFs z, θz z z
Over-Constraints none x, y, θ x , θy x, y, θ x , θy , θz
A.3.2 Secondary (Load-Side) Drive The secondary drive timing pulley drives a preload-adjustable .375" ballscrew with 0.125" lead (PowerTrac 0375, Nook Industries) which is supported on one end by dual back-to-back angular contact bearings, and simply supported on the other end by a singular radial bearing. Given the gear ratio of the planetary gearhead and the timing belt configuration, the ballscrew advances roughly 500 µm for every rotation of the motor. The ballnut is threaded into a custom-machined carriage with integrated flexures to tolerate misalignment between the screw and the guiding rails, described more thoroughly in the following subsection. The carriage features a linear encoder on the underside (Heidenhain AK Lida 409 encoder tape and 47-series optical read-head with 5 µm resolution for x1 quadrature, or just over 1 µm for x4 quadrature) which is used to close the position control loop of the stage. A mechanical breadboard is screwed onto the carriage, with a fastening pattern that features two overlapping 1"x1" grids of 1/4-20 threaded holes. Two photointerruptors and status LEDs define positive and negative travel limits which are wired into a lower-level travel enable/disable loop on the motor amplifier. 183
A high-stiffness, lightweight chassis (CNC-milled out of 6061 Aluminum) supports the entire structure, and also provides alignment features for the linear optical encoder tape and runthroughs for cabling. The chassis is assembled via pin-and slot features that ensure precise alignment between mating components. Bolt through-holes and angle brackets allow the system to be mounted in various configurations to any mechanical or optical breadboard with a 1" x 1" grid of 1/4-20 threaded holes.
Monolithic Carriage Design Any leadscrew/ballscrew system supported by guiderails is inherently overconstrained which can lead to premature system failure if not appropriately designed for. An overconstraint analysis of the proposed system is illustrated in Fig. A.3 and tabulated in Table A.1. If we idealize a ballscrew as a guiderail where the carriage is free to slide up and down (i.e. ignoring the threaded engagement), the system has a mobility of 2 as the ballscrew permits axial translation and rotation along the desired degree-of-freedom but prohibits motion in all other directions. The addition of the first guiderail serves to further constrain the axial rotation DoF, however it also overconstrains all of the DoFs already constrained by the ballscrew itself. The addition of the second guiderail serves only to increase the stiffness of the system and improve positioning accuracy through symmetry and elastic averaging. However in terms of degrees of freedom and constraint, no new degree of freedom is constrained by the additional guiderail, and the system is further overconstrained. Given the substantial overconstraint, it is important to design features into the carriage or the chassis to tolerate overconstraint in the event of misalignment during assembly, while preserving high-stiffness properties in anticipated operational loading directions. Fig. A.4 shows a finite element analysis of the final monolithic carriage design, performed in SolidWorks Simulation. The carriage was designed to be milled from a single block of Aluminum 6061-T6. An inner-threaded axisymmetric flexure bearing threads onto the ballnut and serves to add compliance between the ballscrew and the guiderails by permitting small misalignments in x, y, θ x and θy . A blade flexure serves to add further compliance between the guiderails. The bottom row shows the anticipated performance of the carriage during normal operation,
184
DisplacementL[mm]
FOS:L5.8
δmax:L180Lμm
FOS:L29
δmax:L10Lμm
OperationalLLoad
Misalignment
vonLMisesLStressL[Pa]
Figure A.4: (left) Finite element analysis of monolithic carriage design under misalignment (off-axis) loading and operational (on-axis) loading (the mesh is refined to 0.001" element sizes on the flexural elements (roughly 1/6th of the flexure thickness), and 0.01" element size otherwise), (right) photograph of monolithic carriage integrated into the secondary drive.
where a 100N load is applied in −z and offset from the surface of the carriage by roughly two inches to simulate a cantilevered load that might be encountered during a characterization task. Such a load results in a moment Mx that is transferred to the leadscrew and guiderails via the monolithic carriage which should ideally be stiff under this loading situation to preserve system accuracy. We observe a substantial factor-of-safety of 29 and a maximum displacement of 10 µm, resulting in a stage stiffness of 10,000 N/mm. As such, the designed flexure is compliant in off-axis directions but stiff along the axis of anticipated loading. The manufactured carriage, integrated into the secondary drive subsystem, is shown in Fig. A.4 (right).
A.4
Electrical and Software
The system utilizes a dedicated PC with MATLAB xPC Real-Time Kernel for high-performance target program execution and data acquisition. A Dell Precision T5810 workstation running MATLAB/Simulink xPC target library serves as the host computer which compiles Simulink models and builds them onto the target PC (another A Dell Precision T5810 workstation) for real-time execution. All data acquisition, quadrature decoding and control signalling is performed by a National Instruments PCI-6220 Multifunction DAQ card and its respective control blocks
185
Vref Buffer
Encoder Brake Limit
Limit PCB
Gage V
AccelnetEMotion ControlEPanel
Analog Out Digital I/O
ATIEController F/TEModule
Analog In
Omega SigEConditioning
NationalEInstruments PCIB6220EMultifunctionE DAQ
Ref Halls
SMA+
TargetEScopeEonEDedicatedEMonitor
Simulink Real-Time xPC OS
m for lat
HostEMachine DellEPrecision
EP
MATLAB/ Simulink R2015a
SMAEPower/ SenseEElectronics
VGA Out
Ethernet
SMA-
on ati lid Va
ETargetEMachine DellEPrecision
VGA Out
IRETemp
PWM/Analog Out Current/Self Sense
PCIe
24VEPower Supply
VGA
MotionEControlEGUI
Figure A.5: Schematic of custom evaluation system, configured for thermomechanical analysis of SMA actuators.
in the Simulink real-time target library. A dedicated monitor enables real-time visualization of sensor/system parameters via VGA output from an accessory kit. The motor is driven by a Copley controls Accelnet controller/amplifier which handles motor commutation via the hall-effect sensors and primary-side (rotary) quadrature encoder, and also closes a lower-level current control loop. A custom-designed limit switch PCB provides crisp logiclevel signals from the optical limit switches to the Accelnet controller which closes a lower-level travel limit loop directly on board the Accelnet controller itself (i.e. independent of the real-time target controller). A 24V power supply provides power to the entire system. A schematic of the entire system (configured for SMA thermomechanical characterization) is shown in Fig. A.5.
A.4.1 Software and Controls A custom graphical user interface (GUI) was developed to enable parameter tuning in real time without the need to re-compile the model to the target system during experiments, as shown in Fig. A.6. The GUI enables a user to connect and build the model to the target system, run the model, manually position the stage through jog/home commands, and execute various positionor force-controlled profiles (sinusoidal, square wave, ramp, chirp). The controller has two basic modes: SMA characterization mode (where an SMA current loop is enabled, in addition to sensor feedback from current sensors and resistance sensors) and general characterization mode (where
186
Figure A.6: Graphical user interface for executing experimental protocols.
SMA mode is disabled, and two analog inputs are enabled for data acquisition).
Control Structures Fig. A.7 shows a block diagram of the control structures implemented on the real-time controller. The xPC target provides control for both the evaluation platform subsystem (either position or virtual spring control of the stage) and the SMA drive/control subsystem (either current or temperature control of the SMA specimen), often simultaneously depending on the experimental protocol. A master controller selects the appropriate controllers based on the desired experimental protocol.
Master Control The master Simulink implementation is shown in Fig. A.8. The controller has ten analog inputs described below: • Temp: Analog input from Omega OS100E infrared non-contact temperature sensor • Ss1 , Ss2 : Self-sensing inputs from SMA power electronics (not used) • Cs1 , Cs2 : Current sensor inputs from SMA power electronics 187
Moving Stage
ATI Nanob7
Secondary Transmission Primary Drive Drive xPCVTarget Differential
Moving Clamp
SMA Specimen
Phasing
Copley Amplifier
Experimental Protocol
Position PIDVController
Quadrature Decoder
VirtualVSpring PIDVController
FhT Controller
LowvPass bUUVHz
Current Sensor
LowvPass 5VHz
Current PDVController
IRVTemp Sensor
LowvPass +UVHz
Temperature PIDVController
Ground Clamp
DriveVSignal
MotionVControl Logic Master Controller
Vv
SMAVPower Electronics
V1
xPCVHost Matlabh SimulinkVxPC
GraphicalV UserVInterface
SMAVControl Logic
PWMVDriveVSignal
Figure A.7: Master controller block diagram of evaluation platform, showing the four different controllers embedded on the system’s real-time kernel.
• Jog: Analog input from manual jogging potentiometer • ATI Fz , Tx , Ty : Differential force and torque inputs from ATI Nano17 force sensor • AI: External analog input for general use The system also has eight digital inputs: four for the linear encoder, and four for manual positioning controls (Homing, zeroing, negative and positive jogging). The controller provides analog outputs for controlling the linear stage as well as the current through the specimen being tested. Digital outputs are provided to the power-off break to protect the load cell in the case of overloading, based on force thresholding implemented in the ‘Load Cell Logic’ block.
Velocity Controller The velocity controller shown in Fig. A.9 is implemented in the ‘Velocity Controller for Jogging’ block and is only active when manual positioning mode is selected. The stage position is differentiated and filtered, and the error between the desired and actual velocity is passed into a PI loop that was tuned heuristically.
188
Host Scope Id: 6 Auto Data Host Scope1
TrajEx Auto Switch
wf_select
|u|
u_V
Waveform Select Clock
feedforward
Select Waveform1
Calculate Runtime Time
Execute
-K-
Session Time
Abs
Runtime
Runtime
pos_d
Encoder Position
sec/count
Target Scope Id: 1
Position
Position Scope Scope Trigger
Force Waveform
Data Out
feedforward
PCI-6229 National Instr. Encoder
Host Scope Id: 5
Auto Trajectory Controller (Pos/Force) 0
Encoder Count Zero Signal
PCI 6229 ENC
Zeroed Position Absolute Position
Encoder Zeroing
Auto Data Host Scope
Zeroed Stage Location [mm]
Auto Encoder Position
Absolute Stage Location [mm]
Manual
Hardware Speed u_V Negative Jog DI
Control Effort
u_V
Home
Positive Jog DI
2
Motor Control Effort
Hardware Home
Velocity Controller for Jogging
Hardware Home
Motor Control Effort
Force
PCI-6229 National Instr. Digital Input
6 3 7
Hardware Zero Negative Jog
Mode Selection
OR
GoHome
SMA Control Effort
1 2
PCI-6229 DA Logical Operator1
Go Home
Positive Jog
PCI-6229 DI 0
Home Position
Zeroed Stage Location [mm]
u_V
Force [N]
Host Scope Id: 2
Home Current Position
1
7
Temp
Self-Sensor
Current Sensor
Pot Out
Homing Controller
3 PCI-6229 National Instr. Analog Input
Jog Potentiometer
Nano Fz
Nano Tx
Current Sensor [A]
Cs 1 [V]
Control Effort Force
Self-Sensor
desired position
Pot In
C_filt 1
Current Sensor
Nano Ty
Self Sensor [V]
SMA Target Scope Current, Self-Sense, Control
Stage Transparency/Virtual Spring
Nano Fz [V] Forces
Force [N]
Balanced Force Fz
Target Scope Id: 3
Nano Tx [V]1
Nano Ty [V]1
T [Nmm]
Overload Fault
6
Nano Tz
Futek Fz
Load Cell Logic
Torque [N/mm]
4
Temp [C]
Temperature [C]
FUTEK Fz
AI 1
External Input PCI-6229 AD
>0
8
0
Nano Tz [V]1
Extra AI
14
Stage/Force Scope Numerical
Latch Fault
OutputEnableTrigger Output Enable 5
Target Scope Id: 4
Force [N]
Spring Force
Torques
12
Force Scope Graphical
Self-Sense 1
F [N]
4
Target Scope Id: 7
Jog Potentiometer
Stage Location
13
Force Checking/ Manual Data Host Scope
SMA Sensor Data
converge?
Terminator1
Temp
Self-Sense Adj 1
8
PCI-6229 National Instr. Analog Output
AI Signal Conversion and Filtering
Extra AI
Feedback
control effort 1
Current 1
Sense 1 Out
Self Sense 1
Sense Raw
Temperature
Setpoint
Null Out
PCI-6229 DO
Switch
Self Sensor [V]
PCI-6229 National Instr. Digital Output
-CEnc Check1
NOT
Convert
Not1 Data Type Conversion1
SMA Controller
Figure A.8: Simulink implementation of the master control system.
Position Controller The position controller shown in Fig. A.10 features a PID loop with velocity/acceleration feedforward terms and a feedforward term for gravity compensation. Gains were tuned heuristically. This controller is implemented in the ‘Auto Trajectory Controller’ block and is used to execute deterministic stage position trajectories for characterization.
Force Controller The force controller shown in Fig. A.11 is based on Fz feedback from the ATI Nano17 and features a PID loop with feedforward gravity compensation. Gains were tuned heuristically. This controller is implemented in the ‘Auto Trajectory Controller’ block and is used to execute deterministic stage
189
Position Differentiation
Velocity PI Loop K Ts z-1
1 v_desired
Ki
Discrete-Time Integrator
butter
K (z-1) Ts z
2 p_actual
20
v_actual
.45 Low-Pass Filter
Discrete Derivative1
Kp
1 V_out
Voltage Limiting
Figure A.9: Simulink implementation of high-level velocity controller for manual jogging and positioning. Feedforward
PID Loop with Gravity Compensation butter .001
K (z-1) Ts z Velocity FF
Low-Pass Filter1
.01
Kvff
K Ts z-1
1500
butter
K (z-1) Ts z
.001
Acceleration FF Low-Pass Filter2
Ki
Kaff
gravity term
Discrete-Time Integrator
65
Product
3 Feedforward Modulator
Kp2
1 Pos_d_mm
Kd
butter
K (z-1) Ts z
.35
Voltage Limiting
Discrete Derivative
2 Pos_act_mm
Low-Pass Filter
1 V_out
Dead Zone
Figure A.10: Simulink implementation of high-level position controller (PID control with velocity/acceleration/gravity feed-forward).
PID Loop with Gravity Compensation
Ki 1 Force_d_N
16
2 Force_act_N
1.9
.5
K Ts z-1
9
gravity term
Discrete-Time Integrator
Kp
Kd
K (z-1) Ts z Discrete Derivative
Voltage Limiting
butter Low-Pass Filter
1 u_V
Dead Zone
Figure A.11: Simulink implementation of high-level force controller based on feedback from ATI Nano17 load cell.
force trajectories for characterization.
Virtual Spring Controller The virtual spring controller shown in Fig. A.12 is used to simulate antagonism for SMA characterization. The protocol for virtual spring control is given below: (1) Jog stage to pre-stretch specimen by δ0 and measure the force F0 at this location. 190
Position PID Controller .1 Kd
K (z-1) Ts z Discrete DerivativeDead Zone
10 Kp2 2 Stage Location
10
Constraint Equation Solving Kp1
2 desired delta k_spring Stiffness delta_0 Pre-Stretch F_0 Initial Force
k delta_0 F_0 F_curr last_delta
Voltage Limiting
1 V_out
K Ts z-1 Discrete-Time Integrator
delta
3 Net Force
delta_0_b
Divide
Virtual Spring 1 Force_act
butter Low-Pass Dead Zone1 Filter Memory
Figure A.12: Simulink implementation of virtual spring controller, where a custom MATLAB function computes desired spring position based on force feedback and desired bias spring parameters, which is subsequently fed into a PID position controller.
(2) Configure stage as a virtual spring such that a restoring force Fb = k (δ0,b + ∆) is applied opposite the direction of contraction of the specimen, where k is the desired bias spring linear stiffness and δ0,b is the virtual bias pre-stretch given by F0 /k. (3) As the specimen is actuated, update the position to preserve (quasi-) static equilibrium in the current configuration (i.e. Fb = F). As the specimen is actuated, the virtual spring controller solves for the displacement required such that the actuator force and virtual bias force are equivalent, as per the following algorithm: Algorithm 1 Virtual Spring Controller 1:
Initialize δ0 , F0 , k b , δ0,b
2:
Initialize ∆ = 0
3:
while Controller On do
4:
Measure F
5:
k act = F/(δ0 − ∆); % solve effective linear stiffness of specimen
6:
∆=
7:
end while
k act δ0 −k b δ0,b k act +k b ;
% update displacement
191
Current SMA Controller K (z-1) Ts z
2 Kd1 1 C_actual
500
2 C_desired
1000
Discrete Derivative
Kp1
Ki1
Saturation2
1 SMA1 Out
K Ts z-1 Discrete-Time Integrator
Figure A.13: Simulink implementation of SMA current controller based on feedback from analog current sensors in SMA driver module.
This displacement ∆ ultimately drives a position PID controller, and Algorithm 1 is implemented in the form of a custom MATLAB function. This controller is implemented in the ‘Stage Transparency/Virtual Spring’ block.
Current Controller The current controller shown in Fig. A.13 features a PID loop based on feedback from each ratiometric current sensor in the SMA power electronics module that is aggressively filtered to average out the PWM fundamental. Gains were tuned heuristically. This controller is implemented in the ‘SMA Controller’ block and is used to control the current through the SMA specimen being tested.
Temperature Controller The temperature controller shown in Fig. A.14 features a PID loop based on feedback from an Omega OS100E infrared non-contact temperature sensor, which subsequently provides a reference current for the current PID controller described previously. Gains were tuned heuristically. This controller is implemented in the ‘SMA Controller’ block and is used to control the temperature of the SMA specimen being tested.
192
Temperature PID Loop K (z-1) Ts z
1 Kd1 1 T_actual
Heat-up only
Discrete Derivative
4 Kp1 10 Ki1
2 T_desired
>0
Saturation2 K Ts z-1
0
1 SMA1 Out
Switch
Constant
Discrete-Time Integrator
Figure A.14: Simulink implementation of SMA temperature control based on feedback from non-contact infrared temperature sensor.
A.5
System Performance Benchmarks
A.5.1 Force and Position Control To demonstrate system utility in performing precision position- and force-based tasks, the stage was commanded to execute both smooth (sinusoidal) and non-smooth (triangular) trajectories in both control modes. For position control, the stage was outfitted with a 5N dummy load. For force control, an extension spring with a stiffness of roughly 0.1 N/mm was attached to the stage. The results are shown in Fig. A.15. We observe that, in position control mode, the stage is capable of following dynamic trajectories with less than 40 µm tracking error. In force control mode, the stage is capable of tracking the input force with less than ±100 mN error for a relatively soft load, as shown in Fig. A.16. Force tracking performance is generally better for stiffer loads.
System Identification The system’s position control dynamics were quantified by commanding a chirp signal with δpp =2mm, where the frequency range is 0.01-20Hz. The stage’s actual position was recorded at a rate of 2 kHz, and data were post-processed in MATLAB. The findpeaks() function was used to locate the stage’s actual amplitude compared to the commanded amplitude, and calculate the phase difference between input and output, as shown in Fig. A.17 (a). The resulting Bode plot, shown in Fig. A.17 (b), shows the actual data compared to a model fit to the system. The magnitude drops below -3 dB at about 9 Hz, setting an upper limit on the system bandwidth in 193
Displacement [mm]
Displacement [mm]
Position Control 10 Actual Commanded
0
-10 0
5
10
15
Position Control 10 Actual Commanded
5
0 0
5
40 20 0 -20 -40 0
5
10
10
15
Time [s]
Position Error [um]
Position Error [um]
Time [s]
15
40 20 0 -20 -40 0
5
Time [s]
10
15
Time [s]
(a)
(b)
Figure A.15: Position control example with a 5N dummy load: (a) sine wave (desired vs. actual stage location) and tracking error, (b) triangle wave (desired vs. actual stage location) and tracking error.
Force Control
Force Control 1.5 Actual Commanded
Force [N]
Force [N]
1
0
-1
Actual Commanded
1 0.5 0
0
5
10
15
0
5
0.1
0
-0.1 0
5
10
10
15
Time [s]
Force Error [N]
Force Error [N]
Time [s]
15
0.1
0
-0.1 0
Time [s]
5
10
15
Time [s]
(a)
(b)
Figure A.16: Force control example for a soft elastic load: (a) sine wave (desired vs. actual applied load) and tracking error, (b) triangle wave (desired vs. actual applied load) and tracking error.
position control mode.
A.5.2 Temperature and Current Control For SMA thermomechanical characterization purposes, we are generally interested in providing a step input and seeing how the thermodynamics evolve under (1) fixed current (variable thermody-
194
Magnitude [dB]
1.5
0.5
0 Experimental Model Fit
-200 -400
10-3
10-2
10-1
100
101
100
101
Frequency [Hz]
0 2
Phase [rad]
Amplitude [mm]
1
200
-0.5 Desired Actual Desired Peaks Actual Peaks
-1
-4
-1.5 0
10
20
0
30
40
50
60
70
80
90
100
Experimental Model Fit
-2
10-3
10-2
10-1
Frequency [Hz]
Time [s]
(a)
(b)
Figure A.17: System identification: (a) chirp waveform used in dynamic characterization, from which magnitude and phase information are extracted, (b) System dynamic analysis for position control of a 1mm sine wave, and the resulting model fit. Dynamic analysis shows a bandwidth of around 9 Hz for a 1mm peak-to-peak displacement.
1.4 100
1.2
Temperature [C]
Current [A]
1 0.8 0.6 0.4 0.2 Commanded
80
60
40
20
Actual
Commanded
Actual
0 0
10
20
30
40
50
60
0
2
4
Time [s]
6
8
10
Time [s]
(a)
(b)
Figure A.18: SMA thermodynamics control. where shaded error bar denotes 1σ for n = 5: (a) current control results for maintaining steady current through SMA specimen, (b) temperature control results for maintaining steady temperature through SMA specimen.
namic conditions) or (2) fixed temperature (isothermal conditions). Example current/temperature control performance is shown in Fig. A.18, demonstrating that the controllers shown in Figs. A.13 and A.14 can control current to within 20 mA (1.7% FS) and temperature to within 4 degrees (4% FS) which is adequate for thermomechanical characterization purposes.
195
Relative Resistance R/R0
50 ε=0.2 ε=0.4 ε=0.8 ε=0.8, laser-treated
40
LED 5V 200 Ω Conductive Electrode
30 ΔL
ε=0.0
ε=0.5
ε=2.0
ε=2.5
Conductive Electrode
20 L0
10
0 0
0.5
1
1.5
2
2.5
ε=1.0
Strain ε (a)
ε=1.5 (b)
Figure A.19: Soft sensor characterization: (a) relative change-in-resistance of samples with varying pre-strain and laser treatment (strained until conductivity is lost), (b) a sensor’s performance as an electrical conductor, where the LED’s brightness shows qualitatively the level of electrical conductivity at various application strains.
A.6
Soft Sensor Characterization
In addition to SMA thermomechanical characterization efforts, the system was also used to characterize the performance of soft sensors and high-strain electrical conductors for wearable robotics applications. The sensors described in (Atalay et al., 2017b) consist of pre-strained elastomeric sheets, upon which aluminum is sputtered on the top and bottom face to create a capacitive sensor. The pre-strain is released, causing the surface morphology to buckle, thereby improving electrical conductivity characteristics for large strains. These sensors could have potential applications in human motion monitoring, physiology monitoring, and human-machine interaction. In characterizing the performance of these soft sensors, the PCMEMS evaluation system was preferred over commercial material testers due to its ability to execute high-displacement (20 cm), fast (80 mm/s) tensile profiles for cyclic testing purposes. Fig. A.19 shows an example characterization as well as a conductivity demonstration. In Fig. A.19 (a), samples of varying pre-strain and laser surface microtreatment are stretched until conductivity is lost, showing that more pre-strain results in much better performance over greater application strains. We also observe the effects of laser surface microtreatment which improves performance further. In In Fig.
196
A.19 (b), the sensor is configured in series with an LED, qualitatively demonstrating robustness as a soft conductive elastomer for wearable applications.
A.7
Dynamic Calculations
This section describes the dynamic calculations which contributed to motor and transmission design.
A.7.1 Dynamic Bandwidth In calculating dynamic bandwidth, we assume a first-order approximation of stage displacement:
f =
0.35 tr
(A.1)
Here, tr is the mechanical time constant of the motor/ballscrew assembly, which is given by the following:
tr =
( Jm + Jl + Jls ) R arm N1 Kτ Kv N2
(A.2)
where Jm , Jl and Jls are the motor, load, and shaft/leadscrew inertias (respectively), R arm is the winding impedance of the motor, Kτ is the motor torque constant, N1 /N2 is the transmission ratio and Kv is the motor speed constant. Load and shaft/leadscrew inertias are given by the following:
Jl = m
Jls =
1 2πP
2
πLρr4 2
(A.3)
(A.4)
where m is the mass of the load, P is the pitch of the leadscrew, L is the length of the leadscrew, ρ is the density of the screw/shaft material, and r is the radius.
A.7.2 Oscillation Amplitude Calculating the displacement x (t) as a function of frequency f and amplitude δ, we assume a sinusoidal profile: 197
Table A.2: Comparison of motor requirement based on system desired specifications with manufacturer-specified performance metrics (Note: (*) denotes transmission-dependent specifications).
Parameter Load Torque Acceleration Torque Reflected Motor Torque Maximum Motor Speed Maximum Stage Speed Motor Power Motor Current
System Design 57.6 49.1 28.9 8210 77.6 23.4 0.98
Motor Nominal N/A* N/A* 33.8 7220 N/A* 40.0 1.49
Motor Max N/A* N/A* 160 9250 N/A* 158 6.57
x (t) = 2δ sin(2π f t)
Units mN · m mN · m mN · m RPM mm/s W A
(A.5)
Taking the derivative allows us to compute the (linear) velocity (v(t)) and acceleration (a(t)) profile, which we can use to calculate the required angular velocity (ωm ) and angular acceleration (αm ) of the motor: ωm PN1 dx = 4π f δ cos(2π f t) = dt N2
(A.6)
αm PN1 dv = −8π 2 f 2 δ sin(2π f t) = dt N2
(A.7)
v(t) =
a(t) =
The velocity function is maximized when cos(2π f t) = 1, so we can solve for the maximum displacement at a specific frequency given the maximum velocity of the motor:
δmax =
ωm PN1 4π f N2
(A.8)
We can further extend the analysis to account for the applied load:
δmax =
(ωnoload − (St τm )) PN1 4π f N2
(A.9)
where ωnoload is the no-load speed of the motor, St is the torque/speed gradient of the motor and τm is the required motor torque which is given by: N2 τm = N1 ηgear
sin(α) + µ cos(α) ( Jm + Jl + Jls )(ω f − ω0 ) + L 2πηscrew P t 198
(A.10)
Table A.3: Evaluation system specifications
Specification Applied Force Range (linear, secondary drive) Measured Force Range (ATI Nano17) Force Resolution (ATI Nano17) Stage Stiffness Stage Displacement Range (carriage) Stage Maximum Velocity Stage Displacement Resolution (linear, secondary drive) Displacement Bandwidth (no load, 1mm) Analog Input/Output Resolution System Sample Rate
Value 0-100 0-10 3 10,000 0-20 80 0.5 9 16 2
Units N N mN N/mm cm mm/s µm Hz bit kHz
where ηgear is the efficiency of the motor gearhead, ηscrew is the efficiency of the ballscrew, α is the inclination angle, µ is the coefficient of friction between the guide-rails and linear bearings, and
(ω f − ω0 )/t is the desired angular acceleration of the motor. At maximum load (100N), vertical mounting (α = π/2), and given an acceleration of 2050 rad/s2 (equivalent to ramping up to the maximum speed of the motor in 0.5 seconds), Table A.2 shows that the motor is operating well within its nominal limits. The evaluation system’s overall specifications are tabulated in Table A.3.
199
Appendix B
Disturbance Rejection via Analog Filtering and Machine Learning† B.1 Introduction In Chapter 4 we described the implementation of light-intensity modulation (LIM) transduction techniques to accurately sense deflections and forces. LIM force and proprioception sensors are seeing increasing interest in the field of surgical robotics and flexible systems in particular. However, optoelectronic sensing modalities are notoriously susceptible to ambient effects such as temperature and environmental irradiance which can register as false force and displacement readings if not adequately compensated for. Fig. B.1 (a) shows typical behavior of a fully-integrated optoelectronic multi-axis force sensor under null force, but variable lighting and temperature profiles induced by adjusting the ambient light and applying a substantial thermal profile. The bottom plot shows the emitter-detector pair outputs as they correlate to environmental changes which are being measured simultaneously by temperature and irradiance sensors as shown in the top plot. Observe how each emitter/detector pair has a different response to relative changes in light (denoted by light green shading), whereas temperature dependencies are somewhat uniform among all sensors (denoted by light red shading). We see how these effects could be construed as force readings despite the absence of † Material
from this appendix was published in (Gafford et al., 2016d,a)
200
Relat ive Int ensit y
1.0 0.9 0.8 0.7 0.6 0.5
0
1
2
4 Tim e [ s]
5
6
7
8
PCMEMS Force Sensor
0.4 Out put Volt age
3
180 160 140 120 100 80 60 40 20 0
Tem perat ure [ C]
Environm ent al Sensors
1.1
S1 S2
0.3 0.2
S3 S4
0.1 0.0 0.1 0.2 0.3
0
1
2
3
4 Tim e [ s]
5
6
7
8
(a)
(b)
Figure B.1: Environmental disturbance in optoelectronic sensor systems: (a) PCMEMS Multisensor response to environmental disturbance with null force input: (top) environmental sensor readings (where green shading denotes a change in ambient light and red shading denotes a temperature change), (bottom) corresponding emitter/detector output changes, (b) operational schematic and graphical model demonstrating the complex inter-relationships between inputs, observations and outputs, illustrating how knowledge of s1 , ..., s4 alone is insufficient to accurately reconstruct the input force vector.
applied force; thus, the need for environmental compensation is clear. Fig. B.1 (b) presents a graphical model of how sensor observations are combined to estimate the input force vector, and how these observations can be contaminated by environmental effects. In this appendix, we discuss two approaches to reject disturbances in optoeletronic force and displacement sensor systems (Gafford et al., 2016d,a). The first is through analog signal conditioning, wherein the emitter is driven with a periodic signal which is actively band-passed at the output stage of the detector. The second is the use of regression-based machine learning techniques to dynamically compensate for environmental biases that plague multi-axis optoelectronic force sensors.
B.2 Rejection through Analog Signal Conditioning Some disturbance rejection in optoelectronic sensing systems can achieved via component selection, and infrared phototransistors can be chosen which have peak spectral responses in the infrared regime (700 nm wavelength and higher) to reduce sensitivity to ambient conditions. However, all components have some non-negligible spectral bandwidth that can range from 400 - 1100 nm, and
201
as such, there is usually some overlap between the sensor’s spectral response and the range of visible light. To overcome this, we introduce a frequency component into the conditioning circuitry so that the system only responds to a very narrow band of input frequencies in the temporal domain which are used to drive the LED. This approach was demonstrated experimentally in the design of a single-axis, self-assembling mm-scale force sensor based on LIM (described more thoroughly in (Gafford et al., 2016d), and the conditioning circuit is shown in Fig. B.2. An LM555 timer, configured as an astable oscillator, provides a square wave with a frequency of f 555 = 8kHz. However, a square wave is not an ideal excitation source due to the high frequency content of the rising/falling edges which are difficult to filter in subsequent stages. To remedy this, the square wave is passed through a triple-pole RC filter which produces an approximation of a 8 kHz sine wave with reduced amplitude and an offset of Vo f f set = 0.5Vcc . This wave is AC-coupled through a series capacitance to remove the DC offset and passed through a non-inverting amplifier to produce a sine wave with 3 Vpp . This wave is used to drive the IR LED. The phototransistor collector current, which is also somewhat sinusoidal in nature, is converted into a voltage by a variable resistor R8 , and this voltage is sent through a double-pole band-pass filter with a center frequency at 8kHz and a quality factor of Q = 0.67 (resulting in a loss of -1.2 dB at 8 kHz). The high-pass and low-pass filter stages within the band-pass filter are buffered by an adjustable-gain non-inverting amplifier. A passive peak detector converts the AC-signal into a DC signal that can be processed by a data acquisition unit. Note that there is an additional low-pass filter inherent in the phototransistor response (with a cutoff of roughly 23 kHz, estimated by f c = 0.35/(τr ) where τr is the rise time of the phototransistor (15 µS)). Fig. B.2 (right) shows oscilloscope traces of the signal at various locations in the conditioning architecture. We see how the circuit generates a purely-DC analog signal that is modulated according to the input force. Fig. B.3. shows the frequency response of the filtering circuitry to input sinusoids of varying frequency. The data is compared to a model of the theoretical behavior as given by the following transfer function:
F (s) =
s s + R f 1C f 1
2
R f 2C f 2 s + R f 2C f 2
2
−1 τPT −1 s + τPT
!
(B.1)
Note that the transfer function also includes a term to account for the inherent low-pass
202
5 V1
V2
V3
V4
V1 Filter
ACS
Astable Three-Pole DCSOffset Filter Removal Oscillator
V2 Amplify Amplifier GainS=S5
Filter Sensor
Band-Pass fc1=300 Hz fc2=22 kHz
V3 Amplify
V4 Rectify
Adjustable Gain
Peak Detection
Amplitude [V]
LM555
4 3 2 1 0 0
0.05
0.1
0.15
0.2
Time [ms]
Figure B.2: (left) Electrical architecture the source excitation and filtering of the phototransistor response, and (right) representative scope traces at various locations along the conditioning circuitry. 0.9 Measured Theoretical
0.8
GainM[V out/Vin]
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
103
104
105
FrequencyM[Hz]
Figure B.3: Theoretical and experimental frequency response at the output stage, where the pass band centers at 8 kHz as designed.
filter contained in the phototransistor response (characterized by the transistor rise-time which is roughly τPT = 15µS). As designed, the pass-band centers at around 8 kHz, and we see that the data matches the model quite well near the pass-band, but deviates significantly at higher frequencies. This is likely due to the fact that the phototransistor output no longer resembles a pure sinusoid at higher frequencies, resulting in frequency components that cannot be filtered. The voltage swing expected from the output of the signal conditioning circuitry, pre-multiplied by the filter attenuation factor of 0.87 as given in Fig. B.2, is given by:
Vout
R = 0.87 1 + 11 R10
i PT R8
(B.2)
As such, when designing analog filtering systems, it is important to consider the signal attenuation at the desired frequency due to filter roll-off at the cutoff frequency. Higher quality 203
can be achieved through multiple-pole filters at the cost of increased circuit complexity.
B.3 Rejection through Machine-Learning-Based Regression In addition to analog filtering, additional rejection can be achieved digitally through the use of linear and kernel-based regression techniques assuming there is an estimate of environmental disturbance phenomena either through a sensor reading or a system model. To explore this approach, we fabricated a three-axis LIM force sensor with integrated temperature and ambient irradiance sensing via PCMEMS. We explore machine learning regression techniques to compensate for temperature and ambient light sensitivity using on-board environmental sensor data. We compare batch-based ridge regression, kernelized regression and support vector techniques to baseline ordinary least-squares estimates to show that on-board environmental monitoring can substantially improve sensor force tracking performance and output stability under variable lighting and large (>100C) thermal gradients. By augmenting the least-squares estimate with nonlinear functions describing both environmental disturbances and cross-axis coupling effects, we can reduce the error in Fx , Fy and Fz by 10%, 33%, and 73%, respectively. We assess viability of each algorithm tested in terms of both prediction accuracy and computational overhead, and analyze kernel-based regression for prediction in the context of online force feedback and haptics applications in surgical robotics. Finally, we suggest future work for fast approximation and prediction using stochastic, sparse kernel techniques.
B.3.1 Background of Machine Learning Techniques in Sensor Disturbance Rejection There has been some prior work in employing machine learning techniques to improve sensor performance in the face of nonlinearities, disturbances or other detrimental effects. Neural networks have been used to approximate nonlinearities in multi-axis force sensors (Cao et al., 2009; Lei et al., 2006; Yao et al., 2010). Artificial neural networks (ANN) have been employed to linearize the behavior of capacitive humidity sensors (Chatterjee et al., 2000). We chose not to pursue NNs as they have a tendency to overfit the data, require numerous hyperparameters to be optimized, and are not guaranteed to converge to local minima. Newer techniques, such as Locally-Weighted Projection Regression (LWPR) have been used in force control applications 204
(Vijayakumar and Schaal, 2000); however, like NNs, LWPR typically requires optimal tuning of numerous hyperparameters. Kernel-based support vector machines (SVM) have been employed to fit nonlinear functions to cross-axis coupling terms in multi-axis strain gage-based force sensors (Ma et al., 2012) and to compensate for nonlinear and environmental effects in photoelectric displacement sensors (Guo et al., 2011) and capacitive pressure sensors (Wang, 2008). To the author’s knowledge, there exists no prior literature in applying kernel-based machine learning techniques to actively reject thermal and ambient light disturbances while approximating cross-axis coupling relationships for optoelectronic multi-axis force sensing methodologies. As an experimental platform to test different algorithms, we developed a multi-axis force sensor with on-board temperature and irradiance sensing capabilities using PCMEMS. Building on prior work in monolithic sensor fabrication (Gafford et al., 2014) and discrete-component LIM force sensors (Gafford et al., 2016d), we demonstrated improved regression performance by developing a model of the system that includes both environmental and cross-axis error terms and attempt to approximate these effects using simple linear techniques (ridge regression), kernel techniques (kernelized ridge regression, ǫ−sensitive support vector regression), and stochastic methods (sub-gradient support vector regression). We assess each approach in terms accuracy and computational complexity in both training and prediction. Overall, we demonstrate that the performance of multi-axis optoelectronic sensors can be drastically improved by including onboard estimates of environmental data. We also demonstrate that nonlinear kernel-based methods result in substantial performance improvements, and discuss ways in which these techniques can be applied to online estimation for deployable implementations.
B.3.2 Experimental Multisensor Design A discrete-component, multi-axis, light-intensity-modulated force sensor consists of ls emitterdetector (E/D) pairs (IR LEDs and phototransistors) separated from each-other by an elastic structure, as illustrated in Fig. B.1 (b,left). As a force is applied to the structure, causing it to deflect in m degrees-of-freedom, the relative distance separating each emitter-detector pair is changed, and using the principle of light intensity modulation, we can combine the outputs of each pair to reconstruct the input force vector so long as ls ≥ m.
205
2mm
Input Force
2mm Temp Sensor
Ambient Sensor
LIM Force Sensors (x4)
Resistors (x5)
Figure B.4: Fabricated PCMEMS multisensor: (left) post-fabrication prior to assembly, still attached to assembly scaffold, (middle) assembled with components placed, (right) integrated into 3d-printed casing.
Fabrication The composite laminate consists of four layers of 75 µm 304 stainless steel, two layers of 25 µm Kapton polyimide, two layers of 25 µm Kapton polyimide with 18 µm copper cladding, and seven layers of DuPont Pyralux F0100 adhesive. PCMEMS fabrication as described in Appendix C was employed to fabricate the prototype. Assembly is guided by four axisymmetric Sarrus linkage assemblies which constrain assembly kinematics to the z-axis, enabling trivial assembly into the 3D structure. These linkages also help to prevent torsional and transverse deformations in the structure. The final assembly step includes manually folding four stiffener beams into corresponding slots, thereby transmitting input forces to the flexural members. After assembly is complete, electrical components are reflow soldered into place onto the top and bottom flexible circuit layers. The fabricated sensor, measuring roughly 10 mm in diameter and 3.5 mm in height, is then integrated into a 3D-printed casing for encapsulation. Fig. B.4 shows the fabricated sensor in various stages of integration. The assembly sarrus linkages ‘jam’ against the inner wall of the casing if sufficient load is applied, thereby providing a mechanical ‘stop’ to prevent overloading. A custom excitation and filtering circuit, described more thoroughly in (Gafford et al., 2016d), converts the small collector current from each emitter-detector pair (consisting of a HIR19-21C IR LED and PT19-21C IR phototransistor, both from Everlight) into a DC-level voltage that can be processed by a data acquisition unit. The temperature sensor (MCP9701T-E/TT from Microchip Technology) is an integrated circuit that contains all necessary conditioning circuitry. The ambient sensor (PT19-21C) was low-pass filtered with a cutoff of 1 kHz.
206
ATI6Nano17 Signal6Conditioner
To6NI-PCI6259 DAQ NI6USB-6002 DAQ
PCMEMS Multisensor
ATI6Nano17 Reference6Load6Cell
PCMEMS Signal6Conditioner
Figure B.5: Experimental setup, showing the encapsulated PCMEMS sensor mounted directly onto an ATI Nano16 reference load cell.
Experimental Setup The PCMEMS multisensor was rigidly fastened to an ATI-Nano17 6-axis commercial load cell which serves as ground truth. Data were collected using LabView at a rate of 2 kHz for roughly 135 seconds (using USB-6002 DAQ for the PCMEMS multisensor and PCI6259 DAQ for the ATI Nano17, both from National Instruments). Variable loading, lighting, and temperature conditions were applied through the duration of data capture. A heat gun was used to apply excessive thermal gradients above ambient. The experimental setup is shown in Fig. B.5. The resulting data set was randomly shuffled and split into a training and testing set with a ratio of 75%/25%. The result is a training set of roughly ntrain ∼200,000 data points and a testing set of ntest ∼70,000 data points.
B.3.3 Inference In this section, we consider a simple linear inference model of the sensor to serve as a baseline for comparison. We then develop a model to describe the effects of temperature, irradiance, and cross-axis coupling to augment the baseline model. We explore linear and kernel-based techniques for optimizing the weighting parameters of the augmented model, and discuss trade-offs in terms of accuracy and computational complexity. All subsequent algorithms were implemented in Python, making use of numPy and sciPy packages. 207
Baseline Model (Moore-Penrose) The most common means of calibrating multi-axis force sensors is to apply a calibration profile (i.e. some static or dynamic combination of known forces), and record the sensor outputs to formulate 1 m×l , and perform Moore-Penrose pseudoinversion to an inverse calibration matrix W− MP ∈ R
compute the forward calibration matrix W MP ∈ R l ×m (Penrose, 1954). Moore-Penrose theory states that, for any rectangular matrix C, there exists a unique matrix C+ that satisfies the four Penrose conditions. A corrollary of Moore-Penrose theory says that, if C has full row rank, the matrix C+ reduces to:
C+ = ( C T C ) −1 C
(B.3)
As shown, the calibration matrix C+ ≡ W MP amounts to the ordinary least squares (OLS) solution without regularization. Performing least-squares regression on a time-series of s1 , ..., s4 with N observations (i.e. Xs ∈ R N ×ls ) without accounting for temperature and ambient light yields the following calibration matrix:
Yˆ MP = Xs W MP
= s1 s2 s3
(B.4)
1.11 3.75 −6.36 3.96 6.27 − 6.50 s4 −4.00 −2.89 2.40 −0.44 −4.44 3.58
(B.5)
where so = [so (1), ...so (i ), ...so ( N )] T ∈ R N is a matrix containing all observations from sensor o, and Yˆ MP ∈ R N ×m is a matrix containing the estimated force components for N samples. For three-axis force sensing, Yˆ MP = [fˆ x,MP , fˆ y,MP , fˆ z,MP ] where fˆ p,MP = [ fˆp,MP (1), ... fˆp,MP (i ), ... fˆp,MP ( N )] T ∈ R N is the series of Moore-Penrose estimates for force component p. Timeplots of the component-wise PCMEMS sensor estimates, compared with the reference ATI Nano17 readings (denoted Y = [fx , fy , fz ] ∈ R N ×m ), are shown in Fig. B.6. The shaded area represents the 95% credible interval. We observe that regression on Fx and Fy performs well even using a simple regression model ignoring environmental effects, while prediction for Fz is
208
2.0
1.5
1.5
1.0
0.0
0.0
Fx (Nano17) 5
−1.0
OLS 10 Time [s]
(a)
15
20
−1.5 0
−1 −2
−0.5
−0.5
−1.5 0
0 Force [N]
0.5
−1.0
1
0.5 Force [N]
Force [N]
1.0
2
−3
Fy (Nano17) 5
OLS 10 Time [s]
(b)
15
20
−4 0
Fz (Nano17) 5
OLS 10 Time [s]
15
20
(c)
Figure B.6: Comparison of ground-truth force data with estimates generated via component-wise ordinary leastsquares regression (Moore-Penrose pseudoinversion) using a linear combination of s1 , ..., s4 : (a) ˆfx,MP and fx , (b) ˆf ˆ y,MP and fy , (c) fz,MP and fz . In general, we observe how the differential nature of Fx and Fy results in stable measurements, but the common-mode nature of Fz results in poor performance.
markedly poor. This is a construct of the sensor morphology, where forces applied in x and y ultimately generate differential signals in which common-mode errors get canceled out (i.e. Fx causes s1 and s3 to increase while causing s2 and s4 to decrease). Conversely, the performance of Fz is poor as this is component does not generate differential signals, and as a result, commonmode errors still exist in the final measurement. To correct this common-mode contamination of Fz , we can improve over the baseline by integrating on-board environmental measurements and performing complete-data ridge regression, kernel regression and SVR-augmented OLS as discussed below.
Ridge Regression Improving over the baseline, we combine outputs from ls = 4 E/D pairs (s1 , ..., s4 ) with outputs from le = 2 environmental sensors, denoted st and se , respectively. For convenience, we define feature space sets Xs = {s1 , s2 , s3 , s4 } ∈ R N ×ls and Xe = {st , se } ∈ R N ×le , and note that Xs ∩ Xe = X ∈ R N ×l (the union of the two sets comprises a complete data set) but Xs ∪ Xe = ∅ (E/D data and environmental data are distinct and non-interfering). Ultimately, we seek to find some combinatorial formulation W ∈ R l ×m of all on-board sensor data X to reproduce the most accurate estimate of the input force Y ∈ R N ×m . A simple linear model is proposed to treat the actual force input as some weighted, linear combination of individual sensor voltages X corrupted by some component-wise noise ε ∈ R N ×m with each row entry ε(i ) drawn from the normal distribution
209
N (µ = 0, Σ ∈ R m×m ). Yˆ = XW + ε
(B.6)
2 where σ2 is the worst-case noise variance To further simplify the problem, we re-write Σ = Iσwc wc
of all the sensors. We can use l2 regularized regression techniques to fit a linear model to the data by computing the maximum a posteriori (MAP) estimate of the data given a prior distribution (∼ N (0, (1/τ 2 )I) where I ∈ R l ×l is the identity matrix) on the regression weights to prevent overfitting as we explore higher-dimensional feature spaces. We no longer constrain ourselves to linear inputs, and are free to build a mapping X 7→ (Φ ∈ R N ×lΦ ) where Φ is some nonlinear function of X and lΦ is the dimensionality of the new feature space. Assuming a Gaussian prior with precision τ 2 on the regression weights, and a sensor noise with variance σ2 , we can compute ˆ RR which involves constructing a numerically-stable regularized estimate of the feature weights w ˜ and output matrix y˜ as below: an ‘augmented’ feature space matrix Φ Φ/σ ˜ = Φ √ Λ
where
√
y/σ y˜ = 0
(B.7)
(B.8)
Λ is the Cholesky decomposition of Λ = (1/τ 2 )I. We compute the Cholesky decomposi-
tion of the augmented feature space, and compute the optimum regressor estimate as follows:
˜ = QR Φ
(B.9)
ˆ RR = R−1 Qy˜ w
(B.10)
ˆ RR Yˆ RR = Φw
(B.11)
The output hypothesis is given by:
ˆ RR is the ridge estimate. Note that, by using the Cholesky decomposition, in optimizing where Y
210
2.0
0.8 Testing RMSE
2.5 Normalized Test RMSE
1.0
Fx Fy Fz Mag
1.5 1.0
0
0.4 0.2
0.5 0.0
0.6
1
OLS Linear Ridge Quadratic Ridge Physics Ridge
Force [N]
3.0
1
2
3
4 5 6 Basis Dimension
7
8
9
0.0
(a)
−1 −2 −3
Fz (Nano17)
Fx
Fy Force Component
(b)
Fz
−4 0
5
QRR 10 Time [s]
15
20
(c)
Figure B.7: (a) Parametric sweep for ridge regression with a polynomial feature space transformation where the testing error is plotted as a function of basis dimension (overfittting occurs after order 2), (b) bar plot comparing component-wise test data RMSE for each feature space employed as discussed in Section 3.2, and (c) ˆfz,RR and fz where Φ has a quadratic basis. We observe significant force tracking performance over the Moore-Penrose solution with Xs data only.
the regression weights we only have to invert the triangular matrix R which is O(n2 ), as opposed to inverting the full (Φ T Φ + λI ) matrix which would be O(n3 ). For this paper we consider three different mappings: • Complete Data Linear Mapping: We preserve a linear mapping for all sensor data, i.e. for so ∈ X, so 7→ a + bso . • Complete Data Quadratic Mapping: We map all sensor data to a quadratic feature space: for so ∈ X, so 7→ a + bso + cso ⊙ so ∗ . • Complete Data Physics-Derived Mapping: For so ∈ Xs , so 7→ a + cso ⊙ so . Meanwhile we preserve a linear mapping for temperature and irradiance data, i.e., for so ∈ Xe , so 7→ a + bso . Ridge regression was performed on the training and test data by incorporating data from the temperature and ambient sensor. The initial data set X were transformed into a polynomial feature space Φ ∈ R N ×(ld) where l = ls + le = 6 and d is the order of the polynomial transform. To justify the use of a second-order polynomial feature space, a parametric sweep was performed where, for a given feature space dimension, the root-mean-squared error was computed for each dimension d: ∗ Here,
⊙ refers to an element-wise multiplication operation
211
RMSEm =
r
∑in=1 ( f m,i − yˆ RR,m,i )2 N
(B.12)
where m is the force component being evaluated m ∈ { x, y, z}. The results, shown in Fig. B.7 (a), demonstrate that a quadratic feature space is optimum, after which overfitting occurs. Here, the dotted black line is the RMSE magnitude over all components, given by:
| RMSE| =
q
RMSEx2 + RMSEy2 + RMSEz2
(B.13)
The component-wise testing RMSE for each transform is shown in Fig. B.7 (b), demonstrating that the quadratic and physics-derived feature spaces perform similarly. Force tracking results for a quadratic feature space transform on full sensor data are given in Fig. B.7 (c), showing a much better prediction of Fz . Note that Fx and Fy tracking results are excluded as the results show a negligible improvement in performance over the baseline. Thus it is observed that including environmental sensor data, in addition to preservation of the physics of the system in regression, can greatly improve the performance.
B.3.4 A Modified Model For Kernel Methods To improve performance further, we modify the original model to include the OLS solution augmented with nonlinear function estimates of cross-axis terms, temperature sensitivity terms and ambient sensitivity terms, as follows:
1 Y = Yˆ MP ϕ x,y ϕ x,z Y = Yˆ MP
ϕy,x 1
ϕy,z ϕ Xe ξ
ϕz,x ξ x,t ξ y,t ξ z,t ϕz,y + Xe ξ x,e ξ y,e ξ z,e 1
(B.14)
where Yˆ MP is the original OLS hypothesis with sensor data in Xs (i.e. Equation (B.5 )), Xe = [st , se ] are the temperature and irradiance readings, and ϕ a,b , ξ a,b are nonlinear operators on components a, b. With this model, we nonlinearly approximate cross-axis terms ϕ(·,·) and environmental terms ξ (·,·) which are assumed to contribute to the error between the actual solution and the OLS 212
solution. We define a new (modified) state matrix X¯ = [Yˆ MP
Xe ] ∈ R N ×(m+le ) for subsequent
analysis. As this model introduces several new degrees-of-freedom to the problem, we explore kernelized methods to compute optimum estimates of ϕ and ξ. We make use of the ‘kernel trick’ which enables the use of linear algorithms to implicitly operate in a transformed feature space, resulting in an efficient means of constructing models which are nonlinear in input space. Below we explore full (kernelized ridge regression) and sparse (support vector regression) kernelized regression techniques, and attempt to speed up prediction time using stochastic methods.
Kernelized Ridge Regression Saunders et al (Saunders et al., 1998) present a means of performing Ridge Regression using kernel operators (Kernelized Ridge Regression, or KRR). Given a test point x ∗ , the output hypothesis yˆ is given by:
yˆ KRR = yT (K + λI )−1 k
(B.15)
where y is the training data output, λ is a regularization parameter, K is a kernel matrix with entries given by the following:
Ki,j = k (x¯ i , x¯ j )
(B.16)
where x¯ i , x¯ j are the ith and jth rows in X¯ and k is the kernel vector with entries defined by the following:
k i = k( x ∗ , x¯ i )
(B.17)
The computational complexity is dictated by the matrix inversion of (K + λI ) which is
O(n3 ), where n is the size of the training set. We cannot use the Cholesky decomposition as in ˆ from feature space X¯ in the hypothesis regular ridge regression, as we cannot linearly separate w evaluation. We consider three kernel transformations k ( x, x ′ ) as below: (1) Linear kernel: x · x ′ (2) Polynomial kernel: (1 + x · x ′ )d where d is the order 213
Fy (Nano17) OLS/ KRR
1
0.5 0 -0.5 Fx (Nano17) OLS/ KRR
-1
0
0.5
Force [N]
Force [N]
1
Force [N]
1
1.5
2 1.5
0
-3
5
10
15
20
Fz (Nano17) OLS/ KRR
-4
-1.5 0
-2
-0.5 -1
-1.5
-1
0
T ime [s]
(a)
5
10
T ime [s]
(b)
15
20
0
5
10
15
20
T ime [s]
(c)
Figure B.8: Comparison of ground-truth force data with estimates generated via component-wise kernelized ridge regression with RBF kernel basis, where the data set has been downsampled for tractability (decimation = 100): (a) ˆf ˆ ˆ x,KRR and fx , (b) fy,KRR and fy , (c) fz,KRR and fz . We observe significant performance improvements in force tracking of Fz , as well as noticeable performance improvements in Fx and Fy .
(3) Radial Basis Function (RBF) kernel: exp(−ν|| x − x ′ ||2 ) where ν is a free parameter KRR was performed on the modified model given in Equation (B.14 ) to estimate nonlinear cross-axis and environmental dependencies ϕ and ξ, respectively. Due to memory constraints, the original data set was downsampled by 100 (n = N/100) to make the batch problem tractable. Although KRR was performed on linear, polynomial and RBF kernels, only the RBF results are shown in Fig. B.8 for each component. We observe near perfect force tracking in Fx and Fy and significant improvements in Fz over the baseline. ǫ-SVR and Stochastic Approximation While kernelized ridge regression is adept at fitting nonlinear models for moderately-sized data sets, a lack of sparsity in the hypothesis lends to poor prediction scaling to larger sets in terms of computation time. We introduce sparsity by exploiting support vector regression (SVR) techniques which solve the same kernel problem by minimizing an l1 objective function (Smola and Schölkopf, 2004). The idea is to ‘toss out’ hypotheses that lie within some user-specified ‘tube’ of the truth value, thereby embedding sparsity in the hypothesis that is a function of tube parameter ǫ. Instead of applying the kernel trick directly to ridge regression to minimize an l2 least-squares problem, we utilize sparse ǫ−sensitive support vectors for generating hypotheses. This involves finding an optimum set of multipliers α(∗) to minimize the following quadratic programming problem: 214
−
1 N (α(i ) − α(i )∗ )(α( j) − α( j)∗ )k(x¯ i , x¯ j ) 2∑ i,j N
∗
N
(B.18)
∗
− ǫ ∑ ( α (i ) − α (i ) ) + ∑ yi ( α (i ) − α (i ) ) i =1
i =1
subject to N
∑ ( α (i ) − α (i )∗ ) = 0
(B.19)
α(i ), α(i )∗ ∈ [0, C ]
(B.20)
i
where ǫ denotes the tube sensitivity (i.e. the amount of deviation from yi allowed before penalization), and C is a regularization parameter. We can represent the optimum parameter estimate as a linear combination of the support vectors transformed into a higher-dimensional space Φ using the representer theorem, as given below:
w=
∑ i ∈Dsv
(α(i ) − α(i )∗ )Φ(x¯ i )
(B.21)
where we sum over the set of valid support vectors Dsv . This leads to the following hypothesis for a test point x′ :
f ( x′ ) =
∑ i ∈Dsv
(α(i ) − α(i )∗ )k(x¯ i , x′ )
(B.22)
This dual problem is inherently convex and can be posed as a quadratic programming problem which can be solved in batch form. We evaluated the same kernel transformations as used for KRR. Fig. B.9(a) shows the force tracking performance in z after ǫ−SVR using RBF kernel. Note that Fx and Fy have been omitted for brevity as they are nearly identical to what is shown in Fig. B.8 (a) and (b). We see that the performance is similar to KRR but slightly worse, likely due to sparsity in the hypotheses. Stochastic SVR for linear kernels: A drawback of batch-based SVR is that the approximation time depends on the size of the training set. For example, performing batch-based SGD with RBF kernel on the full dataset takes over two hours to train. To speed up the training process, we can exploit stochastic methods to iteratively minimize the SVR primal objective function via 215
gradient descent. PEGASOS (Primal Estimated sub-GrAdient SOlver) was originally proposed in (Shalev-Shwartz et al., 2011) as an efficient means of using minibatch stochastic gradient descent to solve binary classification problems (y ∈ {−1, 1}) using support vector machines. The general idea is to randomly select minibatches At of size q from the original data set D and perform SGD on the primal objective function l(w; A+ t ) that contains the entries which correspond to valid support vectors in At (the subset A+ t ⊂ At ⊂ D ). We modified the original cost function to enable regression with support over y ∈ R with ǫ-sensitive tube loss, which linearly penalizes points that lie outside a ‘tube’ of width 2ǫ:
l(w; At ) =
1 λ ||w||2 + ∑ max {|hw, x¯ i i − yi | − ǫ, 0} 2 q i ∈ At
(B.23)
For a linear kernel, the gradient with respect to parameter vector w can be easily calculated: 1 ∂l(w; At ) = λw + ∑ sgn (hw, x¯ i i − yi ) x¯ i ∂w q i ∈ At
(B.24)
From this, we formulate a gradient descent algorithm with the following update step using and adaptive learning rate ηt = 1/(λt) as proposed in (Shalev-Shwartz et al., 2011): ∂l(wt ; At ) wt+1/2 = wt − ηt ∂w ηt 1 + sgn (hwt , x¯ i i − yi ) x¯ i = wt 1 − t q i∈∑ At
(B.25)
The final step is an optional projection step:
wt +1
√ ) 1/ λ = min 1, wt+1/2 ||wt+1/2 || (
(B.26)
PEGASOS for a linear kernel converges with complexity O(d/(λǫ)) (d is the number of non-zero features in each training example, therefore the runtime is independent of the training set size) as opposed to O(d2 n) with batch-based SVR on a linear kernel (Chapelle, 2007). Stochastic SVR for nonlinear kernels: To explore subgradient methods for SVR on nonlinear kernels, we use representer theorem to re-write w as a weighted linear combination of support vectors:
216
102
1
Fx Fy Fz
-1 -2
100
10-1
Fz (Nano17) OLS/SVR
-3
10-2
-4 0
5
Fx Fy Fz
100
Test RMSE
101
Test RMSE
Force [N]
0
10
15
20
10-1
100
101
10-1
Execution Time [s]
T ime [s]
(a)
(b)
100
101
Execution Time [s]
(c)
Figure B.9: ǫ-SVR results: (a) ˆfz,SVR and fz using batch-based ǫ-SVR with RBF kernel on a downsampled data set (decimation = 100), (b) linear PEGASOS on raw data set (decimation = 1) using a linear kernel, and (c) kernelized PEGASOS on downsampled data set (decimation = 100) using RBF kernel. Dashed lines indicate RMSE obtained via batch ǫ-SVR.
∑
w=
α(i )Φ(x¯ i )
(B.27)
i ∈Dsv
Rather than replacing w in the primal objective function with the representer form and taking the gradient with respect to α(i ), (Shalev-Shwartz et al., 2011) recommends taking the gradient with respect to w as the problem can be proven to be strictly convex. For this application, we use the ǫ−sensitive tube loss objective function: l (w, x¯ i ) = yit − hwt , φ(x¯ i )i
(B.28)
the update on w can be written as: t
wt+1 = ηt ∑ 1 [|l (w, x¯ i )| > ǫ]sgn(l (w, x¯ i ))φ(x¯ i )
(B.29)
i =1
where 1 is the indicator function. We can treat αt+1 ( j) as a counting variable that, for support vector j, counts up by ηt if the loss function is positive and counts down by ηt if the loss function is negative, so long as the tube sensitivity condition (|l (w, x¯ i )| > ǫ) is met. We can then update wt+1 using representer theorem as in Equation (B.27 ). In performing gradient descent, we now update each individual support vector (if it exists) until convergence.
217
Table B.1: Algorithm performance summary (Note: (*) indicates approximate full data run time based on downsampled data run time)
Algorithm
Fx RMSE (train)
Fy RMSE (train)
Fz RMSE (train)
Fx RMSE (test)
Fy RMSE (test)
Fz RMSE (test)
Baseline (OLS on Xs ) Ridge (Linear) Ridge (Poly) KRR (Linear) KRR (Poly) KRR (RBF) Batch ǫ-SVR (Linear) Batch ǫ-SVR (Poly) Batch ǫ-SVR (RBF) SGD ǫ-SVR (Linear) SGD ǫ-SVR (Poly) SGD ǫ-SVR (RBF)
0.136 0.128 0.114 0.126 0.130 0.103 0.153 0.123 0.119 0.134 0.131 0.121
0.103 0.091 0.088 0.099 0.096 0.076 0.106 0.088 0.075 0.099 0.099 0.096
0.518 0.288 0.251 0.483 0.297 0.227 0.566 0.404 0.305 0.534 0.423 0.332
0.150 0.142 0.164 0.137 0.131 0.135 0.148 0.147 0.141 0.137 0.141 0.145
0.136 0.103 0.121 0.105 0.092 0.090 0.107 0.104 0.102 0.109 0.101 0.102
0.936 0.961 0.411 0.629 0.306 0.262 0.621 0.416 0.409 0.656 0.432 0.414
Training Runtime [s] 0.032 0.201 0.216 129.8* 3586* 2340* 10,700* 15,900* 7900* 6.81 18,100* 3100*
yˆ Pred. Runtime [s] 2.10E-5 2.50E-5 3.00E-5 14.76* 15.06* 14.35* 6.554* 7.168* 6.892* 8.72E-4 0.491* 0.288*
Stochastic SVR Results Fig. B.9 (b) and (c) show the test set RMSE convergence to the batch value as a function of time for both linear and nonlinear kernels. Linear PEGASOS was performed on the full data set, and we observe convergence to the batch RMSE (from the downsampled dataset) after about six seconds. As such, this method is orders of magnitude more efficient than batch support vector regression using a linear kernel. However, for nonlinear kernels, each iteration requires a kernel evaluation over up to n training examples, and as a result, this algorithm is not particularly efficient for nonlinear kernels as the runtime complexity is O(nsv /(λǫ)). Therefore the raw dataset was downsampled for tractability. The resulting convergence curve is shown in Fig. B.9 (c) and we observe convergence to the batch value after about 30 seconds on the subsampled data set. To summarize, while SGD methods are particularly efficient for approximation with linear kernels, nonlinear kernels are preferred due to improved tracking accuracy at the cost of runtime performance. In future work, we discuss nonlinear kernel approximation methods that could potentially speed up convergence of nonlinear stochastic support vector methods.
218
B.3.5 Discussion Training and test set RMSE for each of the methods are summarized in Table B.1. In addition, we also tabulate the runtime performance both in training over the entire dataset or a subsampled version, and prediction of yˆ for a single test point x ∗ . Note that, for algorithms requiring a subsampled data set for tractability, the run time given is the predicted full-data run-time by considering the theoretical algorithmic runtime complexity. For stochastic methods, the prediction time is the amount of time required for the objective function to converge. We observe that kernelized ridge regression with an RBF kernel was the superior performer in terms of force tracking, despite relatively cumbersome computation complexities that were remedied via downsampling. Ridge regression on a quadratic feature space offers the best trade-off in terms of both performance and run-time. Both kernelized ridge regression and support vector regression perform similarly in terms of force tracking capabilities given the same regularization parameter (λ = 0.005), although SVR is predictably less accurate due to sparsity in the hypothesis. We also observe reciprocal performance between the two, where kernel techniques were faster to train but slower to predict. This is to be expected, as the inherent sparsity in support vector approaches leads to more efficient hypothesis prediction. Prediction time can be sped up further by loosening the tube constraints (larger ǫ) at the cost of tracking accuracy. While kernel ridge regression and SVR offer the best tracking performance, there are obvious computational drawbacks that would potentially contraindicate their application in on-line implementations (density of hypothesis/prediction in the former and computationally-intensive quadratic programming in the latter). We showed that sub-gradient ǫ−SVR techniques using linear kernels can substantially improve runtime performance by several orders of magnitude both in training and prediction, however linear kernels do not result in the best tracking performance. We also derived the algorithm necessary for sub-gradient ǫ−SVR for nonlinear kernels; however, this algorithm requires a kernel evaluation for each iteration that hinders runtime performance. It is also clear that SVR offers marginal performance improvements over polynomial ridge regression despite longer runtimes. In the next section, we discuss how the prediction runtime of KRR can be improved via a downsampled training set without significantly affecting the tracking accuracy.
219
1.0
Subsampled KRR
0.9
Mag RMSE [N]
0.8
Baseline (Raw), 47 kHz
0.7 0.6
Poly Ridge (Raw), 33 kHz
Batch SVR RBF, 1.5 Hz
0.5
SGD SVR RBF, 3.5 Hz
0.4 0.3
KRR Raw, 0.07 Hz
0.2 10
-1
10
0
10
1
10
2
10
3
Max Possible Sampling Frequency [Hz] Figure B.10: Tracking performance of kernelized ridge regression (RBF kernel) vs. maximum sampling frequency (1/t predict ). Performance of other methods tested in this paper are provided for reference.
Implications on Real-Time Force and Position Feedback We have seen that a modified model coupled with kernel-based regression techniques can improve tracking on all axes. However, we also observe in Table B.1 rather long prediction runtimes when kernels are constructed from large (n =200,000) training sets. In robotics applications, we typically strive for sample rates on the order of hundreds of Hz for force-feedback and haptics applications (Hu et al., 2006). Therefore it is necessary to drastically reduce the training set such that realtime prediction becomes tractable. To observe the effects of training size on tracking accuracy, we show how kernel ridge regression with an RBF kernel behaves with a subsampled training set. Subsampling the training set reduces the dimensionality of K and k, thereby reducing the runtime complexity t predict of predicting yˆ at a test point x ∗ . In Fig. B.10 we plot the overall RMSE magnitude (Equation (B.13 )) against the maximum sample rate for RBF kernel ridge regression with a subsampled training set. We see how tracking accuracy reduces with a lower-dimension training set, while the maximum possible sampling rate (1/t predict ) increases. We still observe that kernelized ridge regression is the superior performer over the ‘haptic regime’ (300Hz-1000Hz) in terms of tracking accuracy and sample rate. However, for high-speed applications (10 kHz or more), data-complete polynomial ridge regression provides the best compromise.
220
B.4 Future Work There are a few approximation approaches that could enable stochastic ǫ−SVR to be computationally practical in both training and prediction. In terms of training, (Rahimi and Recht, 2007) propose a means of approximating the RBF kernel operator using a linear combination of random Fourier features (‘Random Kitchen Sinks’), reducing the computation time per iteration from
O(nd) to O( B + d) (where B is the number of random features). For prediction, (Claesen et al., 2014) approximate the RBF kernel using a Maclaurin series approximation, reducing the prediction time complexity from O(nsv d) to O(d2 ). Future work could combine both of these techniques in kernelized PEGASOS to enable stochastic ǫ−SVR for RBF kernels with O(( B + d)/(λǫ)) training complexity and O(d2 ) prediction complexity. Additionally, newer techniques, such as Local Gaussian Process Regression, could be explored as well (Nguyen-Tuong et al., 2009).
221
Appendix C
PCMEMS Scalability, Robustness, and Biocompatibility† C.1 Introduction In Chapter 4, we introduce PCMEMS as a means to economically produce high-quality sensors and mechanisms at milli- to meso-scales. In this appendix, we provide a brief justification of PCMEMS as a feasible manufacturing method for fabricating sensors and mechanisms for medical applications. We begin by analyzing process economy and scalability. We present an experimental robustness analysis to derive scaling laws for device design based on common construction/assembly features and anticipated loads. The appendix concludes with an analysis of the biocompatibility of the process, as well as persisting challenges and areas for future work.
C.2 Process Economy An overview of the fabrication process is as follows: (1) Initial Cuts made in a diode-pumped solid-state (DPSS) Nd:YVO4 laser. Laser power and galvonometer speed/frequency are dependent on the material properties. (2) Cleaning and Surface Treatment via immersion in isopropanol in an ultrasonic bath at 80C † Material
from this appendix was published in (Gafford et al., 2013, 2016c)
222
for 10 minutes, followed by an etching process in Argon plasma (4 sccm, 100% forward current for 1 minute). (3) Stacking and Tacking using a custom weight press and a Watlow temperature controller. Flash-curing (50 psi (345 kPa), 130C for one minute) to deposit adhesive layers onto structural layers. (4) Full Cure at 50 psi, 200C for two hours. (5) Release Cuts made in the DPSS laser to release part from sacrificial scaffolding material. (6) Feature-Guided Assembly into the final three-dimensional mechanism. In terms of process economy, the fabrication time scales roughly with the number of layers N included in the laminate. Cleaning, plasma etching, and adhesive tacking are batch processes (with batch sizes given by netch and ntack ), and curing time τcure is roughly independent of the number of layers in the laminate. As such, these processes are relatively invariant regardless of device complexity. The initial and final laser cutting processes suffer the largest dependence on device complexity, as shown in Fig. C.1. The DPSS laser can be modeled as a pulsed Gaussian beam which suffers an exponential loss in power transfer and ablation rate as material is removed 2 P ∝ P0 (1 − e−1/δ ) (assuming the beam is focused on the top-of-stock). This can be corrected to a point by displacing the focal depth as subsequent cut passes are made, but energy diffusion into the sidewalls ultimately limits the total depth-of-cut. Despite the decaying exponential relationship between material depth δ and ablation rate h which can drive up fabrication time for thicker materials, the true process bottleneck lies in the lamination/curing cycle which requires a fixed cure time τcure , but also requires no direct human intervention, and as such, opens up the possibility of streamlining and parallelization. The total processing time t can be approximated by performing a dimensional analysis on the processing variables, as given by the following:
"
N
t = 2 nbatch ∑ i =1
li δi he−αδi f ds
#
+ τetch
N netch
+ τtack
N ntack
+ τcure + τsetup + τpp
(C.1)
where discrete layers in the laminate are denoted by i where i ∈ (1, · · ·, N ), nbatch is the number of devices in the current batch, li is the toolpath length (determined by the CAD package used 223
Normalized Ablation Rate
355 nm DPSS Laser
304 Stainless Steel
ds
Beam Divergence
Ablation Zone
Nlayers
Nlayers
Nlayers
Release
time
Lamination and Curing
time
time
time
Tacking
time
Cleaning, Etching
Initial Cuts
Depth of Cut [✁m]
Nlayers
Nlayers
Figure C.1: PCMEMS fabrication process analysis: (top) 355nm Nd:YVO4 laser modeled as a Gaussian beam, and associated ablation rate dependence on depth of cut, (bottom) manufacturing timing diagram (hatched requires constant supervision/attention, solid requires no intervention) with qualitative graphs showing how the processing time scales with the number of layers in the laminate.
to design the layers), α is an empirically determined energy loss constant, f is the laser pulse frequency in Hz, ds is the laser spot size (which ranges from 5µm to 10 µm depending on the material being cut), and τetch , τtack , τcure , τsetup and τpp are the (relatively invariant) times associated with fabrication steps indicated by the associated subscript (where pp is shorthand for ‘pick-andplace’). Most of the processing time lies in setup (which can be streamlined), batch cycles (such as cleaning, etching and adhesive tacking) and laminate curing time, indicating an overall weak dependence on l and N (metrics of device complexity) which makes the process very amenable to rapid prototyping fabrication cycles. As such, this technology is a platform fabrication process that can be leveraged to prototype multiple new designs in parallel with a relatively short turn-around time (less than a day from start to finish).
224
C.2.1 Applicability to Medical Devices PCMEMS fabrication features a number of characteristics that indicate its applicability to the manufacturing of surgical devices at millimeter scales. 1. The purely two-dimensional fabrication enables batch manufacturing of numerous devices in parallel, resulting in high throughput and significantly driving down fabrication costs for single use or disposable devices. 2. Direct integration of sensors and actuators obviates post-manufacturing alignment, assembly, and bonding. 3. Motion is permitted via folding of flexure-based joints, eliminating concerns of friction and wear. 4. PCMEMS boasts an extensive material catalog encompassing numerous biocompatible materials and medical-grade alloys.
C.2.2 Note on Sterilization and Usage The ultimate goal of this work is to enable the fabrication of single-use, disposable robotic modules, which relaxes some of the requirements imposed by extended use. However, multiple-use devices manufactured via PCMEMS must be robust to sterilization processes. The fabrication process necessitates a laminate curing temperature of 200 C over a duration of 2 hours. Steam sterilization entails immersion in 121 C steam for 30 minutes, followed by a 15 minute cycle at 132 C as per ISO 17665. Therefore, sterilization requirements are much less taxing on the laminate materials than the requirements of the fabrication process itself. Pre- and post-cure processes typically consist of ultrasonic cleansing in an Isopropanol bath which is a common decontamination protocol that takes place prior to sterilization. Therefore, while further study is required to evaluate PCMEMS robustness to common sterilization processes, it is reasonable to suspect that such devices can withstand the mechanical rigors of sterilization.
225
F
F F T=Flmom
Al
Al
w
w
lmom
h
h
w
Figure C.2: Experimental setup for robustness tests on PCMEMS assembly features, where insets show sample renderings and the geometric parameter of interest is circled (note that layer thickness is exaggerated for clarity): (from left) lap shear failure (loading plane parallel to adhesion plane), peel failure (loading plane perpendicular to adhesion plane), and castellated hinge torsion failure (planar torque about hinge).
C.3 PCMEMS Mechanism Failure Analysis Mechanical design of PCMEMS components relies on an implicit understanding of the mechanical properties of composite structures when loaded to ensure device longevity. To evaluate the mechanical robustness of components manufactured using the PCMEMS fabrication process, we experimentally analyzed the failure modes of structural and assembly components commonly employed in PCMEMS designs. We tested the following failure modes: (1) lap shear failure resulting in adhesive delamination due to axial loading (layups both with and without the flexible Kapton layer), (2) delamination via peel failure due to normal loading, and (3) castellated hinge failure via torsional loading.
C.3.1 Experimental Setup A custom aluminum jig with an elastic hinge, capable of friction clamping components > 0, we drop the proportional gain and have large derivative and integral gain to dampen the response for overshoot prevention, and eliminate steady-state error.
234
Table D.1: Fuzzy inference rules for agonist actuator (output sets are for K¯ p− , K¯ d− , and K¯ i− , respectively)
e˙(t) ↓, e(t) → NB NM NS ZE PS PM PB
NB B,VS,VS B,VS,VS B,VS,VS B,VS,VS B,VS,VS B,VS,VS B,VS,VS
NM MS,MS,MS M,S,S BS,S,VS BS,S,VS BS,S,VS MS,S,S MS,MS,MS
NS S,M,M MS,M,MS M,MS,S M,MS,VS M,MS,S MS,M,MS S,M,M
ZE VS,B,B VS,B,M S,M,S M,MS,VS S,M,S VS,B,M VS,B,B
Table D.2: Fuzzy inference rules for antagonist actuator (output sets are for K¯ p+ , K¯ d+ , and K¯ i+ , respectively).
e˙(t) ↓, e(t) → NB NM NS ZE PS PM PB
ZE VS,B,B VS,B,M S,M,S M,MS,VS S,M,S VS,B,M VS,B,B
PS S,M,M MS,M,MS M,MS,S M,MS,VS M,MS,S MS,M,MS S,M,M
PM MS,MS,MS M,S,S B,S,VS B,S,VS B,S,VS M,S,S MS,MS,MS
PB B,VS,VS B,VS,VS B,VS,VS B,VS,VS B,VS,VS B,VS,VS B,VS,VS
• As e(t) → 0 and |e˙(t)| → 0, indicating that the system has converged, we bring up the proportional gain marginally for disturbance rejection. For example, Rule Set {1, 2} in D.2 expresses the following logic: • IF e(t)=PS AND e˙(t)=NB, THEN K¯ p+ =S, K¯ d+ =M AND K¯ i+ =M These rule bases were embedded in the fuzzy inference engine shown in Fig. 6.1 (a) and ultimately used to create the control surfaces shown in Fig. 6.3.
D.3
Software Implementation
The FPID/PWM controller and hysteresis valve controller are integrated into a master controller which processes sensor feedback and contains logic for selecting different control modes and parameters. The master control architecture is shown in Fig. D.1, where the ‘Position PID 235
Control Outputs
Time xPC Session Elapsed Time
Sensor/Control Inputs 6
-K-
Simulation time
sec/count
8
LED 2
Current LED 1 Current LED 2 Handle LED
Current LED Indication 1 OLCL Selector
Current 2
Current 2
Mode Select Current 1
SMA Driver Signal Conversion And Filtering
Pot or V1
ref pot filtered Pot
pos_a
pos_d_traj
pos_d
pos_d Mode Select Clock Tactile Input Joystick Input
Control Select
v_raw
FS2
Force filtered
PWM In
i_d out
v1 Out v2 Out v3 Out In1 v4 Out v5 Out v6 Out v7 Out
fluid logic
i_a out
i_OL
HW ON
fluid out
Output Enable Disable
valve_OL
Gains
relay 1 relay 2 relay 3 relay 4 relay 5 relay 6 relay 7
Position PID controller
13
Temp
Data Acquisition
temp filtered
[id1 id2 ia1 ia2]
Sensor Feedback Signal Conversion and Filtering
Control In
[r1 r2]
RefMode
>0
Target Scope Id: 1 Desired/Actual Current
[F_ref]
Switch
[Kp1 Kd1 Ki1 Kp2 Kd2 Ki2]
Pot Filtered/Scaled
Mode Switch Mode Switch
Mode Out
pos_d Control In CL Select
Tactile 1
[id1 id2 ia1 ia2 r1 r2, pos_d pos_a F_ref, Gains]
[pos_d pos_a]
[r1 r2] [pos_d pos_a]
Tactile 1 Out
[v1 v2]
MM-32 Diamond Digital Input
Host Scope Id: 2 Reference Sensor Data Scope Target Scope Id: 3 Desired/Actual Position
[id1 id2 ia1 ia2]
Input to Velocity or Position Tactile 1
[id1 id2 ia1 ia2]
[pos_d pos_a]
Ref Mode
Pot
2
1 2 3 MM-32 4 Diamond 5 Digital Output 6 7
temp
Sensor and Controller Analog Inputs
1
MM-32 2 Diamond 3 Analog Output Actuator Current Control
External Valve Control
User-Controlled Valves
Open-Loop Controller
f1_raw
12
MM-32 Diamond Digital Output 3 LED Indication Out 2
PWM out
i_OL valve_OL
1
i_d i_d Flush out
Angle filtered
FS1
PWM_out
i_a Mode Select
pos_d_vel
PT
ref load cell filtered Load Cell
FS1
FS2
PWM 1 PWM 2
Controller
Current 1
Current 1
9 MM-32 Load Cell or V2 Diamond Analog Input 10 11
LED 1
CL State
CL Select Handle LED
pos_d
CL Trajectory Selector
Filtered Current
7
PWM In
[id1 id2 ia1 ia2 r1 r2, pos_d pos_a v1 v2, F f1 f2 temp Gains]
Host Scope Id: 4 On-Board Sensor data scope
Force [N]
3
Tactile 2
Tactile 2
[f1 f2]
Tactile 2 Out
temp
4
Control Digital Input
Tactile 3
Tactile 3
Angle [deg] [v1 v2] Force [N] [f1 f2] temp
Tactile 3 Out
Controller Debouncing/ Filtering/Scaling
[pos_a v1 v2 F f1 f2 temp]
Target Scope Id: 5 On-Board Sensor Data
Target Scope Id: 6 Controller States
Figure D.1: Simulink implementation of EndoMODRA master controller
Figure D.2: Simulink implementation of EndoMODRA master controller
Controller’ contains the FPID/PWM control architecture shown in Fig. 6.4. The ‘Controller Debouncing/Filtering/Scaling’ block contains several first-order filters that smooth button presses, as well as deadband filtering for the input potentiometer to make the controller insenstive to
236
Command
(+)
Time [s]
(-) I(-),max
Current [A]
I(+),max
(+) Actuator (-) Actuator
I(+),min t(+),h
t(+),s
I(-),min t(+),h
t(-),s
On
Time [s]
Valve State
(+) Valve (-) Valve
Off t(+),f
t(+),f
Time [s]
Figure D.3: Open-loop (sensorless) control paradigm: (top) control input, (middle) deterministic current profile, and (bottom) fluid cooling logic.
small perturbations. The ‘Input to Velocity or Position’ block converts master controller input into a desired position output based on the position- and rate-based control updates given in Equations 6.8 and 6.9, respectively. This block is active when the controller is configured in teleoperation mode. In automated trajectory execution mode, the user can select from various waveforms (sinusoidal, step, triangular, and stairstep) and parameters (amplitude, frequency). These waveforms are generated in the ‘CL Trajectory Selector’ block.
D.3.1
Open-Loop Control
In the case of distal multisensor failure, the user can default to an open loop control paradigm. The ‘Open-Loop Controller’ block generates deterministic current and fluid cooling profiles based on the scheme shown in Fig. D.3. The current profile is designed to have a high current transient for fast heat up (I(•),max ), which is held for a certain amount of time (t(•),h ) and then slews to a low-current sustain (I(•),min ) at a slew rate of ( I(•),max − I(•),min )/t(•),s to prevent the actuator from overheating. 237
