Human-Machine Collaboration for the Robotic ...

Extended Abstract: Human-Machine Collaboration for the Robotic Harvesting of Fresh Market Apples* Joseph R. Davidson, Changki Mo, Abhisesh Silwal, Manoj Karkee, Jun Li, Kehui Xiao, Qin Zhang, and Karen Lewis

Abstract— The lack of mechanical harvesting technologies is a significant problem that threatens the long-term sustainability of the U.S. tree fruit industry. This paper presents preliminary results from a project designed to address this problem by employing mechanization and human-machine collaboration in the production of high quality tree fruit for the fresh market. Dynamic analysis of hand picking has been conducted in order to determine optimal patterns for fruit removal. Data gathered from this analysis has facilitated the design of an underactuated prototype end-effector. Initial tests results indicate the system is relatively robust to position errors as well as sources of variability, like fruit shape, size, and orientation, present in the highly unstructured orchard environment. We also discuss initial results from efforts to develop an apple identification algorithm with a hierarchical approach to improve the accuracy of apple detection for robotic harvesting.

injury rate due to ladder use, and increasing uncertainty about the availability of farm labor [7, 8], the lack of mechanical harvesting is a critical problem receiving much attention from both federal agencies (e.g. United States Department of Agriculture) and state and local organizations. This paper presents preliminary results from a project designed to address this problem by employing mechanization and human-machine collaboration in the production of high quality fruit for the fresh market. To help improve the accuracy, speed, and robustness of robotic tree fruit harvesting systems, we have studied hand picking patterns to facilitate design of an effective end-effector based on the biological advantages and constraints that define commercial tree fruit production systems. We are also investigating human-machine collaboration for improved fruit detection.

I. INTRODUCTION In the U.S. Pacific Northwest, a large, seasonal laborforce is required for the production of tree fruit crops like fresh market apples, cherries, and pears. The most time and labor-intensive task in fruit crop production is harvesting. In Washington State alone the apple and pear harvest requires the employment of 30,000 additional workers with an estimated harvest cost of $1,100 to $2,100 USD per acre per year [1, 2]. To reduce harvesting costs and dependence on seasonal labor, researchers have developed shake-and-catch systems for harvesting fruits such as berries, cherries, and citrus [3, 4]. These techniques, which apply vibration to the trunk or branch of the tree in order to separate the fruit, are only used for fruit destined for the processing market where there are established tolerances for fruit bruising and external defects. Researchers have also proposed robotic systems for the harvesting of tree fruit [5, 6]. However, the development of robotic harvesting technology has achieved limited success primarily due to inadequate accuracy, speed, and robustness. Because of rising labor costs, a high workplace *Research supported by the United States Department of AgricultureNational Institute of Food and Agriculture (USDA-NIFA) through the National Robotics Initiative (NRI). J.R. Davidson and C. Mo are with the School of Mechanical and Materials Engineering, Washington State University, Richland, WA 99354 USA (phone: 509-372-7296; e-mail: joseph.davidson@ wsu.edu). A. Silwal, M. Karkee, and Q. Zhang are with the Center for Precision and Automated Agricultural Systems, Washington State University, Prosser, WA 99350 USA. J. Li and K. Xiao are with the Center for Precision and Automated Agricultural Systems, Washington State University, Prosser, WA 99350 USA, on leave from South China Agricultural University, Guangzhou, China. K. Lewis is with the Grant-Adams County extension office, Washington State University, Moses Lake, WA 98837 USA.

II. MODELING OF GRASPING FORCES Despite numerous attempts to transfer industrial robotic technology directly to field based, biologically driven environments, the mechanization of specialty crop harvesting has achieved only limited success. Previous robotic harvesting projects have identified end-effector performance as a significant opportunity for improvement. Our preliminary studies suggest that a comprehensive, quantitative study of the motion and forces during hand picking is required for development of an effective endeffector. We have performed field experiments to track and quantify forces developed with different apple picking patterns. Experimental results have been used to develop a model that estimates for a two-fingered picking pattern the grasping forces required to remove fruits at different inclination angles. Fig. 1 shows the force diagram of the apple and stem-branch system. The fruit is in equilibrium under the exerted effects of fruit gravity G, elastic force Fsn, elastic torque of the stem-branch Td(θ), and normal and tangential grasping forces at the given inclination angle θ. Using statistical regression analysis of the experimental data, it is possible to develop a function that predicts the elastic torque of the stem-branch Td(θ) and the detachment force. The calculated curves of tangential forces associated with an inclination angle range of 0 to 40° are compared with experimental data in Fig. 2. The estimation errors between measured data and calculated data yielded values from 0.077 to 0.031 N for the tangential force Ft1 and from -0.03 to 0.034 N for the tangential force Ft2. The reason for the estimation fluctuations of tangential force may result from the nonlinear characteristics of tension and shear of the stembranch joint.

III. MANIPULATOR AND END-EFFECTOR DESIGN

F’stθ ( )

Td θ ( ) F’sn

Fsn Ft1

Fst(θ) α α

Fn1

Ft2

Fn2 θ

G

Fig. 1. Pendulum model of the fruit-branch-stem system used to predict detachment forces.

The calculated normal and tangential forces at different inclination angles correspond to the measured data. The results show that the end-effector grasping forces required to remove an apple with the described finger placement can be predicted using the proposed model. 0.8

Tangential force Ft1 (N)

The knowledge acquired during dynamic analysis of hand picking has been critical during design of the end-effector. Our goal is to develop and evaluate an end-effector inspired by fruit growth habits and the human hand’s manipulation methods during apple harvesting. The end-effector has been optimized to meet a set of five specific design objectives as outlined below: • Detachment success of 90%. Detachment success is defined as the number of successfully harvested ripe fruit per total number of localized ripe fruit present in the manipulator’s workspace.

Actual Data Estimated Data

0.6

In order to reduce cost and increase flexibility for configuration modifications, we have decided to construct a custom manipulator rather than purchase an industrial model. For this project a 6 degree of freedom (DOF), articulated manipulator has been fabricated using custom aluminum frames and Dynamixel Pro actuators (Robotis, Inc., Irvine, CA). The manipulator has revolute joints and a maximum reach of approximately 0.6 m. Numerical solutions to the inverse kinematics problem are determined with the combined optimization method first proposed by Wang and Chen [9]. The inverse kinematics algorithm has been developed in Matlab (Mathworks, Inc., Natick, MA), compiled into a C++ shared library, and integrated with the manipulator’s controller in the Microsoft Visual Studio development environment. The manipulator’s planned trajectory is executed using a simple, open loop look-andmove approach. In order to reduce system complexity and increase speed visual servoing is not used.

0.4

• Maximum cycle time of 6 seconds. This is the time required to pick and store one fruit – it does not include the time required for ripeness determination and fruit localization.

0.2

• Damage rate to the fruit does not exceed 10%. • Can be used for harvesting of multiple apple cultivars. • Relatively lightweight, simple, and cost-effective.

0 0

10

20

30

40

Inclination angle (°) 1

Tangential force Ft2 (N)

Actual Data Estimated Data

0.9

0.8

0.7

0.6 0

10

20

30

Inclination angle (°) Fig. 2. Comparison of predicted and measured tangential forces.

40

The end-effector (Fig. 3) is designed to provide a spherical, enveloping power grasp of the fruit. To reduce complexity a tendon-driven, underactuated design has been selected. Each of the three end-effector fingers, which are arranged symmetrically around a circular palm, is a singleacting cable-driven system with two links and two flexure joints. A disc differential provides underactuation between the fingers and helps provide a shape adaptive grasp. Underactuation and the passive compliance provided by the flexure joints provide several advantages in the unstructured orchard environment. For example, underactuation between the links and fingers helps to ensure a form closure grasp of fruits with variable shapes, sizes, and orientations. Also, the passive compliance of flexure joints has been shown to increase robustness to positioning errors [10]. In the case of unintended collisions, which are expected during harvesting, the flexure joints can sustain out-of-plane deflection and large deflections without damage [10]. The fingers were created with a 3D printer using a fabrication process similar to that described by Ma et al. [11].

Fig. 3. Computer Aided Design (CAD) model of the prototype end-effector. The prototype is an underactuated design with three fingers. Components include tendons, a differential, soft finger pads, flexure joints, and a single actuator.

Our goal at the start of this project was to develop an end-effector with minimal sensors that utilized open-loop, feedforward control. As such, an important initial step in the development of the end-effector prototype was experimental analysis of the normal forces that develop during a grasp. To analyze these forces a plastic sphere with a radius of 4 cm was located symmetrically with respect to the end-effector (i.e. the centerline of the end-effector was coincident with the center of the sphere). Piezoresistive force sensors (Tekscan, Inc., Boston, MA) were then attached to the points of proximal contact on each of the three fingers. To complete a grasp the end-effector’s actuator was operated in torque mode and driven to its stall torque. The quasi-static equilibrium force was then recorded for increasing actuator loads. This same experiment was also repeated for distal link normal forces. The results from the proximal normal force test are shown in Fig. 4. For this particular grasp approximately 10% of the actuator’s maximum torque value was required to produce normal forces representative of those developed during manual picking of apples. Though the normal force distribution was similar for each of the fingers, the grasp was not force-isotropic. Because normal forces are highly configuration dependent, slight variations in the sensor placement can significantly impact the final results.

Fig. 4. Proximal normal forces that develop during a power grasp of a sphere with radius of 4 cm.

An important design criteria for the end-effector is the ability to complete an effective grasp even when provided fruit position data with moderate errors. The objective of a second experiment was to quantify the maximum allowable error in position data provided by the vision system. To complete this study a replica fruit system containing a branch, stem, and apple was established using, respectively, a piece of rubber conduit, extension spring, and plastic sphere (Fig. 5). The centered position of the end-effector relative to the fruit was designated as the point where the normal of the end-effector palm was coincident with the fruit origin and the surface of the fruit was tangent to the palm. Using the inverse kinematics algorithm, the system was then driven to a grid of points in the x-y plane that included position errors relative to the center position. A grasp was considered successful if each of the three fingers made two points of contact with the fruit. Results showed that the endeffector could accommodate position errors of approximately 2.5 cm and still complete a successful power grasp. This allowable error bounds the maximum expected position error from the machine vision system. IV. MACHINE VISION SYSTEM Detecting fruit such as apples in an outdoor environment with varying degree of visibility is a challenging problem that has greatly affected the performance of robotic tree fruit harvesting systems. Fruit clustering presents further complications. A literature review [12, 13] indicates that reasonable accuracy can be achieved for detecting fruit in an orchard environment when the apples are clearly visible or only a small portion of the fruit is occluded. However, limited work has been carried out to detect fruit in clusters, which is critically important as fruit clusters are common in field conditions. This work presents an apple detection algorithm and a hierarchical approach to improve the accuracy of apple detection for robotic harvesting. An overthe-row platform with tunnel structure and artificial lighting was used to acquire images of tree canopies trained in a tall spindle architecture.

Fig. 5. Manipulator and end-effector during a position error test. The endeffector was driven to different points in the horizontal plane to gauge its robustness to position errors.

Curvature

Edge pixels Fig. 6. Conceptual plot of method that uses apple curvature to locate upper hemispheric region and calyx in order to estimate peduncle orientation.

Iterative Circular Hough Transform (CHT) [14] was used to detect clearly visible fruit. Partially occluded apples were detected using blob analysis, and a clustering algorithm was used to merge parts of an apple divided by occlusion. In addition, the issue with clusters and partial to full occlusion was minimized by strategically harvesting the most visible fruits first. Harvesting preference was given to clearly visible and individual fruit in clusters identified by CHT. These prioritized apples were then picked manually for demonstrating the performance of the hierarchical approach. As images were taken again, partially or fully occluded apples were better exposed after some of the clearly visible ones were harvested. The process was repeated until no more apples were identified. For a set of 980 test images, the fusion of CHT and blob analysis yielded 90% detection accuracy with 1.8% false positives. When the hierarchical strategy was applied, 98% of the apples were detected and harvested. This method provided a unique and novel insight into the potential for fruit detection and robotic harvesting in an orchard environment. Another challenging problem in robotic apple harvesting is peduncle detection. The optimum position for the robotic end-effector to grasp a fruit is highly correlated to stem location. Qualitative observation of images taken during field data collection revealed that views to most of the peduncles are obscured by branches, leaves and surrounding apples. Although their visibility is occluded, the curvature of the apples themselves could potentially reveal information critical for approximating peduncle orientation. Preliminary analysis of the curvature of fruit edges has revealed a method to localize the calyx and upper hemispheric regions of the apple. As the centroid, calyx, and stalk are aligned along the same axis, this information would be vital for estimation of apple peduncle location. Fig. 6 presents a conceptual plot of this technique. ACKNOWLEDGMENT The authors would like to thank Cameron Hohimer for his assistance with additive manufacturing of the end-effector components.

REFERENCES [1]

[2] [3]

[4]

[5]

[6]

[7] [8]

[9]

[10]

[11]

[12]

[13]

[14]

R. K. Gallardo, M. Taylor and H. Hinman, "2009 Cost Estimates of Establishing and Producing Gala Apples in Washington (FS005E)," 2010. S. Galinato and R. K. Gallardo, "2010 Estimated Cost of Producing Pears in North Central Washington (FS031E)," 2011. D. L. Peterson, M. D. Whiting and S. D. Wolford, "Fresh-Market Quality Tree Fruit Harvester Part I: Sweet Cherry," Applied Engineering in Agriculture, vol. 19, no. 5, pp. 539-543, 2003. R. Polat, I. Gezer, M. Guner, E. Dursun, D. Erdogan and H. Bilim, "Mechanical Harvesting of Pistachio Nuts," Journal of Food Engineering, vol. 79, pp. 1131-1135, 2007. J. Baeten, K. Donne, S. Boedrij, W. Beckers and E. Claesen, "Autonomous Fruit Picking Machine: A Robotic Apple Harvester," Field and Service Robotics, vol. 42, pp. 531-539, 2008. D. Zhao, J. Lu, W. Ji, Y. Zhang and Y. Chen, "Design and Control of an Apple Harvesting Robot," Biosystems Engineering, vol. 110, pp. 112-122, 2011. S. A. Fennimore and D. J. Doohan, "The Challenges of Specialty Crop Weed Control," Weed Technology, pp. 364-372, 2008. L. Calvin and P. Martin, "The U.S. Produce Industry and Labor: Facing the Future in a Global Economy," Economic Research Service, United States Department of Agriculture (USDA), 2010. L. T. Wang and C. C. Chen, "A Combined Optimization Method for Solving the Inverse Kinematics Problem of Mechanical Manipulators," IEEE Transactions on Robotics and Automation, vol. 7, no. 4, pp. 489-499, August 1991. A. M. Dollar and R. D. Howe, "The Highly Adaptive SDM Hand: Design and Performance Evaluation," The International Journal of Robotics Research, vol. 29, no. 5, pp. 585-597, April 2010. R. R. Ma, L. U. Odhner and A. M. Dollar, "A Modular, Open-Source 3D Printed Underactuated Hand," in IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, 2013. D. Stajnko, J. Rakun and M. Blanke, "Modelling Apple Fruit Yield using Image Analysis for Fruit Colour, Shape, and Texture," European Journal of Horticultural Science, vol. 74, no. 6, pp. 26-267, 2009. R. Linker, O. Cohen and A. Naor, "Determination of the Number of Green Apples in RGB Images Recorded in Orchards," Computers and Electronics in Agriculture, vol. 81, pp. 45-57, 2012. A. Silwal, A. Gongal and M. Karkee, "Apple Identification in Field Environment with Over-the-Row Machine Vision System," Agricultural Engineering International: Agric Eng Intl (CIGR Journal), vol. 16, no. 4, pp. 66-75, 2014.