Redmond Ramin Shamshiri, PhD
[email protected]
Next
Precision Agriculture (Satellite Farming) has faced a breakthrough shift since risings of UAVs Conventional Remote Sensing platforms are being replaced by integrated UAVs UAVs & Robotics are the future of Precision Ag Precision Agriculture of Oil Palm is one of the largest market in Malaysia to be hit by UAVs
Precision Agriculture
GPS GIS
Wireless Sensor Networks
Automation Control
Robotics
PA is a farming Management concept based on Sensing, Measuring and Assessment
RGB Infrared NIR Hyperspectral Multispectral Thermal Mapping software, Mobile Apps
Sensing Components
Sensors , Cameras o o o o o o o
Typical Applications in Agriculture UAV
Remote Sensing
VRT
As a flexible remote sensing platform Crop/ Tree Scouting Health/growth assessment Inventory management Yield estimation / Monitoring Weed and disease detection Mapping (2D, 3D, GIS, NDVI) Risk/Hazard/Safety management Soil condition assessment VRT Robotics harvesting Academic and Research application
Prev
RGB Hyperspectral Multispectral Thermal NDVI Infrared NIR
Platform
Airborne
Ground-based Handheld
Satellite
Piloted Airplanes
Next
Vehicle mounted
UAV
Fixed wing
Multi-rotor
Prev
Traditional Scouting o o o o o o o o o o
Next
Precision Agriculture is about optimizing returns on inputs while preserving resources
Integrated with Commercial sensors/Cameras/Software
Prev
Next
Plant densities are an important and limiting factor for growth, Nutritional status, fruiting and hence for a plantation’s yield.
Traditional scouting requires spending hours and hours of visualizing Involves manual operations, ineffective techniques Limitation of human/labor resources (Not on demand) Repeated dull tasks (i.e., Palm census) Not a pleasant environment to work (hot, humid) Hazard/ Safety (Falling from trees, bugs, snakes, etc) Generates Inaccurate/biased statistics Requires expert knowledge/Post processing (i.e., lab analysis of data) Generates limited information (Does not provide comprehensive result) Ignored parameters due to measurement’s difficulties (i.e., tree height,
Oil palm with good plant density
Oil palm with high mortality
Optimal plant densities depend on different factors, such as cultivars, climate, soil characteristics, land preparation… Refilling of canopy gaps and correction of non-optimal plant densities are of high priority for a good plantation management
canopy diameter, tasks that involves climbing trees)
Satellite remote sensing o Cost o Low resolution o Difficulties of access (Not on-demand)
Yield reduction due to high density palm areas that causes Etiolation*.
Ground sensing o Time consuming o Limited field of view
Need for Automatic identification of potentially etiolated palms or high density areas Conventional method, solely based on visual observation, inaccurate, particularly when coverage is large and dominant topography is hillocky. * Etiolation is a process in flowering plants grown in partial or complete absence of light.
Prev
Next
Prev
Next
1
Main Objective
Specific Objectives
To develop/adapt a flexible remote sensing platform by integrating UAV, Sensors, and robust machine vision system for Smart Management of Oil Palm Plantations
1. To setup and operate a multi-rotor UAV remote sensing system for Oil Palm plantations 2. To produce 2D and 3D visual maps of the fields under study
Smart Management involves:
(including Blocks, rivers, roads, boundaries)
3. To produce NDVI (Vegetation stress) and GIS maps for health and growth assessment
A: Smart Inventory Management
4. To develop a robust, real-time machine vision system for the following tasks
B: Smart Growth/Health Assessment
(4.1) Inventory management Palm census and density, track and record Crown diameter estimation, Canopy size Palm height measurements Plantable, Unplantable, Overplanted areas Palms distance
“Smart” refers to: Autonomous monitoring, data processing and decision making
(4.2) Yield mapping system Detection/Quantification of fresh fruit bunches from UAV images Yield per Palm and Yield per ha Prev
Next
Prev
Next
Prev
Next
Prev
Next
Prev
Next
Prev
Next
Consideration for a flexible design
Small size Light-weight Affordable Autonomous Stable (against wind and other disturbances) Shifting between multiple sensors Payload Flight time Safety (For operator, environment, and platform) Low-altitude flight On-Demand flight (Ease of access operation) Repair and Maintenance costs
2
Prev
Next
Billion US Dollar
Oil Palm contribute to 12 billion USD of Malaysian economy 20 18 16 14 12 10 8 6 4 2 0
Potential for monitoring oil-palm plantations in such a great detail has been never possible
Prev
Next
This research will provide growers/managers with a tool for: Automatic palm detection, counting, size measurements, etc
Palm Oil Export ($)
Calculation of planted areas (for replanting or thinning) Analyzing Palm status based on orthomosaics and digital elevation models Generating valuable information based on each and every individual palm Classification of palms based on crown size, height, vegetation indices, etc Such information can be used for appropriate management decisions (maximize yields)
60
% of the total world
50
Yield Estimation Model development
Palm Oil Export (% of the total)
Correlation between palm height (𝒙𝟏 ) , crown size (𝒙𝟐 ), age (𝒙𝟑 ), vegetation index (𝒙𝟒 ) , …, and yield
40 30
𝒀𝒊𝒆𝒍𝒅 = 𝒇𝒖𝒏𝒄(𝒙𝟏 , 𝒙𝟐 , 𝒙𝟑 , 𝒙𝟒 … )
20 10
A model that is based on a comprehensive information of each palm location, size, and health, will provide managers with an estimation of yield, and make decisions for sustainable practices methods for production increase without necessary needs for expanding the plantation into natural forests
0
Prev
Prev
Next
Prev
Next
Methodology steps
Research phases
Accurate planted area Creation of Palm trees inventory database for specific plot Total land use Palm distances to specific areas Canopy diameter estimation Tree height measurements Calculating palm density for specific plot Creation of 2D, 3D, GIS, NDVI maps for plantation Monitoring Healthy/Unhealthy palms (Stress assessment) Monitoring exposed soil (VRT application) Quantification of FFB, Estimation of mature fruits Calculating yields for each palm from the acquired images Yield monitoring Creation of yield maps Chlorophyll analysis Drought assessment Biomass indication Leaf area index Growth monitoring Weed detection Inventory management decision support systems Yield Model Development Academic and Research application
1. 2. 3. 4.
Platform setup, integrating UAV, sensors and software Creating high quality 2D and 3D maps of the area Developing custom-built programs/algorithms for smart inventory management Developing custom-built programs/algorithms for smart Health/growth assessment
DATA COLLECTION Data Processing Mapping Modeling
UAV images need to be photogrammetrically processed and translated into accurate 2D orthomosaics and maps, 3D models and surface models, and other GIS datasets
UAV Setup Flight Preparation Test and trials Mission planning
Image acquisition Video streaming Camera/Sensor setups Calibration
Image processing Creating 2D, 3D, GIS, NDVI Maps (Pix4Dmapper, Agisoft)
Results / Reports
Image/Map interpretation GIS analysis Custom software
Data analyzing
Management strategies Decision makings
Correlation analysis
Reports generation
Orthomosaics, 3D Models and Digital Surface Models, 3D Flythrough Videos, Multispectral Image Mosaics, Index Maps, (i.e., NDVI) needs to be processed/interpreted
Next
Prev
Next
3
DJI Phantom, a small hobbyist quadcopter
Purchase, Adapt, Or Build from Scratch? Sensors (Cameras)
UAV Platform
Hyperspectral
Application Software
Programming (Processing)
RGB
Multi-rotor
Fixed wing
Factors to be considered
NIR Onboard GPS
Auto flight Controller
LiDAR
Multi-rotor UAVs launch and land vertically are favored where space is tight Fixed-wing UAV Requires suitable space to launch and land Can provide longer flight duration Can carry a heavier payload.
Built from scratch UAV
Commercial UAVs are expensive, Not designed for operation inside oil-palm plantations Might need protection frames
NDVI Thermal
Quadcopter micro UAV 'microdrones md4-200' with the prototype MSMS multispectral sensor
Setting up a UAV Multispectral
Fixed Wing, ebee
The quadrocopter UAV, model md4-1000
Low-cost: affordable by oil palm growers Modular sensor system (shifting different sensors) Higher flight time, more ha coverage Payload Mobile applications Data sharing Wireless networks
MōVI M10 - Digital 3-Axis GyroStabilized , Price: $4,995
The camera system on board the Oktokopter
In-flight stability, flight-time duration and payloads are major paramount concerns
Prev
Next
Visible (RGB)
Prev
Next
Normalized Difference Vegetation Index (NDVI): measurement of the amount of live vegetation in an area
Infrared\ Near Infrared
NDVI - TWIN LUMIX LX7 low cost, rugged, high resolution imaging solution for NDVI, agricultural and archeological data analysis. provides true R and IR information from different sensors, providing the clean photogrammetric information required for vegetation anaysis processing. Combined with a 5cm resolution at 400ft this sensor provides a host of benefits for the agronomist and archeaologist . True Resolution:10.1Mp, Aperture Range:F1.4 - F2.3 Max Shutter Speed: 1 second, ISO:80 - 6400 ISO Image Quality: 82, Resolution (GSD): 5cm @400ft Format:RAW, JPG, Image Stabilisation:roll and anti-shake
SUN
Sony A6000 Visible Sensor
Lumix LX7 Sensor Details
Lumix LX7 Infrared Sensor
Focus Points: 179 True Resolution: 24.3Mp Pixel Size: 15.1 µm² ISO: 1347 ISO Image Quality: 82 Dynamic Range: 13.1 EV Colour Depth:24.1 bits Resolution (GSD):1.5cm – 4.5cm
True Resolution:10.1Mp Aperture Range:F1.4 - F2.3 Max Shutter Speed: 1 second ISO:80 - 6400 ISO Image Quality: 82 Resolution (GSD): 5cm @400ft Format:RAW, JPG Image Stabilisation:roll & anti-shake
True Resolution:10.1Mp Aperture Range:F1.4 - F2.3 Max Shutter Speed: 1 second ISO:80 - 6400 ISO Image Quality: 82 Resolution (GSD): 5cm @400ft Format:RAW, JPG Image Stabilisation:roll and anti-shake
X%
Canon S110-NIR, 12 MP, adapted to be controlled by drones autopilot Acquires image data in the NIR band
Y%
Resolution: 12 MP Ground resolution at 100 m: 3.5 cm/px Sensor size: 7.44 x 5.58 mm Pixel pitch: 1.86 um Image format: JPEG and/or RAW
𝑋% − 𝑌% ≤ +1 𝑋% + 𝑌%
NDVI0.66 Very healthy plants
Healthy plants have a strong near infrared reflectivity, called the "Red Edge".
Prev
−1 ≤ 𝑁𝐷𝑉𝐼 =
RGB
NIR
NDVI Canon PowerShot SX260 12.1 MegaPixel Digital Camera Models: XNiteCanonSX260: UV+Visible+IR XNiteCanonSX260: IR- Only XNiteCanonSX260NDVI: 3-Band Vegetation Stress Remote Sensing Camera
Next
Prev
Next
Object-Oriented Software Framework for Hyperspectral Imaging Thermal Sensors
Multispectral Sensor
High Resolution Uncooled Thermal Camera: Tau 640
To identify Oil Palm stress factors, soil types, fertilizers, or insecticides To identify differentiate plant species or recognize other plant (weeds…) To identify soil or chemical conditions that are, in each case, able to be identified by their unique spectral signature. To graphically illustrate vegetation indices such as NDVI that are defined by relationships of specific narrow-band wavelengths. With each exposure, 4 or 6 separate bands of visible or near-infrared radiation move through each camera's lens and filter to form a separate monochromatic image on the camera's sensor.
Hyperspectral Imaging
RGB Color Image Pixel 3 dimensional data at R, G, B waveband
High Resolution thermal imaging can assist disease detection and water stress in Oil Palms, or for scouting at nights, fire hazard alarm
Optris PI450 Temperature range: -20°C to 900°C Spectral range: 7,5 bis 13 µm Optical resolution: 382 x 288px Frame rate 80 Hz capture single images at a rate one per 2 seconds. Each pixel from each image has an exact temperature associated with it.
Prev
Next
Prev
Next
4
Hyperspectral image contains enormous information. Pixel dozens or hundreds dimensional data
Advantages • We can analyze the target in detail. • We can combine both spatial and spectral analysis. • We can apply it to the wide area of agricultural sensing.
[-] Intensity [-] Intensity
250
200 200
Disadvantages • Data structure is complicated and special. • Data size is large. • There are few software applications and libraries. • We have to develop the software by ourselves. • We have to understand complicated data structure.
150 150 100 100 50 50 0
0 0
25 100
50
150
50 200
Waveband Spectral Band [-] [-]
Prev
Next
Prev
Next
Mechanism of Hyperspectral Camera (ImSpector)
400 nm from http://www.specim.fi/ functions as line sensor
Camera
Wavelength spectral axis
Camera
Camera axis
1000 nm
Target
Spatial axis by Electric moving pan head
by Driving vehicle
Prev
Next
Prev
Next
Dataflow of Sampling Pixel Spectral Data Sensing line frame
Spectral image A
B B-D
A-C
y (x, y)
x Pixel spectral data 250 Intensity [-]
200
C
D
150 100 50 0 0
50
100
150
200
Spectral Band [-]
Prev
Next
Prev
Next
5
Extraction of Desired Waveband Value ("WavebandProcessor" object) Outdoor Fields Hyperspectral Camera
Target
Digital Video Recorder
Indoor Laboratory
IEEE 1394 Interface
Personal Computer
Sugar beet Waveband No.15（500nm）
Prev
Next
Composition of 3 Waveband Values as False Color
("FalseColorProcessor" object)
Sugar beet Waveband No.35（750nm）
Prev
Next
Distinction between Plant and Soil ("PlantSoilProcessor" object)
1: Plant
0: Soil Sugar beet NIR color (NIR, R, G)
Soybean (crop row) Visible rays（R, G, B）
Sugar beet
Prev
Next
Estimation of Chlorophyll Content (SPAD Value) ("SpadProcessor" object)
24.7
Soybean (crop row)
Prev
Next
Plant Classification ("PlantClassificationProcessor" object)
50 - 55 45 -
1: Sugar beet
40 35 -
26.0
0: Soil
30 25 -
36.9
40.2
20 15 -
2: Sugina
10 5-
24.7
0-
Estimated SPAD values of sugar beet
Prev
Estimated plant ID number (Soil, Sugar beet, Sugina) Next
Prev
Next
6
Light Detection and Ranging (LiDAR)
Topographic Survey and Mapping Contour Mapping Cross Section / Longitudinal Analysis 3D Mapping Floodplain Mapping Vegetation Mapping Shoreline Analysis Corridor / Route Studies Slope Analysis
3D Point Cloud of LiDAR
Ortho-mosaic of Aerial Photo
Routescene LidarPod
Potential Application Palm height measurements Prev
Next
Price 8700 USD unlimited use in time 3500 USD /year (rent) 350 USD /month (rent)
Convert thousands of UAV images into Geo-referenced 2D maps Orthomosaics models 3D surface models point clouds Orthomosaics
Prev
3D Models
Next
Price: £400.00
Vegetation, Health, growth analysis SampleVideoAgisoft
Process thousands of aerial images Suitable for a non-specialist operator Generate high- resolution Geo-Ref orthophotos Exceptionally detailed Geo-Ref DEMs* Fully automated workflow Easy integration to the Q-Pods system Create projects using more than one camera and process imagery together (i.e., NIR and RGB)
SampleVideoPIX4D
* DEM: Digital Elevation Model
Prev
Next
Prev
Next
Prev
Next
Creating smart data for decision support systems GIS tasks Essentials of GIS & Aerial Image Interpretation Map Generation Creating Workflows Management of Spatial Data Natural Color Images Multi-spectral Images Digital Elevation Models Multi-temporal Images Images source: Adapted from Terracentra
Prev
Images source: Adapted from Terracentra
Next
7
Plantation Infrastructure Inventory, road mapping, inventory and monitoring is very important for efficient plantation management.
Estimated/Precise Palm tree counts in a selected area of interest
Fertility Mapping, detecting and mapping of oil palm fertility or palm vigorous growth level
Identifying unhealthy palms
Plot size: (ha) Palm counts: 2500 Palm density: 100 trees/ha
Prev
Next
Prev
Next
Identification of poor spots examples
Finding average distance of a plot to the river/road
Prev
Next
Prev
Next
Tree Counting methods
Calculating total unplanted area from Geo-referenced maps Training rules
Commercial Software
Other techniques
S5
S4
S3 S1
S2 S6
Total Unplanted Area =
PLOT B
𝒏 𝒊=𝟏 𝑺𝒊
Images source: Adapted from Terracentra
Prev
Next
Prev
Next
8
Study Area=10.3305 Sqr.Inch Estimated count= ?
Training Area = 0.9 Sqr.Inch Training Count = 4
1 2
7
13
8
14
9 3
15 10
4
16
11
21
24 25 26 27
22
5 6
19 20
17
12
18
23 0.5
30
36 37
31
42 43
38
32
44 39
33 28 29
Banana plantation in Indonesia
open-source software QGIS The Leading Open Source Desktop GIS
40 34 41 35
45
The software allows plant counting, density calculations and the generation of mortality maps by visual inspection of the image products.
46
Pineapple plantation in the Philippines
0.4
Manual count ≅ 47
Study Area
Factor =
______ Training Area
Estimated count = Factor × Training count
10.33 = ---------- = 11.47 0.9
= 45.9 Accuracy: 97.6%
Prev
Next
Prev
Next
ERDAS IMAGINE's Spatial Modeler (NIR thresholding) The overall performance (detection rate) between 0.916 to 0.998.
RGB, NIR
NIR
EDF Image
LPF Image
Threshold Image
Prev
Next
Photogrammetric point clouds Technique
Prev
Next
Photogrammetric point clouds Technique
photogrammetric point clouds
Digital surface model
Local maximum display tree position The mapping accuracy amounts for 86.1% for the entire study area and 98.2% for dense growing palm stands.
Prev
Next
Prev
Next
9
Correlating between palm heights, ages, and yield
Prev
Next
Crown (Canopy) Volume Crown diameter
Correlating between image and mass of FFB
Prev
Next
Geo-referenced
FFB detection/quantification
Prev
Next
Autopilot and mission control Visual servo control mechanism for FFB detection Breakthrough innovative ideas (i.e., Night-mission flights) Making smarter UAV platforms that can learn while flying Techniques for Improving accuracy and resolution
Y=Weight (Kg)
Development of customized sensors with built-in algorithms for specific task Improvement of low altitude flight mission Improvement of flight control over actuator limits and noise (i.e., Controller Design for Stabilizing of an Autonomous fixed wing Crop Surveillance Osprey Drone with Actuator Limits and Sensor Noise)
X=% of Indexed pixels, No. of FFB, etc
Prev
Next
Prev
Next
10
UAV can contribute to mechanization of Oil Palm Agriculture
Reducing labor force on field Developing GIS dabase, 2D, 3D, NDVI, and thermal maps Reducing labor hazards Reducing management time Palm tree tagging Monitoring fungal disease with different sensors
FFB quantification is the first step toward building cost-effective robotic harvesting system for existing palm trees
Potential to be extended to other fields in Malaysia, i.e., rice and rubber Embedded Child Script ROS nodes
Further contribution
Plugins
Smart pesticide control Enhanced pollination Constant track and record of growth condition Drastically help growers in decision making Early warnings for disease A ground for autonomous robotic harvesting Academic application
Control Mechanisms Serial port Remote API
Preliminary design Simulation
Re-design
Prev
Next
Nutrient contents, N, P, K, Mg, B
Prev
Field experiments
Next
Link to draft
RGB NDVI
Evaluation Improvement
Prev
Predicted Yield Disease detection Management decisions
NIR
Building the prototype
Rubber plantation
Link to draft
Palm heights, density, Crown Diameter, Volume, FFB quantification, etc
Next
Prev
Next
Picture of an 8 year Oil Palm that produced 535kgs of FFB in one year
PrecisionHawk will presented its drone platform for early disease detection on February 4, 2016 at Dubai Internet City. Prev
Next
Prev
Next
11
Prev
Next
Prev
Next
Prev
Next
Prev
Next
Prev
Next
Prev
Next
5 LITERS $5,299
10 LITERS $8,399
15 LITERS $10,399
12
(UPM) (UPM)
(Univ of Florida) (Univ of Florida) (Univ of Florida) (Wageningen UR) For their insightful suggestions and ideas
Prev
Next
13
