Advanced Driver Assistance Systems (ADAS) and Data Fusion

TM

September 2013

4th Generation 3rd Generation 2000- 2020

2nd

Visible Intelligence

Generation 1970 - 2000

• Car system are smart and users ‘sees actively’ how the car affects him • Park Assist, ACC, LDW, Ped Detection

Hidden Intelligence

1st Generation 1900 - 1970 Mechanical System Drive-train, mechanical brake and steering

TM

• ABS, EAFS, Fuel injection • Cars make smart decision but driver is not ‘actively’ made aware

3

2020 - 2040 Partial to virtually full Autonomous • User does not drive most of the time • New car-driver interaction model

TM

Sensor Units / CAGR % Vision Sensor (Blue): 82M u/ 22% Radar-Lidar (Red): 59Mu/ 30% Ultra Sonic(Green): 23Mu/ -4% Total: 164Mu/ 16%

•Strong Growth of Sensor •360° Sensing & NCAP •Two major Technology !! •77GHz and MegaPixel

•U-sonic in decrease •No long term LIDAR •Night Vision smaller Camera Sensors •Smallest Size – High system flexibility •Largest FOV + increasing Pixel • density •Power reduction – ISP & ICP •Largest Volume and take rate

Radar Sensors

•Larger Size ⇒ integration of component •Longest range (200 meter+) •Power Reduction – Scalability •Distributed Computational load •Most complete Sensor and good Take rate

IMS Research - Feb 13` TM

5

Ultrasonic Sensors •Small Size – Negative growth - mature •Close range accuracy (few cm) •Large number used per vehicle •Only for park assist and in negative growth

Adult Occupant

Fit Rate

100%

Child Protection

0%

Pedestrian Protection

50%

Safety Assist Blind spot monitoring Lane support systems Speed Alert Systems Emergency Braking Attention assist Etc.

50%

Ex: In 2012 36 Models rated: 66% got 5 Stars incl. all of the Small Family cars Every OEM largely focused to hit 5 stars. AEB is already ‘commoditizing’ and ACC is often added as value added TM

6

TM

• • • •

Camera and Radar @>15kmh Cognition Algorithms to extract features / classify objects No display necessary F. Safety applied to longitudinal motion (braking / Steering)

e.g. • Lane Keep Assist • Adaptive cruise control • Emergency braking • Pedestrian protection

• • • • •

Rear/Side Camera, sat. Radar, Usonic @15kmh 3D Enviornmental Modeling allowing self navigation No Display Hard safety – Longditudinal and Lateral motion Integration of Feature extraction

e.g. • Self-driving Auto • Sensor Fusion

8

Scalable Performance

Non classic approach – Innovation

“Multiple” fusing Sensors ⇒ Dense volatile Memories Large BW for Data exchange Complex Parallel Processing

Every MIPS is needed ⇒ ISO26262 process followed Safe volatile memories Security: which standard ?

Application Tools and libraries

Miniaturization

Software Development Accelerated ⇒ SDK Tools for Image Cognition Scalability of SW

Smaller Size ⇒ Package for Power dissipation Small PCB allowed Integration of “other” dies . TM

9

Front View Camera System (LDW, TSR, PP, Obj det)

Smart Camera Rear – Remote Park Assist Park Assist/Self-Parking

Night Vision/ Surround View Camera

Decision Fusion Center Evasion Assistance

Interior Camera Driver Monitoring

High Beam Control

•

Side Camera Blind spot /Surround View

Key Trends: 1. Flexible solutions to adapt to OEMs fast evolving programs 2. Major focus on Power/Performance ratio and Size reduction 3. Strong SW Enablement to support investment return TM

10

Park Assist

Side Impact Assist Radar Fusion Center

Blind Spot Detection

Emergency Brake System & ACC

Cross Traffic Assist

Key Trends: 1. Large increase of sensors to support 360° sensing 2. Strong Cost Reduction 3. Miniaturization TM

11

TM

Qorivva 57A7 Highly Integrated RADAR μC Analog and Digital Excellence Performance– offers top-performance for intense computational tasks with key integrated digital accelerators

Safe –Built on proven safe technology it

delivers a scalable, well documented,

process compliant safe architecture and safe Software

TM

Integration & Cost – Right balance of

memory, large number of Analogue IP designed for Radar, FFT accelerator. Drive Miniaturization and BOM saving Flexible – can be used in all

applications and with all Front End Radar sensor technology and types.

13

Qorivva MPC577xK •

•

Replaces: −

8 ADC

−

1 DAC

−

1 FPGA

−

External SRAM

−

General purpose MCU

Enables:

Previous Generation − Significant PCB area saving Next − Reduced assembly cost Generation

MR2001 77 GHz Chipset •

•

Replaces: −

Bare Die RF solutions with a RF Chipset based on RCP package technology

−

Discrete Filter Components and Amplifiers

Enables: −

Significantly lower assembly cost

−

Lower PCB cost TM

14

ADC Input

Radar Processing Platform

8 x ΣΔ – 10MHz

Signal processing toolbox

4x 12bit SAR -1MHz

FFT

COPY

Z7 dual is. @266MHz 16kB I-cache 2 way

16kB D-cache 2 way

VFPU & SPE2-SIMD

Z7 dual is. @266MHz

DAC

Scheduler

DMA

2Ms/s 8 bit DAC

16kB I-cache 2 way

16kB D-cache 2 way

Vehicle and ECU Network

CPU Platform

Z4LS @ 133MHz 16kB I-cache 2 way

16kB D-cache 2 way

Z4 LS @ 133MHz

8kB I-cache 2 way

Master Comm Bus 128 msg FlexRay 100base T Ethernet

SFPU

4kB D-cache 2 way

SFPU

Intra ECU connectivity LFAST

VFPU & SPE2-SIMD

Volatile Emb. Memory 1.5MB RAM with ECC

NV Memory 4MB with ECC

Ext. Memory I/F

Fabric 64 bit XBAR with E2E ECC

PMU OSC and PLL

Connectivity

16 bit PDI

2 x Cross Trig Unit

3 x IIC

2 x SENT I/F

Safe Memory

2 x FlexPWM (12 ch)

4 x LinFlex Ctrl

4 x dSPI

MEMU

3 x eTimers – 6 ch. each

4 x Flex CAN (1FD)

SWT & STM

T-Sensor FCCU & CRC Safe DMA DEBUG Nexus 3+

Key Features: ƒ F. Safety: developed as per ISO26262 with target ASIL-D ƒ Safety Enablement: Safety Manual and FMEDA ƒ SPT: Radix4/2, r2c, c2c, 50 MHz 16 bit twiddle, 24 bit results, Copy, Transpose ƒ SRAM: Multi ported SRAM Ctrl and 1.5MB SRAM with ECC ƒ Top of Class Analogue IP: PLL, DAC, OSC and ΣΔ ADC ƒ SW Enablement: QMS MCAL and/or Safe Mcal Asil B (D)

Specifications: ƒ CPU: 3x PPC: 2x Z7 266 MHz Power dual issue with SPE2 and VFPU and Z4 133MHz in permanent lockstep ƒ SPT: FFT Accelerator, DMA ƒ Analog: Octal SD + 4 SAR, Ultra low jitter PLL, precision DAC ƒ Package: 356 PBGA – 0.8 mm pitch – 17 x 17 mm2 body ƒ Temp Range (Ta): -40 to 125°C, 150 °C Tj, AEC-Q100 Grade 1 ƒ Main Supply: 3.3V IO (5V SAR) and 1.2V Core (ext or PMU)

TM

Safety & Support

15

TM

P e r f o r m a n c e

Front LRR + Fusion Front MRR

Front LRR/MRR

Satellite Radar(s) Qorivva 57A7

Single Chip Radar

Value TM

17

TM

Low – Mid Range Mono Front Camera Smart Rear Camera / SVC 9Distance Measurement 9Object Detection 9Pedestrian Detection 9Graphic Overlay TM

9Lane Departure Warning 9Blind Spot Detection 9Smart Headlamp Control 9Cross Traffic Alert 9Forward Collision Warning 9Low Cost Adaptive Cruise Control 9Pedestrian Detection (60kph) 9Traffic Sign Recognition (Full)

i.Mx 58A8

i.Mx 58A5

i.Mx 58A2

High End Front Vision & Fusion solution Mid Range Mono Front Camera Smart Rear Camera / SVC 9Smallest Size achievable 9Low cost Pedestrian Detection coverage - NCAP 9Graphic Overlay

TM

9Cost Optimum for 4-function bundle 9Can do Stitching and offload infotainment unit 9Power & Size optimized 9Tools and Libraries 9ASIL B(D)

20

9Stereo Camera & multi bundle apps 9Multiple stream Stitching capability 9Best in Class Performance/Power 9Highly abstracted Image cognition processing 9Smallest form fit at same power: RCP 9Tools-Kernels-app examples 9Std connection DSP, FPGA for flexible perf. Extension 9Easy ASIL-D with Very high performance Fusion box

Image PreProcessing • ISP • Multi-streams Connectivity

Features Extraction • • • •

Features Classification & Prediction

Corners, • SVM • Adaboost Edges, Intensity gradients • K-nearest Neighbor shapes

Gfx & Display

Image Cognition Proc.

L-mem

L-mem

Sequencer

32 CU

32 CU DMA

Sequencer

APEX2 CL

L-mem 32 CU DMA

CPU CPU I-cache

TM

CPU CPU

D-cache

I-cache

I-cache

D-cache

SIMD

GPU

D-cache

I-cache

Display Controller

D-cache

SIMD

SIMD

SIMD

Video Codec Encoder

APEX2 CL

High BW Operations Scalable MIMD Local Memory Soft ISP

Vision Platform

CPU Platform

Image Cognition Proc.

32 CU

Gfx Overlay & Video Distribution

• Tracking, Motion • Object • Safe Fusion Estimation Recognition/Ped • Graphic Overlay estrian & Display • Optical Flow & Disparity • Augmented • 2D vs. 3D Reality Projection • Stitching • Face Detection

Image Proc. Platform

L-mem

Object Recognition & Fusion

Multi Image Processing

L2 Cache

Scalable RISC – Data Fusion SIMD Co-Processor - Neon Memory Hierarchy and coherency 21

Decoder

Vector Graphic Video Codec Smart Display

TM

Specialized accelerators to provide maximum image processing performance with minimum power APEX-642 Core

• • • • • TM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

Multi Channel DMAs

CMEM

CMEM

Array Control Processor

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CMEM

CU

CMEM

CMEM

CMEM

CMEM

CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU

CMEM

CMEM

CMEM

CMEM

CU

CMEM

CMEM

CMEM

APU 1 (32x CU)

CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU CU

CMEM

CMEM

APU 0 (32x CU) CMEM

•

Array Control Processor

Sequencer

Stream DMAs

Massively Parallel SIMD for vector processing (CU’s + 4KB CMEM per CU Array Control Processor (ACP) for scalar processing, access to external and internal memory MC-DMA for efficient instruction and data movement Stream DMA for pixel data in/out of CMEM Sequencer off-loads ACP for data and instruction and sequencing 23

23

•

Ext.  Memory

APEX‐642 ICP 

• MC‐DMA + Stream DMA

Sequencer

•

32 x

CU

CU

CU

32 x

CU

CU

CU

ACP

CU

•

CU

ACP

APU‐0

APU‐1

TM

24

MC-DMA and Stream DMA move slices into CMEM – no cache misses APU SIMD with CU’s do vector processing ACP can do scalar processing on intermediate results – no requirement to move to external memory Slices to be processed next, and intermediate results, are pipelined

Canny Edge, widely used, robust algorithm for edge detection (Sobel Operation). Named after John Canny.

Canny Edge Detection Comparison 70.0

MHz/fps 60.0

50.0

40.0

Canny Edge Algo

30.0

20.0

10.0

0.0 APEX-642

Cortex A8 Adreno 200 Adreno 205 PowerVR 1GHz SGX535

TM

PowerVR SGX540

25

BCM 2727

•

Automated Pipelining

•

Fast Development

•

Easy Optimization

•

Plug & Play Algorithms

•

Adapted to OpenCL

High Level Algorithm Processing Graph A

B

C

D

ACF DataPath

Memory Mgt

Pipelining

APEX Data & Memory Management TM

26

26

TM

SCP2201 Image Cognition Processors Smart Camera Technology Balances Performance and Power –

Delivers the Right Integration –

APEX™ IP offers industry-leading performance per mwatt and delivers an ideal solution for image cognition processing

Image cognition processor delivers the innovative technology required for smart automotive camera applications

Mainstream Cost-effective Solution –

Speeds Time to Market –

Cost, power and size advantages enables smart camera technology into mainstream vehicles, improving auto safety features

Comprehensive solution including libraries, hardware and software development kits speeds time to market.

TM

28

• • •

•

• •

•

ARM926EJ @ 350 MHz 1V Core, 1.8~3.3V IO APEX supports: − 34 billion operations per second (BOPs), or 2 billion MACs per second plus a 2nd slave scalar RISC (350 MHz) Built-in dedicated overlay engine, rotation, supports alpha-blending and overlay Drives TFT LCDs directly BT656 sensor interface (YUV, 8- and 10-bit with separate H/V Sync) Built in TV encoder/Video DAC (NTSC/PAL)

TM

29

TM

•

Video compression reduces cost − Low latency MJPEG compressed 1.2 Mpixel, 30 fps video stream fits 100 Mbps Ethernet link − Predictive compression control allows high quality factor − Negligible degradation of object detection rate at high quality factor − 12-bit per YUV color space component for high dynamic range • Ethernet simplifies architecture − Camera synchronization via IEEE 1588v2 PTP real-time clock − IEEE 1722 Layer 2 AVB transport protocol

TM

31

• • • • • •

64MHz e200 Core M-JPEG Video Encoder Real-Time Ethernet 512K Flash 64K Data Flash 96K SRAM

… samples available TM

32

System Cost

Phase 1 – 2013 SOP Ethernet Proof of Concept 9LVDS Elimination 9Ethernet proven in mass production

Imager

Qorivva 56A0

Phase 2 – 2015 SOP Systems Cost Reduction Through Integration

Phy

+ISP

9BroadRreach phy integrated through SiP (stacked die wire bond) 9Small low cost package (8x8mm) 9Cost, size, complexity reduction

Qorivva 56A1

Imager

KEY VALUE: SPACE Reduction 1. 2.

+ISP

Primary focus (phase2): High level integration to save cost and footprint Secondary focus (phase3): Image signal processing integrated to save cost and power at sensor location

TM

33

Phase 3 – 2017 SOP Value Added Features 9Image signal processing integrated (monolithic) 9BroadRreach phy integrated through SiP (stacked die wire bond) 9MCU + Sensor bundled

Imager

Qorivva 58A2

+ISP

System Integration

•

Qorivva 56A1

Lane/ Object detection etc.

Qorivva 56A1

Ethernet switch

Qorivva 56A1

1 Gbit Ethernet

Qorivva 56A1

GPU 3D

Infotainment MCU – i.MX A9

APEX

GPU 3D

i.MX6

SCP2201

DRAM(s)

Flash

DRAM(s)

ADAS ECU TM

Head Unit Display

Video encode

ARM926

Flash

IPU

3D model -Image stitching: •Vertex shading •Pixel shading •Resizing •Video encode

Infotainment ECU 34

TM

Autonomous cars shall provide 1. Fun when driving 2. Comfort when living in it 3. Safe environment all the time Sensor data processing and features extractions ⇒ Object

Modeling of environment ⇒ Mapping of reality

Navigation & assisted driving ⇒ Self Driving

Actuation ⇒Steering, braking, accelerating + Head Unit

Fusion of object with the goal of replacing for the longest time possible the driver. Fundamental requisites: • Large Computational Performance and BW to handle massive data • A Simple Safety Concept: not simply a Fail-SAFE but in the essence an Available system (almost Fail Operational) TM

36

• •

•

• •

It can not be approached as a traditional ESC or EPS system. − Lock Step is not the solution as it reduces availability We can learn from other markets (medical or avionic) − multi MCU architecture which requires also a strong SW solution to compare & control the individual MCU results The Safety Target set to ASIL-D can be argued − As long as the control of the vehicle is shared between the various systems (steering, braking ECUs) the safety integrity level of the Fusion ECU could be reduced to C − If not acceptable, then multiple mC architecture is more suited (it all depends on the time to give back control) It is valuable to merge Machine Vision or Radar Decision Unit and fusion but only if the SW/HW complexity is manageable Freescale ADAS portfolio of high performing, easily expandables and strongly networked mC offers a valid selection of SoC to pick from to build a Safe system

TM

37

Ethernet

Ethernet

ASIL B

i.Mx6

Qorivva 57A7

ethernet

CAN /FlexRay - Control

ASIL D CAN /FlexRay - Control

Fusion Module – ASIL D i.Mx6Q . 8000DMIPS 4 x Cortex A9 @800MHz 32K I-cache + 32K D-cache per core 1M unified L2 cache . 4 watts C40 low power process Cortex A9 with Dynamic voltage & frequency scaling Power gating Dynamic clock gating . ASIL B Trustzone MMU Watchdogs Redundant clock generation Clock monitoring Redundant peripheral bridges & peripherals Supply monitoring Die temperature monitoring

Qorivva 57A7

Performance

Power

Safety

TM

Performance 1700 DMIPS 2 x zen e200z7 @266MHz 16K I-cache + 16K D-cache per core Zen e200z4 @133MHz Power 2 watts C55 general purpose process Minimal use of Lvt transistors Dynamic clock gating Safety ASIL D Delayed lockstep z4 1.5M SRAM with ECC (including address) ECC on all RAMs End 2 end ECC on all system buses Built-in Self test of all logic and memories 4M Internal Flash with ECC (including address) Redundant clock generation Clock monitoring Supply monitoring Die temperature monitoring 38

Ethernet

Ethernet

Qorivva 57A7

ASIL D

Zipwire

Qorivva 57A7

CAN /FlexRay - Control

ASIL D CAN /FlexRay - Control

Fusion Module – ASIL D 2 x Qorivva 57A7 Performance 3400 DMIPS 4 x zen e200z7 @266MHz 16K I-cache + 16K D-cache per core 2 x Zen e200z4 @133MHz Power 4 watts C55 general purpose process Minimal use of Lvt transistors Dynamic clock gating Safety ASIL D 2 x Delayed lockstep z4 3M SRAM with ECC (including address) ECC on all RAMs End 2 end ECC on all system buses Built-in Self test of all logic and memories 4M Internal Flash with ECC (including address) Redundant clock generation Clock monitoring Supply monitoring Die temperature monitoring TM

39

Ethernet

Ethernet

ASIL B

i.Mx6

ASIL B

i.Mx6

PCIe

CAN /FlexRay - Control

CAN /FlexRay - Control

Fusion Module – ASIL C/D 2 x i.Mx6 Performance

.

2 x 8000DMIPS 8 x Cortex A9 @800MHz 32K I-cache + 32K D-cache per core 2 x 1M unified L2 cache Power

.

8 watts C40 low power process Cortex A9 with Dynamic voltage & frequency scaling Power gating Dynamic clock gating Safety ASIL C/D Trustzone MMUs Watchdogs Redundant task assignment External Flash with 128 bit AES signature Redundant clock generation Clock monitoring Redundant peripheral bridges & peripherals Supply monitoring Die temperature monitoring TM

40

.

TM

Closed Platform Operating System

• Fast time-to-market • Low invest • Differentiation only on cost • Low OEM differentiation • Low profit • Patent restrictions

Application

Object Classification

Feature Extraction

Image Signal Processing Tier1 IP TM

42

3rd Party IP

Closed Platform

General Purpose Platform

High invest with late return

Operating System

Operating System

• Fast time-to-market • Low invest • Differentiation only on cost • Low OEM differentiation • Low profit • Patent restrictions

Application

Object Classification

Feature Extraction

Application

Object Classification

Feature Extraction

Image Signal Processing

• Differentiation not limited • High initial invest for algorithms and SW • Late return of invest

Image Signal Processing Tier1 IP TM

43

3rd Party IP

Closed Platform OS Partners

Operating System Application

App. SW Partners !!

Object Classification

Feature Extraction

General Purpose Platform

Freescale Platform

CogniVue SW & Services

Operating System

Operating System

Application

Application

Object Classification

Object Classification

Feature Extraction

Feature Extraction

Image Signal Processing

Image Signal Processing

BENEFITS

• Open platform ⇒differentiation • Balanced initial invest • Improved timeto-market • Optimized value creation

Freescale Libraries Image Signal Processing

Tier1 IP TM

44

3rd Party IP

TM

Product Qual Proposal Planning Execution

Machine Vision Front – Side Fusion

Applications Radar & Vision

Samples Production

i.MX58Ax 600/800MHz w/ APEX2 CL Quad & Dual – H.264/GPU

i.MX58Ax

SCP2201/07

360° Seeing & Sensing Park assist

350MHz APEX-1 16 MByte DRAM

MPC560xE 64MHz ENET, MJPEG 512K

Qorivva 5601 64MHz ENET, Phy, MJPEG 512K

MPC567xK

Qorivva 58Ax

180MHz FlexRay, ENET 1M – 2M

Radar Long Mid/Short Front & Side

Qorivva 58Ax

Qorivva 57A7 266MHz, SPT FlexRay, ENET 3M – 4M LRR /MRR Radar

MPC564xL 120MHz FlexRay 1M

Qorivva 58Ax Qorivva 57P4 180-200MHz, Dual EL Mot Ctrl ENET 1.5M – 2.5M

2013

2014

Gen 1

Gen 2 Proposal

TM

2015

Planning

2016 Gen 3

Execution

Production 46

Single Core Dual Core Multi Core

2017

Front Looking and 360° sense radar for performing and system cost optimized solutions Vision Systems with Image cognition on an Open SW Platform Safety without compromising Performance Miniaturization Data BW management - Ethernet

Safety TM

47

TM

48

TM