Visualization at Boeing: Past, Present, Future | GTC 2013 - GPU

Engineering, Operations & Technology

Information Technology

Acquiring Information using Visual Analytics

Visualization: Past, Past Present, and Future at Boeing Dave Kasik Senior Technical Fellow The Boeing Company Dave Kasik and [email protected] Senesac The Boeing Company June, 2011

BOEING is a trademark of Boeing Management Company. Copyright © 2011 Boeing. All rights reserved.

A Bit About Us Engineering, Operations & Technology | Information Technology

 Dave:     

Involved in computer p g graphics p since 1969 Boeing Senior Technical Fellow ACM Distinguished Scientist Stand in on starship bridges Stand-in Known around Boeing as – A curmudgeon about virtual reality in the immersive, stereo sense – An A advocate d t ffor augmented t d reality lit – Leading proponent and expert for broad use of visualization for geometric and non-geometric data

 Chris: Ch i    

Involved in computer graphics since 1990 Boeing Senior Architect Specialty - being able to apply technology to real world problems Passion is to simplify complex problems

Copyright © 2012 Boeing. All rights reserved.

2

Why Are We Here? Engineering, Operations & Technology | Information Technology

 Boeing builds astounding aerospace products  We require hugely varied technology, ranging from      

Basic physics to Networking (on-board & conventional) to Computing (real-time systems & traditional) to Material science to Natural language analysis to Basically, you name it, we have it

 Boeing generates terror-bytes terror b tes of data  Has worked with advanced visual analysis techniques to gain more insight from our data


3

Outline Engineering, Operations & Technology | Information Technology

 Motivation  Past: Give a quick Boeing history of projects that changed computer graphics  Present: Stress current 3D use cases in Boeing  Future: F t How H iis computer t graphics hi evolving l i to t address industrial needs?


4

Data Growth: Terabytes  Zetabytes Engineering, Operations & Technology | Information Technology

Moore’s Law: Transistor capacity it doubles d bl every 24 months

Growth of technology

The human brain brain’s s volume doubles only every 3 x 107 months The amount of data stored digitally was estimated at 180 exabytes in 2006. By 2011 there will be at least 1800 exabytes (1.8 zetabytes) Copyright © 2012 Boeing. All rights reserved.

1974

Present

Growth of human capability

A long time ago

Present

5

Why Visualization? Engineering, Operations & Technology | Information Technology

 Broadest communication channel to the human brain  Acquires 80+% of all data communicated to a human

 Brain B i mustt process d data t to t gain i insight i i ht  What can a user do with interactive visualization alone?


6

Visual Task Analysis Engineering, Operations & Technology | Information Technology

 Find an object in a complex scene  Focus F on the th found f d object bj t to t better b tt understand d t d surface f characteristics (e.g., smoothness, roughness)  Once the object is found, look at objects in the immediately surrounding di volume l  Visually scan the scene  Observe dynamics in the entire scene (conventionally by animation)  Work with multiple versions of the same set of objects to compare the two sets  Find anomalies and patterns  More detail in DJ Kasik, “Strategies for Consistent Image P titi i ” IEEE Multimedia, Partitioning”, M lti di Jan-Mar, J M 2004, 2004 pp. 32 32-41 41 Copyright © 2012 Boeing. All rights reserved.

7

The Past: Boeing g and Computer Graphics Engineering, Operations & Technology | Information Technology

           

Computer graphics Human model B-spline surface rendering Fractals User interface management systems Industrial-strength NURBS Algorithms IGES Augmented reality FlyThru/IVT Voxmap PointShell (collision detection) Visual analytics Massive model visualization


8

Computer Graphics Engineering, Operations & Technology | Information Technology

 According to Wikipedia, computer graphics has made computers easier to interact with, and better for understanding and i t interpreting ti many types t off data. d t Developments D l t in i computer t graphics have had a profound impact on many types of media...  The term came from Boeing-Wichita’s Verne Hudson in 1960. Bill Fetter popularized it. it Animation by photography

3D-2D p plotting g Contour plots Copyright © 2012 Boeing. All rights reserved.

9

Human Model Engineering, Operations & Technology | Information Technology

 Fetter developed BOEMAN, the first computer model of a human y in 1964 body  Used a pen plotter to do reach studies for an aircraft carrier


10

Fractal Animation Engineering, Operations & Technology | Information Technology

 Loren Carpenter  Produced, Produced directed film called Vol Libre  Resulted in standing ovation at SIGGRAPH’80

Courtesy: L. Carpenter Copyright © 2012 Boeing. All rights reserved.

11

Augmented Reality Engineering, Operations & Technology | Information Technology

 Term coined by Boeing’s Tom Caudell in 1990  Applications evaluated (Caudell (Caudell, David Mizell) in Boeing proved impractical  Boeing continues to invest in Augmented Reality


12

Massive Model Visualization Engineering, Operations & Technology | Information Technology

 Dave Kasik started investigating ways to visualize entire aerospace products in 2004 in collaboration with organizations around the world  Monograph ‘Real-Time Massive Model Rendering’ (Y (Yoon, ett al.) l ) 2008


13

Observations Engineering, Operations & Technology | Information Technology

 Boeing has a long tradition in computer graphics and interactive techniques  Visualization has specific value for     

Highly complex products P Processing i h huge amounts t off d data t Widest path into brain Complex system integration Complex non-geometric data exploration and analysis tasks

 Boeing pushed the state-of-the-art to understand state-ofthe-art the art products  It takes a long time for state-of-the-art in aerospace and computing technology to become widespread


14

The Present: Computer Graphics Use Cases Engineering, Operations & Technology | Information Technology

 Dozens of use cases, including  Visual Vi l analytics l ti ffor non-geometric ti d data t  3D geometry visualization


15

What Makes Any Visualization Session Interactive? Engineering, Operations & Technology | Information Technology

 Model load time: instantaneous  For groups, “instantaneous” translates to less than one minute  For an individual, five minutes just seems like an eternity unless the person can effectively time share tasks  Reality: the faster the better

 “Flying” time: New transformation matrices that respond to mouse action  Ideally, 16 Hz (the human flicker fusion threshold for video) or faster  Practically, 10 Hz or faster

 Graphical selection. selection Feedback appears in < .25 25 sec


16

Non-Geometric Data: Analytics Overview Engineering, Operations & Technology | Information Technology

 Analysis vs. Analytics  When Wh dealing d li with ith digital di it l data, d t analysis l i is i the th detailed d t il d examination of any size and number of data collections  Analytics is the science behind analysis  In practice, analytics requires understanding the     

Cognitive processes Data acquisition Tools Techniques & methods R Results lt communication i ti patterns tt

that let people obtain an optimal or reasonable decision based on existing data.


17

Gain Insight through Data Analysis Engineering, Operations & Technology | Information Technology

 Situation awareness  Command and control

 Tracking and visibility  Determine status

 Causal chain analysis  Determine why something happened

 Hypothesis testing  Explore possible explanations

 Detecting D i anomalies li and d correlations l i

Well suited for advanced visual analytics tools that allow interactive exploration and assessment of complex data

 Prevent event occurrences

 Prediction/Forecasting  Improve I quantities titi ordered d d

 What-if studies  Explore alternatives

 Summarizing S i i  Communicate results Copyright © 2012 Boeing. All rights reserved.

18

Tools and Techniques Engineering, Operations & Technology | Information Technology

 Numerous tools and techniques             

Statistical analysis y Text and data mining Control rooms Operations p centers Dashboards Alerting Fish-bone Fish bone charts Matrix analysis Neural nets Competing hypotheses Modeling & simulation Pareto analysis …

 Visual analytics is an emerging class of analysis tool Copyright © 2012 Boeing. All rights reserved.

19

Mapping Types of Analysis to Tools TR RUST/ P-M MATCH

An nalyst Nottebook

S Star Ana alytics

CO OGNOS

SAS S Ana alytics

SPSS S

SAS S

Sp potfire

Ta ableau

Ge eotime

Jigsaw

Sa affron

TAC T

Excel E

Type off T Analysis

Sta arlight

Tools

IN-SPIRE

Engineering, Operations & Technology | Information Technology

Situation Awareness Tracking & Visibility Causal Chain Analysis Hypothesis Testing Detecting Anomalies & Correlations Prediction & Forecasting What-If Studies Summarizing Copyright © 2012 Boeing. All rights reserved.

20

Visual Analytics Engineering, Operations & Technology | Information Technology

Formal definition: “The science of analytical reasoning facilitated by interactive visual i t f interfaces.” ” Translation: Rapidly explore large, complex gain new business insight g using g datasets to g interactive visualization.

Detect the expected and discover the unexpected

References:  Illuminating the Path, nvac.pnl.gov (free download), 2005  Special Issue: Foundations and Frontiers of Visual Analytics, Information Visualization, vol. 8, no. 4, Winter 2009 Copyright © 2012 Boeing. All rights reserved.

21

Bird Strike Analysis Engineering, Operations & Technology | Information Technology

 The threat by the numbers:  Approximately pp y 1 bird strike per p 2,000 , flights g  Around 20 strikes per day on jet transport category aircraft alone  About 1 in 10 strikes are damaging  The reported costs average $123 million per year  However, 80% of strikes go unreported, and the true cost could be as high as $615 million per year

 Other factors:    

Increasing traffic Bigger, quieter engines Twin-engine configurations Increasing bird populations


22

Example Engineering, Operations & Technology | Information Technology

 767-300, 4/2/01, Paris, during climb, 12,000 ft, 250 knots, multiple p ducks


23

Time and Location Engineering, Operations & Technology | Information Technology


24

Factory Safety Analysis Engineering, Operations & Technology | Information Technology

 Look at data to understand impact of repetitive stress injuries, including cost.  777 Body Structures investigated historical data to understand how it aligned with injury data.  Pilot study done with Excel for 777 Forward Body Structures. Structures  Analysis based on three years of historical production data, and tested the limitations of Excel.

 Bird Bi d strike t ik project j t showed h d potential t ti l for f visual i l analytics. l ti


25

Types of Changes Engineering, Operations & Technology | Information Technology

Address back problems

Address multiple problems Copyright © 2012 Boeing. All rights reserved.

26

Geometric Data: Visualization Data Scope Engineering, Operations & Technology | Information Technology

Product Structure

General Info (e.g., GD&T)

Geometry


27

High-end vs Low-end Engineering, Operations & Technology | Information Technology

Use Cases Design Review/ Training/ Eng. & E&M/ Tracing systems 3D Scans/ Sales & Mktg Safety & Survivability Quality Assy & /Maint Instructions Part Dim & Tol/ Part Context P t Catalog Parts C t l Design in Context

1,000

100,000

1,000,000

Number of Parts (log scale) Copyright © 2012 Boeing. All rights reserved.

28

Standard Visualization Applications Engineering, Operations & Technology | Information Technology

C-17? or new 747 787

777

P-8A IVT/SuperViewer

F-18 F 18

essentially unlimited Satellite UG VisMockUp IVT/Standard Edition

Parts in Context

Typical Part

Adobe Reader Browsers UG JT2go

< 100,000 parts

< 1,000 parts

10 Million

100 Million

1 Billion

Polygon-Equivalents (log scale) Copyright © 2012 Boeing. All rights reserved.

29

Low-end Visualization Engineering, Operations & Technology | Information Technology

 A Low End Visualization capability requires solutions to several related problems:  A place to host the visualization data or the ability to generate appropriate data on the fly.  A low cost mechanism with which a supplier or customer can navigate to and retrieve the model they want to view. view  Low cost visualization tool(s).  Identification of a suitable data format.


30

Parts Information Management System Engineering, Operations & Technology | Information Technology

Description: The Parts Information Management System (PIMS) is a web based application designed to deliver 787 Dreamliner customized parts information to our customers in 3D, via the web, to anywhere in the world. PIMS is the next generation implementation of the Illustrated P t C Parts Catalog. t l PIMS will ill be b used d by b airline i li Owners, Operators, Maintenance users world wide. Data

Viewer Functions

 Source data from V5

 3D Navigation

 Delivered Anywhere via the WWW.

 User Text Search

 Surface Definition Only

PIMS View of 787 Data

 Integrated text and graphic  Use of Transparency ( f (reference parts) t )  Animation (parts explosion)

Implementation Architecture  PIMS will be available on board aircraft  Multiple customer hardware & operating systems  Requires Browser plug-in from Lattice Technology  Maintenance Level Views authored using IVT & Delmia


31

787 Engineering Information Delivery (EID) Engineering, Operations & Technology | Information Technology Description: The 787 program uses 3D Model-Based – Definition (MBD) processes to define their engineering type design information. information Suppliers and customers require the distribution, query, display and retrieval of 3D MBD engineering type design data for downstream applications like manufacturing, inspection and maintenance engineering. e g ee g EID supports suppo ts tthe e que query y and a d retrieval et e a o of authoritative 3D MBD in CATIA native format, STEP AP203 and Universal 3D (U3D) PDF formats.

Data  CATIA V5  Product structure  Engineering Authority (Exact Geometry and Topology)  GD&T and notes  Open standards Implementation Architecture

Viewer Functions  Geometry/Product structure associativity  Retrieve product structure  Transparency/color  Measurement M t  Section Cuts

 Multiple hardware & operating systems  Adobe Reader (plug-in or standalone) for viewing and interrogation g of the data,, Right g Hemisphere p for translation  Multiple browsers  REDARS + CDW for geometry + product structure Copyright © 2012 Boeing. All rights reserved.

32

Shop Floor 3D Work Instructions Engineering, Operations & Technology | Information Technology

Description: The Shop floor 3D Work Instruction system is designed to deliver work instructions, instructions via the web web, to any shop floor user. The shop floor user can interactively rotate, dynamically zoom and interrogate geometry and dimensions.

Data

Production View

Viewer Functions

 UGNX, PROE, CATIA V4, limited CATIA V5

 Product structure search for part numbers  Material (e.g., transparency,  PLM/XML (Product structure) material)  Animation (parts explosion)  Surface Definition, Definition BB REP data  Canned views defined by the Manufacturing Engineer  Work instructions  Dimensioning  Access to unique instance Implementation Architecture attributes  Potential to run when disconnected from a network?  Multiple hardware & operating systems  Multiple browsers via JAVA applet, stand-alone application based on UG Vis tools Copyright © 2012 Boeing. All rights reserved.

33

High-end Visualization Engineering, Operations & Technology | Information Technology

 Where performance and capacity are critical  ‘Instant’ ‘ load time (less ( than 1 minute))  Performance to interactively manipulate up to 1 billion polygons at 10Hz or faster  Product structure with 200,000 separate parts and 2 million instances  Selection and feedback in .25 seconds or less


34

Potential Applications Engineering, Operations & Technology | Information Technology

   

Design in context Design reviews/error checking Tracing systems Engineering and manufacturing analysis  Mechanisms, weights, manufacturing simulation, etc.

   

Electro-magnetic g analysis y Safety Survivability Complete part context  Quality Q lit inspection i ti  Assembly instructions

       

Massive 3D scans and points Part dimensions and tolerances Airplane-in-a-box (totally disconnected) Human model (large model done in ICIDO) Part catalogs Training and familiarization Maintenance instructions Sales and marketing (as animation)


35

787 Structure Engineering, Operations & Technology | Information Technology


36

787 Detail Engineering, Operations & Technology | Information Technology


37

787 in Flight Engineering, Operations & Technology | Information Technology


38

787 Product Data Design Review Engineering, Operations & Technology | Information Technology

Description: Many 787 groups, including many outside of engineering rely on IVT to look at collections of parts that range from a few to an entire tail number. In addition to looking at the raw geometry, users can for queries based on Enovia attributes to select parts that have certain characteristics.

Data

Viewer Functions

 CATIA V5

 Partial / whole aircraft

 IVT .ft data

 Product structure from the PDM

 Enovia Product Structure  Enovia E i metadata t d t  No PMI Implementation Architecture  Windows  IVT


 Metadata query G Geometry t and d Attribute Att ib t Viewing  Measurement  Transparency  Stored Views  Animated A i t d sequences

39

787 Ergonomics Analysis Engineering, Operations & Technology | Information Technology Description: Use the ic.ido immersive review workstation and its human modeling software. The immersive workstation features a passive stereo display and interaction system to enhance the virtual 3D effect. The interactive interface is implemented with an ART hand and head tracking system. Specific 787 product usage has focused on visual verification of human and systems interfaces in highly integrated volumes volumes. Additional support has occurred for design reviews, trade studies, and partner collaboration.

Data  Exports from CATIA V5 and IVT in VRML format  MPT managed, simplified product structure  Visual Accuracy  No PMI

Viewer Functions      

Kinematic replay Integrated human model Collision avoidance I t f Interference detection d t ti Flex properties Measurement

Implementation Architecture  Microsoft Windows running a standalone application  Specialized display hardware (3D passive stereo powerwall)  ART hand and head tracking, motion capture, and haptics for input  ic.ido supplied application software Copyright © 2012 Boeing. All rights reserved.

40

787 Airplane-on-Ground (AOG) Engineering, Operations & Technology | Information Technology Description: AOG teams provide critical repair service around the world. Prior to the 787, teams relied on crates of engineering drawings to get engineering detail about damaged areas and support the formal repair plan. A combination of EID and IVT/SuperViewer is replacing the crates. The result is standalone laptop with the full digital definition of a spec specific c 787 8 tail ta and a d all a tthe e EID files es for o that t at tail. ta

Data

672Z11620-1 --G

Viewer Functions

 CATIA V5  Product structure  Engineering Authority (Exact Geometry and Topology)  GD&T and notes  IVT/SuperViewer database

 Geometry/Product structure associativity  Retrieve product structure  Transparency/color  Measurement  Section Cuts H Hyperlink li k between b t IVT and EID data

Implementation Architecture  Must run disconnected from the network  Adobe Reader (plug-in or standalone) for viewing and interrogation of the data  IVT with a SuperViewer license  Data resides on USB-connected drive and local hard drive Copyright © 2012 Boeing. All rights reserved.

41

DEMO Engineering, Operations & Technology | Information Technology


42

The Future: Riding g Waves Engineering, Operations & Technology | Information Technology

 Computer graphics has been riding waves for 50+ years  Wave 1: CAD/CAM (1962 - ~1990)  Wave 2: Arts and Entertainment (~1985 – present)  Wave 3: Doesn’t seem to have started yet. Copyright © 2012 Boeing. All rights reserved.

43

Computer Graphics Drivers Engineering, Operations & Technology | Information Technology

 Possible technology drivers for a third wave:  Display technology beyond pixels – Graphics started with fast phosphors and lines and moved to colored dots

 Primitives that are higher order than geometry – Better to support visualizing non-geometric data

 Emphasis on graphics systems – Ultimate solutions involve complete systems (input, output, applications, data)


44

The Third Wave, a Boeing-Systems View Engineering, Operations & Technology | Information Technology


45

The Third Wave, a Boeing-User’s View Engineering, Operations & Technology | Information Technology


46

Environmental Trends We See Engineering, Operations & Technology | Information Technology

 Geometric data:  Boeing has designed in 3D since late 1980’s  Assembly/manufacturing has moved to 3D – While 777 authority data 2D drawings, 3D interference and engineering analysis had profound impact on accurate, no-shim build – 787 authority data annotated 3D and formatted as 3Dpdf

 Support mixes 2D and 3D – 777 relies on 2D documents – 787 mixes 2D and 3D

 Caveats – Design must pay attention to downstream use – Using g 3D data as authority y has implications p for long-term g data retention ((LOTAR))

 Non-geometric data:  Data has numerous instances that must be integrated – Easy data access a challenge

 Visual vocabulary is still emerging  Text analysis difficult Copyright © 2012 Boeing. All rights reserved.

47

Tech Trends We See Engineering, Operations & Technology | Information Technology

 Immersive virtual reality  Flight simulators are a way of life  Installed a few early CAVEs for training and design reviews – Can’t close business case

 Few large screen installations (video walls, etc.)  Most prefer working at desk  Staff moves frequently and is globally dispersed

 Stereo and immersive trying to penetrate marketing, manufacturing ergonomics and modeling & simulation ergonomics,  Boeing focus has been on 3D capabilities that have widespread usage  Internally developed 3D viewer has 15,000 registered users  787 ‘engineering engineering drawings drawings’ are 3D; delivered with Adobe Reader Reader. Overall user community will exceed 100,000

 Input method gap across device classes is widening  WIMP vs. multi-touch vs. tangibles vs. voice vs. …

 Mobile devices showing up everywhere  Augmented reality and analytics are gaining momentum Copyright © 2012 Boeing. All rights reserved.

48

Technical Challenges Engineering, Operations & Technology | Information Technology

 While cost is coming down, it can always get lower.  Volume V l off data d t just j t keeps k getting tti larger. l  Input techniques across different screen sizes are asymmetric  For small- and mid-sized screens, multi-touch and WIMP have different vocabulary – Affects both software and users – Causes problems with both thick and thin client

 For large screens (immersive or not) – – – –

Wear cumbersome hardware (e.g., wands, gloves, HMDs) Graphically select an object or a command Enter text or numbers Workaround is for a dedicated operator

 Stereo for long sessions  Stereo sensitivity can be an issue Copyright © 2012 Boeing. All rights reserved.

49

Top Challenge Engineering, Operations & Technology | Information Technology

Plastics? Pl ti ? Interaction!


50

Summary Engineering, Operations & Technology | Information Technology

 Develop a better understanding of analysis using visualization  Significant opportunities to expand visualization technology in industry  Moving advanced technology into practice is a contact sport


51