Appearance Capture and Modeling - Computer Graphics Bonn

Appearance Capture and Modeling Michael Weinmann, Dennis den Brok, Stefan Krumpen and Reinhard Klein University of Bonn, Germany

Sponsored by

Motivation

    

SA2015.SIGGRAPH.ORG

Cultural Heritage Visual Prototyping Advertisement Entertainment …

Sponsored by

Motivation

Product advertisment

Cultural Heritage

Food „photography“

Paleontology

SA2015.SIGGRAPH.ORG

Sponsored by

Motivation

Accurate reproduction Product advertisment Cultural Heritage of characteristic „look“ and „feel“

Food „photography“

SA2015.SIGGRAPH.ORG

Paleontology

Sponsored by

Example “Moulages”

Moulage collection at the Department of Dermatology of the University of Bonn Photographs by Beatrice Bieber

3



3



Moulage “Psoriasis of the nails and the hand” 3



Moulage “Psoriasis of the nails and the hand” 3

Motivation Acquisition and faithful reproduction of appearance of 3D objects

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by

Complexity of Visual Material Appearance Material appearance reveals details!

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by

Motivation Pipeline

Acquisition

SA2015.SIGGRAPH.ORG

Modeling

Transmission and Rendering

Sponsored by

Course Outline

1) Introduction

2) Preliminaries of Material Appearance 3) Advances in Appearance Capture

4) Advances in Appearance Modeling 5) Advances in Transmission and Rendering 6) Applications, Novel Trends and Conclusions

SA2015.SIGGRAPH.ORG

Sponsored by

Preliminaries of Material Appearance

Sponsored by

What is Material Appearance? The visual impression of a material on a human observer

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by

Appearance Depends on … Shape Pose Scale

Geometry on different scales

Optical material properties Color Texture Reflectance behavior

Different light direction reveals different details

Illumination Different spectral composition (daylight, different house lamps, etc.)

Observer Focus Adaption to brightness Color constancy (discount chromatic bias of illumination) …

SA2015.SIGGRAPH.ORG

https://www.wikipedia.org/

http://www.wired.com

Sponsored by

Pose Scale

Geometry on different scales

Optical material properties Color Texture Reflectance behavior

Different light direction reveals different details

Illumination Different spectral composition (daylight, different house lamps, etc.)

Observer Focus Adaption to brightness Color constancy (discount chromatic bias of illumination) … SA2015.SIGGRAPH.ORG

https://www.wikipedia.org/

http://www.wired.com

Sponsored by

Object independend!

Shape

Object dependend!

Appearance Depends on …

Complexity of Visual Material Appearance Main aspect: surface appearance Different light/view  different look

Ganesha figurine made from labradorite SA2015.SIGGRAPH.ORG

Sponsored by

Why is it so difficult to model material appearance? Complex mesogeometry (rough textures) Subsurface scattering Varying microscopic reflection parameters View- and light dependent shadows Occlusions Local/global illumination effects

SA2015.SIGGRAPH.ORG

Sponsored by

How to measure faithfulness? Image comparison

varying illuminations

Material +Shape + varying poses Reconstruction

d (orig , copy ) : arg max d image (imageorig , imagecopy ) SA2015.SIGGRAPH.ORG

poses ,illum.

Sponsored by

Complexity of Visual Material Appearance

SA2015.SIGGRAPH.ORG

Sponsored by

Properties of Light

ray direction

ray origin 𝒓: ℝ+

ℝ3

s ↦ 𝒐 + 𝑠 𝒅,

SA2015.SIGGRAPH.ORG

𝒔єℝ

http://www.exo.net/~pauld/summer_institute/summer_day8polarization/day8_pol arization.html

Sponsored by

Properties of Light

ray direction


ℝ3

s ↦ 𝒐 + 𝑠 𝒅,

𝒔єℝ

Monochromatic light vs. polychromatic light

SA2015.SIGGRAPH.ORG


Sponsored by

Properties of Light

ray direction


ℝ3

s ↦ 𝒐 + 𝑠 𝒅,

𝒔єℝ

Monochromatic light vs. polychromatic light Polarized light vs. non-polarized light SA2015.SIGGRAPH.ORG


Sponsored by

Light Interaction at Surfaces

SA2015.SIGGRAPH.ORG

Sponsored by

Object Appearance Impression of reflection of incident light Influenced by features on different scales

SA2015.SIGGRAPH.ORG

Sponsored by

Object Appearance Impression of reflection of incident light Influenced by features on different scales Macroscopic

SA2015.SIGGRAPH.ORG

Sponsored by

Object Appearance Impression of reflection of incident light Influenced by features on different scales Macroscopic Mesoscopic

SA2015.SIGGRAPH.ORG

Sponsored by

Object Appearance Impression of reflection of incident light Influenced by features on different scales Macroscopic Mesoscopic Microscopic

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


Viewpoint and illumination dependent

SA2015.SIGGRAPH.ORG

Sponsored by

Mesoscopic Effects

Self-Occlusion

SA2015.SIGGRAPH.ORG

Sponsored by

Mesoscopic Effects

Self-Occlusion

SA2015.SIGGRAPH.ORG

Sponsored by

Mesoscopic Effects

Self-Occlusion

SA2015.SIGGRAPH.ORG

Self-Masking

Sponsored by

Mesoscopic Effects

Self-Occlusion

SA2015.SIGGRAPH.ORG

Self-Masking

Sponsored by

Mesoscopic Effects

Self-Occlusion

SA2015.SIGGRAPH.ORG

Self-Masking

Interreflection

Sponsored by

Mesoscopic Effects

Self-Occlusion

Self-Masking

Interreflection

Local Subsurface Scattering SA2015.SIGGRAPH.ORG

Sponsored by

Mesoscopic Effects

Self-Occlusion

Self-Masking

Interreflection

Local Subsurface Scattering SA2015.SIGGRAPH.ORG

Sponsored by

A Taxonomy of Surface Classes

[Ihrke et al. 2008]

SA2015.SIGGRAPH.ORG

Sponsored by

A Taxonomy of Surface Classes

[Ihrke et al. 2008]

SA2015.SIGGRAPH.ORG

Sponsored by

Surface Classes

Diffuse Reflectance

Glossy Reflectance

Specular Reflectance

𝐿diffuse = 𝐿𝑖 𝑘diffuse cos 𝜃

𝐿diffuse = 𝐿diffuse + 𝐿specular

𝒅o,ideal = 2 𝒏 𝒏 𝒅𝒊 − 𝒅𝑖

= 𝐿𝑖 𝑘diffuse 𝒏 𝒅𝑖 𝒏

𝒏

𝒏

𝒅𝑜,ideal

𝒅𝑜,ideal

𝜃

SA2015.SIGGRAPH.ORG

𝜃 𝜃

𝜃 𝜃

Sponsored by

Surface Classes

Surfaces with Subsurface Scattering

Refractive Surfaces

𝒏

η1 sin 𝜃1 = η2 sin 𝜃2

𝜃1 η1 η2

𝜃2

SA2015.SIGGRAPH.ORG

Sponsored by

Reflectance Fields Consider reflectance field

𝑅(𝑥𝑖 , 𝜔𝑖 ; 𝑥𝑜 , 𝜔𝑜 ) parametrized on the sphere that transfers incident light field 𝐿𝑖 𝑥𝑖 , 𝜔𝑖

to corresponding outgoing light field 𝐿𝑜 𝑥o , 𝜔o

Note that the scene might be arbitrarily complex

SA2015.SIGGRAPH.ORG

Sponsored by

Reflectance Fields Light transport is linear: 𝐿𝑜,𝐿𝑖1 +𝐿𝑖2 𝑥𝑜 , 𝜔𝑜 = 𝐿𝑜,𝐿𝑖1 𝑥𝑜 , 𝜔𝑜 + 𝐿𝑜,𝐿𝑖2 𝑥𝑜 , 𝜔𝑜 Image-based relighting equation 𝐿𝑜

𝑥𝑜 ,𝜔𝑜

= 𝑆 Ω𝑖 𝑥 𝑖

𝑅 𝑥𝑖 , 𝜔𝑖 ; 𝑥𝑜 , 𝜔𝑜 𝐿𝑖

𝑥𝑖 ,𝜔𝑖

𝑑𝜔𝑖 𝑑𝑥𝑖

eight dimensional reflectance field

SA2015.SIGGRAPH.ORG

Sponsored by

Commonly Used Reflectance Models

general function (12D)

ρ(𝑥𝑖 , 𝑦𝑖 , 𝜃𝑖 , 𝜑𝑖 , 𝜆𝑖 , 𝑡𝑖 , 𝑥𝑟 , 𝑦𝑟 , 𝜃𝑟 , 𝜑𝑟 , 𝜆𝑟 , 𝑡𝑟 ) fixed wavelength and time

RF / BSSRDF (8D)

ρRF / BSSRDF (𝑥𝑖 , 𝑦𝑖 , 𝜃𝑖 , 𝜑𝑖 , 𝑥𝑟 , 𝑦𝑟 , 𝜃𝑟 , 𝜑𝑟 ) spatially inhomogeneous materials

Spatially inhomogeneous materials

BTF (6D)

SVBRDF (6D)

spatially homogeneous materials

BSSDF (6D)

ρBTF (𝑥, 𝑦, 𝜃𝑖 , 𝜑𝑖 , 𝜃𝑟 , 𝜑𝑟 ) ρSVBRDF (𝑥, 𝑦, 𝜃𝑖 , 𝜑𝑖 , 𝜃𝑟 , 𝜑𝑟 ) ρBSSDF (𝑥𝑟 − 𝑥𝑖 , 𝑦𝑟 − 𝑦𝑖 , 𝜃𝑖 , 𝜑𝑖 , 𝜃𝑟 , 𝜑𝑟 ) fixed position fixed lighting fixed lighting fixed view fixed view fixed position opaque materials

SLF (4D)

ρSLF (𝑥, 𝑦, 𝜃𝑟 , 𝜑𝑟 )

SRF (4D)

ρSRF (𝑥, 𝑦, 𝜃𝑖 , 𝜑𝑖 )

BRDF (4D)

ρBRDF (𝜃𝑖 , 𝜑𝑖 , 𝜃𝑟 , 𝜑𝑟 )

diffuse (nearly) flat

Texture Maps / Bump Maps (2D) ρTexture Map / Bump Map (𝑥, 𝑦)

SA2015.SIGGRAPH.ORG

Sponsored by

Calibration

© xritephoto.com

Sponsored by

Calibration Determine relationships between components f

f

f

f

SA2015.SIGGRAPH.ORG

Sponsored by

Calibration Determine relationships between components Geometric relations f

f

f

f

SA2015.SIGGRAPH.ORG

Sponsored by


f

f

f

SA2015.SIGGRAPH.ORG

Sponsored by


f

f

f

SA2015.SIGGRAPH.ORG

Sponsored by


f

f

f

SA2015.SIGGRAPH.ORG

Sponsored by


f

f

Radiometric relations

SA2015.SIGGRAPH.ORG

f

Sponsored by

Geometric Calibration

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image)

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation Measured sample

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation Measured sample

Turntable

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation Measured sample Camera

Turntable

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation Measured sample Camera Light source

Turntable

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation Measured sample Turntable Center of projection, Camera sensor, view frustum, … Light source Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation Measured sample Turntable Center of projection, Camera sensor, view frustum, … Light source

Required for 3 tasks:

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by



Z

1.

Reconstructing surface geometry

X Y

SA2015.SIGGRAPH.ORG

Sponsored by



Z X

1. 2.

Reconstructing surface geometry Mapping: camera pixels ↦ surface

Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Measurement geometry (of each image) Position & orientation

𝜔𝑜 𝜔𝑖

Measured sample Turntable Center of projection, Camera sensor, view frustum, … Light source


Z X

1. 2. 3.

Reconstructing surface geometry Mapping: camera pixels ↦ surface Sampled directions?

Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Model of projection 15 parameters/camera (r1,r2,r3,t1,t2,t3,c,h1,h2,a,s,k1,k2,p1,p2) Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Model of projection 15 parameters/camera (r1,r2,r3,t1,t2,t3,c,h1,h2,a,s,k1,k2,p1,p2)

Feature in captured image Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Model of projection 15 parameters/camera (r1,r2,r3,t1,t2,t3,c,h1,h2,a,s,k1,k2,p1,p2)

Feature in captured image Prediction of camera model

Z X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Model of projection 15 parameters/camera (r1,r2,r3,t1,t2,t3,c,h1,h2,a,s,k1,k2,p1,p2) Z X

Feature in captured image Prediction of camera model Minimize reprojection error

Y

SA2015.SIGGRAPH.ORG

Sponsored by

Camera Calibration Model of projection 15 parameters/camera (r1,r2,r3,t1,t2,t3,c,h1,h2,a,s,k1,k2,p1,p2) Z X Y

SA2015.SIGGRAPH.ORG

Feature in captured image Prediction of camera model Minimize reprojection error Utilize standard solutions: Zhang’s algorithm [Zha00] Bundle Adjustment [LA09]

Sponsored by

Camera Calibration Cam1 0°

… Cam6 0°

Cam6 15°

Cam6 60°

Cam1 1 60°

… SA2015.SIGGRAPH.ORG

26 Sponsored by


1. Identify markers [MS13]

… Cam6 0°

Cam6 15°

Cam6 60°

Cam1 1 60°


27 Sponsored by


1. Identify markers [MS13] 2. Zhang’s algorithm

… Cam6 0°

Cam6 15°

Cam6 60°

Cam1 1 60°


28 Sponsored by


1. Identify markers [MS13] 2. Zhang’s algorithm “single camera sees target from different angles”

… Cam6 0°

Cam6 15°

Cam6 60°

Cam1 1 60°


29 Sponsored by



…

“single camera sees target from different angles”  board is rotated in front of camera

Cam6 0°

Cam6 15°

Cam6 60°

Cam1 1 60°


30 Sponsored by



… Cam6 0°

Cam6 15°


3. Bundle Adjustment

Cam6 60°

Cam1 1 60°


31 Sponsored by



… Cam6 0°

Cam6 15°


3. Bundle Adjustment “many cameras see a common static scene”

Cam6 60°

Cam1 1 60°


32 Sponsored by



… Cam6 0°

Cam6 15°


3. Bundle Adjustment “many cameras see a common static scene”

Cam6 60°

Cam1 1 60°


33 Sponsored by



… Cam6 0°

Cam6 15°


3. Bundle Adjustment “many cameras see a common static scene”  no rotations  forget relation between rotated markers Cam6 60°

Cam1 1 60°


34 Sponsored by

Turntable Calibration Z

Model of turntable X Y

SA2015.SIGGRAPH.ORG

Sponsored by


Model of turntable Axis of rotation Center of rotation

X Y

SA2015.SIGGRAPH.ORG

Sponsored by

Turntable Calibration Z 30°


Some 3D point

SA2015.SIGGRAPH.ORG

X Y

Sponsored by

Turntable Calibration Model of turntable

Z 270°

Axis of rotation Center of rotation

30°

X Y

Some 3D point Different rotations

30° 270° SA2015.SIGGRAPH.ORG

Sponsored by



X 0° Y

Some 3D point Different rotations Prediction of 0° pose


Sponsored by



X 0° Y

Some 3D point Different rotations Prediction of 0° pose Minimize deviation


Sponsored by

Turntable Calibration

300°

300°

SA2015.SIGGRAPH.ORG

Sponsored by


350° 340° 330° 320° 310° 300°

10° 0°

50° 60° 20° 30° 40°

60° 50° 40° 30°

SA2015.SIGGRAPH.ORG

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by


350° 340° 330° 320° Z 310° 300°

10° 0°

50° 60° 20° 30° 40° h1

Y

X

h2

SA2015.SIGGRAPH.ORG

60° 50° 40° 30°

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by


Z

10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

60° 50° 40° 30° SA2015.SIGGRAPH.ORG

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by


Fit plane to all points (PCA) Z

10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

60° 50° 40° 30°

SA2015.SIGGRAPH.ORG

45

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by


Fit plane to all points (PCA)  normal Z

10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

90°

60° 50° 40° 30°

SA2015.SIGGRAPH.ORG

46

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by



10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

90°

60° 50° 40° 30°

SA2015.SIGGRAPH.ORG

47

Axis of rotation

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by



10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

?

Axis of rotation

90°

Center of rotation 60° 50° 40° 30°

SA2015.SIGGRAPH.ORG

48

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by



10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

?

Axis of rotation

90°

Center of rotation Linear Least Squares; closed form solution SA2015.SIGGRAPH.ORG

49

60° 50° 40° 30°

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by



10° 0° 350° Y 340° 330° 320° 310° X 300°

50° 60° 20° 30° 40°

60° 50° 40° 30°

SA2015.SIGGRAPH.ORG

50

20°

0° 10°

300° 310° 320° 330° 340° 350°

Sponsored by

Light Source Calibration

Cam7 15°

SA2015.SIGGRAPH.ORG

Sponsored by

Light Source Calibration Cam1 0°

Cam6 15°

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by


1. Identify LED reflections

Cam6 15°

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by


1. Identify LED reflections 2. Raytracing

Cam6 15°

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by



Cam6 15°

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by



Cam6 15°

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by


1. Identify LED reflections 2. Raytracing 3. Triangulation [HZ04]

Cam6 15°

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by


Cam6 15°

1. Identify LED reflections 2. Raytracing 3. Triangulation [HZ04] 4. Nonlinear optimization [Lev44]

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by


Cam6 15°

1. Identify LED reflections 2. Raytracing 3. Triangulation [HZ04] 4. Nonlinear optimization [Lev44] LED- & ball-positions

Cam11 Cam7 60° 15°

SA2015.SIGGRAPH.ORG

Sponsored by

Calibration Results

SA2015.SIGGRAPH.ORG

Sponsored by

Calibration Results Camera accuracy:

detected

prediction

SA2015.SIGGRAPH.ORG

Sponsored by


detected

0.16 pixels reprojection error prediction

SA2015.SIGGRAPH.ORG

Sponsored by


detected

0.16 pixels reprojection error prediction

SA2015.SIGGRAPH.ORG

Sponsored by

Calibration Results Camera accuracy: 0.16 pixels reprojection error  0.001° direction error

SA2015.SIGGRAPH.ORG

detected

prediction

Sponsored by

Calibration Results detected

Camera accuracy: 0.16 pixels reprojection error  0.001° direction error

prediction

Turntable accuracy: 0.003° pose deviation

0°

0°

SA2015.SIGGRAPH.ORG

Sponsored by



prediction


0°

LED accuracy: 0.4 pixels reprojection error

0°

detected

SA2015.SIGGRAPH.ORG

prediction Sponsored by



prediction


0°

LED accuracy: 0.4 pixels reprojection error  0.08° direction error

0°

detected

SA2015.SIGGRAPH.ORG

prediction Sponsored by

Radiometric Calibration

SA2015.SIGGRAPH.ORG

Sponsored by

Radiometric Calibration Determine the relation between scene and image brightness before light arrives at the image plane: Scene

Scenenradiance L

Lens

Imageirradiance E

Linear relationship!

after:

Imageirradiance E

Camera electronics

measure pixel values I

Nonlinear Relationship! SA2015.SIGGRAPH.ORG

Sponsored by

Radiometric Calibration Determine the relation between scene and image brightness before light arrives at the image plane: Scene

Scenenradiance L

Lens

Imageirradiance E

Linear relationship!

after:

Imageirradiance E

Camera electronics

measure pixel values I

Nonlinear Relationship! SA2015.SIGGRAPH.ORG

Sponsored by photometric calibration

Radiometric Calibration (225,203,216)

Camera Response Curve (OECF): Describes relationship between RGB-values and corresponding luminance and is a-priori unknown (and often non-linear) Gamma correction Image (color) optimizations

(141,25,4)

(40,70,143) What do these RGB values mean?

Direct measurement via test chart Patches with known gray levels Uniform illumination © xritephoto.com

Inversion using OECF leads to pixel values linearly related to luminance values SA2015.SIGGRAPH.ORG

Sponsored by

Radiometric Correction

et = 50ms

SA2015.SIGGRAPH.ORG

Sponsored by

Radiometric Correction 1. Removal of the effect of hot pixels or sensor bias (dark frame subtraction)

et = 50ms

SA2015.SIGGRAPH.ORG

Sponsored by

Radiometric Correction 1. Removal of the effect of hot pixels or sensor bias (dark frame subtraction) 2. Inversion of camera response to obtain energy values from the observed pixel values et = 50ms

f

SA2015.SIGGRAPH.ORG

Sponsored by

Radiometric Correction 1. Removal of the effect of hot pixels or sensor bias (dark frame subtraction) 2. Inversion of camera response to obtain energy values from the observed pixel values et = 500ms

et = 50ms

f

SA2015.SIGGRAPH.ORG

f

Sponsored by

Radiometric Correction 1. Removal of the effect of hot pixels or sensor bias (dark frame subtraction) 2. Inversion of camera response to obtain energy values from the observed pixel values © xritephoto.com

et = 500ms

et = 50ms

f

SA2015.SIGGRAPH.ORG

f

Sponsored by

Radiometric Correction 1. Removal of the effect of hot pixels or sensor bias (dark frame subtraction) 2. Inversion of camera response to obtain energy values from the observed pixel values © xritephoto.com

et = 500ms

et = 50ms

f

f

[sr-1] SA2015.SIGGRAPH.ORG

Sponsored by

Advances in Reflectance Acquisition

Sponsored by

Motivation Both geometry and reflectance have to be acquired

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by

Geometry Acquisition Shape-from-Silhouette

object

SA2015.SIGGRAPH.ORG

Sponsored by


object

SA2015.SIGGRAPH.ORG

Sponsored by


object

SA2015.SIGGRAPH.ORG

Sponsored by


object

SA2015.SIGGRAPH.ORG

Sponsored by


object

visual hull

SA2015.SIGGRAPH.ORG

Sponsored by

Geometry Acquisition Shape-from-Silhouette Coarse geometry reconstruction Misleading shape (blur due to misalignment) Visual Hull [Mueller et al. 2005]

SA2015.SIGGRAPH.ORG

Sponsored by

Geometry Acquisition Shape-from-Silhouette Coarse geometry reconstruction Misleading shape (blur due to misalignment)

Active Triangulation Laser scanning Structured light

SA2015.SIGGRAPH.ORG

Visual Hull [Mueller et al. 2005]

Sponsored by

Geometry Acquisition Shape-from-Silhouette Coarse geometry reconstruction Misleading shape (blur due to misalignment) Visual Hull [Mueller et al. 2005]

Active Triangulation Laser scanning Structured light Highly accurate

Laser Scan

SA2015.SIGGRAPH.ORG

Structured Light Sponsored by [Weinmann et al. 2011]

Geometry Acquisition

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


triangulation [HZ04]

SA2015.SIGGRAPH.ORG

Sponsored by


triangulation [HZ04]

5‰ less accurate

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


0°

SA2015.SIGGRAPH.ORG

45°

90°

Sponsored by


0°

SA2015.SIGGRAPH.ORG

45°

90°

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

uncalibrated turntable

Sponsored by


SA2015.SIGGRAPH.ORG

calibrated turntable


Sponsored by


SA2015.SIGGRAPH.ORG

calibrated turntable


Sponsored by

Geoemtry Acquisition Also possible for flat material samples ...

SA2015.SIGGRAPH.ORG

Sponsored by

A short reminder … Complexity of material appearance

SA2015.SIGGRAPH.ORG

Sponsored by

A short reminder … Complexity of material appearance Multitude of reflectance models

SA2015.SIGGRAPH.ORG

Sponsored by

Reflectance Representation Often: minimalistic model that allows to represent material appearance characteristics

Photo

SA2015.SIGGRAPH.ORG

Texture

Polynomial Texture Map

BTF

Sponsored by


Photo

SA2015.SIGGRAPH.ORG

  

Texture


BTF

Sponsored by


Photo

SA2015.SIGGRAPH.ORG

  

Texture


BTF

Sponsored by

Reflectance Representation Starting point: (known) object geometry, registered images Store reflectance parameterized over the surface

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


Z Y

intersection of viewing ray with 3D geometry + texture mapping

X

SA2015.SIGGRAPH.ORG

Sponsored by


V

Z Y

U

SA2015.SIGGRAPH.ORG


X

Sponsored by


V

Z Y

U

SA2015.SIGGRAPH.ORG


X

Sponsored by


V

Z Y

U

SA2015.SIGGRAPH.ORG


X

Sponsored by


V

Z Y

U

SA2015.SIGGRAPH.ORG


X

Sponsored by

Reflectance Representation Assumption: Known geometry Registered camera images

[Lensch et al. 2001/2003]

SA2015.SIGGRAPH.ORG

Sponsored by


[Lensch et al. 2001/2003]

SA2015.SIGGRAPH.ORG

Sponsored by


For every point:

SA2015.SIGGRAPH.ORG

Sponsored by


For every point: Measurement Tabulated

SA2015.SIGGRAPH.ORG

Sponsored by



In local orientation

SA2015.SIGGRAPH.ORG

Sponsored by



In local orientation Local hemispheres

SA2015.SIGGRAPH.ORG

Sponsored by


For every point: Measurement Tabulated Camera hemispheres In local orientation Local hemispheres

SA2015.SIGGRAPH.ORG

Sponsored by

Appearance Representation

• Gather samples

SA2015.SIGGRAPH.ORG

Sponsored by


• Gather samples

SA2015.SIGGRAPH.ORG

Sponsored by


• Gather samples

SA2015.SIGGRAPH.ORG

Sponsored by


• Gather samples • Not measured

SA2015.SIGGRAPH.ORG

Sponsored by


• Gather samples • Not measured • Irregular

SA2015.SIGGRAPH.ORG

Sponsored by


• Gather samples • Not measured • Irregular • Occlusion SA2015.SIGGRAPH.ORG

Sponsored by


• Gather samples • Not measured • Irregular • Occlusion SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

Texture Representation Texture Map: 𝜌Texture Map 𝒙 no view-/light dependency

SA2015.SIGGRAPH.ORG

Sponsored by


V

U SA2015.SIGGRAPH.ORG

Sponsored by


V


Sponsored by

Texture Representation Texture Map: 𝜌Texture Map 𝒙 no view-/light dependency Dense sampling required! Only per-vertex information might be too coarse

V


Sponsored by

Texture Representation Texture Map: 𝜌Texture Map 𝒙 no view-/light dependency Dense sampling required! Only per-vertex information might be too coarse

Which colors should be stored? V


Sponsored by

Texture Acquisition View selection

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


Single view: Projection of images onto surface geometry Selection of best view based on MRF

SA2015.SIGGRAPH.ORG

Sponsored by


Single view: Projection of images onto surface geometry Selection of best view based on MRF 𝐸 = 𝐸Data + 𝐸Reg

Quality of images: - footprint size of faces in image domain - resolution - gradient magnitudes over footprint - photo-consistency Penalizes seams between patches

SA2015.SIGGRAPH.ORG

Sponsored by


Single view: Projection of images onto surface geometry Selection of best view based on MRF 𝐸 = 𝐸Data + 𝐸Reg

Multiple views: Per-face blending Smoothness of neighboring patches Consideration of discontinuities Inclusion of scale-dependency SA2015.SIGGRAPH.ORG

Quality of images: - footprint size of faces in image domain - resolution - gradient magnitudes over footprint - photo-consistency Penalizes seams between patches

Sponsored by

Texture Acquisition Color adjustment: Trivial for single view acquisition

SA2015.SIGGRAPH.ORG

Sponsored by

Texture Acquisition Color adjustment: Trivial for single view acquisition Multi-view scenarios: Determine single color value per surface point from multiple observations Requires photometric adjustments

SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

BRDF Representation v

BRDF: 𝜌BRDF 𝝎𝑖 , 𝝎𝑟 Suitable for flat, homogeneous materials Various different models: Parametric models Non-parametric models

u 𝜔𝑖 , 𝜔𝑟

Measurements under varying 𝝎𝑖 , 𝝎𝑟

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (1/4) Gonioreflectometer-like setups: Typically 1 camera, 1 light source Sequential for 𝜔𝑖 ,𝜔𝑜

[Hünerhoff et al. 2006]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (1/4) Gonioreflectometer-like setups: Typically 1 camera, 1 light source Sequential for 𝜔𝑖 ,𝜔𝑜 Various flavors:

[Matusik 2003]

Different number of DOFs Robot rotates sample Fixed light, moving camera Fixed camera, robot moves light


Robots move camera and light

[Kim et al. 2008]

[Leloup et al. 2008]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (1/4) Gonioreflectometer-like setups: Typically 1 camera, 1 light source Sequential for 𝜔𝑖 ,𝜔𝑜 Various flavors:

[Matusik 2003]

Different number of DOFs Robot rotates sample Fixed light, moving camera Fixed camera, robot moves light

[Sun et al. 2006]


Robots move camera and light

[Kim et al. 2008]

Parallel measurements [Leloup et al. 2008]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (2/4) Mirror-based setups: 1 camera 1 light source (projector)

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (2/4) Mirror-based setups: 1 camera 1 light source (projector) Clever arrangement of components [Dana et al. 2001/2004]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (2/4) [Ward 1992]

Mirror-based setups: 1 camera 1 light source (projector) Clever arrangement of components

[Mukaigawa et al. 2007/2009]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (2/4) [Ward 1992]

Mirror-based setups: 1 camera 1 light source (projector) Clever arrangement of components

[Mukaigawa et al. 2007/2009]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (3/4) Exploitation of geometric configurations: [Lu and Little, Marschner, Ngan and Durand, etc.]

SA2015.SIGGRAPH.ORG

Sponsored by


Spherical samples

SA2015.SIGGRAPH.ORG

[Marschner 1999/2000]

Sponsored by


Spherical samples

[Marschner 1999/2000] [Marschner 1999/2000]

Cylindrical samples

SA2015.SIGGRAPH.ORG

Sponsored by


Spherical samples

[Marschner 1999/2000] [Marschner 1999/2000]

Cylindrical samples Samples with measured geometry

SA2015.SIGGRAPH.ORG

Sponsored by and Durand 2006] [Ngan

Setup Designs (4/4) Arrays of detectors / light sources

[Dong et al. 2010] [Ben-Ezra et al. 2008]

SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

Representation SVBRDF: 𝜌SVBRDF 𝒙, 𝝎𝑖 , 𝝎𝑟

SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by


SA2015.SIGGRAPH.ORG

Sponsored by

Representation SVBRDF: 𝜌SVBRDF 𝒙, 𝝎𝑖 , 𝝎𝑟 Spatial distribution of mutually independent BRDFs

SA2015.SIGGRAPH.ORG

Sponsored by

Representation SVBRDF: 𝜌SVBRDF 𝒙, 𝝎𝑖 , 𝝎𝑟 Spatial distribution of mutually independent BRDFs Need to be stored on the true geometry

SA2015.SIGGRAPH.ORG

Sponsored by

Representation SVBRDF: 𝜌SVBRDF 𝒙, 𝝎𝑖 , 𝝎𝑟 Spatial distribution of mutually independent BRDFs Need to be stored on the true geometry

Various approaches: Sparse sampling approaches (parametric) Dense sampling approaches (parametric or data-driven)

SA2015.SIGGRAPH.ORG

[Lensch et al. 2005]

Sponsored by

SVBRDF Acquisition Sparse sampling approaches:

SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Acquisition Sparse sampling approaches: Camera at fixed/various [Lensch et al.] viewpoints, moveable light source [Lensch et al. 2001/2003, Goldman et al.]

SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Acquisition Sparse sampling approaches: Camera at fixed/various [Lensch et al.] viewpoints, moveable light source [Lensch et al. 2001/2003, Goldman et al.] Gonioreflectometer-like setups [McAllister 2002]

SA2015.SIGGRAPH.ORG

[McAllister 2002]

Sponsored by

SVBRDF Acquisition Sparse sampling approaches: Camera at fixed/various [Lensch et al.] viewpoints, moveable light source [Lensch et al. 2001/2003, Goldman et al.] Gonioreflectometer-like setups [McAllister 2002] Condenser lens based setups

[McAllister 2002]

[Dong et al. 2010]

[Dong et al. 2010]

SA2015.SIGGRAPH.ORG

Sponsored by


[McAllister 2002]

[Dong et al. 2010]

[Dong et al. 2010]

Arrays of cameras/light sources [Weyrich et al. 2006]

SA2015.SIGGRAPH.ORG

Sponsored by


[McAllister 2002]

[Dong et al. 2010]

[Dong et al. 2010]

Arrays of cameras/light sources [Weyrich et al. 2006]

Light Source

Polarization-based approaches [Ghosh et al. 2009/2010, Zickler et al. 2006]

Polarizer

Camera SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Acquisition Setups for light-weight setups / uncontrolled conditions:

SA2015.SIGGRAPH.ORG

Sponsored by


[Haber et al. 2009]

Collections of internet images [Haber et al. 2009]

SA2015.SIGGRAPH.ORG

Sponsored by


[Haber et al. 2009]


Hand-held video cameras [Palma et al. 2012]

SA2015.SIGGRAPH.ORG

Sponsored by


[Haber et al. 2009]



[Riviere et al. 2014]

Mobile hardware [Riviere et al. 2014, Aittala et al. 2015] Illumination with flash light (extended light source / tablet for specular materials)

SA2015.SIGGRAPH.ORG

Sponsored by


[Haber et al. 2009]



[Riviere et al. 2014]

Mobile hardware [Riviere et al. 2014, Aittala et al. 2015] Illumination with flash light (extended light source / tablet for specular materials) Exploitation of self-similarities in reflectance behavior [Aittala et al. 2015] SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Acquisition Dense sampling approaches:

SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Acquisition Dense sampling approaches: Mirror-based approaches [Dana and Wang 2004]

[Dana et al.]

SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Acquisition [Holroyd2010]

Dense sampling approaches: Mirror-based approaches [Dana and Wang 2004]

Gonioreflectometers

[Dana et al.]

SA2015.SIGGRAPH.ORG

[Marschner et al. 2005, Lawrence et al. 2006, Holroyd et al. 2010]

Sponsored by



Gonioreflectometers

[Dana et al.]


Camera arrays / light source arrays [Köhler et al. 2013, Nöll et al. 2013, Nöll et al. 2015]

[Köhler et al. 2013]

SA2015.SIGGRAPH.ORG

Sponsored by



Gonioreflectometers

[Dana et al.]



Continuous spherical harmonic gradient illumination (LED arc) [Köhler et al. 2013]

SA2015.SIGGRAPH.ORG

[Tunwattanapong et al. 2013]

Sponsored by



Gonioreflectometers

[Dana et al.]



Continuous spherical harmonic gradient illumination (LED arc) [Köhler et al. 2013]

[Tunwattanapong et al. 2013]

Generalized linear light source reflectometry [Chen et al. 2014] SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

Representation Reminder:

Reflectance field: BTF: 𝐵𝑉 𝜔𝑖 ; 𝑥𝑜 , 𝜔𝑜

𝑅𝑉 𝑥𝑖 , 𝜔𝑖 ; 𝑥𝑜 , 𝜔𝑜

[Dana1997]

special reflectance field that transfers

incident directional light field to corresponding

outgoing light field parametrized on an approx. flat surface V

Sample resulting light field 𝐿𝑜,𝑉 via photographs SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (1/3) Gonioreflectometer 1 camera 1 light source Parallel for 𝑥𝑜 Sequential for 𝜔𝑖 ,𝜔𝑜 Various flavors:

SA2015.SIGGRAPH.ORG

105

Sponsored by

Setup Designs (1/3) Gonioreflectometer 1 camera 1 light source Parallel for 𝑥𝑜 Sequential for 𝜔𝑖 ,𝜔𝑜 Various flavors:

[Dana1997]

[Sattler2003]

[Kimachi2006]

Robot rotates sample Fixed light, moving camera [Dana1997], [Sattler2003], [Kimachi2006]

SA2015.SIGGRAPH.ORG

106

Sponsored by

Setup Designs (1/3) [McAllister2000]

Gonioreflectometer 1 camera 1 light source Parallel for 𝑥𝑜 Sequential for 𝜔𝑖 ,𝜔𝑜 Various flavors:

[Dana1997]

[Sattler2003]

[Kimachi2006]

Robot rotates sample Fixed light, moving camera [Dana1997], [Sattler2003], [Kimachi2006] Fixed camera, robot moves light [McAllister2000], [Koudelka2003], [Tsuchida2005]

SA2015.SIGGRAPH.ORG

107

Sponsored by

Setup Designs (1/3) [McAllister2000]

Gonioreflectometer 1 camera 1 light source Parallel for 𝑥𝑜 Sequential for 𝜔𝑖 ,𝜔𝑜 Various flavors:

[Dana1997]

[Sattler2003]

[Kimachi2006]

Robot rotates sample Fixed light, moving camera [Dana1997], [Sattler2003], [Kimachi2006] Fixed camera, robot moves light [McAllister2000], [Koudelka2003], [Tsuchida2005]

[Holroyd2010] [Filip2013]

Robots move camera and light [Holroyd2010], [Filip2013] SA2015.SIGGRAPH.ORG

108

Sponsored by

Setup Designs (2/3) Mirrors / Kaleidoscope 1 camera 1 light source (projector)

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (2/3) [Wang2006]

Mirrors / Kaleidoscope 1 camera 1 light source (projector) Curved mirror

[Dana et al.]

[Wang2006]

Parallel for 𝜔𝑜 Sequential for 𝑥𝑜 , 𝜔𝑖

SA2015.SIGGRAPH.ORG

Sponsored by



[Dana et al.]

[Wang2006]


Piecewise planar

[Tagawa2012]

Parabolic arrangement [Tagawa2012]

SA2015.SIGGRAPH.ORG

Sponsored by



[Dana et al.]

[Wang2006]


Piecewise planar

[Tagawa2012]


Kaleidoscope [Han2003], [Ihrke2012]

[Han2003]

[Ihrke2012]

SA2015.SIGGRAPH.ORG

Sponsored by



[Dana et al.]

[Wang2006]


Piecewise planar

[Tagawa2012]


Kaleidoscope [Han2003], [Ihrke2012]

[Han2003]

No moving parts Parallel for 𝑥𝑜 and 𝜔𝑜 Sequential for 𝜔𝑖 SA2015.SIGGRAPH.ORG

[Ihrke2012]

Sponsored by

Setup Designs (3/3) Camera arrays Multiple cameras Multiple lights (Semi-)parallel for 𝑥𝑜 and 𝜔𝑜 Sequential sampling of 𝜔𝑖

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (3/3) Camera arrays Multiple cameras Multiple lights (Semi-)parallel for 𝑥𝑜 and 𝜔𝑜 Sequential sampling of 𝜔𝑖 Turntable rotates sample

[Furukawa2002] [Matusik2002]

[Tong2005]

[Furukawa2002], [Matusik2002], [Tong2005], [Köhler2013], [Schwartz2013] [Schwartz2013]

[Köhler et al. 2013]

SA2015.SIGGRAPH.ORG

Sponsored by

Setup Designs (3/3) Camera arrays Multiple cameras Multiple lights (Semi-)parallel for 𝑥𝑜 and 𝜔𝑜 Sequential sampling of 𝜔𝑖 Turntable rotates sample

[Furukawa2002] [Matusik2002]

[Tong2005]

[Furukawa2002], [Matusik2002], [Tong2005], [Köhler2013], [Schwartz2013] [Schwartz2013]

No moving parts [Müller2004], [Weyrich2005], [Hu2010], [Wu2011] [Müller2004]

[Köhler et al. 2013] [Hu2010] [Wu2011]

[Weyrich2005]

SA2015.SIGGRAPH.ORG

Sponsored by

Comparison (1/3): Direction Sampling Gonioreflectometers

𝜔𝑖

𝜔𝑜 204 [DVGNK97]

311 / 7,650 [McA02]

6,561 [FVH*13],[SSK03]

Camera arrays

Mirrors 𝜔𝑖

𝜔𝑖

𝜔𝑜

𝜔𝑜 484 [HP03]

SA2015.SIGGRAPH.ORG

1.9×108 [WD06]

568 [KTT06]

10,800 [KMBK03]

2,500 [MTK*10]

4,320 [FKIS02]

12,960 [MPZ*02]

22,801 [MMS*04]

52,272 [SSWK13]

Sponsored by


𝜔𝑖


311 / 7,650 [McA02]

6,561 [FVH*13],[SSK03]

Camera arrays

Mirrors 𝜔𝑖

𝜔𝑖

𝜔𝑜

𝜔𝑜 484 [HP03]

SA2015.SIGGRAPH.ORG

1.9×108 [WD06]

568 [KTT06]

10,800 [KMBK03]

2,500 [MTK*10]

4,320 [FKIS02]

12,960 [MPZ*02]

22,801 [MMS*04]

52,272 [SSWK13]

Sponsored by




𝜔𝑖


311 / 7,650 [McA02]

6,561 [FVH*13],[SSK03]

Camera arrays

Mirrors 𝜔𝑖

𝜔𝑜

  484 [HP03]

SA2015.SIGGRAPH.ORG

1.9×108 [WD06]

568 [KTT06]

10,800 [KMBK03]

2,500 [MTK*10]

𝜔𝑖

 

𝜔𝑜

 

4,320 [FKIS02]

12,960 [MPZ*02]

22,801 [MMS*04]

52,272 [SSWK13]

Sponsored by




𝜔𝑖

𝜔𝑜

 204 [DVGNK97]

311 / 7,650 [McA02]



6,561 [FVH*13],[SSK03]

Camera arrays

Mirrors 𝜔𝑖

𝜔𝑜

  484 [HP03]

SA2015.SIGGRAPH.ORG

1.9×108 [WD06]

568 [KTT06]

10,800 [KMBK03]

2,500 [MTK*10]

𝜔𝑖

 

𝜔𝑜

 

4,320 [FKIS02]

12,960 [MPZ*02]

 22,801 [MMS*04]

52,272 [SSWK13]

Sponsored by




𝜔𝑖

𝜔𝑜

 204 [DVGNK97]

311 / 7,650 [McA02]



6,561 [FVH*13],[SSK03]

Camera arrays

Mirrors 𝜔𝑖

𝜔𝑜

  484 [HP03]

SA2015.SIGGRAPH.ORG

1.9×108 [WD06]

568 [KTT06]

10,800 [KMBK03]

2,500 [MTK*10]

𝜔𝑖

 

𝜔𝑜

 

4,320 [FKIS02]

12,960 [MPZ*02]

 22,801 [MMS*04]

52,272 [SSWK13]

Sponsored by

Comparison (2/3): Spatial Sampling Gonioreflectometers SSK03

McA02

RSK10

DVGNK97

HLZ10

KMBK03

FVH*13

Camera arrays SSWK13

KNRS13

Mirrors

MMS*04

HP03 MTK*10 IRM*12

WD06 SWRK11 SA2015.SIGGRAPH.ORG

122

Sponsored by


McA02

RSK10

DVGNK97

HLZ10

KMBK03

FVH*13


KNRS13

Mirrors

MMS*04

HP03 MTK*10 IRM*12

0.6 x 0.6 cm² WD06 SWRK11

SA2015.SIGGRAPH.ORG

123

Sponsored by


McA02

RSK10

DVGNK97

HLZ10

KMBK03

FVH*13


KNRS13

Mirrors



MMS*04

HP03



IRM*12

MTK*10

0.6 x 0.6 cm²


124

Sponsored by


McA02

  HLZ10

RSK10

DVGNK97 KMBK03

FVH*13


KNRS13

Mirrors



MMS*04



HP03



IRM*12

MTK*10

0.6 x 0.6 cm²




125

Sponsored by


McA02

  HLZ10

RSK10

DVGNK97 KMBK03

FVH*13

Camera arrays

1000 DPI

SSWK13

KNRS13

Mirrors



46 x 46 cm²

MMS*04



HP03



IRM*12

MTK*10

0.6 x 0.6 cm²




126

Sponsored by

Comparison (3/3): Speed Gonioreflectometers

Mirrors

Camera arrays

100 93,5

2.3 years!

90 80 70 60

hours

60 50 40

36

30 18

20

14 10

10

14

13

7

5 1

0,75

1

1,80

0,007

2

0,5

0,95

0

SA2015.SIGGRAPH.ORG

Sponsored by


Mirrors

Camera arrays

100 93,5

2.3 years!

90 80 70 60

hours

60 50 40

36

30 18

20

14 10

10

14

13

7

5 1

0,75

1

1,80

0,007

2

0,5

0,95

0

SA2015.SIGGRAPH.ORG

Sponsored by


Mirrors

Camera arrays

100 93,5

2.3 years!

90 80 70 60

50 40

36

25 seconds

hours

60

30 18

20

14 10

10

14

13 5

1

0,75

1

1,80

0,007

7 2

0,5

0,95

0

SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

Representation BSSRDF: 𝜌BSSRDF 𝒙𝑖 , 𝝎𝒊 , 𝒙𝑟 , 𝝎𝒓 Generalization of spatially varying BRDF 8-dimensional  makes acquisition expensive! Models subsurface scattering

SA2015.SIGGRAPH.ORG

Sponsored by

BSSRDF Acquisition Separation of diffuse and specular components:

SA2015.SIGGRAPH.ORG

Sponsored by

BSSRDF Acquisition Separation of diffuse and specular components: Diffuse/global component: Describes non-local reflection Determined by scattering of light within the material

Global Component SA2015.SIGGRAPH.ORG

Sponsored by


Specular/direct component Describes local reflection, i.e. direct reflection

Global Component SA2015.SIGGRAPH.ORG

Direct Component

Sponsored by


Specular/direct component Describes local reflection, i.e. direct reflection

Approaches based on Dichromatic reflectance models Polarization (e.g. [Wolff and Boult 1991]) High-frequency illumination SA2015.SIGGRAPH.ORG patterns [Nayar et al. 2006]

Global Component

Direct Component

Sponsored by

BSSRDF Acquisition Acquisition of subsurface scattering characteristics: Homogeneous materials: Simple analytical models (single scattering characteristics) + additional dipole models for multiple scattering [Jensen et al. 2001]

Similar model used in [Weyrich et al. 2006]

SA2015.SIGGRAPH.ORG

Sponsored by




Inhomogeneous materials: Use of exponential functions (instead of dipole model) and geometry [Fuchs et al. 2005] SA2015.SIGGRAPH.ORG

Sponsored by




Inhomogeneous materials: Use of exponential functions (instead of dipole model) and geometry [Fuchs et al. 2005] [Jensen et al. 2001] Sponsored by

SA2015.SIGGRAPH.ORG


BSSRDF Acquisition Acquisition of subsurface scattering characteristics: Inhomogeneous materials: Neglection of influence of of 𝝎𝑖 on 𝝎𝒓 [Goesele et al. 2004]  𝜌 𝒙𝑖 , 𝒙𝑟

[Goesele et al. 2004]

SA2015.SIGGRAPH.ORG

Sponsored by



SA2015.SIGGRAPH.ORG

Sponsored by



SA2015.SIGGRAPH.ORG

Sponsored by



SA2015.SIGGRAPH.ORG

Sponsored by



SA2015.SIGGRAPH.ORG

Sponsored by



SA2015.SIGGRAPH.ORG

Sponsored by


Grid of projected points [Peers et al. 2006]


SA2015.SIGGRAPH.ORG

Sponsored by


Grid of projected points [Peers et al. 2006]

Multi-layer models [Donner and Jensen 2005], [Donner et al. 2008], [Ghosh et al. 2008], [Dong et al. 2010]


[Donner et al. 2008]

SA2015.SIGGRAPH.ORG

Sponsored by


Grid of projected points

[Tong et al. 2005]

[Peers et al. 2006]

Multi-layer models [Donner and Jensen 2005], [Donner et al. 2008], [Ghosh et al. 2008], [Dong et al. 2010]

Separate acquisition of global and local scattering [Tong et al. 2005]

SA2015.SIGGRAPH.ORG


[Donner et al. 2008]

Sponsored by

[Tong et al. 2005]

SA2015.SIGGRAPH.ORG

Sponsored by

Representation Reflectance field: 𝑅𝑉 𝑥𝑖 , 𝜔𝑖 ; 𝑥𝑜 , 𝜔𝑜

[Debevec2000]

Appearance of „stuff“ inside V (as a light field 𝐿𝑜,𝑉 𝑥𝑜 , 𝜔𝑜 ) given light from outside V (as a light field 𝐿𝑖,𝑉 𝑥𝑖 , 𝜔𝑖 ) 8-dimensional

SA2015.SIGGRAPH.ORG

Sponsored by

Representation Reflectance field: 𝑅𝑉 𝑥𝑖 , 𝜔𝑖 ; 𝑥𝑜 , 𝜔𝑜

[Debevec2000]

Appearance of „stuff“ inside V (as a light field 𝐿𝑜,𝑉 𝑥𝑜 , 𝜔𝑜 ) given light from outside V (as a light field 𝐿𝑖,𝑉 𝑥𝑖 , 𝜔𝑖 ) 8-dimensional

Related to BSSRDF [Nicodemus1977]

SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

Surface Light Field Acquisition Surface light field: 𝜌SLF 𝒙, 𝝎𝒓 [Wood et al. 2000]

SA2015.SIGGRAPH.ORG

Sponsored by

Surface Light Field Acquisition Surface light field: 𝜌SLF 𝒙, 𝝎𝒓 [Wood et al. 2000]

SA2015.SIGGRAPH.ORG

Sponsored by

Surface Light Field Acquisition Surface light field:

𝜌SLF 𝒙, 𝝎𝒓 Setup designs: Single camera at different viewpoints

[Wood et al. 2000]

[Gortler et al. 1996], [Levoy and Hanrahan 1996], [Wood et al. 2000], Liang et al. 2008], [Palma et al. 2013]

SA2015.SIGGRAPH.ORG

Sponsored by



[Wood et al. 2000] [Wilburn et al. 2005]


Camera arrays [Wilburn et al. 2005]

SA2015.SIGGRAPH.ORG

Sponsored by






SA2015.SIGGRAPH.ORG

Sponsored by






Compressive sensing [Kamal et al. 2012], [Marwah et al. 2013]

SA2015.SIGGRAPH.ORG

Sponsored by

SA2015.SIGGRAPH.ORG

Sponsored by

Reflectance Field Representation Surface reflectance field:

𝜌𝑆RF 𝒙, 𝝎𝑖 Often image-based Does not rely on knowledge about surface geometry Representations: Model-driven Data-driven

SA2015.SIGGRAPH.ORG

Sponsored by

Reflectance Field Representation Surface reflectance field:

𝜌𝑆RF 𝒙, 𝝎𝑖 Often image-based Does not rely on knowledge about surface geometry Representations: Model-driven Data-driven

SA2015.SIGGRAPH.ORG

Sponsored by

Surface Reflectance Field Acquisition Single viewpoint (1 camera) Multiple light directions Different variants:

SA2015.SIGGRAPH.ORG

Sponsored by

Surface Reflectance Field Acquisition Single viewpoint (1 camera) Multiple light directions Different variants: Moving light source [Cultural Heritage Imaging]

[Cultural Heritage Imaging]


SA2015.SIGGRAPH.ORG


Sponsored by

Surface Reflectance Field Acquisition [HP Labs]

Single viewpoint (1 camera) Multiple light directions Different variants: Moving light source Light source arrays http://culturalheritageimaging.org/ What_We_Offer/Gear/Lighting_Array/ [Debevec et al. 2000]

[Goerghiades et al. 1999] SA2015.SIGGRAPH.ORG

[Malzbender et al. 2001]

Sponsored by

References The respective report and a list of references can be found in:

Advances in Geometry and Reflectance Acquisition Michael Weinmann und Reinhard Klein SIGGRAPH Asia 2015 Courses, ACM, 2015 http://dl.acm.org/citation.cfm?id=2818165

SA2015.SIGGRAPH.ORG

Sponsored by

Advances in Appearance Modeling

Sponsored by

Appearance Compression

SA2015.SIGGRAPH.ORG

Sponsored by

Motivation Compression mostly solved for 2D textures Good codecs exists Level-of-Detail

But for e.g. BTFs… GBs or TBs per material / object Should be consistent in the angular domain

SA2015.SIGGRAPH.ORG

Sponsored by

Motivation Compression facilitates… Rendering Offline (data should fit RAM) Online (data must fit GPU RAM)

Editing / synthesis Transmission

SA2015.SIGGRAPH.ORG

Sponsored by

Motivation Do we need that many samples?

Photograph

Resampled BTF with interpolation

That, or a good model for interpolation

SA2015.SIGGRAPH.ORG

Sponsored by

Model-Driven BRDF Compression

SA2015.SIGGRAPH.ORG

Sponsored by

Lambert 𝑓𝐿𝑎𝑚𝑏𝑒𝑟𝑡 = 𝑐𝑜𝑛𝑠𝑡 Simple analytical model for mostly diffuse surfaces

SA2015.SIGGRAPH.ORG

Sponsored by

Blinn-Phong [Blinn 1977] 𝑓𝐵𝑙𝑖𝑛𝑛−𝑃ℎ𝑜𝑛𝑔 = 𝑘𝑠 ⋅ ℎ, 𝑛

𝑛

𝑘𝑠 - specular color & intensity 𝑛 - specularity

SA2015.SIGGRAPH.ORG

Sponsored by

Ward [Ward 1992]

𝑓𝑊𝑎𝑟𝑑 =

𝑘𝑠 4𝜋𝛼𝑥 𝛼𝑦 cos 𝜃𝑖 cos 𝜃𝑜

⋅𝑒

− tan2

𝜃ℎ ⋅

cos2 𝜙ℎ sin2 𝜙ℎ + 𝛼2 𝛼2 𝑥 𝑦

𝑘𝑠 - specular color & intensity 𝛼𝑥 - lobe width in tangential direction 𝛼𝑦 - lobe width in bi-tangential direction

SA2015.SIGGRAPH.ORG

Sponsored by

Lafortune 𝑓𝐿𝑎𝑓𝑜𝑟𝑡𝑢𝑛𝑒 = 𝑘𝑠 ⋅

𝑙𝑡

⋅𝑀⋅𝑣

𝑚

𝑘𝑠 - specularity 𝑀 - symmetric 3x3 matrix

SA2015.SIGGRAPH.ORG

Sponsored by

Torrance-Sparrow [Torrance et al. 1967] No Occlusion

𝑓𝑇𝑜𝑟𝑟𝑎𝑛𝑐𝑒−𝑆𝑝𝑎𝑟𝑟𝑜𝑤 =

𝑘𝑠 4

⋅

𝐷 𝑚 ⋅𝐹(𝜂)⋅𝐺 𝑛,𝑙 ⋅〈𝑛,𝑣〉 Masking

D 𝑚 𝐺 𝐹 𝜂

- distribution of normal directions - RMS slope of micro-facets - geometric attenuation factor - Fresnel term - Complex refractive index

SA2015.SIGGRAPH.ORG

Shadowing

Sponsored by

Model Fitting BRDF models highly non-linear Non-linear optimization necessary E.g. Levenberg-Marquardt (least-squares) or similar

Error term? [Löw et al. 2012] ∑(log(1 + cos⁡𝜃𝑖 ⋅ 𝑓𝑏𝑟𝑑𝑓 ) ⁡ − log 1 + cos 𝜃𝑖 ⋅ 𝑓𝑚𝑒𝑎𝑠𝑢𝑟𝑒𝑑 )2 Logarithm reduces dynamic range

Linear SA2015.SIGGRAPH.ORG

Ground truth

Logarithmic Sponsored by

MERL BRDF Database [Matusik 2003]

100 isotropic materials 90x90x180x3 = 4,374,000 samples / material Ruzinkiewicz parameterization

In/Out

SA2015.SIGGRAPH.ORG

Half/Diff

Sponsored by

Example: Dark Blue Paint

Acquired data

SA2015.SIGGRAPH.ORG

Bp

Ward

Blinn-Phong

Ward

Laf

Lafortune

Cookt

Torrance

Sponsored by

Evaluation [Ngan et al. 2006] Observed order of quality (roughly): Torrance Lafortune Ward Blinn-Phong

SA2015.SIGGRAPH.ORG

Sponsored by

Composite Models Some materials impossible to represent by single lobe Layering e.g. can produce multiple lobes

Acquired data SA2015.SIGGRAPH.ORG

Torrance, 1 lobe

Torrance, 2 lobes

Sponsored by

genBRDF [Brady et al. 2014] Novel analytical BRDF models from measured data via genetic algorithm User-specified amount of free parameters

SA2015.SIGGRAPH.ORG

Sponsored by

Spherical Harmonics Orthonormal basis for 𝐿2 (𝑆 2 ) Fitting: simple basis projection per fixed viewing direction:

𝑐𝑖𝑣 = ∫ 𝑓𝑏𝑟𝑑𝑓 𝑙 , 𝑣 ⋅ 𝑦𝑖 𝑙 𝑑𝑙 Enable real-time environmental lighting Useful for SVBRDFs and BTFs, too

SA2015.SIGGRAPH.ORG

Sponsored by

BRDF Models - Overview

Model

# parameters

Lambert

1 (RGB)

Blinn-Phong

2 (RGB) + 1

Ward

2 (RGB) + 3

Lafortune

2 (RGB) + 4

Torrence-Sparrow

2 (RGB) + 2

genBRDF

User-defined, < 5 typical, possibly RGB

Spherical harmonics

< 100 per view typical

 Excellent compression, even with multiple lobes, quality often satisfactory

SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDFs Extension of BRDFs to spatial domain Per-texel BRDFs

Appropriate for materials with little non-local effects Smooth, connected meso-scale geometry Little subsurface scattering

SA2015.SIGGRAPH.ORG

[Lensch et al. ]

Sponsored by

SVBRDF Fitting Much more involved Needs to deal with: Typically much lower #samples (not homogeneous); [Aittala et al. 2015] combine sparse BRDFs from similar texels for robust fitting from two images

SA2015.SIGGRAPH.ORG

Sponsored by

SVBRDF Fitting Possibly meso-scale surface structure (rotated normals, occlusion, parallax, interreflections)

Can be accounted for to some extent ([Ruiters et al. 2009]) Original

BRDF Optimization Iteration

Heightmap Optimization

Iteration

Weightmap Optimization

Our Technique (with light exchange)

Remove Interreflections 0.030 / 0.007 mm

 One set of BRDF parameters / texel  Very good compression, but…

SA2015.SIGGRAPH.ORG

Mean relative BRDF error

8.9%

Sponsored by

2.7%

Limitations Meso-structure: BRDF  A(pparent)BRDF

Expensive to re-create from SVBRDF

SA2015.SIGGRAPH.ORG

Sponsored by

Limitations

ωo Retro-reflection

Meso-structure: BRDF  A(pparent)BRDF

Specular reflection

ωi


SA2015.SIGGRAPH.ORG

Sponsored by

Limitations



Specular reflection

ωi ωo


SA2015.SIGGRAPH.ORG

ωi Sponsored by

Limitations



Specular reflection

ωi ωo Influence from neighborhood


SA2015.SIGGRAPH.ORG

ωi Sponsored by

Limitations



Specular reflection

ωi ωo Influence from neighborhood


SA2015.SIGGRAPH.ORG

ωi

Impossible to describe with analytical model

Sponsored by

Model-Driven BTF Compression

SA2015.SIGGRAPH.ORG

Sponsored by

BTF Sparse Parametric Mixture Models [Wu et al. 2011] Represent ABRDFs as mixture of analytical models + residual Clustering on ABRDFs 1 residual / cluster  Compression ratios of 1:70 – 1:300

SA2015.SIGGRAPH.ORG

Sponsored by

Spherical Harmonics ABRDFs also representable in SH basis Enables real-time rendering of environmentally-lit BTFs More SHs necessary to represent high-frequency effects Coarse sampling of view hemisphere  not much compression, but: Can be applied after certain other compression methods

SA2015.SIGGRAPH.ORG

Sponsored by

Data-Driven BRDF Compression

SA2015.SIGGRAPH.ORG

Sponsored by

Data-Driven Approaches Idea: Instead of pre-designed models, … … use models derived from data

h0

= reconstructed ABRDF

SA2015.SIGGRAPH.ORG

h1

+ g1

h2

+ g2

h3

+ g3

+ ...

mean

Sponsored by

Linear Models For BRDFs [Matusik et al. 2003a] Principal Component Analysis (PCA) on MERL database: 𝐷 = mean + EVs ⋅ Coeffs Performed in log(RGB) color space Less dynamic range (highlights!) HVS more sensitive to ratios

SA2015.SIGGRAPH.ORG

Eigenvalue plot

Sponsored by

Linear Models For BRDFs Result: 45 eigen-BRDFs suffice (error

Appearance Capture and Modeling - Computer Graphics Bonn

Appearance Capture and Modeling - Computer Graphics Bonn

Suggest Documents

Paper - Computer Graphics Bonn

IEEE Computer Graphics and Applications - Computer Graphics Bonn

Toward Image-Based Facial Hair Modeling - Computer Graphics Bonn

Supplemental Material - Computer Graphics Bonn - UniversitÃ¤t Bonn

Near Optimal Hierarchical Culling - Computer Graphics Bonn

computers &graph|c - Computer Graphics Bonn

Light Scattering from Filaments - Computer Graphics Bonn

Interactive Steering of Mesh Animations - Computer Graphics Bonn

Action Graph: A Versatile Data Structure for ... - Computer Graphics Bonn

Computing Static Electricity on Human Hair - Computer Graphics Bonn

Time-Varying Surface Appearance - UC Berkeley Computer Graphics ...

Advanced Modeling Techniques for Computer Graphics

Filling Long-Time Gaps of Motion Capture Data - Computer Graphics ...

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Computer Graphics

Web-based collaborative feature modeling - Computer Graphics and ...