Image Processing Prof. Christian Bauckhage

outline lecture 10 recap space domain filters and convolutions properties of convolutions space domain filters in image processing summary

recap

the 2D Fourier transform and its inverse

1 F(µ, ν) = 2π

∞ Z

∞ Z

f (x, y) e−i (µx+νy) dx dy −∞ −∞

1 f (x, y) = 2π

∞ Z

∞ Z

F(µ, ν) ei (µx+νy) dµ dν −∞ −∞

recap

the discrete 2D Fourier transform and its inverse

F[m, n] =

M−1 X N−1 X

mx

ny

f [x, y] e−i 2π ( M + N )

x=0 y=0

M−1 N−1 ny mx 1 XX f [x, y] = F[m, n] ei 2π ( M + N ) MN m=0 n=0

recap

30

30

20

20

10

10

0

ν

ν

periodic images f (x, y) and their spectra |F(µ, ν)|

−10

−20

−20

−30

−20

−10

0

µ

10

20

−30

30

30

30

20

20

10

10

0

ν

ν

−30

0

−10

−10

−20

−10

−30

−20

−10

0

10

20

30

0

10

20

30

µ

0

−10

−20

−30

−30

−20

−30

−20

−10

0

µ

10

20

30

−30

µ

recap

30

30

20

20

10

10

0

ν

ν

more periodic images f (x, y) and their spectra |F(µ, ν)|

−10

−20

−30

0

−10

−20

−30

−20

−10

0

µ

10

20

30

−30

−30

−20

−10

0

µ

10

20

30

recap

natural image and its logarithmic frequency spectrum

f (x, y)

log|F(µ, ν)|

recap

frequency domain filtering given a signal f (x, y), compute its Fourier transform F(µ, ν) = F f (x, y)

choose a filter function G(µ, ν) and compute ˜ F(µ, ν) = F(µ, ν) · G(µ, ν) finally, obtain the filtered signal as ˜f (x, y) = F−1 F(µ, ˜ ν)

recap

low-pass filter

F

− →

·G

l −→

↓ F−1

recap

high-pass filter

F

− →

·G

h −−→

↓ F−1

recap

band-pass filter

F

− →

·G

b −−→

↓ F−1

recap

low-pass filtering of a noisy image

f (x, y)

log|F(µ, ν)|

˜ log|F(µ, ν)|

˜f (x, y)

space domain filters and convolutions

question which operation in the space domain corresponds to multiplication in the frequency domain ?

F

f (x), g(x)? −−−−→ F(ω), G(ω) · y? y ˜f (x)

F −1

←−−−−

˜ F(ω)

answer let’s see . . .

frequency domain filtering (dropping the constant (2π)−

∞ Z

F(ω) · G(ω) = −∞

∞ Z

f (x) e−iωx dx · −∞

g(x) e−iωx dx

1 2

)

frequency domain filtering (dropping the constant (2π)−

∞ Z

f (x) e−iωx dx ·

F(ω) · G(ω) = −∞

g(x) e−iωx dx

∞ Z −iωv

f (v) e −∞

−∞

∞ Z

=

∞ Z

g(u) e−iωu du

dv · −∞

1 2

)

frequency domain filtering (dropping the constant (2π)−

∞ Z

f (x) e−iωx dx ·

F(ω) · G(ω) = −∞

g(x) e−iωx dx

∞ Z −iωv

f (v) e −∞ ∞ Z

−∞

∞ Z

=

∞ Z

g(u) e−iωu du

dv · −∞

∞ Z

f (v) e−iωv g(u) e−iωu dv du

= −∞ −∞

1 2

)

frequency domain filtering (dropping the constant (2π)−

∞ Z

f (x) e−iωx dx ·

F(ω) · G(ω) = −∞

g(x) e−iωx dx

∞ Z −iωv

f (v) e −∞ ∞ Z

−∞

∞ Z

=

∞ Z

g(u) e−iωu du

dv · −∞

∞ Z

f (v) e−iωv g(u) e−iωu dv du

= −∞ −∞ ∞ Z

∞ Z

f (v) g(u) e−iω(v+u) dv du

= −∞ −∞

1 2

)

frequency domain filtering

to eliminate the sum v + u in the complex exponential, we substitute u=x−v

frequency domain filtering

to eliminate the sum v + u in the complex exponential, we substitute u=x−v

so that du d x−v =1 = dx dx

frequency domain filtering

to eliminate the sum v + u in the complex exponential, we substitute u=x−v

so that du d x−v =1 = dx dx and thus du = dx

frequency domain filtering

∞ Z

∞ Z

f (v) g(u) e−iω(v+u) dv du

F(ω) · G(ω) = −∞ −∞ ∞ Z

∞ Z

f (v) g(x − v) e−iωx dv dx

= −∞ −∞

frequency domain filtering

∞ Z

∞ Z

f (v) g(u) e−iω(v+u) dv du

F(ω) · G(ω) = −∞ −∞ ∞ Z

∞ Z

f (v) g(x − v) e−iωx dv dx

= −∞ −∞ ∞ Z

=

∞ Z

−∞

−∞

f (v) g(x − v) dv e−iωx dx

frequency domain filtering

in other words ∞ Z

F(ω) · G(ω) = F

f (v) g(x − v) dv −∞

convolution

the convolution of f (x) and g(x) is given by ∞ Z

f (x) ∗ g(x) = f ∗ g (x) =

f (ξ) g(x − ξ) dξ −∞

space domain filtering

we therefore have

F

f (x), g(x)? −−−−→ F(ω), G(ω) · ∗ y y F −1

f (x) ∗ g(x) ←−−−−

˜ F(ω)

note

multiplication in the frequency domain ⇔ convolution in the space domain

the convolution theorem

F f ∗ g = c · F f · Fg

properties of convolutions

significance

convolutions occur all over mathematics

significance

convolutions occur all over mathematics, for instance in probability theory (the sum of 2 independent random variables is distributed as the convolution of the individual distributions) in algebra (the product of two polynomials can be computed using convolutions)

algebraic properties

commutativity f ∗g=g∗f associativity f ∗ g∗h = f ∗g ∗h distributivity f ∗ g+h = f ∗g + f ∗h c·f ∗g=c· f ∗g

linearity

the convolution is a linear operation

complex conjugation and differentiation

f ∗g=f ∗g

d d d f ∗g = f ∗g=f ∗ g dx dx dx

identity element

there is an identity element δ(x) such that

f (x) ∗ δ(x) = f (x)

question can we already say something about δ(x) ?

question can we already say something about δ(x) ?

answer consider this F

f (x), δ(x) −−−−→ F(ω), ∆(ω) ∗ · y y f (x)

F −1

←−−−−

F(ω)

question can we already say something about δ(x) ?

answer consider this (observing that ∆(ω) = 1 ∀ ω) F

f (x), δ(x) −−−−→ F(ω), ∆(ω) ∗ · y y f (x)

F −1

←−−−−

F(ω)

2D convolution

∞ Z ∞ Z

f ∗g (x, y) =

f (ξ, η) g(x−ξ, y−η) dξ dη −∞ −∞

discrete 2D convolution

∞ X ∞ X f i, j g x − i, y − j f ∗ g [x, y] = i=−∞ j=−∞

note

in the real world, every signal (⇔ measurement) is necessarily discrete and of finite support

note

in the real world, every signal (⇔ measurement) is necessarily discrete and of finite support ⇔ strictly speaking, there is no infinitesimally fine grained measurement and neither are there infinitely extended measurement domains

discrete 2D convolution

m

n

2 2 X X f ∗ g [x, y] = f x − i, y − j g i, j

i=− m2 j=− 2n

illustration m

n

2 2 X X h[x, y] = f ∗ g [x, y] = f x − i, y − j g i, j n i=− m 2 j=− 2

=

2 1 X X f x − i, y − j g i, j i=−2 j=−1

f

M m

y

g n N

x

space domain filters in image processing

principle

a “small” filter mask is shifted over the image f g y

x

principle

a “small” filter mask is shifted over the image f

f g

g y

y

x

x+1

principle

a “small” filter mask is shifted over the image f

f y

y

x

f

g

g

g

···

y

x+1

x+2

principle

a “small” filter mask is shifted over the image f

f y

y

f

y+1

x

.. .

···

g

···

y

x+1

x

g

f

g

g

x+2

theory and practice

arrays

functions f

f

(0, 0)

(0, 0)

g (0, 0)

(0, 0)

g

theory and practice

arrays

functions m

h[x, y] =

2 X

n

2 X

n i=− m 2 j=− 2

f x − i, y − j g i, j

h[y,x] = 0.0 for j in range(-n/2,n/2): for i in range(-m/2, m/2): h[y,x] += f[y+j,x+i]*g[n/2-j,m/2-i]

theory and practice

arrays

functions m

h[x, y] =

2 X

n

2 X

n i=− m 2 j=− 2

f x − i, y − j g i, j

h[y,x] = 0.0 for j in range(-n/2,n/2): for i in range(-m/2, m/2): h[y,x] += f[y+j,x+i]*g[n/2-j,m/2-i]

import scipy.ndimage as img f = ... # N x M image array g = ... # n x m filter array h = img.filters.convolve(f,g)

summary

we now know about

the convolution of two functions the convolution of discrete functions of finite support

exercises

show that

f (x) ∗ g(x) = g(x) ∗ f (x)

show that d d f (x) ∗ g(x) = f (x) ∗ g(x) dx dx

exercises

in the lecture, we proved that F f ∗ g = c · F f · Fg

now, show that F f · g = c · F f ∗ Fg

outline lecture 10 recap space domain filters and convolutions properties of convolutions space domain filters in image processing summary

recap

the 2D Fourier transform and its inverse

1 F(µ, ν) = 2π

∞ Z

∞ Z

f (x, y) e−i (µx+νy) dx dy −∞ −∞

1 f (x, y) = 2π

∞ Z

∞ Z

F(µ, ν) ei (µx+νy) dµ dν −∞ −∞

recap

the discrete 2D Fourier transform and its inverse

F[m, n] =

M−1 X N−1 X

mx

ny

f [x, y] e−i 2π ( M + N )

x=0 y=0

M−1 N−1 ny mx 1 XX f [x, y] = F[m, n] ei 2π ( M + N ) MN m=0 n=0

recap

30

30

20

20

10

10

0

ν

ν

periodic images f (x, y) and their spectra |F(µ, ν)|

−10

−20

−20

−30

−20

−10

0

µ

10

20

−30

30

30

30

20

20

10

10

0

ν

ν

−30

0

−10

−10

−20

−10

−30

−20

−10

0

10

20

30

0

10

20

30

µ

0

−10

−20

−30

−30

−20

−30

−20

−10

0

µ

10

20

30

−30

µ

recap

30

30

20

20

10

10

0

ν

ν

more periodic images f (x, y) and their spectra |F(µ, ν)|

−10

−20

−30

0

−10

−20

−30

−20

−10

0

µ

10

20

30

−30

−30

−20

−10

0

µ

10

20

30

recap

natural image and its logarithmic frequency spectrum

f (x, y)

log|F(µ, ν)|

recap

frequency domain filtering given a signal f (x, y), compute its Fourier transform F(µ, ν) = F f (x, y)

choose a filter function G(µ, ν) and compute ˜ F(µ, ν) = F(µ, ν) · G(µ, ν) finally, obtain the filtered signal as ˜f (x, y) = F−1 F(µ, ˜ ν)

recap

low-pass filter

F

− →

·G

l −→

↓ F−1

recap

high-pass filter

F

− →

·G

h −−→

↓ F−1

recap

band-pass filter

F

− →

·G

b −−→

↓ F−1

recap

low-pass filtering of a noisy image

f (x, y)

log|F(µ, ν)|

˜ log|F(µ, ν)|

˜f (x, y)

space domain filters and convolutions

question which operation in the space domain corresponds to multiplication in the frequency domain ?

F

f (x), g(x)? −−−−→ F(ω), G(ω) · y? y ˜f (x)

F −1

←−−−−

˜ F(ω)

answer let’s see . . .

frequency domain filtering (dropping the constant (2π)−

∞ Z

F(ω) · G(ω) = −∞

∞ Z

f (x) e−iωx dx · −∞

g(x) e−iωx dx

1 2

)

frequency domain filtering (dropping the constant (2π)−

∞ Z

f (x) e−iωx dx ·

F(ω) · G(ω) = −∞

g(x) e−iωx dx

∞ Z −iωv

f (v) e −∞

−∞

∞ Z

=

∞ Z

g(u) e−iωu du

dv · −∞

1 2

)

frequency domain filtering (dropping the constant (2π)−

∞ Z

f (x) e−iωx dx ·

F(ω) · G(ω) = −∞

g(x) e−iωx dx

∞ Z −iωv

f (v) e −∞ ∞ Z

−∞

∞ Z

=

∞ Z

g(u) e−iωu du

dv · −∞

∞ Z

f (v) e−iωv g(u) e−iωu dv du

= −∞ −∞

1 2

)

frequency domain filtering (dropping the constant (2π)−

∞ Z

f (x) e−iωx dx ·

F(ω) · G(ω) = −∞

g(x) e−iωx dx

∞ Z −iωv

f (v) e −∞ ∞ Z

−∞

∞ Z

=

∞ Z

g(u) e−iωu du

dv · −∞

∞ Z

f (v) e−iωv g(u) e−iωu dv du

= −∞ −∞ ∞ Z

∞ Z

f (v) g(u) e−iω(v+u) dv du

= −∞ −∞

1 2

)

frequency domain filtering

to eliminate the sum v + u in the complex exponential, we substitute u=x−v

frequency domain filtering

to eliminate the sum v + u in the complex exponential, we substitute u=x−v

so that du d x−v =1 = dx dx

frequency domain filtering

to eliminate the sum v + u in the complex exponential, we substitute u=x−v

so that du d x−v =1 = dx dx and thus du = dx

frequency domain filtering

∞ Z

∞ Z

f (v) g(u) e−iω(v+u) dv du

F(ω) · G(ω) = −∞ −∞ ∞ Z

∞ Z

f (v) g(x − v) e−iωx dv dx

= −∞ −∞

frequency domain filtering

∞ Z

∞ Z

f (v) g(u) e−iω(v+u) dv du

F(ω) · G(ω) = −∞ −∞ ∞ Z

∞ Z

f (v) g(x − v) e−iωx dv dx

= −∞ −∞ ∞ Z

=

∞ Z

−∞

−∞

f (v) g(x − v) dv e−iωx dx

frequency domain filtering

in other words ∞ Z

F(ω) · G(ω) = F

f (v) g(x − v) dv −∞

convolution

the convolution of f (x) and g(x) is given by ∞ Z

f (x) ∗ g(x) = f ∗ g (x) =

f (ξ) g(x − ξ) dξ −∞

space domain filtering

we therefore have

F

f (x), g(x)? −−−−→ F(ω), G(ω) · ∗ y y F −1

f (x) ∗ g(x) ←−−−−

˜ F(ω)

note

multiplication in the frequency domain ⇔ convolution in the space domain

the convolution theorem

F f ∗ g = c · F f · Fg

properties of convolutions

significance

convolutions occur all over mathematics

significance

convolutions occur all over mathematics, for instance in probability theory (the sum of 2 independent random variables is distributed as the convolution of the individual distributions) in algebra (the product of two polynomials can be computed using convolutions)

algebraic properties

commutativity f ∗g=g∗f associativity f ∗ g∗h = f ∗g ∗h distributivity f ∗ g+h = f ∗g + f ∗h c·f ∗g=c· f ∗g

linearity

the convolution is a linear operation

complex conjugation and differentiation

f ∗g=f ∗g

d d d f ∗g = f ∗g=f ∗ g dx dx dx

identity element

there is an identity element δ(x) such that

f (x) ∗ δ(x) = f (x)

question can we already say something about δ(x) ?

question can we already say something about δ(x) ?

answer consider this F

f (x), δ(x) −−−−→ F(ω), ∆(ω) ∗ · y y f (x)

F −1

←−−−−

F(ω)

question can we already say something about δ(x) ?

answer consider this (observing that ∆(ω) = 1 ∀ ω) F

f (x), δ(x) −−−−→ F(ω), ∆(ω) ∗ · y y f (x)

F −1

←−−−−

F(ω)

2D convolution

∞ Z ∞ Z

f ∗g (x, y) =

f (ξ, η) g(x−ξ, y−η) dξ dη −∞ −∞

discrete 2D convolution

∞ X ∞ X f i, j g x − i, y − j f ∗ g [x, y] = i=−∞ j=−∞

note

in the real world, every signal (⇔ measurement) is necessarily discrete and of finite support

note

in the real world, every signal (⇔ measurement) is necessarily discrete and of finite support ⇔ strictly speaking, there is no infinitesimally fine grained measurement and neither are there infinitely extended measurement domains

discrete 2D convolution

m

n

2 2 X X f ∗ g [x, y] = f x − i, y − j g i, j

i=− m2 j=− 2n

illustration m

n

2 2 X X h[x, y] = f ∗ g [x, y] = f x − i, y − j g i, j n i=− m 2 j=− 2

=

2 1 X X f x − i, y − j g i, j i=−2 j=−1

f

M m

y

g n N

x

space domain filters in image processing

principle

a “small” filter mask is shifted over the image f g y

x

principle

a “small” filter mask is shifted over the image f

f g

g y

y

x

x+1

principle

a “small” filter mask is shifted over the image f

f y

y

x

f

g

g

g

···

y

x+1

x+2

principle

a “small” filter mask is shifted over the image f

f y

y

f

y+1

x

.. .

···

g

···

y

x+1

x

g

f

g

g

x+2

theory and practice

arrays

functions f

f

(0, 0)

(0, 0)

g (0, 0)

(0, 0)

g

theory and practice

arrays

functions m

h[x, y] =

2 X

n

2 X

n i=− m 2 j=− 2

f x − i, y − j g i, j

h[y,x] = 0.0 for j in range(-n/2,n/2): for i in range(-m/2, m/2): h[y,x] += f[y+j,x+i]*g[n/2-j,m/2-i]

theory and practice

arrays

functions m

h[x, y] =

2 X

n

2 X

n i=− m 2 j=− 2

f x − i, y − j g i, j

h[y,x] = 0.0 for j in range(-n/2,n/2): for i in range(-m/2, m/2): h[y,x] += f[y+j,x+i]*g[n/2-j,m/2-i]

import scipy.ndimage as img f = ... # N x M image array g = ... # n x m filter array h = img.filters.convolve(f,g)

summary

we now know about

the convolution of two functions the convolution of discrete functions of finite support

exercises

show that

f (x) ∗ g(x) = g(x) ∗ f (x)

show that d d f (x) ∗ g(x) = f (x) ∗ g(x) dx dx

exercises

in the lecture, we proved that F f ∗ g = c · F f · Fg

now, show that F f · g = c · F f ∗ Fg