Learning Sparse Partial Differential Equations for ...

Learning Sparse Partial Differential Equations for Vector-valued Images Yuanyuan Jiao1,2 , Xiaogang Pan1,2 , Zhenyu Zhao3 , and Chenping Hou3

⋆

1

2 3

State Key Laboratory of Astronautic Dynamics, Xi’an, Shaanxi, 710043. College of Nine, National University of Defense Technology, Changsha, 410073. College of Science, National University of Defense Technology, Changsha 410073. [email protected], panxiaogang [email protected], [email protected], [email protected].

Abstract. Learning Partial Differential Equations (LPDEs) from training data for particular tasks has been successfully applied to many image processing problems. In this paper, we propose a more effective LPDEs model for vector-valued image tasks. The PDEs are also formulated as a linear combination of fundamental differential invariants, but have several distinctions. First, we simplify the current LPDEs system by omitting a PDE which works as an indicate function in current ones. Second, instead of using L2 -norm, we use the L1 -norm to regularize the coefficients with respect to the fundamental differential invariants. Third, as the objective function is not smooth, we resort to the alternating direction method (ADM) to optimize it. We illustrate the properties of our LPDEs system by several examples in denoising and demosaicking of RGB color images. The experiments demonstrate the advantage of the proposed method over other PDE-based methods.

Keywords: Partial differential equations, L1 -norm regularization, ADM, Color images

1

Introduction

Partial Differential Equations (PDEs) have shown their superiority in computer vision and image processing [2, 32], e.g., denoising [30], enhancement [29], segmentation [18], stereo and optical flow computation [3]. However, designing a PDE system for a particular task usually requires high mathematical skills and good insight into the problem. Basically, there are two kinds of methods for designing PDEs. The first kind is to write down PDEs directly. This requires good mathematical understandings on the properties of the PDEs. The second kind is to define an energy functional first [28], which collects the wish list of the desired properties of the output image or video, and then derive the evolution equations ⋆

Thanks the NSFC support (No. 61471369, 61503396) and the open Research Foundation of State Key Laboratory of Astronautic Dynamics (No. 2013ADL-DW0101 and 2014ADL-DW0102). Chenping Hou is the corresponding author.

2 PDEs evolution

Coef.

Coef.

. . .

. . . Input image

Invariants

Goal

t=T/N

Invariants

t=T

Expect output image

Fig. 1. Pipeline of the proposed approach. It aims to learn the PDEs evolution process from the input image to the expect output image.

by computing the Euler-Lagrange variation of the energy functional. For example, the ROF model [31] and TV-L1 [8] for image denoising are designed directly, while the Nambu model [33] and the PL model [12] for color image processing are designed in the variational way. Recently, Liu et al. [25] proposed a method that learned partial differential equations (LPDEs) from training image pairs (i.e., the input image and the expect output image). In [24, 25], they successfully apply LPDEs to image restoration, debluring and denoising tasks. They also use LPDEs to solve some mid-andhigh-level tasks that the traditional PDE-based methods cannot. In [25], they apply LPDEs to object detection, color2gray, and demosaicking. For saliency detection, Liu et al. [23] propose to learn the boundary of a PDEs system. Zhao et al. [36] extend the LPDEs method to natural scene text detection tasks. Although they have made great progress, they mainly focus on gray images or particular tasks. In this paper, we focus on the learning PDEs for vector-valued images. Figure 1 shows the pipeline of the approach. We assume that the image process is a PDEs evolutionary type. The time-dependent operations of the evolutionary PDEs resemble the different steps of the process. Our aim is to learn the PDEs evolution process when given the input/output image pairs (the first and the last columns in Figure 1). There are three coupled PDEs with respect to each channel (i.e., R, G, B three channels, see in Figure 1) and the governing functions are linear represented by the fundamental translational and rotational invariants. We believe that for color images the coefficients with respect to the fundamental invariants should be in a sparse representation. So here we use L1 -norm regularization (also called sparsity regularization) instead of L2 -norm regularization. When change to L1 -norm regularization, the optimal control approach proposed by [25] is no longer available as the objective function is not smooth enough. Alternating direction method (ADM) [19, 7] has been widely used to solve such problems and has a great performance. So here we adopt

3

ADM to learn the combination coefficients. In summary, our contributions are as follows: 1. A new framework Learning Sparse Partial Differential Equations is proposed which is more suitable for designing PDEs. For most computer vision tasks on vector-valued images, the invariants are redundant and the coefficients should be sparse, so we use L1 −norm regularization instead of L2 −norm regularization. 2. We apply the alternating direction method (ADM) for training PDEs. Fed with pairs of input and output images, the proposed method can automatically learn the combination coefficients in the PDEs system. That means one can solve more difficult problems in computer vision. The rest of the paper is structured as follows. First of all, we give a brief review on intelligent LPDEs system in Section 2. Then we present the main idea of learning sparse PDEs in Section 3. In Section 4, we apply the alternating direction method (ADM) for training PDEs. Then in Section 5 we compare LSPDE with other PDE methods by using some computer vision and image processing problems. Finally, we give concluding remarks and discussions on the future work in Section 6.

2

Intelligent LPDEs system

In this section, we briefly review the framework of intelligent LPDEs system for computer vision and image processing problems. More details can be found in [25, 20, 21]. 2.1

Notations

We provide a brief summary of the notations used throughout the paper. The evolutionary function of the image is denoted as u = {uc , c = 1, 2, 3}, and the original image is denoted as I = {I c , c = 1, 2, 3}. We focus on the learning of the evolutionary PDEs of u = {uc , c = 1, 2, 3}. Vectors are bold lowercase and scalars or entries are not bold. For instance, x is a vector and xi its i−th component. The entries of vector are ∫ functions which usually define on Q, Ω, Γ or [0, T ]. We denote by ⟨f, g⟩ = Q f g dQ,∥f ∥2 = ∫ 2 f dQ the standard inner product and norm in L2 (Q). Other notations can Q be found in Table 1. 2.2

LPDEs

The current LPDEs system is grounded on the translational and rotational invariance of computer vision and image processing problems. Namely, when the input image is translated or rotated, the output image should be translated or rotated accordingly. Then it can be proven that the governing equations are functions of fundamental differential invariants, which form “bases” of all differential

4 Table 1. Notations Notations Ω ∂Ω Hu Q XT

Description An open bounded region in R2 Boundary of Ω Hessian of u Ω × [0, T ] Transpose of matrix (or vector)

Notations tr(.) ∇u Γ P

Description Trace of martrix Gradient of u ∂Ω × [0, T ] {(0, 0), (0, 1), (1, 0), (2, 0), (1, 1), (0, 2)}, index set for partial differentiation

Table 2. Fundamental differential invariants up to the second order, where tr is the trace operator and ∇f and Hf are the gradient and the Hessian matrix of function f , respectively. j 0,1,2 3,4 5 6,7 8 9 10 11 12 13 14 15 16

invj (u, v) 1, v, u ||∇v||2 = vx2 + vy2 , ||∇u||2 = u2x + u2y (∇v)T ∇u = vx ux + vy uy tr(Hv ) = vxx + vyy , tr(Hu ) = uxx + uyy (∇v)T Hv ∇v = vx2 vxx + 2vx vy vxy + vy2 vyy (∇v)T Hu ∇v = vx2 uxx + 2vx vy uxy + vy2 uyy (∇v)T Hv ∇u = vx ux vxx + (vx uy + ux vy )vxy + vy uy vyy (∇v)T Hu ∇u = vx ux uxx + (vx uy + ux vy )uxy + vy uy uyy (∇u)T Hv ∇u = u2x vxx + 2ux uy vxy + u2y vyy (∇u)T Hu ∇u = u2x uxx + 2ux uy uxy + u2y uyy 2 2 2 tr(H2v ) = vxx + 2vxy + vyy tr(Hv Hu ) = vxx uxx + 2vxy uxy + vyy uyy tr(H2u ) = u2xx + 2u2xy + u2yy

invariants that are invariant with respect to translation and rotation. We assume that the evolution of the image u is guided by an indicator function v, which collects large scale information. As shown in Table 2, there are 17 fundamental differential invariants {invi (u, v), i = 0, · · · , 16} up to the second order. The simplest function of fundamental differential invariants is a linear combination of them. Therefore, learning the PDEs can be transformed into learning the linear combination coefficients among the fundamental differential invariants, which are functions of time t only and independent of spatial variables [25]. To this end, one may prepare a number of input/output training image pairs. By minimizing the difference between the output of PDEs and the ground truth. We set the initial function as the input image. This results in a PDEs constrained optimal control problem: M ∫ 1 ∑ min E(a(t), b(t)) = (Om − um (x, y, T ))2 dΩ a,b 2 m=1 Ω 16 ∫ T 16 ∫ T ∑ ∑ b2i (t) dt, a2i (t) dt + λ2 + λ1 i=0

0

i=0

0

(1)

5

 ∂u ∑16 m  i=0 ai (t)invi (um , vm ) = 0, (x, y, t) ∈ Q, ∂t −    u (x, y, t) = 0, (x, y, t) ∈ Γ,  m   um (x, y, 0) = Im , (x, y) ∈ Ω, ∑16 ∂vm   i=0 bi (t)invi (um , vm ) = 0, (x, y, t) ∈ Q, ∂t −    v (x, y, t) = 0, (x, y, t) ∈ Γ, m   vm (x, y, 0) = Im , (x, y) ∈ Ω,

s.t.

(2)

where {(Im , Om ), m = 1, · · · , M } denote the M input/output training image pairs, um (x, y, t) is the evolution image at time t with respect to the input image Im , vm (x, y, t) is the corresponding indicator function, Ω ⊂ R2 is the (rectangular) region occupied by the image4 , T is the temporal span of evolution which can be normalized as 1, Q = Ω × [0, T ], Γ = ∂Ω × [0, T ], and ∂Ω denotes the boundary of Ω. The last two terms in (3) are regularization terms on the coefficients ai (t) and bi (t). For vector-valued images, by the fundamental differential invariants involving all channels and the extra indicator function, the channels are naturally coupled and the correlation among the channels is implicitly encoded in the control parameters. Accordingly, there are much more fundamental differential invariants, which are invariant to translation and rotation. The set of such invariants up to second order is {1, fr , (∇fr )T ∇fs , (∇fr )T Hfm ∇fs , tr(Hfr ), tr(Hfr Hfs ) |fr , fs , fm ∈ {u1 , · · · , uC , v}}. If C = 3, there are 69 fundamental differential invariants up to the second order. For more details, please refer to [25].

3

Learning sparse PDEs

In this paper, we first simplify the learning-based PDEs model for color images by omitting the PDE which works as an indicate function. As shown in Table 3, when omitting v there are 37 fundamental differential invariants {invi (u), i = 0, · · · , 36} up to the second order (C = 3). Then the problem changes to min E (a(t)) = a

M 3 ∫ 1 ∑∑ 2 (Oc − ucm (x, y, T )) dΩ 2 m=1 c=1 Ω m

 ∂uc ∑ c  ∂tm − 36 i=0 ai (t)invi (um ) = 0, (x, y, t) ∈ Q, s.t. ucm (x, y, t) = 0, (x, y, t) ∈ Γ,  c c um (x, y, 0) = Im , (x, y) ∈ Ω,

(3)

where um = {ucm (x, y, t), c = 1, 2, 3} is the evolution image at time t with respect to the input image Im , Ω ⊂ R2 is the (rectangular) region occupied by 4

The images are padded with zeros of several pixels width around them, so that the Dirichlet boundary conditions, um (x, y, t) = 0, vm (x, y, t) = 0, (x, y, t) ∈ Γ , are naturally fulfilled.

6 Table 3. Fundamental differential invariants up to the second order ( ) invj (u) u = {u1 , u2 , u3 } 1, ui ; i ∈ {1, 2, 3}. ( i )T ∇u ∇uj = uix ujx + uiy ujy ; i, j ∈ {1, 2, 3} and j ≥ i. tr (Hui ) = uixx + uiyy ; i = 1, 2, 3. ( i )T ) ( ∇u Hvk ∇uj = uix ujx ukxx + uix ujy + uiy ujx ukxy +uiy ujy ukyy ; i, j, k ∈ {1, 2, 3} and j ≥ i. 31-36 tr (Hui Huj ) = uixx ujxx + 2uixy ujxy + uiyy ujyy ; i, j ∈ {1, 2, 3} and j ≥ i.

j 0-3 4-9 10-12 13-30

the image5 , T is the temporal span of evolution which can be normalized as 1, Q = Ω × [0, T ], Γ = ∂Ω × [0, T ] and ∂Ω is the boundary of Ω. We denote the coefficients as ac (t) = [ac0 (t), ac1 (t), · · · , ac36 (t)]T , c = 1, 2, 3 for brevity. There are two reasons we omit the PDE with respect to v. – When extending the PDEs system to vector-valued images we find that the PDE with respect to v is redundant. In [25], they use the luminance of the input image as the initial function of the indicator function. It is used to collect the couple information. From Table 3, we can see that the three color channels are naturally coupled and we do not need to propose a new function to collect the couple information. – Considering computation difficulty and complexity, omitting the PDE which works as an indicate function is quite reasonable. For difficulty, Learning the PDE which works as an indicate function directly is extremely hard, so Liu et al. [25] assume that v is vanishing at t = T . Hoverer, this is also extremely hard to preserve. For complexity, when omitting the PDE which works as an indicate function, the number of invariants is decreased from 69 to 37, which can save a lot of time. 3.1

Problem formulation with L1 -norm regularization

For most color image problems, we observe that a few fundamental invariants can be enough to represent the governing functions. That means the coefficient functions {aci (t)|i = 0, 1, · · · , 36, c = 1, 2, 3} should be sparse. So we use L1 -norm instead of L2 -norm [25] to regularize the coefficients a, which is more suitable for color images. Then the whole problem formulation can be stated as follows: M 3 ∫ 1 ∑∑ 2 E(a(t)) = (Oc − ucm (x, y, T )) dΩ 2 m=1 c=1 Ω m 36 ∑ 3 ∫ T ∑ |aci (t)| dt. +λ i=0 c=1 5

(4)

0

The images are padded with zeros of several pixels width around them such that the Dirichlet boundary conditions ucm (x, y, t) = 0, (x, y, t) ∈ Γ , are naturally fulfilled.

7

The two reasons we use sparsity regularization are as follows. – For one task, the active invariants are in the minority. For example, the mostly used diffusion equations F = div(cu) just has one term tr(Hu ) if c is a constant. For anisotropic diffusion, we have that div(c∇u) = tr(cHu ) + (∇u)T ∇c, which can be considered as a combination of tr(Hu) and ∇u with the pre-defined function c. – When there are more terms in the governing functions, the numerical stability of PDEs is extremely harder to preserve. An additional reason is that L1 norm regularization can make the model more robust to the outliers (some bad samples).

4

Learning PDEs via ADM

For constrained optimization with L1 norm regularization, the constraints have to be added to the objective function as penalties, resulting in approximated solutions only. The alternating direction method (ADM) [19, 7] has regained a lot of attention recently and is also widely used. It is especially suitable for nonsmooth function with L1 norm regularization like (4) because it fully utilizes the separable structure of the objective function. Another method is the split Bregman method [16], which is closely related to ADM [13] and is influential in image processing. Recently, the linearized ADM (LADM) [35] was proposed to make a improvement. Deng and Yin [10] further propose the generalized ADM that makes both ADM and LADM as its special cases and prove its globally linear convergence by imposing strong convexity on the objective function or full-rankness on some linear operators. Besides, He and Yuan [5] and Tao [34] considered the multi-block LADM and ADM, respectively. Here, we adopt in this paper the alternating direction method (ADM), which minimizes the following augmented Lagrangian function: (

) M ∫ 1 ∑ c 2 L(u, a) = (ucm |t=1 − Om ) dΩ + λ∥ac ∥1 2 Ω c=1 m=1 ( M ) (5) M 3 ∑ ∑ ∑ µ c c c 2 ⟨Ym , D(um , a )⟩ + ∥D(um , a )∥ . + 2 c=1 m=1 m=1 3 ∑

where ||ac ||1 =

36 ∫ ∑ 1 i=0

0

|aci (t)| dt, D(um , ac ) =

∂ucm ∂t

−

∑36 i=0

aci (t)invi (um ), m =

1, 2, · · · , M . The problem (5) is unconstrained. So it can be minimized with respect to a, and u, respectively, by fixing the other variables and then update the Lagrange multipliers Ym and where µ is a penalty parameter. We here use the same initialization method as LPDEs did [25]. A heuristic method was used to initialize the control functions, which initialized acj (t) successively in time, where j = 0, 1, · · · , 36. At each time step,

∂ucm ∂t

is expected to be

c Om −ucm (t) 1−t

so that

8

um (t) can move toward the expected output Om . Then acj (t) can be computed by solving a least squares optimization problem. The ADM for learning sparse PDEs is outlined in Algorithm 1. In the following subsections, we present how to solve the u-sub problem and the a-sub problem.

Algorithm 1 Solving Problem (5) by ADM Input Training image pairs {(Im , Om )}M m=1 . Initialize Ym = 0, µ = 1, µmax = 106 , ε = 10−8 and a. while not converged do 1. fix a = {ac , c = 1, 2, 3} and update um = {uc , c = 1, 2, 3}, m = 1, · · · , M by 3 ∫ 1∑ c 2 ) dΩ (ucm |t=1 − Om 2 c=1 Ω um µ c + ⟨Ym , D(um , ac )⟩ + ∥D(um , ac )∥2 . 2

um =arg minL(um , ac ) =

(6)

2. fix um = {uc , c = 1, 2, 3}, m = 1, · · · , M and update a = {ac , c = 1, 2, 3} by ac =arg minL(uc , ac ) = λ∥ac ∥1 + ac

M ∑

c ⟨Ym , D(ucm , ac )⟩

m=1

(7)

M ∑ µ + ∥D(ucm , ac )∥2 . 2 m=1

3. update the multipliers: c c Ym = Ym + µD(ucm , ac ), c = 1, 2, 3.

4. check the convergence conditions: 3 ∑

∥D(ucm , ac )∥∞ < ε.

c=1

end while

4.1

Solving the u-sub problem

When fixing a,the updating of um = {u1 , u2 , u3 }, m = 1, · · · , M is to solve the problem as follows 1∑ 2 c=1 3

um = arg minJ(um , a) = um

+

∫ c 2 (ucm |t=1 − Om ) dΩ Ω

3 ∑ c=1

⟨Ymc , D(um , ac )⟩ +

3 ∑ µ c=1

2

∥D(um , ac )∥2 . (8)

9

It equals to minimize the function 3 ∫ 3 1∑ µ∑ Yc c 2 L(um ) = (ucm |t=1 − Om ) dΩ + ∥D(um , ac ) + m ∥2 . 2 c=1 Ω 2 c=1 µ

(9)

With variational approach [22] we can derive that the Gˆ ateaux derivative for each channel is as follows ( ) ∂ Ymc DL c =− D(um , a ) + Ducm ∂t µ [ ( )] p+q ∂ Ymc p+q c c − (−1) σpq (um ) D(um , a ) + , ∂xp ∂y q µ ∑36 ∂uc where D(um , ac ) = ∂tm − i=0 aci (t)invi (um ) and  36 ∑  ∂invj (ucm )  c c   σ (u ) = a (t) , pq m j  ∂(ucm )pq j=0 (10)  p+q c  ∂ u    (ucm )pq = p mq . ∂x ∂y Also the boundary conditions must satisfy the equation as follows ( ) 36 ∂ucm ∑ c Ymc µ − ai (t)invi (um ) + |t=1 = Om − ucm |t=1 . ∂t µ i=0

(11)

When the function get the minimum, it should satisfy the following PDE with c , the initial condition ucm |t=0 = Im  ∂ Yc Yc ∂ p+q   (D(um , ac ) + m ) = −(−1)p+q p q {σpq (ucm )(D(um , ac ) + m )}    µ ∂x ∂y µ  ∂t c Y 1 c (D(um , ac ) + m )|t=1 = (Om − ucm |t=1 ),    µ µ    c . ucm |t=0 = Im (12) We sketch the existing theory of optimal control governed by PDEs for the deduction. Details can be found in the appendix. To solve the equation (12), we derive it into two coupled PDEs. We denote ∂uc Yc as φ = ucm and ∂tm − F (ucm , a) + µm = ψ. Then we can get the equations for ψ and φ  ∂ p+q ∂ψ   = −(−1)p+q p q {σpq (φ)ψ},   ∂t ∂x ∂y     36  ∑  Yc  ∂φ = aci (t)invi (φ) − m + ψ, ∂t µ (13) i=0     1 c   ψ|t=1 = (Om − φ|t=1 ),   µ    c φ|t=0 = Im .

10

Because the equations are nonlinear and coupled, it is hard to get the analytic solution. We propose an iterate numerical method to solve the equations and summarize it in Algorithm 2.

Algorithm 2 Solving Problem (13) Input Training image pairs {(Im , Om )}, Ym , m = 1, · · · , M and µ. Initialize Set ψ0 = 0 and get φ = φ0 by solving the PDE { ∂φ ∑ c Ym = 36 i=0 ai (t)invi (φ) − µ , ∂t c . φ|t=0 = Im while not converged do 1. Solve the PDE {

p+q

∂ψi ∂t

ψi |t=1 Get the solution ψi . 2. Solve the PDE {

∂φi ∂t

∂ = −(−1)p+q ∂x p ∂y q {σpq (φi−1 )ψi }, 1 = µ (Om − φ|t=1 ),

φi |t=0

∑ c = 36 i=0 ai (t)invi (φ) − = Im .

Ym µ

+ ψi ,

Get the solution φi and i = i + 1, 3. check the convergence conditions: ∥ψi+1 − ψi ∥∞ < ε. end while

4.2

Solving the a-sub problem

Given u, a = {a1 , a2 , a3 }-sub problem is as follows min L(ac ) = λ∥ac ∥1 +

M ∑

⟨Ymc , D(um , ac )⟩ +

m=1

where ∥ac ∥1 =

36 ∫ ∑ 1 0

i=0

|aci (t)| dt, D(um , ac ) =

1, 2, · · · , M . The problem equals to min L(a) =λ

36 ∫ ∑ i=0

M ∑ µ ∥D(um , ac )∥2 2 m=1

∂ucm ∂t

−

36 ∑ i=0

(14)

aci (t)invi (um ), m =

1

|aci (t)| dt 0

( )2 M ∫ 36 ∑ ∂ucm ∑ c Ymc + − ai (t)invi (um ) + dQ. ∂t µ m=1 Q i=0

(15)

11 ∂uc

Yc

For brevity, we denote as F (ucm ) = ∂tm + µm . Because of compactly supported, the integration and summation can exchange. So we can get ∫

36 1∑

c

L(a ) = λ 0

∫

1

=λ 0

|aci (t)|

dt +

( M ∫ ∑ m=1

i=0

F (ucm )

−

Q

36 ∑

)2 aci (t)invi (um )

dQ

i=0

  ( )2 68 M ∫ 36 ∑ ∑ ∑  |aci (t)| + F (ucm ) − aci (t)invi (um ) dΩ  dt. i=0

m=1

Ω

i=0

That means to minimize the following function ( )2 36 M ∫ 36 ∑ ∑ ∑ c c c L(a (t)) = λ |ai (t)| + F (um ) − ai (t)invi (um ) dΩ. m=1

i=0

Ω

i=0

at each time t. T T c Let ac (t) = [a0 (t), a1 (t), · · · , a36 (t)] , gc (t) = [g0c (t), g1c (t), · · · , g36 (t)] , G = {Gl,k (t)|l, k = 0, 1, · · · , 36} and each element can be gotten by the following formulation  M ∫ ∑   c   g (t) = invi (um (t)) F (ucm ) dΩ,   i Ω m=1 (16) M ∫  ∑   T  invl (um (t)) invk (um (t)) dΩ.   Glk (t) = m=1

Ω

Then the problem can be rewritten as follows ac (t) = arg min λ∥ac (t)∥1 − µ (gc )T ac (t) + ac (t)

µ c T (a (t)) Gac (t). 2

When we implement the framework, we need to do the discretization with respect to temporal variable t first. For example, we discretize t with a step size ∆t and denote ti = i · ∆t, i = 0, · · · , N . At each time step tn+1 we define . ac (tn+1 ) = a, then we can get µ a = arg min λ||a||1 − µb′ a + aT Ga. (17) 2 a It is a least squares optimization with L1 -norm regularization. Many efficient algorithms that guarantee globally optimal solutions have been proposed, such as Gradient Projection (GP) [15], Homotopy [27], Iterative ShrinkageThresholding (IST) [9], Accelerated Proximal Gradient (APG) [4].

5 5.1

Experimental results Implementation

To complete the implementation details we first describe how we discretized our PDEs, see equation (3). Here we use an image extension so as to satisfy the

12 ∂f vanishing normal derivatives and the derivatives ∂f ∂t , ∂x and  f (t+∆t)−f (t) ∂f  ,  ∂t = ∆t f (x+1)−f (x−1) ∂f , ∂x = 2   ∂2f = f (x + 1) − 2f (x) + f (x − 1). ∂x2 2

∂2f ∂x2

are as follows:

(18)

2

∂ f ∂ f The discrete forms of ∂f ∂y , ∂y and ∂x∂y can be defined similarly. In addition, we discretize the integrations as ∑ ∫ f (x, y) dΩ = N1 f (x, y),   Ω (x,y)∈Ω (19) T ∫ ∑   0T f (t) dt = ∆t f (i · ∆t). i=0

where N is the number of pixels in the spatial area, ∆t is a properly chosen time T + 0.5⌋ is the index of the expected output time. We use the step size and ⌊ ∆t forward scheme to approximate the PDEs in (3). For all of our experiments, we choose λ = 10−4 . It is very important to note that we are not proposing an algorithm but a framework to learn PDEs for color images. This is independent of the actual algorithm which is proposed to minimize the corresponding objective function. 5.2

Denoising

In practical image processing scenarios, one major obstacle is posed by noise in images. Denoising is one of the most fundamental low-level vision problems. For this task, experiments are verified on images with unknown natural noise, compared with the existing PDE-based methods, color total variation (CTV [6]), Parallel Level Sets (PL [12]) and LPDEs [25]. As test data we have chosen some color images of the Berkeley Segmentation Database [1]. For better comparison we used publicly available noisy versions of those which are degraded by additive, uncorrelated Gaussian noise of standard deviations of 5, 10, 15, 25 and 35 [14]. Then we compare the methods on images of the remaining objects. For LPDEs [25] we use the implementation available at [26] and we use the implementation available at [11] for CTV [6] and PL [12]. We use the same initialization method as [25] did. The experiment results are shown in Figure 2. Overall, we can see that color total variation (CTV) [6] performs much worse than the proposed method, LPDEs [25] and PL [12]. PL can remove noise effectively, but it gives more blur results than the proposed method and LPDEs. Although LPDEs gives sharper result than PL, it fails to remove some noise. Our method has done the best balance at the two aspects. Figure 3 shows the mean PSNRs by different methods taken over the ten test images. It can be seen that the mean PSNR of our method are higher than those of other methods. After carefully examining the experiment results , we conclude that our distribution is mainly due to two factors: 1) the PDEs system, which learned the coefficients from many training

13

GT

Std=5

Our

CTV

PL

LPDEs

Std=10

Our

CTV

PL

LPDEs

Std=15

Our

CTV

PL

LPDEs

Std=25

Our

CTV

PL

LPDEs

Std=35

Our

CTV

PL

LPDEs

Fig. 2. The results of denoising images with natural noise. (a-b) Noiseless and noisy images. (c-f) Denoised images using CTV [6], PL [12], LPDEs [25] and Our method, respectively. The PSNRs are presented below each image.

images while traditional PDEs methods need to set the parameters with experience and 2) traditional PDEs methods need to choose the parameters for each image separately. 5.3

Demosaicking

Demosaicking is the reconstruction of a color image which is obtained by acquiring image data only at positions described by the Bayer filter. This means that at each position either the intensity for red, green or blue is acquired. This technique enables either to acquire a lot less data for a given resolution or to enhance the resolution by using the same amount of data. A detailed discussion of demosaicking is given in [17]. The same data sets for demosaicking as for denoising is used. We select the first 60 images with the richest texture, measured in their variances. Then we

14 42 Our LPDE PL CTV

40 38 36 34 32 30 28 26 24

5

10

15

20

25

30

35

Fig. 3. The figure shows the denoising performance with respect to the noise level of the methods under comparison. The solid line is the mean PSNR taken over the ten test images.

downsample the images into Bayer CFA raw data and use bilinear interpolation results as the input images of the training pairs. The left images are using for testing. Some results are given in Figures 4 and 6. Figure 4 shows that Our framework can be used to fully recover Bayer filtered images with low noise levels. The difference between the reconstructed images and the original images are hardly visible. Figure 6 gives a comparison with CTV, PL and LPDEs. While color total variation gives blurry results with many color artefacts, PL and LPDEs are capable of eliminating most of these. It is clear that the proposed method gives the sharpest images with only a little color artefacts in high noise levels. Figure 5 shows the mean PSNRs by different methods taken over the ten test images. It can be seen that the mean PSNRs of our method are higher than those of other methods.

6

Conclusion

In this paper, a more effective LPDEs model for color image tasks is proposed. We simplify the current LPDEs system by omitting a PDE which works as an indicate function in current ones. Then, instead of using L2 -norm regularization, L1 -norm is used to regularize the coefficients with respect to the fundamental differential invariants. Last, as the objective function is not smooth, we resort to the alternating direction method (ADM) to optimize it, which ensures convergence in an at least sub-linear rate. In the future, we would like to apply

15

Std=5

GT

Bayer

Our

Std=5

GT

Bayer

Our

Std=5

GT

Bayer

Our

Fig. 4. The results of demosaicking for noise with low standard deviation (std=5).

34 Our LPDE PL CTV

32

PSNR

30

28

26

24

22

20

5

10

15

20 Std

25

30

35

Fig. 5. The figure shows the demosaicking performance with respect to the noise level of the methods under comparison. The solid line is the mean PSNR taken over the ten test images.

LPDEs system for more difficult problems in computer vision, such as image understanding and recognition.

16

GT

Std=5

Our

CTV

PL

LPDEs

Std=10

Our

CTV

PL

LPDEs

Std=15

Our

CTV

PL

LPDEs

Std=25

Our

CTV

PL

LPDEs

Std=35

Our

CTV

PL

LPDEs

Fig. 6. The results of denoising images with natural noise. (a-b) Noiseless and noisy images. (c-f) Denoised images using CTV [6], PL [12], the method proposed by Liu et al. [25] and Our method, respectively. The PSNRs are presented below each image.

Appendix

When each of them get the minimum, the functional J(um , a) (8) get the minimum. We simply write ucm as φ and fix ukm (k ̸= c) when we derive the formulation next. We divide J(φ, a) into two parts {

L1 (φ) = L2 (φ) =

∫ 1 c 2 2 Ω (φ|t=1 − Ocm ) Ym 2 µ 2 ∥D(φ, a) + µ ∥ .

dΩ,

(20)

17

For the first part,we have ∫ [ ] 1 c 2 c 2 L1 (φ + εδφ) − L1 (φ) = ((φ + εδφ)|t=1 − Om ) − (φ|t=1 − Om ) dΩ 2 Ω ∫ c =ε δφ|t=1 (φ|t=1 − Om ) dΩ + o(ε). Ω

∑36 For the second part, first define F (φ, a) = i=0 aci (t)invi (φ), Then can get )2 ( )2 ] ∫ [( µ Ymc Ymc L2 (φ + εδφ) − L2 (φ) = − D(φ, a) + dQ D(φ + εδφ, a) + 2 Q µ µ )2 ∫ [( µ ∂φ ∂δφ Yc = +ε − F (φ + εδφ, a) + m 2 Q ∂t ∂t µ ] ( ) 2 Yc − D(φ, a) + m dQ µ ) ∫ [( ∂δφ =µ ε − F (φ + εδφ, a) + F (φ, a) · ∂t Q )] ( Ymc dQ + o(ε), D(φ, a) + µ where Q = Ω × [0, T ] and dQ = dΩ dt. We first compute F (φ + εδφ, a) − F (φ, a), it equals ( ) ∂F ∂F ∂δφ ∂F ∂ 2 δφ F (φ + εδφ, a) − F (φ, a) = ε (δφ) + + ··· + ∂φ ∂φx ∂x ∂φyy ∂y 2 ∑ ∂ p+q δφ σpq (φ) p q + o(ε). =ε ∂x ∂y (p,q)∈P

where

   σpq (φ) =  φ

pq

=

∂F ∂φpq

=

36 ∑ j=0

acj (t)

∂invj (φ) ∂φpq ,

∂ p+q φ ∂xp ∂y q .

From the above formulations we can get δL = L(φ + εδφ, a) − L(φ, a) ∫ c =ε δφ|t=1 (φ|t=1 − Om ) dΩ Ω [( ) ∫ ∂δφ +µ ε − F (φ + εδφ, a) + F (φ, a) ∂t Q ( )] Yc D(φ, a) + m dQ + o(ε). µ

(21)

18

As the perturbation δφ should satisfy that δφ|Γ = 0 and δφ|t=0 = 0. Integrating by parts,we have ∫ c δL = ε δφ|t=1 (φ|t=1 − Om ) dΩ Ω ( ) ∫ Yc +εµ δφ D(φ, a) + m t=1 dΩ µ Ω ( ) ∫ Yc −εµ δφ D(φ, a) + m t=0 dΩ µ Ω ( ) ∫ ∂ Yc δφ −εµ D(φ, a) + m dQ ∂t µ Q ( ) ∫ p+q ∑ ∂ δφ Yc −εµ σpq (φ) p q D(φ, a) + m dQ ∂x ∂y µ Q (p,q)∈P ∫ c =ε δφ|t=1 (φ|t=1 − Om ) dΩ Ω ) ( ∫ Ymc dΩ +εµ δφ D(φ, a) − t=1 µ Ω ( ) ∫ ∂ Yc −εµ δu D(φ, a) − m dQ ∂t µ Q ( ) ∫ ∫ ∑ Ymc σpq (φ)δφ D(φ, a) + −εµ dΓ dt µ T Γ (p,q)∈P [ ( )] ∫ ∫ ∂ p+q Yc −(−1)p+q εµ δφ p q σpq (φ) D(φ, a) + m dΩ dt ∂x ∂y µ T Ω ∫ c )dΩ =ε δφ|t=1 (φ|t=1 − Om Ω ( ) ∫ Ymc +εµ δφ D(φ, a) + dΩ t=1 µ Ω ( ) ∫ ∂ Yc −εµ δφ D(φ, a) + m dQ ∂t µ Q [ ( )] ∫ p+q ∂ Yc −(−1)p+q εµ δφ p q σpq (φ) D(φ, a) + m dQ. ∂x ∂y µ Q If

DL Dφ

exists, the boundary conditions must satisfy the equation as follows ) ( Ymc c = Om − φ|t=1 . (22) µ D(φ, a) + t=1 µ

Then we get the Gˆ ateaux derivative ( ) DL ∂ Yc =− D(φ, a) + m Dφ ∂t µ [ ( )] p+q ∂ Yc −(−1)p+q p q σpq (φ) D(φ, a) + m . ∂x ∂y µ

19

When the functional get the minimum, it should satisfy the follow PDE with c the initial condition ucm |t=0 = Im ,  ( ) [ ( )] c c Ym Ym ∂ p+q ∂ p+q  D(φ, a) + = −(−1) σ (φ) D(φ, a) +  pq ∂xp ∂y q µ  (∂t )µ c Ym 1 c (23) = µ (Om − φ|t=1 ),  D(φ, a) + µ t=1   c φ|t=0 = Im .

References 1. Arbelaez, P., Fowlkes, C., Martin, D.: The Berkeley segmentation dataset [Online]. Available: http://www.eecs.berkeley.edu/Research/Projects/CS/ vision/bsds/ (2013) 2. Aubert, G., Kornprobst, P.: Mathematical problems in image processing: partial differential equations and the calculus of variations, vol. 147. Springer (2006) 3. Beauchemin, S.S., Barron, J.L.: The computation of optical flow. ACM Computing Surveys (CSUR) 27(3), 433–466 (1995) 4. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences 2(1), 183–202 (2009) 5. Bingsheng, H., Xiaoming, Y.: Linearized alternating direction method of multipliers with gaussian back substitution for separable convex programming. Numerical Algebra Control and Optimization 22(2), 313–340 (2013) 6. Blomgren, P., Chan, T.F.: Color TV: total variation methods for restoration of vector-valued images. IEEE Transactions on Image Processing 7(3), 304 – 309 (1998) 7. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3(1), 1–122 (2011) 8. Chan, T.F., Esedoglu, S.: Aspects of total variation regularized L1 function approximation. SIAM Journal on Applied Mathematics 65(5), 1817–1837 (2005) 9. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Modeling & Simulation 4(4), 1168–1200 (2005) 10. Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. Journal of Scientific Computing 66(3), 1–28 (2016) 11. Ehrhardt, M.J., Arridge, S.R.: Vector-valued image processing by parallel level sets [online]. Available: http://www0.cs.ucl.ac.uk/staff/ehrhardt/software.html (2013) 12. Ehrhardt, M.J., Arridge, S.R.: Vector-valued image processing by parallel level sets. Image Processing, IEEE Transactions on 23(1), 9–18 (2014) 13. Esser, E.: Applications of lagrangian-based alternating direction methods and connections to split bregman. Cam Report (2009) 14. Estrada., F.J.: Image denoising benchmark [online]. Available: http://www.cs. utoronto.ca/∼strider/Denoise/Benchmark/ (2010) 15. Figueiredo, M.A., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE Journal of Selected Topics in Signal Processing 1(4), 586–597 (2007)

20 16. Goldstein, T., Osher, S.: The split bregman method for l1-regularized problems. Siam Journal on Imaging Sciences 2(2), 323–343 (2009) 17. Gunturk, B.K., Altunbasak, Y., Mersereau, R.M.: Color plane interpolation using alternating projections. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 11(9), 997–1013 (2002) 18. Li, C., Xu, C., Gui, C., Fox, M.D.: Level set evolution without re-initialization: a new variational formulation. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. vol. 1, pp. 430–436. IEEE (2005) 19. Lin, Z., Liu, R., Su, Z.: Linearized alternating direction method with adaptive penalty for low-rank representation. Advances in Neural Information Processing Systems pp. 612–620 (2011) 20. Lin, Z., Zhang, W., Tang, X.: Learning partial differential equations for computer vision. Tech. rep., Technical report, Microsoft Research, MSR-TR-2008-189 (2008) 21. Lin, Z., Zhang, W., Tang, X.: Designing partial differential equations for image processing by combining differential invariants. Tech. rep., Technical report, Microsoft Research, MSR-TR-2009-192 (2009) 22. Lions, J.L.: Optimal control of systems governed by partial differential equations, vol. 170. Springer Verlag (1971) 23. Liu, R., Cao, J., Lin, Z., Shan, S.: Adaptive partial differential equation learning for visual saliency detection. In: CVPR (2014) 24. Liu, R., Lin, Z., Zhang, W., Tang, K., Su, Z.: Learning PDEs for image restoration via optimal control. In: ECCV (2010) 25. Liu, R., Lin, Z., Zhang, W., Tang, K., Su, Z.: Toward designing intelligent pdes for computer vision: An optimal control approach. Image and Vision Computing 31(1), 43–56 (2013) 26. Liu, R., Lin, Z., Zhang, W., Tang, K., Su, Z.: Toward designing intelligent PDEs for computer vision: An optimal control approach. Available: http://www.cis. pku.edu.cn/faculty/vision/zlin/zlin.htm (2013) 27. Malioutov, D.M., Cetin, M., Willsky, A.S.: Homotopy continuation for sparse signal representation. In: ICASSP. vol. 5, pp. v–733. IEEE (2005) 28. Martin, B., Andrea, C.M., Stanley, O., Martin, R.: Level Set and PDE Based Reconstruction Methods in Imaging. Springer (2013) 29. Osher, S., Rudin, L.I.: Feature-oriented image enhancement using shock filters. SIAM Journal on Numerical Analysis 27(4), 919–940 (1990) 30. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12(7), 629–639 (1990) 31. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60(1), 259–268 (1992) 32. Sapiro, G.: Geometric partial differential equations and image analysis. Cambridge university press (2006) 33. Sochen, N., Kimmel, R., Malladi, R.: A general framework for low level vision. IEEE Transactions on Image Processing 7(3), 310–318 (1998) 34. Tao, M.: Some parallel splitting methods for separable convex programming with the o(1/t) convergence rate. Pacific Journal of Optimization 10(2), 359–384 (2014) 35. Yang, J., Yuan, X.: Linearized augmented lagrangian and alternating direction methods for nuclear norm minimization. Mathematics of Computation 82(281), 301–329 (2011)

21 36. Zhao, Z., Fang, C., Lin, Z., Wu, Y.: A robust hybrid method for text detection in natural scenes by learning-based partial differential equations. Neurocomputing 168, 23–34 (2015)