Object Proposal with Kernelized Partial Ranking. Jing Wang. Joint work with Jie Shen and Ping Li. Rutgers University. Au
Object Proposal with Kernelized Partial Ranking Jing Wang Joint work with Jie Shen and Ping Li Rutgers University
August 15, 2017
Jing Wang
PR
August 15, 2017
1 / 15
Problem Setup
Object proposals An ensemble of bounding boxes with high potential to contain objects
Goal Determine a small set of proposals with a high recall
Jing Wang
PR
August 15, 2017
2 / 15
Existing solutions and shortcomings
Existing solutions Step 1 Extract multiple features of boxes Step 2 Usually choose a ranking algorithm, such as Ranking SVM
Shortcomings of Ranking SVM: 1 High time complexity due to pairwise constraint 2 Linear kernels are usually utilized in ranking algorithm due to the computational and memory bottleneck of training a kernelized model.
Jing Wang
PR
August 15, 2017
3 / 15
Our solution
Our method A kernelized partial ranking model Benefits 1 Reduce the number of constraints from O(n2 ) to O(nk ) (n is the number of all potential proposals for an image, we are only interested in the top-k) 2 Permit non-linear kernels 3 Introduce a consistent weighted sampling (CWS) paradigm
Jing Wang
PR
August 15, 2017
4 / 15
Problem Formulation For each image, we have an ensemble of candidates B = {b1 , b2 , · · · , bn } a vector y = {y1 , y2 , · · · , yn }, with each yi being the IoU to the ground truth of the candidate bi . Learning the prediction function f :
X → Y.
(1)
The mapping function f is formulated as follows f (X ; w ) = (w · φ(x 1 ), · · · , w · φ(x n )),
(2)
where w is the weight vector we aim to learn,“·” denotes the inner product and the potential φ(x) maps x to a new feature space. Jing Wang
PR
August 15, 2017
5 / 15
None-Linear Kernel
Given u and v are any d-dimensional vectors with non-negative components, Min-Max Kernel is defined as Pd
Min-Max: gmm (u, v) = Pdi=1
min{ui , vi }
i=1 max{ui , vi }
(3)
,
where ui and vi denote the ith component of u and v respectively.
Jing Wang
PR
August 15, 2017
6 / 15
Consisted Weighted Sampling Approximates the min-max kernel by linear functions Algorithm 1 Consistent Weighted Sampling (CWS)1 Require: Feature vector u ∈ Rd with non-negative elements, number of trials S. Ensure: Consistent uniform samples (i1∗ , i2∗ , · · · , iS∗ ) and (t1∗ , t2∗ , · · · , tS∗ ). 1: for s = 1, 2, · · · , S do 2: for all i = 1, 2, · · · , d do 3: ri ∼ Gamma(2, 1), ci ∼ Gamma(2, 1), βi ∼ Uniform(0, 1). 4: ti ← blog(ui )/ri + βi c, yi ← exp(ri (ti − βi )), ai ← ci /(yi exp(ri )). 5: end for 6: is∗ = arg mini ai , ts∗ = tis∗ . 7: end for 1
Ioffe Sergey. Improved Consistent Sampling, Weighted Minhash and L1 Sketching, ICDM, 2010. Jing Wang
PR
August 15, 2017
7 / 15
CWS -theoretical guarantee Theorem (Collision probability) ∗ , t ∗ ) and (i ∗ , t ∗ ) For any two non-negative vectors u and v, let (is,u s,u s,v s,v be the consistent samples produced by Algorithm 1 at the s-th trial. Then we have ∗ ∗ ∗ ∗ Pr (is,u , ts,u ) = (is,v , ts,v ) = gmm (u, v). (4)
a
a
Ioffe Sergey. Improved Consistent Sampling, Weighted Minhash and L1 Sketching, ICDM, 2010.
Due to the above theorem, we immediately have the following result: ∗ ∗ ∗ ∗ E 1{(is,u , ts,u ) = (is,v , ts,v )} = gmm (u, v), (5) where the indicator function 1{event} outputs 1 if event happens and 0 otherwise. Jing Wang
PR
August 15, 2017
8 / 15
Partial Ranking Model Given a training set {(X j , y j )}N j=1 where N denotes the number of training images. Assumptions 1 y j = (y1j , · · · , ynj ): in a non-ascending order 2 φ(·): the feature map for the min-max kernel The convex optimization problem min w
s.t.
1 kw k22 , 2 w · φ(x jp ) ≥ w · φ(x jq ), ∀ j ∈ [N], p ∈ [k ], q ∈ [n]\[k],
(6)
where [N] denotes the integer set of {1, · · · , N} and likewise for [n] and [k ].
Jing Wang
PR
August 15, 2017
9 / 15
Learning with Large Margin Model
A soft-margin formulation N
min
w ,ξ1 ,...,ξN
s.t.
X 1 kw k22 + C ξj , 2 j=1 j w · φ(x p ) − φ(x jq ) ≥ 1 − ξj , ∀ j ∈ [N], ∀ p ∈ [k], ∀ q ∈ [n]\[k].
where ξj is a non-negative slack variable, C is a non-negative trade-off parameter.
Jing Wang
PR
August 15, 2017
10 / 15
Overview of learning procedure
Positive top k
Negative
1. Image
2. Object proposals
last n-k
3. Training samples
Consistent Weighted Sampling
Partial Ranking model
5. Kernel Linearization
6. Training
4. Features
Figure 1: Overview of the learning procedure.
Jing Wang
PR
August 15, 2017
11 / 15
Experiment
Dataset PASCAL VOC2007 Evaluation Metrics Recall, Average Recall (AR) Baselines BING, CPMC, GOP, EB, Endres, MCG, OBJ, Rigor, Rantalankila, RS, M-MCG, RP, SS
Jing Wang
PR
August 15, 2017
12 / 15
Experimental Results 1
1 BING CPMC EB Endres GOP MCG M-MCG OBJ Rantalankila Rigor RP RS SS PR-EB PR-GOP PR-MCG PR-OBJ PR-Rigor PR-RS PR-SS
0.6 0.4 0.2 0 0.5
0.6
0.7
0.8
0.9
BING CPMC EB Endres GOP MCG M-MCG OBJ Rantalankila Rigor RP RS SS PR-EB PR-GOP PR-MCG PR-OBJ PR-Rigor PR-RS PR-SS
0.8
Recall
Recall
0.8
0.6 0.4 0.2 0 0.5
1
0.6
0.7
0.8
0.9
1
IoU overlap threshold
IoU overlap threshold
(a) 100 proposals per image.
(b) 500 proposals per image. BING
1
1
0.8
0.8
CPMC EB Endres GOP
0.6
Average recall
Recall
MCG
BING CPMC EB Endres GOP
0.4
MCG M−MCG OBJ Rantalankila Rigor
0.2
M−MCG OBJ Rantalankila Rigor RP
0.6
RS SS PR−EB PR−GOP PR−MCG
0.4
PR−OBJ PR−Rigor PR−RS PR−SS
0.2
RP RS SS PR−EB
0 0
PR−GOP
200
400
600
# proposals
PR−MCG
800
PR−OBJ PR−Rigor
1000
PR−RS PR−SS
(c) Recall at 0.7 IoU.
0 0
200
400 600 # proposals
800
1000
(d) Average recall.
Figure 2: Comparison results with all baselines in terms of Recall versus IoU threshold. Jing Wang
PR
August 15, 2017
13 / 15
Experimental Results on each class
Table 1: Average Recall on each 20 class of VOC 2007 test set with 300 proposals per image Algorithms BING CPMC EB Endres GOP MCG M-MCG OBJ Rantalankila Rigor RP RS SS VGG ZF PR-EB PR-GOP PR-MCG PR-OBJ PR-Rigor PR-RS PR-SS
aero 0.27 0.55 0.51 0.49 0.41 0.52 0.60 0.33 0.45 0.37 0.54 0.04 0.62 0.42 0.40 0.51 0.42 0.56 0.34 0.50 0.50 0.69
bicycle 0.27 0.44 0.51 0.53 0.43 0.49 0.52 0.33 0.35 0.29 0.38 0.14 0.48 0.48 0.46 0.55 0.43 0.55 0.34 0.38 0.45 0.60
Jing Wang
bird 0.22 0.44 0.45 0.40 0.34 0.42 0.47 0.27 0.35 0.27 0.33 0.12 0.41 0.41 0.36 0.49 0.38 0.45 0.29 0.32 0.41 0.54
boat 0.16 0.33 0.37 0.33 0.26 0.32 0.39 0.23 0.25 0.22 0.28 0.08 0.33 0.34 0.32 0.41 0.30 0.38 0.25 0.30 0.35 0.45
bottle 0.13 0.17 0.20 0.18 0.12 0.25 0.25 0.13 0.16 0.09 0.13 0.02 0.14 0.29 0.23 0.33 0.17 0.28 0.17 0.09 0.20 0.27
bus 0.27 0.61 0.58 0.59 0.55 0.63 0.62 0.41 0.42 0.40 0.50 0.13 0.56 0.44 0.41 0.59 0.54 0.64 0.41 0.47 0.53 0.66
car 0.20 0.43 0.41 0.47 0.38 0.45 0.46 0.28 0.35 0.29 0.34 0.08 0.39 0.45 0.41 0.48 0.43 0.50 0.29 0.34 0.44 0.52
cat 0.40 0.76 0.61 0.74 0.72 0.73 0.73 0.45 0.63 0.62 0.66 0.19 0.72 0.54 0.51 0.62 0.63 0.72 0.45 0.68 0.59 0.79
chair 0.19 0.34 0.32 0.39 0.27 0.39 0.41 0.21 0.33 0.17 0.31 0.12 0.33 0.34 0.27 0.42 0.33 0.41 0.24 0.21 0.39 0.51
cow 0.22 0.52 0.50 0.49 0.37 0.52 0.55 0.29 0.42 0.30 0.38 0.13 0.44 0.50 0.45 0.53 0.48 0.58 0.30 0.35 0.46 0.57
PR
table 0.32 0.56 0.50 0.60 0.58 0.53 0.57 0.42 0.44 0.35 0.55 0.08 0.60 0.47 0.47 0.52 0.50 0.61 0.44 0.48 0.45 0.70
dog 0.34 0.71 0.62 0.70 0.63 0.67 0.70 0.41 0.59 0.52 0.59 0.18 0.68 0.55 0.53 0.63 0.57 0.68 0.42 0.60 0.55 0.76
horse 0.28 0.57 0.57 0.56 0.50 0.57 0.58 0.37 0.40 0.37 0.43 0.16 0.50 0.52 0.49 0.57 0.43 0.59 0.38 0.47 0.44 0.61
mbike 0.27 0.51 0.52 0.55 0.47 0.52 0.55 0.31 0.41 0.36 0.42 0.16 0.49 0.49 0.44 0.57 0.42 0.57 0.34 0.43 0.46 0.61
person 0.24 0.37 0.37 0.37 0.32 0.42 0.43 0.26 0.25 0.20 0.28 0.09 0.34 0.48 0.43 0.45 0.33 0.46 0.28 0.22 0.35 0.47
plant 0.20 0.31 0.32 0.33 0.26 0.32 0.35 0.21 0.25 0.16 0.24 0.13 0.27 0.36 0.32 0.43 0.27 0.40 0.25 0.19 0.36 0.44
sheep 0.21 0.47 0.47 0.46 0.33 0.47 0.52 0.26 0.35 0.24 0.37 0.14 0.41 0.45 0.40 0.53 0.42 0.52 0.27 0.29 0.45 0.55
sofa 0.37 0.71 0.57 0.75 0.67 0.70 0.71 0.46 0.59 0.55 0.67 0.10 0.71 0.52 0.50 0.58 0.58 0.71 0.47 0.62 0.56 0.80
August 15, 2017
train 0.34 0.62 0.57 0.66 0.59 0.63 0.65 0.41 0.44 0.41 0.52 0.14 0.61 0.50 0.48 0.56 0.50 0.66 0.41 0.56 0.46 0.68
tv 0.24 0.53 0.55 0.49 0.39 0.59 0.58 0.30 0.51 0.31 0.48 0.19 0.50 0.47 0.38 0.52 0.48 0.56 0.30 0.28 0.53 0.65
14 / 15
Thank you!
Jing Wang
PR
August 15, 2017
15 / 15
