Foundations of Machine Learning Lecture 1

Foundations of Machine Learning Introduction to ML Mehryar Mohri Courant Institute and Google Research [email protected]

Foundations of Machine Learning

page

1

Logistics Prerequisites: basics in linear algebra, probability, and analysis of algorithms. Workload: about 3-4 homework assignments + project (topic of your choice). Mailing list: join as soon as possible.


page 2

Course Material Textbook

Slides: course web page. http://www.cs.nyu.edu/~mohri/ml17


page 3

This Lecture Basic definitions and concepts. Introduction to the problem of learning. Probability tools.


page 4

Machine Learning Definition: computational methods using experience to improve performance. Experience: data-driven task, thus statistics, probability, and optimization. Computer science: learning algorithms, analysis of complexity, theoretical guarantees. Example: use document word counts to predict its topic.


page 5

Examples of Learning Tasks Text: document classification, spam detection. Language: NLP tasks (e.g., morphological analysis, POS tagging, context-free parsing, dependency parsing). Speech: recognition, synthesis, verification. Image: annotation, face recognition, OCR, handwriting recognition. Games (e.g., chess, backgammon). Unassisted control of vehicles (robots, car). Medical diagnosis, fraud detection, network intrusion. Foundations of Machine Learning

page 6

Some Broad ML Tasks Classification: assign a category to each item (e.g., document classification). Regression: predict a real value for each item (prediction of stock values, economic variables). Ranking: order items according to some criterion (relevant web pages returned by a search engine). Clustering: partition data into ‘homogenous’ regions (analysis of very large data sets). Dimensionality reduction: find lower-dimensional manifold preserving some properties of the data. Foundations of Machine Learning

page 7

General Objectives of ML Theoretical questions:

• • •

what can be learned, under what conditions? are there learning guarantees? analysis of learning algorithms.

Algorithms:

• • •

more efficient and more accurate algorithms. deal with large-scale problems. handle a variety of different learning problems.


page 8

This Course Theoretical foundations:

• •

learning guarantees. analysis of algorithms.

Algorithms:

• •

main mathematically well-studied algorithms. discussion of their extensions.

Applications:

•

illustration of their use.


page 9

Topics Probability tools, concentration inequalities. PAC learning model, Rademacher complexity, VC-dimension, generalization bounds. Support vector machines (SVMs), margin bounds, kernel methods. Ensemble methods, boosting. Logistic regression and conditional maximum entropy models. On-line learning, weighted majority algorithm, Perceptron algorithm, mistake bounds. Regression, generalization, algorithms. Ranking, generalization, algorithms. Reinforcement learning, MDPs, bandit problems and algorithm. Foundations of Machine Learning

page 10

Definitions and Terminology Example: item, instance of the data used. Features: attributes associated to an item, often represented as a vector (e.g., word counts). Labels: category (classification) or real value (regression) associated to an item. Data:

• • •

training data (typically labeled). test data (labeled but labels not seen). validation data (labeled, for tuning parameters).


page 11

General Learning Scenarios Settings:

• •

batch: learner receives full (training) sample, which he uses to make predictions for unseen points. on-line: learner receives one sample at a time and makes a prediction for that sample.

Queries:

• •

active: the learner can request the label of a point. passive: the learner receives labeled points.


page 12

Standard Batch Scenarios Unsupervised learning: no labeled data. Supervised learning: uses labeled data for prediction on unseen points. Semi-supervised learning: uses labeled and unlabeled data for prediction on unseen points. Transduction: uses labeled and unlabeled data for prediction on seen points.


page 13

Example - SPAM Detection Problem: classify each e-mail message as SPAM or nonSPAM (binary classification problem). Potential data: large collection of SPAM and non-SPAM messages (labeled examples).


page 14

Learning Stages labeled data

algorithm

prior knowledge

training sample

A(⇥)

features

validation data test sample


A(⇥0 )

parameter selection

evaluation

page 15



page 16

Definitions Spaces: input space X , output space Y. Loss function: L : Y ⇥Y ! R .

• • •

L(b y , y) : cost of predicting yb instead of y .

binary classification: 0-1 loss, L(y, y 0 ) = 1y6=y0 . regression: Y ✓ R, l(y, y 0 ) = (y 0

y)2.

Hypothesis set: H ✓ Y X, subset of functions out of which the learner selects his hypothesis.

• •

depends on features. represents prior knowledge about task.


page 17

Supervised Learning Set-Up Training data: sample S of size m drawn i.i.d. from X ⇥Y according to distribution D : S = ((x1 , y1 ), . . . , (xm , ym )).

Problem: find hypothesis h 2 H with small generalization error.

• •

deterministic case: output label deterministic function of input, y = f (x). stochastic case: output probabilistic function of input.


page 18

Errors Generalization error: for h 2 H , it is defined by

R(h) =

E

(x,y)⇠D

[L(h(x), y)].

Empirical error: for h 2 H and sample S, it is m

X 1 b R(h) = L(h(xi ), yi ). m i=1

Bayes error:

R? =

•

inf

h h measurable

R(h).

in deterministic case, R? = 0.


page 19

Noise Noise:

•

in binary classification, for any x 2 X , noise(x) = min{Pr[1|x], Pr[0|x]}.

•

observe that E[noise(x)] = R⇤ .


page 20

Learning ≠ Fitting

Notion of simplicity/complexity. How do we define complexity?


page 21

Generalization Observations:

• • • •

the best hypothesis on the sample may not be the best overall. generalization is not memorization. complex rules (very complex separation surfaces) can be poor predictors. trade-off: complexity of hypothesis set vs sample size (underfitting/overfitting).


page 22

Model Selection General equality: for any h 2 H , R(h)

best in class

R⇤ = [R(h) R(h⇤ )] + [R(h⇤ ) R⇤ ] . | {z } | {z } estimation

approximation

Approximation: not a random variable, only depends on H. Estimation: only term we can hope to bound.


page 23

Empirical Risk Minimization Select hypothesis set H. Find hypothesis h 2 H minimizing empirical error: b h = argmin R(h). h2H

• •

but H may be too complex.

the sample size may not be large enough.


page 24

Generalization Bounds 

Definition: upper bound on Pr sup |R(h) h2H

b R(h)| >✏ .

Bound on estimation error for hypothesis h0 given by ERM: R(h0 )

R(h⇤ ) = R(h0 )  R(h0 )

b 0 ) + R(h b 0) R(h b 0 ) + R(h b ⇤) R(h

 2 sup |R(h) h2H

R(h⇤ ) R(h⇤ )

b R(h)|.

How should we choose H? (model selection problem)


page 25

error

Model Selection

estimation approximation upper bound ⇤

H=

[

H .

2


page 26

Structural Risk Minimization

(Vapnik, 1995)

Principle: consider an infinite sequence of hypothesis sets ordered for inclusion, H 1 ⇢ H2 ⇢ · · · ⇢ H n ⇢ · · ·

b h = argmin R(h) + penalty(Hn , m). h2Hn ,n2N

• •

strong theoretical guarantees.

typically computationally hard.


page 27

General Algorithm Families Empirical risk minimization (ERM): b h = argmin R(h). h2H

Structural risk minimization (SRM): Hn ✓ Hn+1 , b h = argmin R(h) + penalty(Hn , m). h2Hn ,n2N

Regularization-based algorithms:

0,

b h = argmin R(h) + khk2 . h2H


page 28



page 29

Basic Properties Union bound: Pr[A _ B]  Pr[A] + Pr[B]. Inversion: if Pr[X ✏]  f (✏) , then, for any > 0 , with probability at least 1 , X  f 1( ). Jensen’s inequality: if f is convex, f (E[X])  E[f (X)] . Z +1 Expectation: if X 0 , E[X] = Pr[X > t] dt . 0


page 30

Basic Inequalities Markov’s inequality: if X Pr[X

0 and ✏ > 0 , then ✏]  E[X] ✏ .

Chebyshev’s inequality: for any ✏ > 0 ,

Pr[|X


E[X]|

✏] 

2 X ✏2

.

page 31

Hoeffding’s Inequality Theorem: Let X1 , . . . , Xm be indep. rand. variables with the same expectation µ and Xi 2 [a, b], ( a < b ). Then, for any ✏ > 0, the following inequalities hold:  ✓ ◆ m 2 X 1 2m✏ Pr µ Xi > ✏  exp m i=1 (b a)2



m 1 X Pr Xi m i=1


µ > ✏  exp

✓

2

2m✏ (b a)2

◆

.

page 32

McDiarmid’s Inequality (McDiarmid, 1989)

Theorem: let X1 , . . . , Xm be independent random variables taking values in U and f : U m ! R a function verifying for all i 2 [1, m] , sup |f (x1 , . . . , xi , . . . , xm ) f (x1 , . . . , x0i , . . . , xm )|  ci .

x1 ,...,xm ,x0i

Then, for all ✏ > 0 , h

i

Pr f (X1 , . . . , Xm ) E[f (X1 , . . . , Xm )] > ✏  2 exp


✓

2✏ Pm

2

2 i=1 ci

◆

page 33

.

Appendix


page

34

Markov’s Inequality Theorem: let X be a non-negative random variable with E[X] < 1 , then, for all t > 0 , 1 Pr[X tE[X]]  . t Proof: X Pr[X t E[X]] = Pr[X = x] x tE[X]

 

X

x t E[X]

X x



x Pr[X = x] t E[X]

x Pr[X = x] t E[X]

X 1 =E = . t E[X] t Foundations of Machine Learning

page 35

Chebyshev’s Inequality Theorem: let X be a random variable with Var[X] < 1 , then, for all t > 0, 1 Pr[|X E[X]| t X ]  2 . t Proof: Observe that Pr[|X

E[X]|

t

X ] = Pr[(X

E[X])2

t2

2 X ].

The result follows Markov’s inequality.


page 36

Weak Law of Large Numbers Theorem: let (Xn )n2N be a sequence of independent random variables with the same mean µ and variance Pn 1 and let X n = n i=1 Xi , then, for any ✏ > 0 , lim Pr[|X n

n!1

µ|

2

0 , E[etX ]  e

t2 (b a)2 8

.

Proof: by convexity of x 7! etx , for all a  x  b , e

tx

b  b

x ta x e + a b

a tb e . a

Thus, E[etX ]

E[ bb

X ta ae

+

with, (t) = log( b b a eta + Foundations of Machine Learning

X a tb b a e ] a tb b ae )

=

b b

ta e + a

a tb b ae

= ta + log( b b a +

=e

(t)

,

a t(b a) ). b ae

page 39

Taking the derivative gives: 0

aet(b

(t) = a

b b

a a

b

Note that: (0) = 0 and 00

(t) =

= u(1

b b

ae

t(b

a)

.

a b

a

(0) = 0. Furthermore,

a 2 b a] t(b a) t(b

(b a)2 a) + ↵]2

↵ [(1

0

a)

t(b a)

↵(1 ↵)e = [(1 ↵)e =

t(b ae

a

=a

t(b a)

abe [b bae

a)

↵)e u)(b

(1

t(b a)

↵)e t(b a) (b ↵)e t(b a) + ↵]

+ ↵] [(1 (b a)2 2 a)  , 4

a)2

a . There exists 0  ✓  t such that: with ↵ = b a t2 00 a)2 0 2 (b (t) = (0) + t (0) + (✓)  t . 2 8


page 40

Hoeffding’s Theorem Theorem: Let X1 , . . . , Xmbe independent random variables. Then for Xi 2 [ai , bi ], the following inequalities hold Pm for Sm = i=1 Xi , for any ✏ > 0 ,

Pr[Sm

Pr[Sm

E[Sm ]

E[Sm ] 

✏]  e

2✏2 /

✏]  e

Pm

2✏2 /

2 (b a ) i i i=1

Pm

i=1 (bi

a i )2

.

Proof: The proof is based on Chernoff’s bounding technique: for any random variable X and t > 0 , apply Markov’s inequality and select t to minimize

Pr[X


✏] = Pr[etX

tX E[e ] t✏ e ] . t✏ e

page 41

Using this scheme and the independence of the random variables gives Pr[Sm E[Sm ] ✏] e

t✏

E[et(Sm

=e

t✏

t(Xi ⇧m E[e i=1

(lemma applied to Xi E[Xi ])  e

t✏

=e

choosing t = 4✏/

Pm

i=1 (bi

e

E[Sm ])

]

E[Xi ])

]

m t2 (bi ai )2 /8 ⇧i=1 e P 2 t✏ t2 m i=1 (bi ai ) /8

e

2✏2 /

Pm

i=1 (bi

ai )2

,

ai ) 2 .

The second inequality is proved in a similar way.


page 42

Hoeffding’s Inequality Corollary: for any ✏ > 0 , any distribution D and any hypothesis h : X ! {0, 1}, the following inequalities hold: b Pr[R(h) b Pr[R(h)

R(h)

✏]  e

R(h) 

2m✏2

✏]  e

2m✏2

.

Proof: follows directly Hoeffding’s theorem. Combining these one-sided inequalities yields h

b Pr R(h) Foundations of Machine Learning

R(h)

i

✏  2e

2m✏2

.

page 43

Chernoff’s Inequality Theorem: for any ✏ > 0 , any distribution D and any hypothesis h : X ! {0, 1}, the following inequalities hold: Proof: proof based on Chernoff’s bounding technique.

b Pr[R(h)

(1 + ✏)R(h)]  e

b Pr[R(h)  (1


✏)R(h)]  e

m R(h) ✏2 /3 m R(h) ✏2 /2

.

page 44

McDiarmid’s Inequality (McDiarmid, 1989)

Theorem: let X1 , . . . , Xm be independent random variables taking values in U and f : U m ! R a function verifying for all i 2 [1, m] , sup |f (x1 , . . . , xi , . . . , xm ) f (x1 , . . . , x0i , . . . , xm )|  ci .

x1 ,...,xm ,x0i

Then, for all ✏ > 0 , h

i

Pr f (X1 , . . . , Xm ) E[f (X1 , . . . , Xm )] > ✏  2 exp


✓

2✏ Pm

2

2 i=1 ci

◆

page 45

.

Comments:

• •

Proof: uses Hoeffding’s lemma. Hoeffding’s inequality is a special case of McDiarmid’s with m 1 X |bi ai | f (x1 , . . . , xm ) = xi and ci = . m i=1 m


page 46

Jensen’s Inequality Theorem: let X be a random variable and f a measurable convex function. Then, f (E[X])  E[f (X)].

Proof: definition of convexity, continuity of convex functions, and density of finite distributions.


page 47