Mar 11, 2008 - Predictive Distribution. 35. Outline. ⢠Introduction. ⢠Simple linear regression. ⢠Simple regressi
11/3/2008
Regression (Part I)
[email protected]
The regression problem Example: how does gas consumption depend on external temperature? depend on external temperature? (Whiteside, 1960s). weekly measurements of • average external temperature • total gas consumption (in 1000 cubic feet) • How does gas consumption depend on external temperature? • How much gas is needed for a given temperature ? 2
1
11/3/2008
Linear model
3
Outline • • • • •
Introduction Simple linear regression Simple polynomial regression Maximum likelihood estimation Maximum posterior estimation
4
2
11/3/2008
Is linear the only option?
5
Variable types numerical / interval‐scaled / quantitative where differences and quotients etc. are meaningful, e.g., temperature, size, weight i i h nominal / discrete / categorical / qualitative where differences and quotients are not defined, usually with a finite, enumerated domain, e.g., X := {red, green, blue} ordinal / ordered categorical where levels are ordered, but differences and quotients are not h l l d d b t diff d ti t t defined, usually with a finite, enumerated domain, e.g., X := {small, medium, large}
6
3
11/3/2008
Definitions: predictors and target
7
Definitions: regression classification
8
4
11/3/2008
Outline • • • • •
Introduction Simple linear regression Simple polynomial regression Maximum likelihood estimation Maximum posterior estimation
9
Simple linear regression
10
5
11/3/2008
Parameters
11
Parameter estimation (example)
12
6
11/3/2008
Parameter estimation (example)
13
Parameter estimation (example)
14
7
11/3/2008
Least Squares Estimation (LSE)
15
LSE: proof
16
8
11/3/2008
LSE: proof
17
LSE: example
18
9
11/3/2008
Outline • • • • •
Introduction Simple linear regression Simple polynomial regression Maximum likelihood estimation Maximum posterior estimation
19
Simple polynomial regression
20
10
11/3/2008
Sum‐of‐Squares Error Function
21
0th Order Polynomial
22
11
11/3/2008
1st Order Polynomial
23
3rd Order Polynomial
24
12
11/3/2008
9th Order Polynomial
25
Outline • • • • •
Introduction Simple linear regression Simple regression with polynomials Maximum likelihood estimation Maximum posterior estimation
26
13
11/3/2008
The Gaussian Distribution
27
Gaussian Mean and Variance
28
14
11/3/2008
The Multivariate Gaussian
29
Gaussian Parameter Estimation Likelihood function Likelihood function
30
15
11/3/2008
Maximum (Log) Likelihood
31
Properties of and
32
16
11/3/2008
Polynomial regression re‐visited
33
Maximum Likelihood
Determine Determine by minimizing sum‐of‐squares error, . by minimi ing sum of squares error
34
17
11/3/2008
Predictive Distribution
35
Outline • • • • •
Introduction Simple linear regression Simple regression with polynomials Maximum likelihood estimation Maximum posterior estimation
36
18
11/3/2008
Maximum posterior estimation (MAP)
Determine by minimizing regularized sum‐of‐squares error, .
37
19
