Machine Learning, Module 7

Machine Learning, Module 7

Mattias Ohlsson December 2, 2013

CBBP, Theoretical Physics

Machine Learning

one of many definitions

Machine Learning is the study of computer algorithms that learn and improve through experience (data). CBBP, Theoretical Physics

Machine learning tools are (usually) constructed with the use of a labeled database Actual diagnosis

Database of labeled images

Database of medical images

Criteria

Extraction of features

Expert knowledge Construction of machine learning tool

We talk about data-driven learning CBBP, Theoretical Physics

A few examples

Classification of times series data (Electrocardiogram)


Sequences of letters (protein structure classification)

LSADQISTVQASFDKVKGDPVGILYAVFKADPSIMA KFTQFAGKDLESIKGTAPFETHANRIVGFFSKIIGEL PNIEADVNTFVASHKPRGVTHDQLNNFRAGFVSYM KAHTDFAGAEAAWGATLDTFFGMIFSKM

DIQMTQSPSSLSASVGDRVTITCQASQDIIKYLNWY QQTPGKAPKLLIYEASNLQAGVPSRFSGSGSGTDY TFTISSLQPEDIATYYCQQYQSLPYTFGQGTKLQIT


Medical images (ischemia/infarction diagnosis)


Medical images (finding metastatic hotspots)


The overall function of the machine learning tool

”Grey” box 1.2 45.4 -34.2 1.2 7.7 -30.7 21.4


Machine learning tool

0.12 0 No

Healthy

Outline Monday

Artificial neural networks MLP Model selection and generalization performance K-fold cross validation Bootstrap Self-Organizing Feature Map (SOFM) (Ensemble methods)

Wednesday

Ensemble methods Random forest classification Support vector machines


Artificial neural networks

Very simple model

Input CBBP, Theoretical Physics

∑

Output

The building blocks

Xk

y Output

Input

X0= 1

ωk

Perceptron CBBP, Theoretical Physics

The perceptron is a linear classifier

x2

o

o

o x

x

x

x

x

x x x

o

o o

x x

o

x

x x x

Class 2

x

o

o

o o o

o

x

o

o

o o

Class 1

o

o

x

o

x x

o

o

x x x

o

x x

x

o

x1


Build larger networks for more complex mappings

...

...

... ... ... ...

...

Multi-Layer Perceptron


...

...

...


Activation functions


Hidden nodes

Output nodes when classification Output nodes when function approximation


What can it do?



How to “train” the network?

Training data

Form an error function


Maximum Likelihood Principle gives

Function approximation tasks


Classification problems (two classes)


Classification problems (many classes)


Training is “simply minimizing the error function!

Use your favorite minimization method: Gradient descent (Backpropagation algortithm) Conjugate gradient methods (usually better than GD) Quazi Newton methods Levenberg Marquardt

Can be a complex minimization problem!!


How does the MLP work?

Hidden node output

hidden nodes divides the space into two parts with

as the dividing plane




Interpretation of outputs Start with


If minimization works, complex enough network and large dataset!

For classification problems, one gets

Outputs are probabilities! CBBP, Theoretical Physics

Generalization performance

Generalization performance = the performance of a trained model on new data, i.e. data that was not part of the training procedure.

The goal is to have as large as possible generalization performance


Overfitting can limit the generalization performance


How can we understand overfitting?



Overfitting can be avoided using network regularization

One common approach is weight decay


How to detect overfitting? Training data Data Validation data

Error

Validation error

Training error Number of iterations


How to estimate generalization performance? Use the idea of a validation set

All data

Training

Validation

Hold-out method


Use data better All data Split into K parts

V V V

... V

K-fold crossvalidation CBBP, Theoretical Physics

All data Bootstrap sample 1 Bootstrap sample 2

Bootstrap sample B

Bootstrap


Model selection

Error

Training error

Generalization error (estimate)

α* CBBP, Theoretical Physics

α

Estimate generalization performance with a model that needs model selection

All data Construction data

Split into K parts

Split into M parts

Test

V

Construction Test

V

Construction


Training

... Training

...

Construction

Training V

Construction Test

Training

Test

V

Self-Organizing Feature Maps (SOFM)

“SOFM is a kind of clustering tool”

How many clusters and where? CBBP, Theoretical Physics

Clustering can be difficult


K-means – A simple clustering algorithm


SOFM – some features

●

●

●

Artificial neural network trained using unsupervised learning (Only inputs, no targets) Input space is represented using a discretized low-dimensional (often 2-dim) space. A neighborhood function is used to preserve the topological properties of the input space (see also multidimensional scaling (MDS)).


Winning node A number of simple perceptrons

Define the winning node, as the one with the largest output.


Winning node Update the weights


Neighborhood functions

Note!


Some examples (2-D to discrete 2-D)




