Color Measurement. • Part 2 : MatLab – Intro. • Computer Vision techniques. –
Color measurement in CIE Lab units „polygonal‟. – Color segmentation ...
Computer Vision Based Image Analysis for Color Measurement and Segmentation
Vural Gökmen
Outline • Part 1 : Color in Foods – Intro • Color Measurement • Part 2 : MatLab – Intro • Computer Vision techniques – Color measurement in CIE Lab units „polygonal‟ – Color segmentation
Need to measure color • The color is the first sensation that the consumer perceives and uses as a tool to accept or reject, because the color observation allows the detection of certain anomalies or defects of a product. • Color changes during processing/storage may be indicative!
Color is important variable of foods • We can basically explain the importance of color in the food industry by following steps; – Color as a quality indicator of foods – Determining the defects and anomalities on food surface according the color differences, – Color as a process control variable in food manufacturing.
Color is important variable in “thermally processed foods” • Color can be used as a process control variable in thermal processes. • Browning in foods is a usual consequence of thermal processes such as frying, baking and roasting. • Degree of browning may be indicative !
to
t1
t2
t3
t4
Lo ao bo
L1 a1 b1
L2 a2 b2
L3 a3 b3
L4 a4 b4
Maillard Reaction and Color + amino compound
Aldose Sugar
N-substituted glycosylamine
+ H2O
Amadori rearrangement Amadori rearrangement product (ARP) 1-amino-1-deoxy-2-ketose -2H2O > pH 7
-3H2O ≤ pH 7
> pH 7 Fission products (acetol, diacetyl, pyruvaldehyde, etc)
Reductones -2H +2H
Schiff’s base of hydroxymethylfurfural (HMF) or furfural + H2O
Dehydroreductones
- amino compound
+ amino compound -CO2 Strecker degradation +α amino acid
Aldehydes
HMF or Furfural
Aldols and N-free polymers
+ amino compound
Degree of browning may be indicative for early and/or intermediate MRPs. This makes color measurement critical in thermally processed foods.
+ amino compound
Aldimines and Ketimes
Melanoidins (brown nitrogenous polymers)
Browning
Color is an indicator ! •
•
Color formation, in terms of browning, is mainly a surface phenomenon. This is somehow true for acrylmide formation. This fact simply proposes that these two process variables can be correlated.
However; • •
Color is not homogenously distributed on the surface, Rates of color and acrylamide formation may be similar or different,
~300 µg/kg
~2500 µg/kg
Color Measuring Devices • Color spectrometers
• Portable devices & color mouses
• Computer-vision systems
Color Measurement Traditional color measurement
• Color of foods is usually measured in units L*a*b* which is an international standard for color measurements, adopted by the CIE (Commission Internationale d'Eclairage). • The lightness ranges between 0 and 100 while chromatic parameters (a, b) range between -120 and 120.
CIE Color Space
Color models - conversion Among the color space models, CIE Lab model is the most complete one. It is a device independent model.
Once color is measured in any color space model, it can be converted to other color spaces. It is a built-in function in MatLab.
Conditions affect color measurement
The conditions should be constant during the color measurement to make different results comparable. In color measuring devices, this is instrumentally taken under control.
In computer-vision systems, the conditions should be fixed when taking digital images.
Layout of an image acqusition system Camera (megapixel) Light Source (D65)
Image Transfer (USB) Light Source (D65)
90o
45o
45o
Object
Image Analysis Station
Simple setup
D65 Lamps How different is the color at the end of baking?
ΔE There is no indutry standard for COLOR of processed foods. We, as food scientist, mostly deal with color changes during processing. This means DIFFERENCE or ΔE !
Difference (ΔE) •
Uniform changes of components in the L*a*b* color model aim to correspond to uniform changes in perceived color. So the relative perceptual differences between any two colors in L*a*b* can be approximated by treating each color as a point in a three dimensional space (with three components: L*, a*, b*) and taking the Euclidean distance between them. This Euclidean distance in L*a*b* space is ΔE (often called “Delta E”).
E ( L*2 L*1 ) 2 (a2* a1* ) 2 (b2* b1* ) 2
Thermally processed foods non-homogeneously colored surfaces •
•
Color measuring devices cannot measure the whole surface. To take representative mean values, the measurements should be repeated.
Sample to be measured
specimen hole and sample geometries do not fit !!! Specimen hole
Image processing
>> RGB=imread(„crisp.jpg‟);
>> Imshow(RGB); MatLab >> renklab Lab __Value = 73.7869 7.4724 59.4035
Define the region of interest as you wish !
Result - Perception Similarity
Which „mean‟ value represents the sample better?
Outline • Part 1 : Color in Foods – Intro • Color Measurement • Part 2 : MatLab – Intro • Computer Vision techniques – Color measurement in CIE Lab units „polygonal‟ – Color segmentation
Introduction to Matlab
What is MATLAB? •
MATLAB is a program for doing numerical computation. It was originally designed for solving linear algebra type problems using matrices. It‟s name is derived from MATrix LABoratory.
•
MATLAB has built-in functions for solving problems requiring data analysis, image processing, curve fitting, optimization, and several other types of scientific computations. It also contains functions for 2-D and 3-D graphics and animation.
Matlab Desktop
Desktop Tools (MATLAB v7) • Command Window – type commands
• Workspace – view program variables – clear to clear – double click on a variable to see it in the Array Editor
• Command History – view past commands – save a whole session using diary
• Launch Pad – access tools, demos and documentation
Matlab Files (.m) •
Use predefined functions or write your own functions
•
Reside on the current directory or the search path – add with File/Set Path
•
Use the Editor/Debugger to edit, run
Vector / Matrix • a vector
x = [1 2 5 1]
x = 1
2
5
• a matrix
1
x = [1 2 3; 5 1 4; 3 2 -1]
x =
1 5 3
2 1 2
3 4 -1
Matrices y=x(2,3)
• x(i,j) subscription
y = 4
x = 1 5 3
2 1 2
3 4 -1
• whole row
y=x(3,:)
y = 3
• whole column
y=x(:,2)
y = 2 1 2
2
-1
Calculations at the Command Line MATLAB as a calculator
»
-5/(4.8+5.32)^2 ans = -0.0488 » (3+4i)*(3-4i) ans = 25 » cos(pi/2) ans = 6.1230e-017 » exp(acos(0.3)) ans = 3.5470
Assigning Variables » a = 2; » b = 5; » a^b ans = 32 » x = 5/2*pi; » y = sin(x)
Semicolon suppresses screen output Results assigned to “ans” if name not specified
y = 1
» z = asin(y) z =
1.5708
() parentheses for function inputs
Basic Matrix Operations Matrix Multiplication >> a=[ 2 3 4; 5 6 7]; >> b=[1 2 3]; >> a*b ans = 35 44 53 65 83 101
Array Multiplication >> a= [1 2 3 4; 5 6 7 8]; >> b= [1 2 3 4; 1 2 3 4]; >> c= a.*b c= 1 5
4 9 16 12 21 32
Toolboxes • Collections of functions to solve problems from several application fields. – – – – – – – –
Signal Processing Toolbox Image Processing Toolbox Curve Fitting Toolbox Neural Network Toolbox Fuzzy Logic Toolbox Statistics Toolbox Partial Differential Equations Toolbox And many others…
Image Processing with MATLAB
What is image processing? • Any form of information processing for which the input is an image, such as photographs or frames of video; the output is not necessarily an image, but can be for instance a set of features of the image. • Most image-processing techniques involve treating the image as a two-dimensional signal and applying standard signal-processing techniques to it.
Image Formats • The following image formats are suppoorted by MATLAB: – – – – – –
BMP HDF JPEG / JPG PCX TIFF XWB
Reading the Image with MATLAB • If an image is stored as a JPEG-image on the disc we first read it into Matlab. RGB=imread(‘Filename.jpg’)
• The picture is read as RGB format pixel-by-pixel between 0 - 255.
Case Studies • Case 1 : Color measurement in CIE Lab units – Matlab code „renklab.m‟
• Case 2 : Color segmentation – Matlab code „VectorQuantize.m‟
Renklab.m • Execute “renklab” at Command Prompt 1. Renklab converts the picture into a different format : Normalization between 0 and 1. (Divided each pixel by 255) A= im2double(A)
Renklab.m (cont.)
2. Selection of area in a polygonal shape Z=roipoly(A) 3. Renklab converts
it from RGB form to Lab form cform = makecform('srgb2lab'); lab = applycform(Z,cform);
Result
Calculated means of Lab values >> Lab__value = L
a
b
55.8246 20.8705 53.6507
Mean Lab of all pixels defined in this area !
Possible Implication of Renklab • Renklab measures the surface color non-invasively, or non-destructively. • It can be integrated into food processing lines for online monitoring the changes in color.
VectorQuantize.m • VectorQuantize is a MatLab code that segments any digital images based on predefined color references. • It is developed for the segmentation of fried potato images, to find a correlation with acrylamide level. • In its current form, it assumes that there are three typical regions in a fried potato image.
• These regions are tentatively determined!
Region 3 Region 1
Region 2
Image Segmentation • Read the image : Z=imread(‘Filename.jpg’) • Convert it into double form: Z=im2double(Z) • Before segmentation 4 different reference areas should be selected: – a=yellowish – b=brownish – c=dark brown – d= background
Reference Values a= yellowish
b= brownish
a=imcrop(Z);
b=imcrop(Z);
Select reference yellowish area
Select reference brownish area
d= background
c= dark brown
c=imcrop(Z);
d=imcrop(Z);
Select reference dark brown area
Select reference background area
Mean Values for Reference Areas •
Take double mean of the reference values: >>a=mean(a); (takes the mean value of columns) >>a=mean(a); (takes the mean value of raws)
Mean 1
•
Mean 2
Repeat it for all reference values.
Reference Matrix
Obtaining matrix of reference values >> u=[a(:,:,1) a(:,:,2) a(:,:,3);b(:,:,1) b(:,:,2) b(:,:,3); c(:,:,1) c(:,:,2) c(:,:,3); d(:,:,1) d(:,:,2) d(:,:,3)] >>u = 0.7523 0.5597 0.4966 0.8166
>> u=u' 0.5260 0.2568 0.1922 0.8063
0.0145 0.0069 0.0090 0.7713
u= 0.7523 0.5597 0.4966 0.8166 0.5260 0.2568 0.1922 0.8063 0.0145 0.0069 0.0090 0.7713
Segmented Images >> VectorQuantize(Z,u);
>> 0.7959 79.59 % brownish area
>> 0.1658 16.58 % brownish area
Practical Implications of Segmentation • VectorQuantize can practically be applied to the image any thermally processed food to monitor „browning‟. • The reference mean values are easily modified to find out any potential correlation with process contaminants. • Online application is possible for inspection purpose.
Perceptible Numerical Info !
50% BROWNING
Correlation with acrylamide Case Study : Potato Chips •
There are three different colors (equivalently three different kinds of pixels) in a fried potato image; – bright yellow (region 1) – brownish yellow (region 2) – darker brown (region 3)
Hint ! It was experimentally observed that Region-2 has a high probability of containing the highest concentration of acrylamide. Number of Pixels in Set 2 NA2 Number of Pixels in Entire Image
Region 3 Region 1 Region 2
Correlation with acrylamide Looking at the correlation between NA2 and acrylamide level 6000
Acylamide level, ng/g
5000
y = 5264.4x r = 0.989
4000 3000 2000 1000
FIGURE The correation between NA2 and acrylamide in potato chips
0 0
0.2
0.4
0.6 NA2
0.8
1
Further readings
Further readings
