Apr 8, 1998 - This document discusses the file structure of the database files, and describes .... create_vbdb(uchar **image, uchar **bitmap, int cols, int rows, int id, ... Convert the symbolic information contained in 'vdb' into raster image.
Documentation of the software package for the vehicle database for the vertical-view Ft. Hood images Gang Liu April 8, 1998
1 Introduction This is the documentation for the software package developed for the vehicle database built for the vertical-view Ft. Hood images. The ground-truthing procedure and the logical description of the organization of the information stored in the database can be found in [1]. This document discusses the le structure of the database les, and describes some subroutines and utility programs that can be useful to users.
2 Database le format speci cation A database le can appear in two formats: text format and binary format. In the text format, all information contained is expressed in English text readable by human beings, and can be manipulated easily using a text editor. A binary format database le contains the representation of the same information in binary numbers. It is a more compact representation, but not directly readable to humans. It is mainly for computer programs.
2.1 Binary format speci cation
A binary database le is a sequential binary le which stores the binary representations of all the data elds in the hierarchy of a vb_db type variable (see Appendix) sequentially in the order by which they appear in the vb_db structure. Speci cally, the elds appear as follows: (See [1] for a description of the elds.) id rows cols num_marked_pts marked_pts[0] marked_pts[1] ..... marked_pts[num_marked_pts-1] num_corners corners[0] corners[1] ..... corners[num_corners-1] num_lines lines[0] lines[1] ..... lines[num_lines-1]
1
num_vhcl vhcl[0] vhcl[1] ..... vhcl[num_vhcl-1] num_gaps gaps[0] gaps[1] ..... gaps[num_gaps-1] avg_inside_gray var_inside_gray avg_bound_gray var_bound_gray avg_inside_grad_mag_canny avg_inside_grad_dir_canny var_inside_grad_mag_canny var_inside_grad_dir_canny avg_inside_grad_mag_facet avg_inside_grad_dir_facet var_inside_grad_mag_facet var_inside_grad_dir_facet avg_bound_grad_mag_canny avg_bound_grad_dir_canny var_bound_grad_mag_canny var_bound_grad_dir_canny avg_bound_grad_mag_facet avg_bound_grad_dir_facet var_bound_grad_mag_facet var_bound_grad_dir_facet avg_gap_gray var_gap_gray avg_gap_grad_mag_canny avg_gap_grad_dir_canny var_gap_grad_mag_canny var_gap_grad_dir_canny avg_gap_grad_mag_facet avg_gap_grad_dir_facet var_gap_grad_mag_facet var_gap_grad_dir_facet avg_outer_gray var_outer_gray avg_outer_grad_mag_canny avg_outer_grad_dir_canny var_outer_grad_mag_canny var_outer_grad_dir_canny avg_outer_grad_mag_facet avg_outer_grad_dir_facet var_outer_grad_mag_facet var_outer_grad_dir_facet
where each of the marked_pts is a data chunk with the elds id r c
each of the corners is a data chunk with the elds id r c vhcl_id prec_pt succ_pt prec_corner succ_corner prec_line succ_line inner_angle prec_pt_ndx succ_pt_ndx vhcl_ndx prec_corner_ndx succ_corner_ndx prec_line_ndx succ_line_ndx
each of the lines is a data chunk with the elds id start end length dir_angle prec_line succ_line start_ndx end_ndx prec_line_ndx succ_line_ndx avg_gray var_gray avg_grad_mag_canny avg_grad_dir_canny var_grad_mag_canny var_grad_dir_canny avg_grad_mag_facet avg_grad_dir_facet var_grad_mag_facet var_grad_dir_facet
2
each of the vhcl is a data chunk with the elds id num_marked marked[0] marked[1] ..... marked[num_marked-1] num_corners corners[0] corners[1] ..... corners[num_corners-1] num_lines lines[0] lines[1] ..... lines[num_lines-1] cenr_marked cenc_marked area_marked cenr cenc area marked_ndx[0] marked_ndx[1] ..... marked_ndx[num_marked-1] corners_ndx[0] corners_ndx[1] ..... corners_ndx[num_corners-1] lines_ndx[0] lines_ndx[1] ..... lines_ndx[num_lines-1] boundrect avg_inside_gray var_inside_gray avg_inside_grad_mag_canny avg_inside_grad_dir_canny var_inside_grad_mag_canny var_inside_grad_dir_canny avg_inside_grad_mag_facet avg_inside_grad_dir_facet var_inside_grad_mag_facet var_inside_grad_dir_facet avg_bound_gray var_bound_gray avg_bound_grad_mag_canny avg_bound_grad_dir_canny var_bound_grad_mag_canny var_bound_grad_dir_canny avg_bound_grad_mag_facet avg_bound_grad_dir_facet var_bound_grad_mag_facet var_bound_grad_dir_facet
where boundrect is a structure of the type vb_boundrect, each element of the lists marked, corners, lines, marked_ndx, corners_ndx, lines_ndx is a binary integer. each of the gaps is a data chunk with the elds id vhcl1 vhcl2 line1 line2 r1 c1 r2 c2 r3 c3 r4 c4 cenr cenc area vhcl1_ndx vhcl2_ndx line1_ndx line2_ndx boundrect avg_inside_gray var_inside_gray avg_inside_grad_mag_canny avg_inside_grad_dir_canny var_inside_grad_mag_canny var_inside_grad_dir_canny avg_inside_grad_mag_facet avg_inside_grad_dir_facet var_inside_grad_mag_facet var_inside_grad_dir_facet
where boundrect is a structure of the type vb_boundrect. Although in the above the entries are listed in lines, and the ones in the same line are separated by blank spaces, there is no \line" structure or \space" in the binary le, and each entry follows immediately the previous one. Subroutines save_vbdb() and read_vbdb() are used to create a disk le with this structure and to read such a le into memory, respectively. 3
2.2 Text format speci cation
A text database le is a text le containing only human readable characters in the basic ASCII set. The text in this le is a representation of all the data elds in the hierarchy of a vb_db type variable (see Appendix). The elds appear sequentially in the order by which they appear in the vb_db structure. The line structure is a main feature of text les, and is supported on virtually all computer systems. A text database le consists of a number of lines. We impose some further requirements for a text le to be a valid database le. First, every line in a database le can be either a data entry line or a comment line, and cannot be both. There is no line-continuation mechanism, for example the backslash (n) used in the C programming language and UNIX shell scripts. Therefore, the text format database les can have lines that are much longer than the width of typical text editor display window. Lines may wrap around a few scan lines on certain editors or in certain printouts, but that is only its appearance on those media, and the original line structure should be recognized. We decide to pay the cost of this possible nuisance when displaying or printing the les for simpler program code for handling the les. A comment line is a line whose rst non-empty character, a character other than SPACE or TAB, is the hash sign (#). The comment line is to assist human understanding of the data entries, or to hold some notes. The information contain in comment lines does not a ect the information contained in the database. Hence, any text can be put in a comment line, and will be ignored by computer programs when retrieving the information for the database. There is no correspondence of the information presented in the comment lines in the binary format database le. Any number of comment lines with any content can appear at any line-starting point of the text database le. Any text line in the database le that is not a comment line is taken as a data entry line. The format of a data entry line depends on which data entries it holds, and varies from place to place. However, the format for every speci c data entry in the database is fully speci ed. Also, one data line might contain the data for more than one data entry. Secondly, the elds in the database appear in the same order as they appear in the binary database le. These are now divided into a number of lines. See the documentation for the subroutine rebuild_vbdb() for details.
3 Major subroutines provided in the package This section describes the major subroutines that are provided in the package for basic manipulation of the database. For each subroutine explained are the calling parameters, the action taken by the subroutine, and its possible return value showing the returning status. int
create_vbdb(uchar **image, uchar **bitmap, int cols, int rows, int id, vb_db *vdb)
/* Parameters:
4
image bitmap cols rows id vdb
---> ---> ---> ---> --->
cols ---> rows ---> mk_truth mk_corner mk_line mk_cent mk_inside mk_vbr
vehicle database 2D array of image pixel values; should have been allocated before this function is called; number of columns of the image number of rows of the image ---> mark ground-truth points? 0 for no, others yes ---> mark corner points? 0 for no, others yes ---> mark line segments? 0 for no, others yes ---> mark vehicle centroids? 0 for no, others yes ---> mark inside vehicle? 0 for no, others yes ---> mark vehicle bounding rectangle?
5
mk_gap mk_gbr
---> --->
mk_outer truth ---> corner ---> line ---> cent ---> inside ---> vbr ---> gap ---> gbr ---> outer --->
---> value value value value value value value value value
0 for no, others yes mark the gap regions? 0 for no, others yes mark gap bounding rectangle? 0 for no, others yes mark the outer regions? 0 for no, others yes to be used to mark ground-truth points to be used to mark corner points to be used to mark line segments to be used to mark vehicle centroids to be used to mark the regions inside vehicles to be used to mark vehicle bounding rectangle to be used to mark the gap regions to be used to mark gap bounding rectangle to be used to mark the outer regions
Action: Convert the symbolic information contained in 'vdb' into raster image format. The values of some pixels in the input 'image' are changed if that change is specified by the input parameters. Any combination of the following can be chosen by properly specifying the 'mk_*' variables and their corresponding pixel values to be changed to: The values of the pixels that are on some ground-truth vehicle boundary are changed to 'truth', those decided to be corners changed to 'corner', those decided to be on line segments changed to 'line', the centroids of the vehicles changed to 'cent', any pixel within the vehicle boundaries changed to 'inside', those on the vehicle bounding rectangles changed to 'vbr', any pixel within gap areas changed to 'gap', pixels on the gap bounding rectangles of changed to 'gbr', any pixel in the outer region (defined by the union of the vehicle and gap regions morphologically dilated by a 7x7 structuring element with the origin at the center) changed to 'outer'. All other pixels remain unchanged. Note: Since it is very common that some pixels are common to more than one entities, e.g., pixels that are corner points are also on line segments, and also can be on the bounding rectangle, when more than one of the entities are desired, those pixels can only take the value of one of the entities. No convention of which entity is to be visible is set up here. Check the code to see what happens.
6
Return values: 0 -1 -2 -3 -4 */
success invalid input parameter(s) image buffer size not big enough not enough memory some lower level routines failed
int save_vbdb(char *fname, vb_db vdb) /* Parameters: fname ---> contains the name of the disk file to be created vdb ---> the vehicle database Action: Put the information stored in the variable 'vdb' into a sequential binary disk file whose name is contained in 'fname'. Return values: 0 -1 -2 contains the name of the disk file to be created vdb the vehicle database Action: Free up the memory allocated to store the information in 'vdb'. Return values: None */
void init_empty_vbdb(vb_db *vdb) /* Parameters: vdb ---> the vehicle database Action: Initialize the counters and pointers in the data structure to be 0's and NULL's. Return values: None */
int print_vbdb(FILE *fp, vb_db vdb) /* Parameters: fp ---> the output stream to which the information about the database is to be printed using fprintf() in plain text vdb ---> the vehicle database Action:
8
Print in plain text, which conforms to the format set up in the documentation for rebuild_vbdb(), the information stored in the variable 'vdb' into an output stream, which can be, for example, stdout, stderr, or one associated with a disk file opened by fopen() for writing. Return values: 0 success fprintf() is assumed to be successful and the return value is not checked. */
int rebuild_vbdb(FILE *fp, vb_db *vdb, FILE *msgstrm) /* Parameters: fp ---> the stream from which the information about the database is to be read using fscanf() in plain text vdb the stream to which diagnostic information is to be printed if any error is encountered; use NULL such information if not of interest; Action: Read from an input stream the information for a vehicle database and put the information in the structure 'vdb'. 'fp' should be a stream of plain text characters in the format specified below. If for some reason the program cannot proceed, a brief diagnostic message is printed to 'msgstrm' if it is not NULL. Be advised that this message gives the final fatal reason for the failure of reconstruction, and the actual cause leading to it might (and often) have happened before the location reported in the diagnostics. 1. each line started by a hash sign (#) is treated as a comment line and will be ignored, hence can contain any text characters as comments; comment line can appear at any position of the stream, provided that the entire line is a comment line; 2. the first line that is not a comment line contains an integer number ID for the 'id' field of 'vb_db';
9
the next non-comment line contains the number of columns and rows in the format "%dx%d" of the original image size based on which this database file is constructed; 3. the information about the marked points in the groundtruthed image comes next: 3a. the next non-comment line contains an integer number for the 'num_marked_pts' field of 'vb_db'; 3b. information about the marked points (num_marked_pts of them) follows one after another; 3c. one non-comment line for each marked point; 3d. each non-comment line contain 3 integer numbers in the format "%d (%d,%d)" for the 'id', 'r', 'c' fields in 'vb_ipoint', in that order; 4. the information about the corners comes next: 4a. the next non-comment line contains an integer number for the 'num_corners' field of 'vb_db'; 4b. information about the corners (num_corners of them) follows one after another; 4c. one non-comment line for each corner; 4d. each non-comment line contain 18 numbers, in the format "%d (%e,%e) %d (%d,%d) (%d,%d) (%d,%d) %e \ %d (%d,%d) (%d,%d) (%d,%d)"; (the same as fmtstr_corner); which are taken to be the values for the fields in 'vb_corner'; in the order the fields are arranged in 'vb_corner'; 5. the information about the line segments comes next: 5a. the next non-comment line contains an integer number for the 'num_lines' field of 'vb_db'; 5b. information about the line segments (num_lines of them) follows one after another; 5c. one non-comment line for each line segment; 5d. each non-comment line contain 21 numbers, in the format "%d (%d,%d) (%e,%e) (%d,%d) (%d,%d) (%d,%d) \ (%e,%e) (%e,%e,%e,%e) (%e,%e,%e,%e)"; (the same as fmtstr_line); which are taken to be the values for the fields in 'vb_line'; in the order the fields are arranged in 'vb_line'; 6. the information about the vehicles comes next: 6a. the next non-comment line contains an integer number
10
for the 'num_vhcl' field of 'vb_db'; 6b. information about the vehicles (num_vhcl of them) follows one after another; 6c. 12 non-comment lines for each vehicle; 6d. the first non-comment line contains the 'id' field in 'vb_vhcl'; 6e. the next non-comment line contains an integer number for the 'num_marked' field in 'vb_vhcl'; 6f. the next non-comment line contains 'num_marked' integer numbers, for the ID's of the marked points; 6g. the next non-comment line contains an integer number for the 'num_corners' field in 'vb_vhcl'; 6h. the next non-comment line contains 'num_corners' integer numbers, for the ID's of the corners; 6i. the next non-comment line contains an integer number for the 'num_lines' field in 'vb_vhcl'; 6j. the next non-comment line contains 'num_lines' integer numbers, for the ID's of the line segments; 6k. the next non-comment line contains 6 numbers, in the format "(%e,%e) %d (%e,%e) %e", (the same as fmtstr_vhcl), for the fields 'cenr_marked', 'cenc_marked', 'area_marked', 'cenr', 'cenc', 'area'; 6l. the next non-comment line contains 'num_marked' integer numbers for the indices of the points in the 'marked_pts' list in 'vb_db', whose ID's are in (6f); 6m. the next non-comment line contains 'num_corners' integer numbers for the indices of the corners in the 'corners' list in 'vb_db', whose ID's are in (6h); 6n. the next non-comment line contains 'num_lines' integer numbers for the indices of the line segments in the 'lines' list in 'vb_db', whose ID's are in (6j); 6o. the next non-comment line contains the information for the bounding rectangle of the vehicle; 15 numbers in the format "(%e,%e) (%e,%e) (%e,%e) (%e,%e) %e %e %e (%e,%e) \ %e %e\n", (fmtstr_boundrect); 6p, the gray scale value and gradient statistics; 20 numbers in the format "(%e %e) (%e %e %e %e) (%e %e %e %e) \ (%e %e) (%e %e %e %e) (%e %e %e %e)\n", (fmtstr_20e); 7. the information about the gaps comes next: 7a. the next non-comment line contains an integer number for the 'num_gaps' field of 'vb_db'; 7b. each non-comment line contain 45 numbers, in the format "%d (%d,%d) (%d,%d) (%e,%e) (%e,%e) (%e,%e) (%e,%e) \
11
(%e,%e) %e (%d,%d) (%d,%d) \ (%e,%e) (%e,%e) (%e,%e) (%e,%e) %e %e %e (%e,%e) %e %e \ (%e,%e) (%e,%e,%e,%e) (%e,%e,%e,%e)\n"; (the same as fmtstr_gap); which are taken to be the values for the fields in 'vb_gap'; in the order the fields are arranged in 'vb_gap'; (the field 'boundrect' gets expanded into its component fields in the order they appear in that structure;) 8. finally the overall gray scale value and gradient statistics; 40 numbers in the format "(%e %e) (%e %e %e %e) (%e %e %e %e) \ (%e %e) (%e %e %e %e) (%e %e %e %e) \ (%e %e) (%e %e %e %e) (%e %e %e %e) \ (%e %e) (%e %e %e %e) (%e %e %e %e)\n"; (fmtstr_40e); Return values: 0 success -1 failure in allocating memory to store the information -2 failure in reading from the input stream, most likely because input stream contains data which does not comply with the specified format Bug: A fixed buffer is used for reading in an entire text line from the input stream. The size of the buffer is set by the macro BUFLEN. If the actual length of the text line is greater than this size, the text line will be truncated in reading, and this routine will complain about invalid data format. */
int check_vbdb(vb_db vdb, FILE *fp) /* Parameters: vdb ---> the vehicle database fp ---> the output stream to print diagnostic information to if inconsistency is found; use NULL if such information is of no interest; Action: Check the validity of all the entries in the database as well as that of the relationship among them. If anything invalidity is found, some
12
relevant, fairly complete, diagnostic messages will be printed to the specified output stream to help locate the problem. Return values: 0 all entries and the relationship are valid -1 there are invalid entries and/or relationships */
Upon successful compilation of the package (see Section 5), these subroutines are all available in the library le libvbdb.a, and their function prototypes are declared in libvbdb.h. In writing their own applications using some of the routines described above, the users just need to include the header le libvbdb.h in their C source code les, and make normal function calls. When building the nal binary executables, they need to link their object les to the library le libvbdb.a. Note: Two binary executables pre-compiled on SunOS 4.1.4 systems, facetgrad and ji corner are included in the package and is required by create vbdb() to do the linear facet model based gradient estimation and corner detection of the vehicle boundaries, respectively. No source code for them is included in this package to reduce its size and complexity.
4 Utility programs for database manipulation To facilitate using, studying, and modifying the database constructed from the groundtruthed dataset, a package of software written in the C programming language was developed. The Makefile for building the executable commands of these utilities is given in the Appendix. For all of the utility commands, typing the command name only prints out the usage message as to the meaning of the required command line arguments and how to specify them. The following is a listing of them and a description of their functions. extracts the ground-truth information and calculates the interested statistics from the original and ground-truth images, and puts them in a binary database le, whose format is speci ed in Section 2. See Appendix B for the details of the database construction procedure. Here is the usage message from this utility:
createvbdb
Usage: createvbdb where name of the original image file (PGM); name of the labeled image file (PGM); an integer number to fill the ID field of the database; name of the binary database file to be created.
13
creates an image le in PGM format from a database le. In the image, various combinations of choices among the pixel locations of the marked points, the straight line segments and corners from the tted polygon models for annotated vehicles, the centroids of the vehicles, the inside areas of the vehicles, the bounding rectangles of the vehicles, the gap regions, the bounding rectangles of the gaps, and the area (called outer regions here) not occupied by vehicle and gap regions, are marked using the values speci ed by the user. The purpose of this utility is for visual inspection of the information contained in the database. Here is the usage message from this utility:
drawvbdb
Usage: drawvbdb \ where is the name of the binary vehicle database file; (0~255) is the value to mark boundary points; (0~255) is the value to mark the corners; (0~255) is the value to mark the fitted lines; (0~255) is the value to mark the marked centroid; (0~255) is the value to mark the vehicle area; (0~255) is the value for vehicle bounding rectangles; (0~255) is the value to mark the gap regions; (0~255) is the value for the gap bounding rectangles; (0~255) is the value to mark the outer regions; is the name of the output PGM format image file. Note: Use -1 for any of , , , , , , , , or if you do not want to mark it.
gives a thorough description of the content of the database le in a plain text format. The output text complies to the text format speci cation in Section 2, and hence can be the input to command rebuildvbdb, to be described below, to rebuild the binary database le. The purpose of this utility is three-fold. First, it allows an examination of the total information contained in the binary database, which is unreadable directly to human beings. Secondly, along with printvbdb it allows easy modi cation of the database. The user creates a text format description of a database using printvbdb, and makes the desired modi cations while keeping the text le comply to the format speci cation. Then a binary database le with the modi cations will be obtained from the modi ed text le with rebuildvbdb to be the converter. Thirdly, it is very important to have a way to by-pass the incompatibility of binary data les across di erent computer platforms. Text les have the most versatility in this aspect. Moving a database to a di erent computer platform is then a trivial job using printvbdb and rebuildvbdb.
printvbdb
14
Here is the usage message from this utility: Usage: printvbdb where is the name of the binary vehicle database file.
takes as its input a text stream that conforms to the text format speci cation in Section 2 and creates a binary database le containing all the information supplied by the input. Since the format speci cation guarantees the completeness of all required elds in a database, the created binary database le will have all required elds. The validity of every data entry, especially that of the relationship between data entries, is not examined. However, a totally valid binary database le will be created if all the information in the input text stream is correct. This utility is mainly for working with printvbdb to modify database les. Here is the usage message from this utility:
rebuildvbdb
Usage: rebuildvbdb where is the name of the input file in text format; use minus sign (-) if the input is from stdin; is the name of the output file in binary format.
checks the validity of each and every data entry, and the relationship among entries in a binary database le. Appendix C gives the requirements that a database has to satisfy. If no invalidity is detected, the program will exit silently with status value 0. Otherwise when some invalidity is detected, a fairly complete diagnostic message is printed out for the user to locate and x the problem, and the program will exit with status value -1. Here is the usage message from this utility:
chkvalidvbdb
Function: Examine the validity of binary vehicle database file. If no validity is found, this utility quits silently. Otherwise, on the first invalidity encountered, it prints to the standard output a diagnostic message and quits to the system with exit status -1. Usage: chkvalidvbdb [filename 2] .... where 's are the name of binary vehicle database files.
15
5 Compilation of the source code Most possibly the user will get a compressed UNIX tar le for this software package. The user needs to use uncompress or gunzip to decompress the le if the le name ends in a \.Z" or \.gz", respectively. If the le name ends with a \.tar", it is already decompressed. The les contained in the package are then extracted from the tar- le using the UNIX command tar with options xvf. For a description of the les extracted, see the README le among the extracted les. The GNU C compiler gcc is assumed by default. However, any ANSI compatible C compiler can be used. Modify the macro CC in the le Makefile and set its value to the name of the compiler to be used. The source le can be compiled and a UNIX archive library le libvbdb.a will be built from the object les by issuing the command make. This also builds the binary executables of the utility programs described in Section 4.
References [1] G. Liu et al., \Ground-truthing the vehicles in the vertical-view Ft. Hood images," tech. rep., Intelligent Systems Laboratory, Department of Electrical Engineering, University of Washington, Seattle, WA 98195, Dec 1997.
16
Appendix A C programming language description of the database #ifndef _libvbdb_h_ #define _libvbdb_h_ #include #ifndef _uchar_defined_ #define _uchar_defined_ typedef unsigned char uchar; #endif #ifndef _ushort_defined_ #define _ushort_defined_ typedef unsigned short ushort; #endif #ifndef _uint_defined_ #define _uint_defined_ typedef unsigned int #endif
uint;
#ifndef _ulong_defined_ #define _ulong_defined_ typedef unsigned long ulong; #endif typedef struct { int id, r, c; } vb_ipoint; typedef struct { int id; float r, c; } vb_fpoint; /* structure for describing a corner of a vehicle boundary */ typedef struct { int id; /* identification of the corner */ float r, c; /* location of the corner */ int vhcl_id; /* ID of the vehicle it is located on */ int prec_pt, succ_pt; /* ID's of the preceding and succeeding
17
}
marked vehicle boundary pixels */ int prec_corner, succ_corner; /* ID's of the preceding and succeeding corners on the same vehicle boundary */ int prec_line, succ_line; /* ID's of the line segments joining this corner to 'prec_corner' and 'succ_corner' */ float inner_angle; /* inner angle formed by 'prec_line' and 'succ_line' */ int vhcl_ndx, prec_pt_ndx, succ_pt_ndx, prec_corner_ndx, succ_corner_ndx, prec_line_ndx, succ_line_ndx; /* indices of the vehicle, points, corners, and lines given above */ vb_corner;
/* structure for describing a straight line segment of a vehicle boundary */ typedef struct { int id; /* identification of the line segment */ int start, end; /* ID's of the corners at the two end points */ /* The corners are ordered clockwise, so the inside of the vehicle lies on the "right side" of the line segment. */ float length, dir_angle; /* length and direction angle from 'start' to 'end' */ int prec_line, succ_line; /* ID's of the preceding and succeeding line segments, connected to this line segment at 'start' and 'end', respectively */ int start_ndx, end_ndx, prec_line_ndx, succ_line_ndx; /* indices of the starting and ending corner point and the preceding and succeeding line segments in the respective lists */ float avg_gray, var_gray; /* the average and the variance of the pixel gray scale values along the line segment */ float avg_grad_mag_canny, avg_grad_dir_canny, var_grad_mag_canny, var_grad_dir_canny, avg_grad_mag_facet, avg_grad_dir_facet, var_grad_mag_facet, var_grad_dir_facet; /* the average and the variance of the
18
magnitude and direction angle of the gradient estimated for the pixels along the line segment, from the Canny's edge detector and the facet edge detector, respectively */ }
vb_line;
/* structure for describing the bounding rectangle for a vehicle or a gap */ typedef struct { float r1, c1, r2, c2, r3, c3, r4, c4; /* the locations of the four vertices of the bounding rectangle; organized clockwise; (r1,c1) is the topmost vertex; if there is a tie for the topmost vertex, it is the one that is on the left; */ float length, width, dir; /* 'dir' is the clockwise direction angle (in [0,pi) ) along which the dimension of the bounding rectangle is greater; in the case of a tie (the bounding rectangle is a square), choose the one with smaller angle value; 'length' is the dimension of the bounding rectangle along the direction 'dir' and 'width' is the dimension along the perpendicular direction; */ float cenr, cenc, area, rectangularity; /* (cenr,cenc) is the location of the center of the bounding rectangle, and 'area' is its area; 'rectangularity' is the ratio of the marked area for a vehicle or the area of a gap over the area of its bounding rectangle, i.e., the 'area' here; */ } vb_boundrect; /* structure for describing a vehicle */ typedef struct { int id; /* identification of the vehicle */ int num_marked; /* number of points on the vehicle's ground-truth boundary */ int *marked; /* array of ID's of the points on the vehicle's ground-truth boundary; has 'num_marked' elements */ int num_corners; /* number of corners on the boundary */ int *corners; /* array of the ID's of the corner on the
19
vehicle's boundary; has 'num_corners' elements */ int num_lines; /* number of line segments on the boundary */ int *lines; /* array of the ID's of the straight line segments on the vehicle's boundary; has 'num_lines' elements */ float cenr_marked, cenc_marked; /* centroid of the area within the boundary marked in the ground-truth*/ int area_marked; /* area within the ground-truth boundary */ float cenr, cenc; /* centroid of the area within the fitted polygonal boundary */ float area; /* area within the fitted polygonal boundary */ int *marked_ndx, *corners_ndx, *lines_ndx; /* indices of the marked points, corners, and line segments in the respective lists */ vb_boundrect boundrect; /* the bounding rectangle of the vehicle */ float avg_inside_gray, var_inside_gray; /* average and variance of the pixel gray scale values inside of the marked boundary. (Note: not fitted boundary.) */ float avg_inside_grad_mag_canny, avg_inside_grad_dir_canny, var_inside_grad_mag_canny, var_inside_grad_dir_canny, avg_inside_grad_mag_facet, avg_inside_grad_dir_facet, var_inside_grad_mag_facet, var_inside_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels within the marked boundary, from the Canny's edge detector and the facet edge detector, respectively */ float avg_bound_gray, var_bound_gray; /* average and variance of the pixel gray scale values along the marked boundary. */ float avg_bound_grad_mag_canny, avg_bound_grad_dir_canny, var_bound_grad_mag_canny, var_bound_grad_dir_canny, avg_bound_grad_mag_facet, avg_bound_grad_dir_facet, var_bound_grad_mag_facet, var_bound_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels along the marked boundary, from the Canny's edge detector and the facet edge detector,
20
respectively */ }
vb_vhcl;
/* structure for describing a gap region */ typedef struct { int id; /* identification of the gap */ int vhcl1, vhcl2; /* ID's of the vehicles the gap lies between */ int line1, line2; /* ID's of the line segments between which the gap lies; 'line1' belongs to 'vhcl1', and 'line2' belongs to 'vhcl2' */ float r1, c1, r2, c2, r3, c3, r4, c4; /* the 4 vertices of the gap region; (r1,c1) and (r2,c2) lie on 'line1', and (r3,c3) and (r4,c4) lie on 'line2'; (r1,c1) lies to the left and up of (r2,c2) with left-right taking precedence over up-down; the same rule is followed for (r3,c3) and (r4,c4) */ float cenr, cenc; /* centroid of the gap region */ float area; /* area within the gap region */ int vhcl1_ndx, vhcl2_ndx, line1_ndx, line2_ndx; /* indices of the vehicles and lines in their respective lists */ vb_boundrect boundrect; /* the bounding rectangle of the gap */ float avg_inside_gray, var_inside_gray; /* average and variance of the pixel gray scale values inside of the gap region */ float avg_inside_grad_mag_canny, avg_inside_grad_dir_canny, var_inside_grad_mag_canny, var_inside_grad_dir_canny, avg_inside_grad_mag_facet, avg_inside_grad_dir_facet, var_inside_grad_mag_facet, var_inside_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels within the gap region, from the Canny's edge detector and the facet edge detector, respectively */ } vb_gap; /* structure for describing a database of vehicles */ typedef struct { int id; /* identification of the database */ int rows, cols; /* the dimension of the original image for
21
which this database file is constructed */ num_marked_pts; /* number of points on the ground-truth vehicle's boundaries */ vb_ipoint *marked_pts; /* array of the pixels on the ground-truth boundary of the vehicles; has 'num_marked_pts' elements */ int num_corners; /* number of corners on all boundaries */ vb_corner *corners; /* array of the vehicle boundary corners on all vehicles in the image; has 'num_corners' elements */ int num_lines; /* number of line segments on all boundaries */ vb_line *lines; /* array of all the straight line segments fitted to the vehicle boundaries in the image; has 'num_lines' elements */ int num_vhcl; /* number of vehicles in this database */ vb_vhcl *vhcl; /* array of vehicles; 'num_vhcl' elements */ int num_gaps; /* number of the "gap" area between closely parked in parallel vehicles */ vb_gap *gaps; /* array of the gap regions */ float avg_inside_gray, var_inside_gray; /* average and variance of the pixel gray scale values inside of the marked boundaries. (Note: not fitted boundary.) */ float avg_inside_grad_mag_canny, avg_inside_grad_dir_canny, var_inside_grad_mag_canny, var_inside_grad_dir_canny, avg_inside_grad_mag_facet, avg_inside_grad_dir_facet, var_inside_grad_mag_facet, var_inside_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels within the marked boundaries, from the Canny's edge detector and the facet edge detector, respectively */ float avg_bound_gray, var_bound_gray; /* average and variance of the pixel gray scale values along the marked boundaries. */ float avg_bound_grad_mag_canny, avg_bound_grad_dir_canny, var_bound_grad_mag_canny, var_bound_grad_dir_canny, avg_bound_grad_mag_facet, avg_bound_grad_dir_facet, var_bound_grad_mag_facet, var_bound_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels along the marked boundaries, from the Canny's edge int
22
float
float
float
float
detector and the facet edge detector, respectively */ avg_gap_gray, var_gap_gray; /* average and variance of the pixel gray scale values inside of the gap regions */ avg_gap_grad_mag_canny, avg_gap_grad_dir_canny, var_gap_grad_mag_canny, var_gap_grad_dir_canny, avg_gap_grad_mag_facet, avg_gap_grad_dir_facet, var_gap_grad_mag_facet, var_gap_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels within the gap regions, from the Canny's edge detector and the facet edge detector, respectively */ avg_outer_gray, var_outer_gray; /* average and variance of the pixel gray scale values in regions of non-interest (neither belonging nor located close to either marked regions or gap regions) */ avg_outer_grad_mag_canny, avg_outer_grad_dir_canny, var_outer_grad_mag_canny, var_outer_grad_dir_canny, avg_outer_grad_mag_facet, avg_outer_grad_dir_facet, var_outer_grad_mag_facet, var_outer_grad_dir_facet; /* the average and the variance of the magnitude and direction angle of the gradient estimated for the pixels in regions of non-interest, from the Canny's edge detector and the facet edge detector, respectively */
}
vb_db;
void int int void
init_empty_vbdb(vb_db *vdb); read_vbdb(char *fname, vb_db *vdb); save_vbdb(char *fname, vb_db vdb); free_vbdb(vb_db *vdb);
int int int int
print_vbdb(FILE *fp, vb_db vdb); rebuild_vbdb(FILE *fp, vb_db *vdb, FILE *msgstrm); check_vbdb(vb_db vdb, FILE *fp); draw_vbdb(vb_db vdb, uchar **image, int cols, int rows, char mk_truth, char mk_corner, char mk_line, char mk_cent,
23
int
char mk_inside, char mark_vbr, char mk_gap, char mk_gbr, char mk_outer, char truth, char corner, char line, char cent, char inside, char vbr, char gap, char gbr, char outer); create_vbdb(uchar **image, uchar **bitmap, int cols, int rows, int id, vb_db *vdb);
#endif
24
B Description of the subroutine create vbdb() Inputs:
a 2D array of unsigned char's for the image pixel gray scale values; call it image; a 2D array of unsigned char's for the bitmap of the pixels within the marked
vehicle outer boundaries; non-zeros show the pixels within the boundary; call it bitmap; size of the arrays in terms of numbers of columns and rows; call them cols and rows; an integer number for to be used as the identi cation of the input image; call it id;
Output:
the vehicle database constructed from the inputs; call it vdb;
Action:
The interested information to be obtained from the input images speci ed in [1] is obtained and the elds of the vb_db type variable vdb are lled in with the obtained information.
Procedure:
obtaining the connected components;
allocate the bu er for a 2D array the of size of the image; each element is of type int; call it components; feed bitmap to a connected component analysis subroutine with 8-connectivity with non-zero pixels as the foreground, and let components take back the connected component labeled image; in component 0 is used for the background and consecutive positive integers starting from 1 are used for labels of pixels of the connected components; the number of connected components is also obtained; call it N ; obtaining the gradients; allocate four bu ers for 2D arrays of the size of the image, each element being of type float; call them rdd_gauss, cdd_gauss, rdd_facet, cdd_facet; call directional derivative estimation subroutines for the Gaussian based facet based methods, and let the arrays to store the directional derivative values along the row and column axes from the two estimators; convert the directional derivative into gradient magnitude and direction angle; the direction angle being in radians and fall in the range [?; ); allocate an N -element array, each element being of type vb_vhcl to store the information of the vehicles; this array is vdb->vhcl; 25
initialize three link lists, marked_list,
corner_list, line_list for the marked points, corners, and lines to be empty; for each list, there is a counter keeping track of the number of items in the list; for each connected component in component, nd a point on its boundary to serve as the starting point of its boundary point list; raster scan will be used and the rst point encountered for a connected component is picked up; the location as well as its label is stored in an N -element array start_pts; in the same scan, nd the areas (number of pixels) of all connected components and store them in an N -element array cc_areas; allocate an N -element array of pointers; each element is to be used as an array itself; call this array of pointers inside_locs; for each connected component, use its pointer to allocate an array with the size of its area; the elements of this latter array are of type vb_ipoint, and are used to store all pixel locations belonging to that connected component; also set up a counter for each connected component to keep track of the number of elements already lled in the array for that connected component; FOR n=0 TO N ? 1
BEGIN { follow the boundary of the (n+1)-th connected component starting at start_pts[n]; { { { {
{ {
call this list of 2D locations thismarked; add the points in thismarked to the link list marked_list with n for the id eld; nd corners along thismarked and put them in an array thiscorners; for each of them, ll in the elds prec_pt and succ_pt; nd lines connecting corners in thiscorners and put them in an array thislines; for each of these lines, nd the set of pixel locations along them, and calculate the average and variance of the gradient magnitude and direction; also don't forget to calculate other elds of the structure for a line these corners in the link list corner_list; ll in the preceding and succeeding relationships between corners in thiscorners, between lines in thislines, between corners and lines; only the id part of the relationships are lled in; be careful here; since thiscorners and thislines depend on each other on the ID elds, their ID's and the cross-reference using the ID's have to be assigned together to get things right; since the corners and lines are not put in their nal lists and hence the index of any of them is not speci ed yet, the *_ndx elds have to be lled in later; add the corners and lines in thiscorners and thislines in the link lists corner_list and line_list; the bounding rectangle of the vehicle is obtained by calling bounding() using the data in thismarked; other measurements of the bounding rectangle are also calculated; 26
{ obtain statistics on thismarked; { obtain statistics on inside_locs[n], include area, centroid, average and variance of gradient; { put this vehicle to the array of vehicles; END
now all marked points, corners, and lines are in their respective link lists. An
array is allocated for each of them to hold the information in the link list; these arrays are vdb->marked_pts, vdb->corners, vdb->lines; then the link lists are no longer useful and hence freed; gathering information about the gaps; { initialize an empty link list gap_list, set num_gaps=0, for the gap regions; { FOR i=0 TO vdb->num_lines-2
BEGIN FOR j=i+1 TO vdb->num_lines-1 BEGIN line1=vdb->lines[i]; line2=vdb->lines[j]; IF line1 and line2
come from di erent vehicles, are roughly in parallel, and are close to each other;
THEN BEGIN
this is a tentative gap region; nd the location of the four
vertices of the gap and check that if it signi cantly overlaps with any of the vehicle regions; if it does, it is not a valid gap region; the rest of the operations are skipped and the iteration goes to the next j; this is a valid gap region; note the location of the four vertices of the gap in thisgap; the area and centroid of thisgap is obtained by analytical calculation of the quadrilateral, instead of using pixel locations inside the gap as used below for getting the gray scale value and gradient statistics for the gap; the bounding rectangle of the gap is obtained using the four vertices of the gap; other measurements of the bounding rectangle are also calculated; add thisgap to gap_list; num_gaps=num_gaps+1;
END END END
27
{ allocate two num_gaps-element arrays, with gap_areas being an integer array { {
and gap_locs being a pointer array, for the pixel locations in gap regions; set total_gap_area=0; FOR i=0 TO num_gaps-1
BEGIN
thisgap=gap_list[i]; allocate a memory bu er slightly bigger than the bounding box of
END
the thisgap; (2 pixels larger on every side;) set the pixel values to 1; draw the four straight line segments in the bu er using value 0; run connected component analysis on the bu er with 1 being the foreground and 0 the background; initialize an empty link-list thisgaplocs; pixels having labels 0 or 2 belong to the gap region; add the pixel locations to thisgaplocs, and put the number of the pixels to gap_areas[i]; obtain statistics of gray scale value and gradients on pixel locations in thisgaplocs; allocated gap_areas[i] units for gap_locs[i]; put the elements in thisgaplocs to gap_areas[i]; free thisgaplocs; total_gap_area= total_gap_area+ gap_areas[i];
{ put the elements in gap_list to vdb->gaps and free up the memory for
; get the statistics for the region of non-interest; { get the region of interest (ROI) by dilating by a 7 7 square with the origin at the center to union of pixel locations in inside_locs and gap_locs; { get the set of pixel locations of non-interest by taking out pixel locations in the ROI; { obtain the statistics of gray scale values and gradients on the region of noninterest; put all pieces of information into vdb, free all temporary memory bu ers, and return; gap_list
28
C Validity checking of a database The database le is a logical structure in which various data elds relate to each other in a certain manner. For a database vdb to be valid, its elds have to satisfy the following necessary conditions:
vdb.rows, vdb.cols
have to be positive; have to be non-negative; , both of which have to be less
vdb.num_marked_pts, vdb.num_corners, vdb.num_lines vdb.num_corners vdb.num_lines vdb.num_marked_pts
and has to equal to than or equal to ; for i; j in [0; vdb.num_marked_pts ? 1], { if p=vdb.num_marked_pts[i], (p.r; p.c) 2 [0; vdb.rows ? 1] [0; vdb.cols-1]; { for i 6= j , vdb.marked_pts[i].id 6= vdb.marked_pts[j].id; each element c in vdb.corners has to satisfy the following: { (c.r, c.c) in [0; vdb.rows ? 1] [0; vdb.cols-1]; { indices within bounds: c.vhcl_ndx in [0; vdb.num_vhcl ? 1]; c.prec_pt_ndx in [0; vdb.num_marked_pts ? 1]; c.succ_pt_ndx in [0; vdb.num_marked_pts ? 1]; c.prec_corner_ndx in [0; vdb.num_corners ? 1]; c.succ_corner_ndx in [0; vdb.num_corners ? 1]; c.prec_line_ndx in [0; vdb.num_lines ? 1]; c.succ_line_ndx in [0; vdb.num_lines ? 1]; { indices are correct: c.vhcl_id = vdb.vhcl[c.vhcl_ndx] ; c.prec_pt = vdb.marked_pts[c.prec_pt_ndx].id ; c.succ_pt = vdb.marked_pts[c.succ_pt_ndx].id ; c.prec_corner = vdb.corners[c.prec_corner_ndx].id ; c.succ_corner = vdb.corners[c.succ_corner_ndx].id ; c.prec_line = vdb.lines[c.prec_line_ndx].id ; c.succ_line = vdb.lines[c.succ_line_ndx].id ; { inner_angle 2 [?; ); { for i 6= j , vdb.corners[i].id 6= vdb.corners[j].id; each element l in vdb.lines has to satisfy the following: { l.start_ndx, l.end_ndx have to be in [0; vdb.num_corners ? 1]; 29
{ l.start = vdb.corners[l.start_ndx].id ; l.end = vdb.corners[l.end_ndx].id ; l.start
6= l.end;
{ vdb.corners[l.start_ndx].vhcl_id = vdb.corners[l.end_ndx].vhcl_id ; { l.length > 0; l.dir_angle 2 [?; ); { for i 6= j , vdb.lines[i].id 6= vdb.lines[j].id; { if l1.id 6= l2.id, the unordered pair of start and end points for l1 has to be di erent from that for l2;
{ l.prec_line_ndx, l.succ_line_ndx have to be in [0; vdb.num_lines ? 1]; { l.prec_line = vdb.lines[l.prec_line_ndx].id ;
= vdb.lines[l.succ_line_ndx].id ; { l.avg_gray 2 [0; 255]; l.var_gray > 0; { l.*_mag_* > 0; l.avg_grad_dir_* 2 [?; ); l.var_grad_dir_* > 0; each element v in vdb.vhcl has to satisfy the following: { v.num_marked in [1; vdb.num_marked_pts]; for i in [0; vdb.num_marked_pts ? 1], v.marked_ndx[i] in [0; vdb.num_marked ? 1]; v.marked[i] = vdb.marked_pts[v.marked_ndx[i]].id ; { v.num_corners in [1; vdb.num_corners]; for i in [0; vdb.num_corners ? 1], v.corners_ndx[i] in [0; vdb.corners ? 1]; v.corners[i] = vdb.corners[v.corners_ndx[i]].id ; { v.num_lines in [1; vdb.num_corners]; for i in [0; vdb.num_lines ? 1], v.lines_ndx[i] in [0; vdb.lines ? 1]; v.lines[i] = vdb.lines[v.lines_ndx[i]].id ; { all v.num_marked add up to vdb.num_marked_pts; all v.num_corners add up to vdb.num_corners; all v.num_lines add up to vdb.num_lines; { (v.cenr_marked, v.cenc_marked) and (v.cenr, v.cenc) in [0; vdb.rows ? 1] [0; vdb.cols-1]; { v.area_marked > 0, v.area > 0; { v.avg_*_gray 2 [0; 255], v.var_*_gray > 0; { v.*_mag_* > 0; v.avg_*_dir_* 2 [?; ); v.var_*_dir_* > 0; { for i 6= j , vdb.vhcl[i].id 6= vdb.vhcl[j].id; each element g in vdb.gaps has to satisfy the following: l.succ_line
30
{ g.vhcl1_ndx,g.vhcl2_ndx in [0; vdb.num_vhcl ? 1];
= vdb.vhcl[g.vhcl1_ndx].id ; = vdb.vhcl[g.vhcl2_ndx].id ; { g.line1_ndx,g.line2_ndx in [0; vdb.num_lines ? 1]; g.line1 = vdb.lines[g.line1_ndx].id ; g.line2 = vdb.lines[g.line2_ndx].id ; { (g.r1, g.c1), (g.r2, g.c2) fall on vdb.lines[g.line1_ndx]; (g.r3, g.c3), (g.r3, g.c3) fall on vdb.lines[g.line2_ndx]; (g.r1, g.c1), (g.r2, g.c2), (g.r3, g.c3), (g.r4, g.c4) in that order form a quadrilateral; { (g.cenr, g.cenc) in [0; vdb.rows ? 1] [0; vdb.cols ? 1]; { g.area > 0; { g.avg_inside_gray 2 [0; 255], g.var_inside_gray > 0; { g.*_mag_* > 0; g.avg_*_dir_* 2 [?; ); g.var_*_dir_* > 0; { for i 6= j , vdb.gaps[i].id 6= vdb.gaps[j].id; gdb.avg_*_gray 2 [0; 255], gdb.var_*_gray > 0; g.*_mag_* > 0; g.avg_*_dir_* 2 [?; ); g.var_*_dir_* > 0; g.vhcl1 g.vhcl2
31
