Crowdsourcing Rapid Assessment of Collapsed Buildings ... - MDPI

remote sensing Article

Crowdsourcing Rapid Assessment of Collapsed Buildings Early after the Earthquake Based on Aerial Remote Sensing Image: A Case Study of Yushu Earthquake Shuai Xie 1,2 , Jianbo Duan 1, *, Shibin Liu 1 , Qin Dai 1 , Wei Liu 1 , Yong Ma 1 , Rui Guo 1 and Caihong Ma 1 1

2

*

Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing 100094, China; [email protected] (S.X.); [email protected] (S.L.); [email protected] (Q.D.); [email protected] (W.L.); [email protected] (Y.M.); [email protected] (R.G.); [email protected] (C.M.) University of Chinese Academy of Sciences, Beijing 100049, China Correspondence: [email protected]; Tel.: +86-10-8217-8151

Academic Editors: Roberto Tomás, Zhenhong Li, Gonzalo Pajares and Prasad S. Thenkabail Received: 14 July 2016; Accepted: 9 September 2016; Published: 14 September 2016

Abstract: Remote sensing (RS) images play a significant role in disaster emergency response. Web2.0 changes the way data are created, making it possible for the public to participate in scientific issues. In this paper, an experiment is designed to evaluate the reliability of crowdsourcing buildings collapse assessment in the early time after an earthquake based on aerial remote sensing image. The procedure of RS data pre-processing and crowdsourcing data collection is presented. A probabilistic model including maximum likelihood estimation (MLE), Bayes’ theorem and expectation-maximization (EM) algorithm are applied to quantitatively estimate the individual error-rate and “ground truth” according to multiple participants’ assessment results. An experimental area of Yushu earthquake is provided to present the results contributed by participants. Following the results, some discussion is provided regarding accuracy and variation among participants. The features of buildings labeled as the same damage type are found highly consistent. This suggests that the building damage assessment contributed by crowdsourcing can be treated as reliable samples. This study shows potential for a rapid building collapse assessment through crowdsourcing and quantitatively inferring “ground truth” according to crowdsourcing data in the early time after the earthquake based on aerial remote sensing image. Keywords: crowdsourcing; building collapse assessment; earthquake; aerial image; EM algorithm

1. Introduction Building collapse is one of the most serious types of earthquake damage. Most casualties from earthquakes are associated with collapsing buildings [1]. The extent of buildings damage reflects seismic intensity, which is important information to assess the losses of life and property in an earthquake-hit area [2]. Rapid assessment of collapsed buildings early after the earthquake can be instrumental in search and rescue during an emergency. It is hard to obtain the whole in-situ information of building damage in a short time after the earthquake, because the earthquake-damaged zones are not accessible in most cases. However, the aerial or satellite remote sensing can provide the image of the whole disaster area, making it possible to estimate the building damage of large disaster-affected regions in the early time. Many methods to visually interpret or automatically extract the building damage after the earthquake were proposed based on high-resolution aerial or satellite remote sensing image over the past ten years, which made a great contribution in estimating damage Remote Sens. 2016, 8, 759; doi:10.3390/rs8090759

www.mdpi.com/journal/remotesensing

Remote Sens. 2016, 8, 759

2 of 16

extent of buildings caused by earthquake using remote sensing data. Chiroiu uses post-earthquake Ikonos imagery to assess the collapsed buildings by visual interpretation [3]. Saito et al. demonstrated that more collapsed buildings were recognized using pre- and post-earthquake QuickBird imagery, because the pre-earthquake imagery is a good reference of the building outlines [4]. Vu et al. uses the region-independent edge detection algorithm to detect the collapsed buildings based on Ikonos imagery. The results of “very heavy damage” and “destroyed” are consistent with visual interpretation and site survey [5]. Huyck et al. uses texture change detection algorithm based on pre- and post-earthquake imagery of mono-sensor and multi-sensor, respectively, finding that the results are quite different and only “hardest hit area” is recognized consistently [6]. Hutchinson et al. firstly extracts building boundary based on pre- and post-earthquake satellite imagery, then calculated the boundary compactness index defined as the ratio of the number of boundary pixels in the post- and pre-earthquake house, finally identifies the damage buildings through a threshold [7]. Chen and Hutchinson proposed a probabilistic classification framework by means of a multiclass classifier based on bitemporal satellite images to address the major limitation in past attempts which is the use of deterministic approaches to classify damage levels [8]. Geiß et al. quantitatively evaluates the suitability of multi-sensor remote sensing to assess the seismic vulnerability of buildings showing potential for a rapid screening assessment of large areas [1]. Due to the complex image characteristics of post-disaster ground objects and the limitation of resources, automated damage detection techniques with remote sensing are still in the preliminary stage [9]. The last decade has seen a proliferation of sophisticated sensors and technology capable of capturing, transferring and storing immense amounts of data, like remote sensing image, increasing the importance and demand for fast and reliable methods of analysis [9]. Web2.0 technology has resulted in changes in the way that data are created. Individuals, who have the characteristics of large quantity, flexible time and uncertain location, now provide vast amounts of information to websites and online databases, much of which is spatially referenced [10]. This phenomenon called “crowdsourcing” is the product of the network society, which is an online and distributed pattern of problem-solving and producing. Therefore, remote sensing combined with crowdsourcing was used to quickly and accurately analyze large data sets by creating and leveraging a distributed network of human analysts. Crowdsourcing geographic information for disaster response has become a research frontier [11]. The Virtual Disaster Viewer, which was a pilot project following the May 2008 Wenchuan, China earthquake, provided damage assessments through crowdsourcing by having experts interpret pre- and post-event satellite imagery [9]. After the 2010 earthquake in Haiti, the GEO-CAN initiative utilized crowdsourcing through the recruitment of experts to make critical damage assessments based on high-resolution post-event satellite and aerial imagery [12]. Building upon the GEO-CAN effort in Haiti, damage assessment after the Christchurch 2011 was improved by asking participants, including non-experts, to delineate damaged buildings use a polygonal tool, in order to making crowdsourcing damage assessments of disaster areas faster and more accurate [13]. The above studies have clearly demonstrated the power of crowdsourcing for damage assessment to improve disaster response. As such, it offers substantial advantages, but suffers from a general lack of quality assurance [14]. The participants with different professional background and knowledge level have different understanding of remote sensing image. The interpretation results of the same image may be different. It is necessary to quantitatively evaluate the quality of crowdsourcing data to ensure the accuracy of damage assessment. In this paper, Yushu earthquake is chosen as a case study. A web-based platform is built to collect the post-earthquake building damage assessment results contributed by public participants. High-resolution aerial remote sensing images are used due to the advantage of being captured and processed faster and higher spatial resolution than satellite imagery [15]. The problem we focus on is damage “extent” identification for buildings which is relatively straightforward and fast by means of RS images, instead of damage “level” assessment which is based on ground evaluations requiring a considerable amount of time and effort [16]. The next section describes the details of


3 of 16

study The8,data Remotearea. Sens. 2016, 759 used for this research and overview of processing are introduced in the3 third of 17 section. The probabilistic model is applied to estimate individuals’ error-rates and infer ground truth. experiment results areare presented in in thethe fourth The experiment results presented fourthsection. section.Following Followingthe theresults, results,some somediscussion discussionisis providedregarding regarding accuracy of algorithm EM algorithm compared the “majority” and the provided thethe accuracy of EM compared to the to “majority” methodmethod and the variation variation distribution of assessment results among participants on each In building. Inthe addition, the distribution of assessment results among participants on each building. addition, features of features of each crowdsourcing-derived damage type arewhich analyzed, which can beasregarded reliableto each crowdsourcing-derived damage type are analyzed, can be regarded reliable as samples samples to train machine learning toobjects recognize objects Our of interest. Our main conclusions are presented train machine learning to recognize of interest. main conclusions are presented in the final in the final sectionwork and future work is proposed. section and future is proposed. 2.2. Materials Materials and Methods 2.1. 2.1.Study StudyArea Areaand andData Data On magnitude earthquake occurred near Yushu, China, at 7:49 a.m.a.m. locallocal time. On1414April April2010, 2010,a 7.1 a 7.1 magnitude earthquake occurred near Yushu, China, at 7:49 The epicenter was located at 33.1 degrees north latitude, 96.6 degrees east longitude and focal depth time. The epicenter was located at 33.1 degrees north latitude, 96.6 degrees east longitude and focal was 14 km. terrain mainlyismountainous, with an average 4493 m. of The earthquake depth was The 14 km. Theisterrain mainly mountainous, with anelevation average of elevation 4493 m. The caused a large number of casualties collapsed The site survey data of Chinese scholars earthquake caused a large number ofand casualties andhouses. collapsed houses. The site survey data of Chinese after the earthquake demonstrate that houses in houses the central area of Jiegu are Town mainlyare with brick scholars after the earthquake demonstrate that in the central areaTown of Jiegu mainly and concrete structure, other while housesother are mainly brick and structure. of theSome brick with brick and concretewhile structure, houses with are mainly withcivil brick and civilSome structure. of the brick and concrete in thearea townsuffered center area suffered serious and all and concrete structures in structures the town center serious damage, anddamage, almost all thealmost brick and the brick and civil structures thesouthern western regions and southern of the towndamaged were totally damaged civil structures in the western in and of theregions town were totally [17]. Figure 1 [17]. Figure below shows location Jiegu the Town, where the earthquake happened. below shows1the location of the Jiegu Town,ofwhere earthquake happened.

Figure1.1. Maps Maps about about administrative administrative areas areas and terrains: terrains: (a) Figure (a) the the geographical geographical location location of of Qinghai Qinghai Province,China; China; the geographical location of County Yushu in County inProvince; Qinghai (c) Province; (c) the Province, (b) (b) the geographical location of Yushu Qinghai the geographical geographical location of Jiegu Town in Yushu County; and (d) the topographic map of Jiegu Town8 location of Jiegu Town in Yushu County; and (d) the topographic map of Jiegu Town made by Landsat made by Landsat 8 OLI image (true color). OLI image (true color).

The data used for this application are a 0.4 m resolution multispectral aerial image of the The data used for this application are a 0.4 m resolution multispectral aerial image of the damaged damaged zones of Jiegu Town captured on 14 April 2010, provided by Institute of Remote Sensing zones of Jiegu Town captured on 14 April 2010, provided by Institute of Remote Sensing and Digital and Digital Earth, Chinese Academy of Sciences (Figure 2). Earth, Chinese Academy of Sciences (Figure 2).

Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759

4 of 16 4 of 17

Figure 2. The aerial image on 14 14April April2010, 2010,which which was severely Figure 2. The aerial imageofofJiegu JieguTown Town captured captured on was thethe areaarea severely damaged by the earthquake. damaged by the earthquake.

Methods 2.2. 2.2. Methods 2.2.1. Architecture Processing 2.2.1. Architecture of of Processing The architecture consists of four parts, which are, in order, basic imagery preparation, damage The architecture consists of four parts, which are, in order, basic imagery preparation, assessment collection, data quality evaluation and damage map export (Figure 3).

damage assessment collection, data quality evaluation and damage map export (Figure 3). (1) Basic imagery preparation

(1) Basic imagery preparation

In this part, we take the post-event high resolution aerial image (Section 2.1 mentioned) as basic In this part, wecrowdsourcing take the post-event high use resolution aerial imagecollapsed (Section buildings. 2.1 mentioned) as basic imagery, which participants to visually interpret We obtain imagery, whichthat crowdsourcing participants usestudy to visually interpret collapsed buildings. We obtain two images cover the different part of the area, respectively, but having the overlapping twozone. images that cover different ofimage the study area, respectively, but having the overlapping To generate thethe entire region part image, registration, mosaic and clipping are applied to the two we the obtain. zone. Toimages generate entire region image, image registration, mosaic and clipping are applied to the

two images we obtain. (2) Damage assessment collection (2) Damage assessment collection image online that is accessible to participants. Each visible We publish the pre-processed building in the image is assigned one of the damage types according to the observed damage.

We publish the pre-processed image online that is accessible to participants. Each visible building (3) Data quality evaluation in the image is assigned one of the damage types according to the observed damage. Because of theevaluation different professional background of participants, the quality of the data provided (3) Data quality by the individual is uneven. The term crowdsourcing has three distinct meanings proposed by Because of[14]. theThe different professional of participants, thereferring quality of thea data provided Goodchild first meaning refersbackground to the solution of a problem by it to number of by the individual is uneven. The term crowdsourcing has three distinct meanings proposed people, without respect to their qualifications. The second meaning refers to the ability of a group to by Goodchild The first refers to the solution of a problem referring it refers to a number validate[14]. and correct themeaning errors that an individual might make. The thirdby interpretation to the of people, without respect their qualifications. second meaning the ability of athe group ability of the crowd to to converge on the truth. WeThe could not get the field refers data into a short time after earthquake. weerrors could that inferan ground truth from subjective of interpretation the post-event highto validate andHowever, correct the individual might make. labeling The third refers to resolution aerial image participants. In truth. the next the data details how we useafter the ability of the crowd toby converge on the Wesection, could we notintroduce get the field inofa short time maximum likelihood estimation, Bayes’ theorem and EM algorithm to estimate the ground truth and the earthquake. However, we could infer ground truth from subjective labeling of the post-event individual error rate. high-resolution aerial image by participants. In the next section, we introduce the details of how we

use maximum likelihood estimation, Bayes’ theorem and EM algorithm to estimate the ground truth (4) Damage map export and individual error rate. In the last part, the results integrated by the previous part are visualized, generating a damage map that shows theexport spatial distribution of damage types. The damage distribution map is generated (4) Damage map with individual buildings rendered with colors representing the type of damage.

In the last part, the results integrated by the previous part are visualized, generating a damage map that shows the spatial distribution of damage types. The damage distribution map is generated with individual buildings rendered with colors representing the type of damage.


5 of 16


5 of 17

Figure Thearchitecture architecture of Figure 3. 3.The ofprocessing. processing.

The process above enables assessing the damage of buildings rapidly and flexibly through

The process above enables assessing the damage of buildings rapidly and flexibly through crowdsourcing without considering the constraint conditions when using algorithm to process crowdsourcing without considering the conditions when using algorithm tointo process image. Each participant is regarded as constraint a “classifier”, which classifies post-disaster buildings three image. categories according to human’s understandingwhich of the image, and then we evaluatebuildings the accuracy of three Each participant is regarded as a “classifier”, classifies post-disaster into each person and integrate the multiple people’s results into a final reliable result. categories according to human’s understanding of the image, and then we evaluate the accuracy of each person and integrate the multiple people’s results into a final reliable result. 2.2.2. Probabilistic Model

Note thatModel there are W participants to assess I buildings, which may be damaged using K damage 2.2.2. Probabilistic types. It is assumed that all responses given by a single participant are independent and all the

Note that there are Windependently. participants to assess I buildings, may bethe damaged usingmore K damage participants interpret In addition, a participantwhich may interpret same building (all ) responses given by a single participant are independent and all the types. than It is once. assumed that Note that (k = 1,…, K; l = 1,…,K; w = 1,…, W) are the probability that a participant w ( ) participants interpret In addition, participant the same (i = 1,…, I; lbuilding = 1,…,K; more will label l given kindependently. is the true type, which are calledathe individualmay error interpret rate. (w)

(k = 1,…, K) are the w = Note 1,…,W) areαthe number of times participant w label building i as l, and than once. that kl (k = 1, . . . , K; l = 1, . . . , K; w = 1, . . . , W) are the probability that a participant probability that the true damage type of building is k. Let

(k = 1,…, K) be a binary (w) variable of

w will label l given k is the true type, which are called the individual error rate. nil ≠ (it),=namely, 1, . . . , I; l = 1, building i. If t is the true damage type of building i, then = 1 and = 0 (k . . . , K; w = 1, . . . , W) are the number of times participant w label building i as l, and p 1, . . . , K) k (k =and p( = 1) = p . We follow a general model for subjective labeling originally proposed by Dawid are the probability that the true damage type of building is k. Let G (k = 1, . . . , K) be a binary variable Skene [18] and apply it to the building damage labeling problem. The ik data from all participants are assumed and all the true types of buildings assumed to 0 be(kavailable. of building i. Iftot be is independent the true damage type of damage building i, then Git = are 1 and Gik = 6= t), namely, the fullfor data is p( Gik =Generally, 1) = pk .the Welikelihood follow afunction generalfor model subjective labeling originally proposed by Dawid and ( ) Skene [18] and apply it to the building damage∏ labeling problem. The data from all participants are (1) ∏ ∏ ∏ ( ( )) . assumed to be independent and all the true damage types of buildings are assumed to be available. Using maximumfunction likelihood estimation, and we Generally, the likelihood for the full data is obtain estimators

∏

I i =1

∏

K k =1

{ pk ∏

W w =1

∏

K l =1

Gik

(w)

(w) nil (αkl )

}

.

(1)

Using maximum likelihood estimation, and we obtain estimators (w)

(w) αˆ kl

=

∑i Gik nil

(w)

∑l ∑i Gik nil

.

(2)


6 of 16

When pk (k = 1, . . . , K) are unknown, these can be estimated: pˆ k =

∑i Gik . I

(3)

At this point, the true damage types of buildings are unknown. We using Bayesian theory to estimate the binary variable Gik (k = 1, . . . , K), p ( Gik = 1| data) =

p (data | Gik = 1 ) p ( Gik = 1) p (data | Gik = 1 ) p ( Gik = 1) = K . p (data) ∑t=1 p (data | Git = 1 ) p ( Git = 1)

(4)

Therefore, (w)

(w) nil

K ∏W w=1 ∏l =1 ( αkl )

p ( Gik = 1| data) =

pk

.

(w)

∑tK=1

∏W w =1

∏lK=1

(w) nil (αtl )

(5)

pt

Then, we use EM algorithm for finding maximum likelihood estimates of parameters in the model above, due to the dependency of the hidden variables Gik . EM algorithm is short for Expectation Maximization algorithm, which was described by Dempster et al. in 1977 [19]. It is an iterative optimization method for maximum likelihood estimation of parameters, which can estimate the parameters from incomplete data set. In this problem, we treat Gik as missing data then the conditions of the EM algorithm are satisfied. The iterative procedure is as follows: 1. 2. 3. 4.

Give initial estimates of the Gs. Use Equations (2) and (3) to obtain estimates of the ps and αs. Use Equation (5) and the estimates of the ps and αs to calculate new estimates of the Gs. Repeat Steps 2 and 3 until the results converge. In Step 1, we use the equation below to calculate initial estimates of Gs, (w)

Gˆ ik =

∑w nik

(w)

∑w ∑l nil

3. Experiment Results Our project asked the participants to classify the post-earthquake damage buildings into one of three damage types: (1) basically intact; (2) partially collapsed; and (3) completely collapsed. These type numbers are used in subsequent tables. Here, the Yushu earthquake case study was selected to illustrate the results. The experiment area was a sub-region of Jiegu Town, which had visible various types of damage extent, shown in Figure 4, and contained 3456 data points labeled by 27 volunteers, describing the damage buildings at 127 locations. As can be seen in Figure 4, “basically intact”, “partially collapsed” and “completely collapsed” are represented by green, yellow and red points, respectively. The system consists of a database for damage assessment accessed through a browser-based interface built using the ArcGIS API. Data from the participants are collected in the browser and transferred into the database through AJAX and PHP. One selects a damage type and then draws a point on the corresponding building. When a participant assesses a building’s damage, the record is stored along with the longitude and latitude of the point he draw, the damage type he select and his id. The data we collected were used later to estimate the error-rate of each participant in identifying the damage extent of each building and infer the ground truth. Figure 5 gives the variation tendency of marginal probabilities of the three damage types with the iteration of EM algorithm. As shown in Figure 5, the results converge when the iteration is


7 of 16

12 times. Table A1 (see Appendix A) gives the estimates of the individual error-rates (α) of the 27 participants. Remote Sens. 2016, 8,The 759 diagonal elements of each matrix are the estimate of the probability of a7 correct of 17 allocation by a participant. Table A2 (see Appendix A) gives the estimated probabilities for the Gs each building. ForFor most buildings, the the posterior probability is 1isfor oneone damage type, andand thethe for each building. most buildings, posterior probability 1 for damage type, consensusappears appears obvious. obvious. consensus Remote Sens. 2016, 8, 759

7 of 17

each building. For most buildings, the posterior probability is 1 for one damage type, and the consensus appears obvious.

Figure4. 4. The The4. experiment area and the location points, which were contributed Figure experiment areaarea andand thethe location of the 3456data data points, which were contributed Figure The experiment locationof ofthe the3456 3456 data points, which were contributed by by by 27 participants online. In the map, the green, yellow and red points indicate the damage type of of 27 participants online. In the map, the andred red points indicate the damage 27 participants online. In the map, thegreen, green, yellow yellow and points indicate the damage type oftype “basically intact”, “partially collapsed” and “completely respectively. “basically intact”, “partially collapsed” and “completelycollapsed”, collapsed”, respectively. “basically intact”, “partially collapsed” “completely collapsed”, respectively.

5. The red, blue andlines greendemonstrate lines demonstrate the fluctuation of marginal probability of Figure 5.Figure The red, blue and green the fluctuation of marginal probability of “basically “basically intact”, “partially collapsed” and “completely collapsed”, respectively, with the increase of intact”, “partially collapsed” and “completely collapsed”, respectively, with the increase of the number the number of iterations. When the number of iterations reaches 12, EM algorithm converges. 5. The red, the blue and green lines demonstrate of marginal probability of ofFigure iterations. When number of iterations reaches 12, the EM fluctuation algorithm converges. “basically intact”, “partially collapsed” and “completely collapsed”, respectively, with the increase of the number of iterations. When the number of iterations reaches 12, EM algorithm converges.

Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759

8 of 16 8 of 17

4. 4.Discussion Discussion OfOf thethe 3456 damage assessments received for for the the experiment region, we find that that “basically intact” 3456 damage assessments received experiment region, we find “basically annotations made up 52.14%, “partially up 34.64% and made intact” annotations made up 52.14%,collapsed” “partiallymade collapsed” made up“completely 34.64% andcollapsed” “completely upcollapsed” 13.22%. There no13.22%. clear bias towards one or two damage if using the EMS-98 madeisup There is no clear bias towards onetypes. or twoHowever, damage types. However, if using EMS-98 scale, the distribution of annotations overall bias to assess a building as or scale, thethe distribution of annotations reveals an overallreveals bias toan assess a building as “No Damage” “No Damage” [13]. In order demonstrate thealgorithm advantageinofterms EM algorithm in terms “Destroyed” [13].orIn“Destroyed” order to demonstrate the to advantage of EM of inferring ground of inferring ground true, we make a comparison with “majority” method. Figure 6a,b shows theEM true, we make a comparison with “majority” method. Figure 6a,b shows the assessment results of assessment results of EM algorithm and “majority” method, respectively. algorithm and “majority” method, respectively.

Figure thethe results between the two methods:methods: (a) the result EMresult algorithm; Figure 6.6. Comparison Comparisonofof results between thedifferent two different (a)ofthe of EM and (b) the result of “majority” method, which are generated with individual buildings rendered with algorithm; and (b) the result of “majority” method, which are generated with individual buildings colors representing type of damage. rectangles represent buildings,represent and green,the yellow and rendered with colorstherepresenting the The type of damage. Thethe rectangles buildings, red indicate “basically “partially collapsed” and “completely collapsed”, respectively. The and green, yellow and redintact”, indicate “basically intact”, “partially collapsed” and “completely collapsed”, rectangles with id numberwith on them have the different results between the two results methods. respectively. Thethe rectangles the id number on them have the different between the

two methods.

As shown in Figure 6, there are 11 buildings that have different results between the two methods: 21, 24, 29, 33, 61, 66, 67, 75, 78, 105 and 107. The first nine buildings are completely collapsed in EM As shown in Figure 6, there are 11 buildings that have different results between the two methods: results while are partially collapsed in “majority” method. The No. 105 building is basically intact in 21, 24, 29, 33, 61, 66, 67, 75, 78, 105 and 107. The first nine buildings are completely collapsed in EM results while is partially collapsed in “majority” method. The situation of No. 107 building is EM results while are partially collapsed in “majority” method. The No. 105 building is basically opposite to No. 105. The “majority” method does not take into account the accuracy of the intact in EM results while collapsed “majority” The situation No. 107 participants, and EM may is notpartially choose the majorityindamage types method. as the final result of one of building building is opposite to No. 105. The “majority” method does not take into account the accuracy


9 of 16

of the participants, and EM may not choose the majority damage types as the final result of one building due to the low accuracy of participants who make the assessment. For example, the No. 33 building received the assessment results of 26 participants. Among them, there are 19 participants who gave the “partially collapsed” result and six participants who assessed the No. 33 building as “completely collapsed”. Consequently, the “majority” method regards the No. 33 building as “partially collapsed”. According to Table A1, we calculate the average accuracy of 19 participants who label as “partially collapsed” when the true is partially collapsed and the average accuracy of six participants who label as “completely collapsed” when the true is completely collapsed. The calculation results are shown in Table 1. We also calculated the corresponding incidence defined as the product of individual accuracy and marginal probability of damage type, as seen in Table 2. Obviously, the average accuracy of the latter is larger than the former and so is the average incidence, indicating that the participants who label the No. 33 building as “completely collapsed” have more “weight”. Table 1. The accuracy of 19 participants who label No. 33 building as “partially collapsed” and 6 participants who label No. 33 building as “completely collapsed”. The averages are calculated in the last line of the table. Participant ID

Accuracy of 2

1 2 3 4 5 6 8 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 26 27 Average

0.5021 0.2699 0.3825

Accuracy of 3

0.6200 0.6806 0.7419 0.5761 0.8492 0.6132 0.8465 0.7924 0.6200 0.5246 0.6887 0.5912 0.6015 0.5011 0.7608 0.7306 0.7602 0.1905 0.3056 0.9523 0.2712 0.5359 0.5570

0.7211

To demonstrate the variation distribution of assessment results that participants give on each building, the percentage of each damage type on each building is presented in Figure 7, in which the building ids are sorted by the percentage of “basically intact” from small to large. Besides, the standard deviation of participants’ assessment results on each building is shown in Figure 8. No clear agreement between participants on each building apart from the No. 99 that all participants label as “basically intact”. This is because participants with different professional background have different cognition, or due to the limitation of image inherent characteristics such as spatial resolution and angle of view. Whatever limitations the professional faces naturally also apply to volunteers [20]. However, no obvious bias towards extreme value indicates that a majority of participants worked without malice.


10 of 17

Table 2. The incidence of 19 participants who label No. 33 building as “partially collapsed” and 6 Remoteparticipants Sens. 2016, 8,who 759 label No. 33 building as “completely collapsed”. The averages are calculated in the 10 of 16

last line of the table. Table 2. The incidence ofParticipant 19 participants who label of No. buildingof as3“partially collapsed” and ID Incidence 2 33 Incidence 6 participants who label No. 33 building as “completely 1 0.1030 collapsed”. The averages are calculated in the last line of the table. 2 0.0553 3 0.0784 Participant ID Incidence of 2 Incidence 4 0.1024of 3 51 0.1395 0.1030 0.0553 62 0.1521 3 0.0784 8 0.1181 4 0.1024 95 0.1402 0.1395 10 0.1257 6 0.1521 8 0.1181 11 0.1736 9 0.1402 12 0.1625 10 0.1257 13 0.1024 11 0.1736 14 0.0866 12 0.1625 15 0.1412 13 0.1024 14 0.0866 16 0.1212 15 0.1412 17 0.1233 16 0.1212 18 0.1027 17 0.1233 20 0.1560 18 0.1027 20 0.1560 21 0.1498 21 0.1498 22 0.1255 22 0.1255 23 0.0391 23 0.0391 24 0.0627 24 0.0627 25 0.1572 25 0.1572 26 0.0556 26 0.0556 27 0.1099 27 0.1099 Average 0.1142 0.1190 Average 0.1142 0.1190

In remote “ground truth” datadata are often used used as theas basis training pattern remotesensing sensingapplications, applications, “ground truth” are often thefor basis for training recognition algorithms to detect of interest [21]. Many semiautomatic techniques have have been pattern recognition algorithms to objects detect objects of interest [21]. Many semiautomatic techniques designed to exploit Earth Observation (EO) data for earthquake damage assessment to the maximum been designed to exploit Earth Observation (EO) data for earthquake damage assessment to the possible extent, yet extent, visual inspection remainsstill theremains best way achieve results [16]. maximum possible yet visual still inspection thetobest waymeaningful to achieve meaningful Combined multiplewith participants, probability model-derived final results are used as theare reliable results [16].with Combined multiple participants, probability model-derived final results used samples to analyze the features of each damage type. as the reliable samples to analyze the features of each damage type.

Figure 7. The on each each building. building. The bars Figure 7. The percentage percentage of of each each damage damage type type on The green, green, yellow yellow and and red red bars represent the percentage of “basically intact”, “partially collapsed” and “completely collapsed”, represent the percentage of “basically intact”, “partially collapsed” and “completely collapsed”, respectively, oneach eachbuilding. building.The Thebuilding buildingids idsdepicted depictedbybyX-axis X-axisare areinin ascending order according respectively, on ascending order according to to percentage of “basically intact” each building. thethe percentage of “basically intact” on on each building.

Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 8, 759 Remote 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 8, 759 Remote Sens. 2016, 759 Remote Sens. 2016, 8, 759

11 of 16 11 of 17 1111 of 17 11 of 1717 11 of 17 11 of 17 of 17 11 of of 17 11 of 17 11 of 17 11 of 17 11 of 17 11 of 17

Figure 8. The standard deviation of participants’ assessment results on each building. The building Figure 8. The standard deviation of participants’ assessment results on each building. The building Figure 8. standard deviation of participants’ assessment results on each building. The building Figure 8.The The standard deviation of participants’ assessment on each building. The building Figure 8.The The standard deviation of participants’ assessment results on each building. The building Figure 8.8. The standard deviation participants’ assessment results building. The building Figure 8. The standard deviation of participants’ assessment results on each building. The building Figure standard deviation of participants’ assessment results on each building. The building Figure 8. The standard deviation of participants’ assessment results on each building. The building Figure 8. The standard deviation ofof participants’ assessment results onon each building. The building Figure 8. standard deviation of participants’ assessment results on each building. The building Figure 8. The standard deviation of participants’ assessment results on each building. The building Figure 8.The standard deviation of assessment results building. The building Figure 8.The The standard deviation ofparticipants’ participants’ assessment results oneach each building. The building ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted byby X-axis are also inin ascending order according toto the percentage ofof “basically intact” onon ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted by X-axis are also in ascending order according to the percentage of “basically intact” on ids depicted are also order according percentage intact” ids depicted byX-axis X-axis are also inascending ascending order according tothe the percentage of“basically “basically intact” on each building. each building. each building. each building. each building. each building. each building. each building. each building. each building. each building. each building. each building. each building.

We select some samples of each damage type from the results of EM algorithm, as shown in We select some samples of each damage type from the results of EM algorithm, as shown in We select some samples of each damage type from the results of EM algorithm, as shown in We select some samples ofof each damage type from the results ofof EM algorithm, asas shown inin We select some samples ofeach each damage type from the results ofEM EM algorithm, asshown shown in We select some samples of damage type from the results of algorithm, as in We select some samples each damage type from the results EM algorithm, shown We some samples ofeach each damage type from the results ofof EM algorithm, asas shown inin We select some samples ofof damage from the results of EM algorithm, as shown in Table 3. We some samples each damage type from the results of EM algorithm, as shown in We select some samples each damage type from results of EM algorithm, as shown in We select some samples damage type from the results algorithm, We select some samples ofeach each damage type from the results ofEM EM algorithm, asshown shown in Table 3. The number 1, 2 and 3ofof represent thetype damage type ofthe basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and Table 3. The number 1, 2 and 3 represent the damage type of basically intact, partially collapsed and The number 1,collapsed, 2number and 1, 3 1, represent damage type offeatures basically intact, partially collapsed and completely Table 3.3.The number 21, 3 3represent damage type of basically intact, partially collapsed and Table 3. The number 22and and 33the represent the damage type basically intact, partially collapsed and Table number 2and represent the damage type basically intact, partially collapsed and Table 3.The The 1, and represent the damage type of basically intact, partially collapsed and completely respectively. The the common ofofof damage type 1 have a clear outline and completely collapsed, respectively. The common features of damage type 11have have aand clear outline and completely collapsed, respectively. The common features of damage type 11 11have aa aaclear outline and completely collapsed, respectively. The common features ofof damage type have clear outline and completely collapsed, respectively. The common features ofdamage damage type have aclear clear outline and completely collapsed, respectively. The common features of type outline and completely collapsed, respectively. The common features damage type 1 have a clear outline and completely collapsed, respectively. The common features of damage type have clear outline and collapsed, respectively. The common features of damage type 1 have a clear outline regular shape, completely collapsed, respectively. The common features of damage type 1 have a clear outline and completely collapsed, respectively. The common features of damage type 1 have a clear outline and completely collapsed, respectively. The common features of damage type 1 have a clear outline and completely collapsed, respectively. The common features of damage type 1 have a clear outline and regular shape, and an intact shadow. Damage type 2 has fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type 2 has fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type 2 has fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type 2 has fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type 2 has fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type has fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type has fuzzy boundaries, irregular shape, offset regular shape, and anan intact shadow. Damage type has fuzzy boundaries, irregular shape, offset and an intact shadow. Damage type 2 has fuzzy boundaries, irregular shape, offset orientation, and loss regular shape, and an intact shadow. Damage type 22 22has fuzzy irregular shape, offset regular shape, and an intact shadow. Damage type 2223has has fuzzy boundaries, irregular shape, offset regular shape, and shadow. Damage type fuzzy boundaries, irregular shape, offset regular shape, and an intact shadow. Damage type has fuzzy boundaries, irregular shape, offset orientation, and loss ofintact shadow effects. Damage type has no boundaries, visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made orientation, and loss of shadow effects. Damage type 3 has no visible characteristics of man-made oforientation, shadow effects. Damage typeeffects. 3effects. has no visible characteristics ofvisible man-made objects. ofofof orientation, and loss ofofof shadow Damage type 3 3has nono visible characteristics man-made orientation, and loss shadow effects. Damage type 33has has no visible characteristics man-made and loss Damage type characteristics orientation, and loss ofshadow shadow effects. Damage type has no visible characteristics ofman-made man-made objects. objects. objects. objects. objects. objects. objects. objects. objects. Some limitations objects. objects. objects. Some limitationsofofour ourexperiment experimentare are discussed discussed below. Some limitations of our experiment are discussed below. Some limitations of experiment are discussed below. Some limitations ofour our experiment are discussed below. Some limitations ofour our experiment are discussed below. Some limitations of experiment are discussed below. Some limitations of our experiment are discussed below. Some limitations ofof our experiment are discussed below. Some limitations of our experiment are discussed below. Some limitations of our experiment are discussed below. Some limitations experiment are discussed below. Some limitations ofour our experiment are discussed below. Table samplesof ofeach eachdamage damagetypes. types. Table3.3.The Thecrowdsourcing-derived crowdsourcing-derived samples Table 3. The crowdsourcing-derived samples of each damage types. Table 3. The crowdsourcing-derived samples of each damage types. Table 3. The crowdsourcing-derived samples of each damage types. Table 3.The The crowdsourcing-derived samples ofeach each damage types. Table 3. The crowdsourcing-derived samples of each damage types. Table 3. crowdsourcing-derived samples of damage types. Table 3. The crowdsourcing-derived samples of each damage types. Table 3. The crowdsourcing-derived samples of each damage types. Table 3. 3. The crowdsourcing-derived samples ofof each damage types. Table 3. The crowdsourcing-derived samples of each damage types. Table crowdsourcing-derived samples damage types. Table 3.The The crowdsourcing-derived samples ofeach each damage types. Damage Types Sample 1 Sample 2 Sample 3 Sample 4 4 Damage Types Sample 1 Sample 2 Sample 3 Sample

Damage Types Damage Types Damage Types Damage Types Damage Types Damage Types Damage Types Damage Types Damage Types Damage Types Damage Types 1 11 1111 1 1 1 111 2 22 22222 22 222 3 3 3 3 3333333 33

Sample Sample 1 11 Sample Sample Sample Sample Sample Sample 11 111111 Sample Sample Sample

11




11 11 11 11 11 11 1111 11 11 11 11

98 98 98 98 98 98 98 9898 98 98 98 98

109 109 109 109 109 109 109 109 109 109 109 109 109

110 110 110 110 110 110 110 110 110 110 110 110 110

15 15 15 15 15 15 15 1515 15 15 15 15

19 19 19 19 19 19 19 1919 19 19 19 19

36 36 36 36 36 36 36 3636 36 36 36 36

96 96 96 96 96 96 96 9696 96 96 96 96

0 2 76 8 76 00 00000 2 22 76 76 8 88 76 76 76 7676 22 2222222 76 76 88 8888888 000 76 76 00 The study area in this paper is one part of the whole damage area, which we used as an experimental area toin apply our methods on itof and present the results clearly. Inwe addition, the The study area in this paper is one part of the whole damage area, which we used as an The study area in this paper is one part of the whole damage area, which used as an The study area inin this paper isis one part ofof the whole damage area, which we used as an The study area in this paper isone one part ofthe the whole damage area, which we used as an The study area this paper is part whole damage area, which we used as an The study area this paper one part the whole damage area, which we used as an The study area in this paper isis one part of the whole damage area, which we used asas anan The study area in this paper is one part of the whole damage area, which we used as an The study area this paper is one part the whole damage area, which we used as an The study area inin this paper part ofof the whole damage area, which we used The study area in this paper isone one part of the whole damage area, which we used as an The study area into this paper ismethods one part of the whole damage area, which we used asInIn an experimental approach is also applicable in the wide area, which is the most important purpose of the experimental area to apply our methods on it and present the results clearly. addition, the experimental area to apply our on it and present the results clearly. In addition, the experimental area apply our methods on it and present the results clearly. addition, the experimental area to apply our methods on it and present the results clearly. In addition, the experimental area toto apply our methods on itand and present the results clearly. In addition, the experimental area to apply our methods on ititand and present the results clearly. In addition, the experimental area toto apply our methods onon it it and present the results clearly. InIn addition, the experimental area to apply our methods on it present the results clearly. In addition, the experimental area apply our methods on it and present the results clearly. In addition, the experimental area apply our methods present the results clearly. addition, the experimental area to apply our methods on and present the results clearly. In addition, the area to apply our methods oninin itin and present thewhich results In addition, thepurpose approach is also crowdsourcing damage assessment. Because the model isclearly. for the general circumstance approach is also applicable the wide area, which is the most important purpose of the approach is also applicable the wide area, the most important of the approach isis also applicable in the wide area, which isproposed the most important purpose of the approach is also applicable in the wide area, which is the most important purpose of the approach is also applicable the wide area, which is the most important purpose of the approach also applicable in the wide area, which is the most important purpose of the approach is also applicable in the wide area, which is the most important purpose of the approach isisis also applicable ininin the wide area, which isisis the most important purpose ofofof the approach also applicable the wide area, which the most important purpose the approach applicable the wide area, which most important purpose the approach isalso also applicable in the wide area, which isthe the most important purpose of the applicable in the wide area, isBecause the most important purpose offor the crowdsourcing damage without the limits of area, wewhich will extend the application wide interpretation in future work. The crowdsourcing damage assessment. Because the model is proposed for the general circumstance crowdsourcing damage assessment. Because the model is for the general circumstance crowdsourcing damage assessment. Because the model isproposed proposed for the general circumstance crowdsourcing damage assessment. Because the model is proposed for the general circumstance crowdsourcing damage assessment. the model is proposed the general circumstance crowdsourcing damage assessment. Because the model is proposed for the general circumstance crowdsourcing damage assessment. Because the model isto proposed for the general circumstance crowdsourcing damage assessment. Because the model is proposed for the general circumstance crowdsourcing damage assessment. Because the model is proposed for the general circumstance crowdsourcing damage assessment. Because the model is for the general circumstance crowdsourcing damage assessment. Because the model isproposed proposed for the general circumstance raw data we collected show that most participants once, although method allows assessment. Because the model isextend proposed for theinterpret general circumstance without the limits ofThe area, without the limits of area, we will extend the application to wide interpretation in future work. The without the limits of area, we will the application to wide interpretation in future work. without the limits ofof area, we will extend the application toto wide interpretation inin future work. The without the limits of area, we will extend the application toonly wide interpretation infuture future work. The without the limits of area, we will extend the application to wide interpretation in work. The without the limits area, we will extend the application wide interpretation future work. The without the limits ofof area, we will extend the application toto wide interpretation inthe future work. The without the limits of area, we will extend the application to wide interpretation in future work. The without the limits of area, we will extend the application to wide interpretation in future work. The without the limits we will extend the application interpretation in work. The without the limits ofarea, area, we will extend the application towide wide interpretation infuture future work. The participants to interpret multiple times. We could not determine the minimum number of the we will extend the application to wide interpretation in future work. The raw data we collected raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows raw data we collected show that most participants interpret only once, although the method allows participants, because the web-based interface iscould open to the public and any case could appear. We show that most participants interpret only once, although the method allows participants to interpret participants to interpret multiple times. We could not determine the minimum number of the participants to interpret multiple times. We could not determine the minimum number of the participants to interpret multiple times. We not determine the minimum number of the participants to interpret multiple times. We could not determine the minimum number of the participants toto interpret multiple times. We could not determine the minimum number ofof the participants to interpret multiple times. We could not determine the minimum number of the participants toto interpret multiple times. We could not determine the minimum number ofof the participants to interpret multiple times. We could not determine the minimum number of the participants interpret multiple times. We could not determine the minimum number the participants interpret multiple times. We could not determine the minimum number the participants to interpret multiple times. We could not determine the minimum number of the could calculate a result based on a span from the start to the time we choose, for example a week. The multiple times. We could not determine the minimum number of the participants, because the participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface is open to the public and any case could appear. We participants, because the web-based interface isisopen open the public and any case could appear. We participants, because the interface is the participants, because theweb-based web-based interface opentoto to thepublic publicand andany anycase casecould couldappear. appear.We We method would give result based on any data collection phase. web-based interface isaopen toon the public and any case could appear. We could calculate result based could calculate result based on span from the start to the time we choose, for example week. The could calculate aa aaresult based aa aaspan from the start to the time we choose, for example aaaaaweek. The could calculate result based on span from the start to the time we choose, for example week. The could calculate result based on span from the start to the time we choose, for example week. The could calculate result based on span from the start to the time we choose, for example week. The could calculate aaaaresult result based on aaaaspan span from the start to the time we choose, for example aaaaweek. week. The could calculate result based onon span from the start toto the time we choose, for example week. The could calculate a result based on a span from the start to the time we choose, for example a week. The could calculate a result based on a span from the start to the time we choose, for example a week. The could calculate a based a from the start the time we choose, for example a The could calculate result based on span from the start to the time we choose, for example week. The Although “actual” ground truths of building damage are necessary order towould truly discuss the on a span from the start to the time we choose, for example a week. Theinmethod give a result method would give aresult result based on any data collection phase. method would give a aaresult based on any data collection phase. method would give result based on any data collection phase. method would give result based on any data collection phase. method would give based on any data collection phase. method would give result based on any data collection phase. method would give result based onon any data collection phase. method would give aa proposed based on any data collection phase. method would give aaaaresult result based on any data collection phase. method would give aresult based any data collection phase. method would give result based on any data collection phase.photographs or broadcasted videos applicability of the method, we could not find ground based on any data collection phase. Although “actual” ground truths of building damage are necessary in order to truly discuss the Although “actual” ground truths of damage are necessary in to discuss the Although “actual” ground truths ofbuilding building damage are necessary inorder order totruly truly discuss the Although “actual” ground truths ofbuilding building damage are necessary inorder order totruly truly discuss the Although “actual” ground truths of damage are necessary in to discuss the Although “actual” ground truths of building damage are necessary in order to truly discuss the Although “actual” ground truths ofof building damage are necessary inin order toto truly discuss the ground truths of building damage are necessary in to truly discuss the Although “actual” ground truths of building damage are necessary in order to truly discuss the Although “actual” ground truths damage are necessary order discuss the Although “actual” ground truths ofbuilding building damage are necessary in order totruly truly discuss the onAlthough TV of the“actual” damaged buildings in the target area, only text information ororder papers are available. We applicability of the proposed method, we could not find ground photographs or broadcasted videos applicability of the proposed method, we could not find ground photographs or broadcasted videos applicability of the proposed method, we could not find ground photographs or broadcasted videos Although “actual” ground truths of building damage are necessary in order to truly discuss the applicability of the proposed method, we could not find ground photographs or broadcasted videos applicability ofof the proposed method, we could not find ground photographs or broadcasted videos applicability of the proposed method, we could not find ground photographs or broadcasted videos applicability ofof the proposed method, we could not find ground photographs oror broadcasted videos applicability of the proposed method, we could not find ground photographs or broadcasted videos applicability the proposed method, we could not find ground photographs or broadcasted videos applicability proposed method, we could not find ground photographs videos applicability ofthe the proposed method, we could not find ground photographs orbroadcasted broadcasted videos on TV of the damaged buildings in the target area, only text information or papers are available. We on TV of the damaged buildings in target area, only text information or papers are available. We on TV ofof the damaged buildings inthe the target area, only text information oror papers are available. We applicability of the proposed method, we could not find ground photographs or broadcasted videos on on TV of the damaged buildings in the target area, only text information or papers are available. We on TV of the damaged buildings in the target area, only text information or papers are available. We on TV the damaged buildings in the target area, only text information papers are available. We on TV of the damaged buildings in the target area, only text information or papers are available. We onon TV ofofof the damaged buildings ininthe target area, only text information oror papers are available. We on TV the damaged buildings in the target area, only text information or papers are available. We TV damaged buildings target area, only text information are available. We on TV ofthe the damaged buildings inthe the target area, only text information orpapers papers are available. We


12 of 16

TV of the damaged buildings in the target area, only text information or papers are available. We focus on using the wisdom of the public to find more damaged buildings in the early stage of the earthquake. The assessment results of the target area in this paper are highly consistent with results of the existing study published by Dou et al. [17] in the corresponding area. 5. Conclusions and Future Work Building damage assessment in the early time after an earthquake is a very crucial problem. Knowing where the collapsed buildings are and to what extent buildings have been damaged are closely related to life-saving for emergency response. However, it is hard to survey the whole in-situ information in a short time after an earthquake. Satellite or aerial remote sensing technology has the capability of earth observation, and becomes a useful tool for damage estimation without being physically present in disaster area. Aerial remote sensing images, which are captured and processed faster and have higher spatial resolution, make it possible to rapidly assess collapsed buildings early after the earthquake. A web-based interface was built. Anyone who accessed our website was required to assign one of the three damage types for each buildings based on the aerial image. The 3456 data points from 27 participants on the experimental area, which is a sub-region of Yushu, were collected. MLE, Bayes’ theorem and EM algorithm were applied to estimate the individual error-rates and infer “ground truth” according to the 3456 data points of 27 participants. The results suggest that EM algorithm is better than “majority” method and there is no clear bias towards extreme value among damage types contributed by participants. This study shows that the variation of image understanding among participants exists, due to their different professional background. We demonstrated how to collect and store the data created by individuals online, how to make them contribute their results flexibly and easily, drawing a point instead of a polygon, and how to quantitatively estimate individuals’ accuracy and the “ground truth” of each building, using a probabilistic model. By means of a sequential procedure of RS image pre-processing, publishing RS image online, collecting crowdsourcing building damage assessment data, evaluating the quality of data contributed by crowdsourcing and damage mapping, the building collapse can be rapidly assessed with viable accuracies in the early time after the earthquake. We conclude that RS data combined with crowdsourcing have a high capability to support large-area assessments of building collapse, meeting the need of disaster emergency response. A new processing framework is proposed to establish the connection between remote sensing image and crowdsourcing, demonstrating potential for crowdsourcing rapid assessment of building collapse early after the earthquake based on aerial remote sensing image. Future efforts will focus on providing multi-source and multi-temporal remote sensing image. Multi-source RS data, such as oblique images, can provide more views of buildings on the ground. Multi-temporal RS data, such as pre-earthquake images, are considered to be very useful as a reference in identifying the damage in the post-event image [4]. Although these adjustments will refine the final results towards rapid damage assessment, a problem should be considered: how to balance the operational complexity of system and the improvement of results, because ordinary people are more inclined to use easy-to-operate systems without spending too much time. In summary, the aim is that the rapid assessment results through crowdsourcing could meet the needs of deploying rescue forces in the early time after the earthquake. More details about the damage level of buildings surveyed on the spot afterwards are beyond the scope of this paper. Acknowledgments: The paper is funded by “Research on the model of remote sensing disaster monitoring and assessment based on crowdsourcing” project, Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences. And we thank Airborne Remote Sensing Center, Chinese Academy of Sciences a lot for providing post-earthquake aerial remote sensing image of Yushu. Author Contributions: Jianbo Duan conceived and designed the experiments; Jianbo Duan, Rui Guo and Caihong Ma built the experimental platform; Shuai Xie prepared and processed the aerial data, performed the experiments, analyzed the data and wrote the paper; Shibin Liu supervised the research; and Qin Dai, Wei Liu and Yong Ma gave comments and revised the manuscript. Conflicts of Interest: The authors declare no conflict of interest.


13 of 16

Appendix A Table A1. The estimates of the individual error-rates. Participant 1 True 1 2 3

1 0.64 0.38 0

Observed 2 3 0.35 0.01 0.5 0.11 0.35 0.65

Participant 15 True 1 2 3


1 0.83 0.5 0

Observed 2 3 0.12 0.05 0.27 0.23 0.36 0.64

1 0.9 0.62 0.24

Observed 2 3 0.08 0.03 0.38 0 0.53 0.24

True 1 2 3

1 0.66 0.23 0.09

Observed 2 3 0.3 0.04 0.73 0.04 0.29 0.62

True 1 2 3

1 0.94 0.32 0.05

Observed 2 3 0.06 0 0.68 0 0.8 0.15

True 1 2 3

1 0.46 0.12 0

Observed 2 3 0.5 0.04 0.74 0.14 0.32 0.68

True 1 2 3

1 0.47 0.46 0.25

Observed 2 3 0.47 0.06 0.54 0 0.35 0.4

True 1 2 3

1 0.9 0.42 0.08

Observed 2 3 0.09 0.01 0.58 0 0.61 0.3

1 0.76 0.25 0

Observed 2 3 0.24 0 0.5 0.25 0.37 0.63

1 0.91 0.39 0.11

Observed 2 3 0.09 0 0.61 0 0.37 0.53

1 0.46 0 0

Observed 2 3 0.29 0.24 0.76 0.24 0.11 0.89



Observed 2 3 0.01 0 0.6 0 0.8 0.1

Participant 20


1 0.99 0.4 0.1

Participant 19


Observed 2 3 0.3 0 0.59 0.34 0.11 0.83

Participant 18


1 0.7 0.07 0.06

Participant 17


Observed 2 3 0.48 0 0.69 0.16 0.29 0.62

Participant 16


1 0.52 0.15 0.1

1 0.83 0.27 0.05

Observed 2 3 0.15 0.02 0.73 0 0.67 0.29


1 0.54 0 0

Observed 2 3 0.45 0.01 0.46 0.54 0.24 0.76


14 of 16

Table A1. Cont. Participant 9 True 1 2 3

1 0.26 0.04 0

Participant 23

Observed 2 3 0.56 0.17 0.4 0.56 0.15 0.85

True 1 2 3


1 0.73 0.08 0

1 0.91 0.15 0

Observed 2 3 0.27 0 0.61 0.31 0.3 0.7

True 1 2 3

1 0.36 0 0.05

Observed 2 3 0.09 0 0.85 0 0.73 0.27

True 1 2 3

1 0.77 0.23 0

Observed 2 3 0.06 0 0.31 0 0.48 0.26

1 0.48 0 0

Observed 2 3 0.49 0.02 0.77 0.23 0.05 0.95

Participant 26

Observed 2 3 0.64 0 0.79 0.21 0.68 0.26

True 1 2 3


1 0.94 0.69 0.26

Participant 25


Observed 2 3 0 0 0.19 0 0.43 0.14

Participant 24


1 1 0.81 0.43

1 0.98 0.69 0.2

Observed 2 3 0.03 0 0.27 0.04 0.35 0.45

Participant 27

Observed 2 3 0.18 0.05 0.5 0.27 0.38 0.62

True 1 2 3

1 0.76 0.35 0.05

Observed 2 3 0.19 0.05 0.54 0.12 0.38 0.57


1 0.67 0.09 0.05

Observed 2 3 0.33 0 0.91 0 0.43 0.52

Table A2. The estimated probabilities for the Gs.

Building ID 1 2 3 4 5 6 7 8 9 10 11 12 13

Damage Types 1

2

3

0 0.002 0 1 1 0.006 1 0.996 0 1 1 1 0.01

0 0.998 0 0 0 0.994 0 0.004 0 0 0 0 0.99

1 0 1 0 0 0 0 0 1 0 0 0 0

Building ID 65 66 67 68 69 70 71 72 73 74 75 76 77

Damage Types 1

2

3

1 0.024 0 0 1 1 1 0.956 0.999 0 0 0 0

0 0.976 0 0.003 0 0 0 0.044 0.001 1 1 0 0

0 0 1 0.997 0 0 0 0 0 0 0 1 1


15 of 16

Table A2. Cont.

Building ID 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

Damage Types 1

2

3

0 1 0 1 1 0 0 0 0 0 1 0 1 1 1 0 0 0 0 1 0 0 1 0 1 1 1 1 0 0 1 1 1 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 0 0 1

1 0 1 0 0 1 1 0 0 1 0 0.072 0 0 0 1 0 1 1 0 0 0 0 0.998 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 1 1 0 0 0.928 0 0 0 0 1 0 0 0 1 1 0 0.002 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0

Building ID 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127

Damage Types 1

2

3

1 0 0 1 1 1 1 1 1 1 0 1 1 0 0.001 1 0 0 0 0 1 1 1 1 0 1 1 0.999 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1

0 0 0 0 0 0 0 0 0 0 0.97 0 0 1 0.999 0 1 1 1 0.99 0 0 0 0 1 0 0 0.001 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 1 1 0 0 0 0 0 0 0 0.03 0 0 0 0 0 0 0 0 0.01 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0

References 1.

Geiß, C.; Taubenböck, H.; Tyagunov, S.; Tisch, A.; Post, J.; Lakes, T. Assessment of seismic building vulnerability from space. Earthq. Spectra 2014, 30, 1553–1583. [CrossRef]


2. 3. 4. 5. 6.

7. 8. 9. 10.

11. 12.

13.

14. 15.

16. 17.

18. 19. 20. 21.

16 of 16

Lei, L.; Liu, L.; Zhang, L.; Bi, J.; Wu, Y.; Jiao, Q.; Zhang, W. Assessment and analysis of collapsing houses by aerial images in the wenchuan earthquake. J. Remote Sens. 2010, 14, 333–344. Chiroiu, L. Damage assessment of the 2003 Bam, Iran, earthquake using Ikonos imagery. Earthq. Spectra 2005, 21, 219–224. [CrossRef] Saito, K.; Spence, R.; de C Foley, T.A. Visual damage assessment using high-resolution satellite images following the 2003 Bam, Iran, earthquake. Earthq. Spectra 2005, 21, 309–318. [CrossRef] Vu, T.T.; Matsuoka, M.; Yamazaki, F. Detection and animation of damage using very high-resolution satellite data following the 2003 Bam, Iran, earthquake. Earthq. Spectra 2005, 21, 319–327. [CrossRef] Huyck, C.K.; Adams, B.J.; Cho, S.; Chung, H.-C.; Eguchi, R.T. Towards rapid citywide damage mapping using neighborhood edge dissimilarities in very high-resolution optical satellite imagery—Application to the 2003 Bam, Iran, earthquake. Earthq. Spectra 2005, 21, 255–266. [CrossRef] Hutchinson, T.C.; Chen, Z.Q. Optimized estimated ground truth for object-based urban damage estimation using satellite images from the 2003 Bam, Iran, earthquake. Earthq. Spectra 2005, 21, 239–254. [CrossRef] Chen, Z.; Hutchinson, T.C. Probabilistic urban structural damage classification using bitemporal satellite images. Earthq. Spectra 2010, 26, 87–109. [CrossRef] Barrington, L.; Ghosh, S.; Greene, M.; Har-Noy, S.; Berger, J.; Gill, S.; Lin, A.Y.M.; Huyck, C. Crowdsourcing earthquake damage assessment using remote sensing imagery. Ann. Geophys. 2011, 54, 680–687. See, L.; Comber, A.; Salk, C.; Fritz, S.; van der Velde, M.; Perger, C.; Schill, C.; McCallum, I.; Kraxner, F.; Obersteiner, M. Comparing the quality of crowdsourced data contributed by expert and non-experts. PLoS ONE 2013, 8, e69958. [CrossRef] [PubMed] Goodchild, M.F.; Glennon, J.A. Crowdsourcing geographic information for disaster response: A research frontier. Int. J. Digit. Earth 2010, 3, 231–241. [CrossRef] Ghosh, S.; Huyck, C.K.; Greene, M.; Gill, S.P.; Bevington, J.; Svekla, W.; DesRoches, R.; Eguchi, R.T. Crowdsourcing for rapid damage assessment: The global earth observation catastrophe assessment network (GEO-CAN). Earthq. Spectra 2011, 27, S179–S198. [CrossRef] Huynh, A.; Eguchi, M.; Lin, A.Y.-M.; Eguchi, R. Limitations of crowdsourcing using the EMS-98 scale in remote disaster sensing. In Proceedings of the 2014 IEEE Aerospace Conference, Big Sky, MT, USA, 1–8 March 2014. Goodchild, M.F.; Li, L. Assuring the quality of volunteered geographic information. Spat. Stat. 2012, 1, 110–120. [CrossRef] Ofli, F.; Meier, P.; Imran, M.; Castillo, C.; Tuia, D.; Rey, N.; Briant, J.; Millet, P.; Reinhard, F.; Parkan, M. Combining human computing and machine learning to make sense of big (aerial) data for disaster response. Big Data 2016, 4, 47–59. [CrossRef] [PubMed] Dell’Acqua, F.; Gamba, P. Remote sensing and earthquake damage assessment: Experiences, limits, and perspectives. Proc. IEEE 2012, 100, 2876–2890. [CrossRef] Aixia, D.; Xiaoqing, W.; Xiang, D.; Xiaoxiang, Y.; Long, W.; Yanfang, D.; Dingjian, J. Quantitative methods of rapid earthquake damage assessment using remote sensing and its application in yushu earthquake. J. Catastr. 2012, 27, 75–80. Dawid, A.P.; Skene, A.M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 1979, 20–28. [CrossRef] Dempster, A.P.; Laird, N.M.; Rubin, D.B. Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B 1977, 39, 1–38. Kerle, N.; Hoffman, R.R. Collaborative damage mapping for emergency response: The role of cognitive systems engineering. Nat. Hazards Earth Syst. Sci. 2013, 13, 97–113. [CrossRef] Smyth, P.; Fayyad, U.; Burl, M.; Perona, P.; Baldi, P. Inferring Ground Truth from Subjective Labelling of Venus Images; MIT Press: Cambridge, MA, USA, 1995. © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).