Design and implementation of the real-time GIS data model and Sensor Web service platform for environmental big data management with the Apache Storm Zeqiang Chena,b, Nengcheng Chena,b,*, Jianya Gonga,b Collaborative Innovation Center of Geospatial Technology, Wuhan 430079, China b State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
[email protected] a
Abstract—An abstract real-time GIS data model and Sensor Web service platform was proposed to manage real-time environmental data. With the development of sensor technology, more and more sensor networks are deployed to monitor our environment, and then generate environmental big data. How to improve the real-time GIS data model and Sensor Web service platform for real-time environmental big data manage is a problem. In this paper, the Apache Storm is adopted to deal with the question. A design and implementation of the real-time GIS data model and Sensor Web service platform for environmental big data management with Apache Storm is proposed. The main studied contents include integrating the Apache Strom with the Sensor Web service as the Sensor Observation Service, and processing the environmental big data timely. To test the feasibility of the design and implementation, two use cases of real-time air quality monitoring and real-time soil moisture monitoring based on the real-time GIS data model in the Sensor Web service platform are realized and demonstrated. The experimental results show that the implementation of real-time GIS data model and Sensor Web Service Platform with the Apache Storm is an effective way to manage real-time environmental big data. /Keywords—real-time GIS data environmental big data; Apache Storm;
I.
model;
Sensor
Web;
INTRODUCTION
A real-time GIS data model considering sensor, observation, geographic object, object, spatio-temporal process, simulation, event, state, and change function was proposed [1]. A Sensor Web service platform is an information service infrastructure based on Sensor Web technologies. Integrating the real-time GIS data model and Sensor Web service platform was an effective way to manage real-time environmental data [1]. With the development of sensor technology, more and more sensor networks are deployed to monitor our environment, and then environmental big data is generated. However, the method of integrating real-time GIS data model and Sensor Web service platform for environmental big data management needs further study. Because, the Sensor Web service platform did not have enough capability to deal with big data.
In this paper, the Apache Storm is adopted to improve the method of the real-time GIS data model and Sensor Web service platform for environmental big data management. The main studied contents include integrating the Apache Strom with the Sensor Web service as the Sensor Observation Service, receiving real-time environmental big data, and processing the environmental big data timely. II.
METHOD
The method of integrating the real-time GIS data model and Sensor Web service platform for environmental data management has been mentioned in the paper [1]. Apache Storm is a distributed real-time computation system. Storm reliably processes unbounded streams of data and does realtime processing what Hadoop did for batch processing. Storm manipulates and transforms streams as tuples, where a tuple is a named list of values. Tuples can contain objects of any type. Storm consists of three abstractions as spouts, bolts, and topologies. A spout is a source of streams in a computation. A bolt encapsulates a logical computation to process any number of input streams and produces any number of new output streams. A topology is a network of spouts and bolts, with each edge in the network representing a bolt subscribing to the output stream of some other spout or bolt. To improve the method with Apache Storm, the key issues are: 1) how to integrate the Storm with Sensor Web services, as the Sensor Observation Service (SOS), and 2) how to mapping the realtime GIS data model to the Storm topology model. For these two issues, we propose method integrating SOS with Storm, and the method of mapping the real-time GIS data model to the Storm topology model. A. Integrating Sensor Observation Service with Apache Storm The SOS, a part of Sensor Web, provides standard interfaces to manage and retrieve metadata and observations from heterogeneous sensor systems. SOS is a specification. The concrete implementation of SOS depends on developers. Integrating SOS with Storm seamlessly is to design a system architecture that the “shell” is SOS interfaces and the “kernel”
is storm. After study carefully, the architecture of SOS integrating Storm is as Figure 1. The architecture refers to sensors, users, and SOS. The SOS has three parts: interface controller, Storm, and database. The sensors and users interact SOS by the standard interfaces as GetCapabilities, DescribeSensor, InsertObservation, and GetObservation and so on. The interface controller receives standard requests and returns their standard responses, but does not any processing. The Storm is responsible for the logical processing. The Storm saves real-time data in memory and store data in database. The database can be a SQL database or NoSQL database (for example, the NoSQL database MongoDB). MongoDB is an open-source document database, and the leading NoSQL database. MongoDB can be clustered in distribution environment for storing big data.
Fig. 2. The mapping model between real-time GIS model and Storm
III.
Fig. 1. The architecture of SOS integrating Storm
B. Mapping the real-time GIS data model to the Storm topology model In an application, Storm topology is a programming model. To realize to real-time GIS data, it is need map real-time GIS data model and the Storm topology model. we map the two models as follows: Real-time GIS data model mapping to topology: The abstraction of a real-time GIS data model can be seen as a topology in Storm. Sensor mapping to spout: A sensor derives observation like a spout. Spatiotemporal process mapping to spout: A spatiotemporal process generates simulation, thence a spatiotemporal process is like a spout. Observation mapping to tuple: observation is a data stream, therefore an abservation can be seen as a tuple. Simulation mapping to tuple: simulation is the result of a spatiotemporal process. The simulation can be seen as a tuple. GeoObject mapping to tuple: The data, states, attributions, and related information of a geoobject can been transferred among the spouts and the bolts. So it is with the object. Change function mapping to bolt: A change function is a processing and it can been encapsulated with a bolt.
PROTOTYPE SYSTEM
A Sensor Web Service Platform add the notion of the realtime GIS data model consistent with the framework of Figure 3 was implemented by the Sensor Web group Wuhan University, China [2] with the Sensor Web technologies. The Sensor Web Service Platform integrates sensor registration service, Sensor Observation Service, Sensor Planning Service, real-time mapping service, satellite positioning service and other services to obtain real-time sensor information, observational data, data product, and other information resources, as well as demonstrates these information resources in the Map World [24] with graphics, text, tables, and video vividly. The Sensor Web Service Platform mainly consists of six major functional modules: Sensor Retrieval Module, Sensor Observational Data Retrieval Module, Sensor Control Module, Sensor Planning Module, Thematic Map Module, and Sensor Registration Module. Currently, dozens of sensors and plenty of real-time environmental data are managed by the Sensor Web Service Platform with the real-time GIS data model notion, such as meteorological data (wind speed, wind direction, sunshine duration, solar radiation, atmospheric pressure, air temperature, air humidity, rainfall), air quality data (Air quality index, pm2.5, pm10, O3, NO2, SO2, CO), soil moisture data, soil temperature data, landslide data and so on. The sensors were deployed in five experimental field as Fushun experimental field, Yemaomian of The Three Gorges experimental field, Huazhong Agricultural University experimental field, Taiyuan experimental field, and Baoxie experimental field (see more in website: http://gsw.whu.edu.cn:9002/SensorWebPro).
Fig. 3. Portal of Sensor Web service platform
IV.
USE CASE
To demonstrate the proposed method for environmental data management, two use cases as real-time air quality monitoring, and real-time soil moisture monitoring in the Wuhan city, China are shown. A. Real-time air quality monitoring The air quality affects people’s health. Governments and citizens pay more attentions on air quality than ever before. The Air Quality Index (AQI) a dimensionless index is quantitative description of the air quality status as one indicator to provide health guidelines for the public. The United States Environmental Protection Agency has been released the Technical Assistance Document for the Reporting of Daily Air Quality – the Air Quality Index (AQI) for standardizing the AQI calculation method [3]. In this use case, the real-time environmental data as SO2, NO2, PM10, CO, O3, and PM2.5, came from the Wuhan Environmental Monitoring Center, are managed timely by the Sensor Web Service Platform. Manage the information or knowledge of the real-time data by real-time GIS data model. The platform can show data in a time point and data series in a time interval. For example, Figure 4 depicts the pollutants at the month of January 2015. From the Figure 4, some facts can be seen: 1) during the period, the maximal AQI is 433 at January 5, and the minimal AQI is 53 at January 28. 2) During the period, the AQI values are between 140 and 300. This means the air quality is “unhealthy for sensitive groups” (AQI is from 101 to 150), “unhealthy” (AQI is from 151 to 200), or “very unhealthy” (AQI is from 200 to 300) [3]. Caution inform to people related. From the results, it can see that the real-time data are obtained and processed by the Storm in the Sensor Web Service Platform.
Fig. 4. Real-time air quality monitoring result
B. Real-time soil moisture monitoring Soil moisture is an important environmental indicator to reflect the degree of agricultural drought. Real-time soil moisture monitoring guides agricultural irrigation timely. An automatic observation station was deployed in Baoxie experimental field (centre location at 114°31'35.61"E 30°28'12.98"N). In the field, the number of the deployed soil moisture sensors only more than 20 discrete points, and the number is limited. If want to know the conditions of soil moisture in the whole experiment area, it needs find the soil moisture value at every unobserved point. Interpolation method is often used to evaluate the unsampled points with the nearest sampled points. Inverse Distance Weighted Interpolation (IDWI), one of the most frequently used spatial interpolation method, is relatively fast and easy to compute, and straightforward to interpret [4]. Therefore, IDWI is adopted to find the unobserved point in the experiment. The Sensor Web Service Platform manages the observed soil moisture and generate real-time map. A soil moisture thematic map is generated online by the IDWI method during the period from 2013-11-21 10:22:45 to 2013-11-22 10:22:38, as an example. The soil moisture thematic map is draw as soon as the data observed, as Figure 5. Meanwhile, a sensor observation values are draw as a curve during the queried time period. From the soil moisture thematic map, the drought degree of any place in the area can be found. It can see that the soil moisture condition of the area is not balance, the northeast of the area is a bit drought, while the west and the southwest of the area is relative wet.
the Apache Strom with the Sensor Web service as the Sensor Observation Service, receiving real-time environmental big data, and processing the environmental big data timely. The integration of real-time GIS data model and Sensor Web Service Platform for environmental big data management with Storm is shown in the air quality monitoring and soil moisture monitoring in Wuhan. The integration is feasible and effective. Future work will focus on evaluating the managing capabilities of the platform for environmental big data. ACKNOWLEDGMENT
Fig. 5. Real-time soil moisture mapping result
V.
DISCUSSION
From the theoretical analysis and the experimental results, it can see that the integrating Apache storm with the real-time GIS data model and Sensor Web Service Platform is feasible and effective way to manage real-time environmental big data. The apache Storm is a distributed real-time stream data processing framework, and it can built a big data processing environment. Many literatures have been proofed the points [5-6]. Meanwhile, in the two use cases, with the similar test environments as the paper [1], the real-time observations are processed by the added Storm timely as Figure 4 and Figure 5. In the use cases, the Storm is deployed in five servers as one name node and four data nodes. It is a small distributed cluster. As the observations managed by the Sensor Web service platform are becoming bigger and bigger, the observations has big data characteristic [7]. Therefore, it is feasible that the real-time GIS data model and Sensor Web service platform can manage environmental big data with the Apache timely. VI.
CONCLUSION AND FUTURE WORK
This paper adopts apache Storm to improve the real-time GIS data model and Sensor Web Service Platform for environmental big data. The core work of this paper integrating
This work was supported by grants from the National Basic Research Program of China (973 Program) (no. 2011CB707101), the National Nature Science Foundation of China program (nos. 41301441, 41171315), the Fundamental Research Funds for the Central Universities (no.2042014kf0200), and China Postdoctoral Science Foundation funded project (no. 2014M562050). REFERENCES [1]
[2]
[3]
[4]
[5]
[6]
[7]
J. Gong, J. Gong, and Z. Chen, “Real-time GIS data model and sensor web service platform for environmental data management,” International Journal of Health Geographics, vol. 14, no. 2, 2015. N. Chen, X. Yang, X. Wang, “Design and implementation of geospatial sensor web information public service platform,” J Geo-information Sci, vol 15, no. 16, pp. 887-894, 2013. Technical Assistance Document for the Reporting of Daily Air Quality – the Air Quality Index (AQI). Online available from: http://www.epa.gov/airnow/aqi-technical-assistance-documentdec2013.pdf. G. Lu and D. Wong, "An adaptive inverse-distance weighting spatial interpolation technique," Computers & geosciences, vol. 34, no. 9, pp. 1044-1055, 2008. A.H.A. Rahnama, "distributed real-time sentiment analysis for big data social streams," 2014 international conference on control, decision and information technologies, pp. 789-794, 2014. D.H. Im, C.H. Cho and L. Jung, "Detecting a large number of objects in real-time using apache storm," international conference on ICT Convergence, pp. 836-838, 2014. S. Nativi, P. Mazzetti, M. Santoro, F. Papeschi, M. Craglia and O. Ochiai, “big data challenges in building the global observation system of systems,” environmental modelling and software, vol. 68, pp. 1-26, 2015.
