Paper - Computer Graphics Bonn

International Conference on Design & Decision Support Systems in Architecture and Urban Planning (2010)

PROBADO3D - Indexing and Searching 3D CAD Databases: Supporting Planning through Content-Based Indexing and 3D Shape Retrieval Ina Blümel1 , René Berndt2 , Sebastian Ochmann3 , Richard Vock3 , Raoul Wessel3 1 German

National Library of Science and Technology, Hannover, Germany e-mail: [email protected] 2 Institute of Computer Graphics and Knowledge Visualization, Graz University of Technology, Austria e-mail: [email protected] 3 Institute of Computer Science II Computer Graphics, University of Bonn, Germany e-mail: {ochmann, vock, wesselr}@cs.uni-bonn.de

Abstract When modeling in 3D CAD, architects often search for already existing 3D content to complete their own design, e.g. environmental models, furniture models or detailed window profiles. 3D models are normally indexed and accessed based on textual metadata. As this metadata is expensive to obtain, PROBADO3D aims to develop workflows and tools for semi-automatic indexing of 3D models. Another goal is the development of intuitive visual search and presentation interfaces that face the needs of architects looking for 3D content. PROBADO3D is a part of the PROBADO framework designed to support multimedia objects of different domains. Keywords: Content-Based Indexing, 3D Shape Retrieval, Visual Search, Digital Library

1. ARCHITECTURAL PLANNING Architects are “...working from abstract problem formulations to concrete solutions and splitting problems into subproblems iterative and recursive processes...” [CR92]. Within complex planning tasks a part of the decision-making processes can be adopted by design methods to obtain a higher effectiveness. [Sch93] is making decision possibilities explicit by characterizing different relevant approaches for designing. He classifies several methods that can be inserted into a general modelling environment and support different aspects of planning, sketching and building. These methods offer a solution to draft problems by the use of search mechanisms. Within this context we will describe the usage of indexed 3D CAD model collections. Architectural CAD models are becoming more heterogeneous and complex. Trades taking part in planning provide highly detailed individual layers of the whole building model. Only few of them are developed completely from scratch. For efficiency reasons architects and specialized planners use case-based reasoning and search for already existing 3D models that A)

serve for inspiration or B) fit best into the given conditions within the actual building model. Case-based reasoning is a method long-known in architecture. It consists of finding and adapting a similar problem definition and the appropriate architectural solution for solving a new problem. The adjustment process or the adaptation of existing architecture on new problems is a complex procedure. The simplest stage is the direct take over. In the next higher stage parts of architecture solutions are transferred, others are adapted geometrically or regarding materials. In the most complicated form of the adaptation topological changes are made. Challenges are 1. the completeness of the architectural case database, 2. methods for indexing the cases so that they can be found, 3. methods for formulating the search so that all cases are found which correspond to the given requirements with regard both to geometrical and topological aspects. The first point is scribed in Section 2.1, however in this paper we will especially go into 2 and 3, expounded in Section 3, and show how automatic indexing of 3D content can assist

I. Blümel, R. Berndt, S. Ochmann, R. Vock, R. Wessel / Supporting Planning through Content-Based Indexing and 3D Shape Retrieval

the architect in using 3D model databases for facilitating the planning process.

2. 3D MODELS AND INDEXING 2.1. Content in PROBADO3D The German National Library of Science and Technology, which references relevant scientific material for all areas of engineering, is setting up a repository for 3D models, which currently contains index files of approximately 7000 models of the architectural domain. Actually it works satisfying as basis for testing query engine functionality, however, it cannot be considered to be a complete database for architectural design cases yet. One result of a survey among architects in the project context, see [BS09], is the gained information about usage of architectural 3D models. First of all we classified five model types, which are buildings, environmental models, components like A) construction units and B) furniture and 3D model details. While building models are rather used as a source for inspiration due to the one-of-a-kind building process, the other model types are only slightly transformed or directly integrated into new drafts, and the demand for this model types is thus higher in architectural practices. Nevertheless building models are first of all acting as precedents in architectural education and design, and the potential of 3D models over images or plans provided for instance in architectural magazines is high if access to the respectively searched content within the model can be assured. As a first step, we concentrate on buildings and components, because of the enduring scientific interest in building configuration and the cumulative need for components to be directly taken over into the own building model. 3D model contributors in PROBADO3D are either students of architecture and component manufacturers or the models are part of architectural CAD application libraries or public databases for architectural CAD models. The control over the original files in most cases still resides with external servers, including access to the files (e.g. pay-per-view or IP-based access for certain groups). Only the index- and preview data are stored within the PROBADO3D system. Users requesting a model will be redirected to the server hosting the original file. So PROBADO3D does not have to struggle with legal problems when offering 3D models and can concentrate on developing search engines for indexing models and making their content searchable. The 3D models to be integrated in the PROBADO3D index are of various file formats and bring along very little metadata, see [BWK08]. The developed pipeline for automatic processing of architectural 3D models and deduction of technical metadata is described in [BBW10]. We will now have a closer look at content-based indexing, which is a prerequisite for different user queries on 3D model collections.

Figure 1: 3D model and underlying geometry.

2.2. Content-based Indexing Content-based shape retrieval methods rely on an abstract mathematical characterization of the underlying 3D model geometry, which is usually given as a set of (unstructured) polygons, see e.g. Figure 1. In general, the mathematical characterization can be derived automatically. By that, content-based shape retrieval does not rely on any usergenerated textual annotations, allowing for fast automatic indexing of even large databases. Most approaches on contentbased shape retrieval (for a detailed introduction see [TV08]) concentrate on query-by-example, i.e. when presented a 3D object, they search an indexed database for geometrically similar shapes. This technique is suitable for scenarios in which the user can provide actual 3D content as a query by either uploading an existing model to a search engine or by sketching a new model using a graphical interface that is connected to the search engine. However, as another result of the survey mentioned in Section 2.1, it became clear that apart from visual-interactive-driven queries, users are highly interested in text-based search relying on metadata. As this metadata is usually either not available or ambiguous, PROBADO3D aims at automatically extracting it from the underlying object geometry. For environmental, furniture, and buildings this requires automatic model classification according to certain shape taxonomy. For buildings, automatic extraction of information about the number of storeys, room areas, gross floor area, window areas per room and per floor, number of rooms per floor are additionally of great interest to the user. The currently supported contentbased indexing modules are described in Section 3. 2.3. Related Work Within this field there are related scientific initiatives for 3D search engines. There are the Princeton Shape Retrieval group [SMKF04] with content-based search engines and Aim@Shape [Fal04] with content-based and metadata based search engines.


Figure 4: Global shape descriptor as a vector. The coefficients encode the 3D model geometry. Figure 3: PROBADO3D accessed from a CAD program.

3. QUERY ENGINES AND INTERFACES 3.1. Interface Overview Design and usability of the search interface are a crucial aspect for the acceptance of a service like PROBOADO3D. Especially query-by-example requires an object [BHF09], which in the case of PROBADO3D is a 3D model. Up to now there is no build-in browser-support for 3D content, so providing these functionality can only be archived using an additional plugin (The upcoming HTML 5 will include WebGL (Web Graphics Library), which will provide support for 3D without the need for a browser plugin).

sponsible for exporting the 3D model to a file and to start this WPF application. Using this mechanism, basically every CAD program, which provides a plugin API can be easily extended to access Probado3D. Different result representations, e.g. the 2D thumbnail cloud (see Figure 2), allow the user to interactively explore the result sets, e.g. use one result as a new query object or view a 3D preview. 3.2. Query-by-example and Browsing for Environmental Elements

The interface of PROBADO3D is built on Microsoft Silverlight, which is available for Windows, Apple OS X and Linux/Unix (using the open source Moonlight). Silverlight as other Rich Internet Applications (RIA) like JavaFX or Adobe Flex offer a rich user interface similar to desktop applications including support for communication over the internet (e.g. consuming web services). One main advantage of Silverlight is the use of the same declarative language (XAML) for describing the user interface as Windows Presentation Foundation (WPF). This provides an easy and very flexible way to guarantee the same rich user experience both in web browsers and classic applications.

In this query-by-example scenario the user is looking for environmental objects or furniture that is similar to a given query object. This query object is either uploaded or generated using one of the PROBADO3D sketch interfaces (see Section 3.1). When presented the 3D content, the query engine first generates a global shape descriptor that represents the object. It consists of a vector of fixed size, the coefficients encode the object geometry (see Figure 4). In a second step, this vector is compared to those associated to the models contained in the PROBADO database. Finally, the objects providing the largest similarity to the query object are presented to the user.

The current prototype supports for the query of either buildings or components different interfaces for interactive sketching of 3D models and uploading query models (queryby-example, see Section 3.2), searching in the textual metadata and browsing using different filters (category, contributor, etc., see Figure 2 / Section 3.3), and furthermore for constructing RCG query graphs (see Section 3.4). For details regard [BBWS09].

Note that global shape descriptors are currently also used for browsing our 3D database. Once the user has selected a model from the query-by-example result list, this object and the according shape descriptor can again be used as a query object. Due to their global character, the shape descriptors used for this task provide a rather coarse representation of the underlying object. However, they are very fast to compute which is crucial in a query-by-example scenario to keep response times short.

In addition to the web based user interfaces, 3rd party modeling tools like GoogleTM SketchUp can also be used for accessing the PROBADO3D search services (see Figure 3). For this purpose a WPF desktop application was developed, which sends a 3D model file to the PROBADO3D web-service and presents the result using the same look and feel as the Silverlight interface. A SketchUp-Plugin is re-

3.3. Textual Search for Objects and Browsing the Database In order to make 3D collections of environmental objects and furniture searchable using keywords, the contained models


Figure 2: Exploring results via browsing in PROBADO3D: categories or similar shapes.

We first generate a comprehensive characterization of the underlying 3D geometry using local shape descriptors. The object is first segmented into somewhat meaningful elements corresponding to parts of primitive geometric shapes like planes, cylinders, cones, sphere, and tori (see Figure fig:Segmentation). For each detected primitive, we then compute a shape descriptor characterizing its geometry. The resulting collection of local shape descriptors serves as input for object classification [WK10]. Figure 5: Segmentation of the 3D object for applying local shape descriptors. The colors code which part of a primitive was detected: red - plane, green - cylinder, grey - torus, yellow - sphere, purple - cone.

must be automatically classified according to an architectural shape taxonomy. While usually low-level mathematical descriptions are used to characterize the geometry of 3D models, a shape taxonomy designed by experts includes high-level descriptions of object categories. Classification schemes for example might contain classes including objects that have a similar function (high-level) although their shape (low-level) might strongly vary (e.g. dining chairs). This is one of the manifestations of the semantic gap, i.e. the gap between the abstract, high level user intention and low level data representation and processing [NPWK05]. In PROBADO3D we aim at bridging this gap. Starting with a low-level mathematical description of the 3D content, we use state-of-the art supervised learning schemes to incorporate architectural expert knowledge into the classification decision.

To predict object categories, we use a supervised learning approach. Based on a number of manually classified objects and their extracted local shape descriptors, we train an algorithm to learn which constellations of local shape features typically indicate a certain category affiliation. When presented new unknown objects, the algorithm predicts their probable class membership according to the automatically extracted local features and stores this information in the metadata database. Note that this approach only requires manual object classification for the training step. Once the algorithm has been trained, no further interaction for classifying new objects is required. For a more detailed introduction to this approach we refer to our work presented in [WBK09, WBK08a]. For the above described supervised learning framework, an object classification scheme tailored to architectural requirements is crucial. As architecture can be considered in several ways (historical, constructional, etc.) we initially had to find suitable classification schemes for our type of content and for the diverse views on the content within certain stages during the planning process, whenever architects search for


3.4. Retrieval and Metadata Generation with Room Connectivity Graphs

Figure 6: Retrieval results on form- (green) and functionoriented (blue) benchmark for 3D components.

Buildings and their designated use are particularly characterized by the topology of contained rooms and storeys than by their overall shape. While the above described global and local shape descriptors represent efficient means to characterize environmental objects and furniture they can hardly describe this topology. Our tests have shown that building models are inferior to categorize only by shape descriptors. To alleviate this problem, we developed the concept of Room Connectivity Graphs (RCGs), see [WBK08b]. RCGs are a data structure that consists of a graph in which rooms are represented as nodes and connections between rooms (e.g. doors or staircases) are represented as edges, see for example Figure 7. When constructing the RCG of a building model, room nodes and the connecting edges are additionally enriched by certain attributes. For example, we extract room and window areas or gross floor area.

3D models. First, there are two kinds of models to be classified: components and buildings, see Section 2.1. Second, architects use both form- and functional approach during their planning process, which is one result of another survey among architects started in the project context. In design planning, 57% of the participating architects are thinking rather form-oriented, 14% function-oriented and 29% consider both concepts to be important. In execution planning, 75% regard the function-oriented approach as much more important, 8% the form-oriented and 12% both concepts. Taking this into consideration, we provide classifications that are both form- and function-oriented to allow intuitive searching within any stage of drafting. More detailed information to the classification schemes for component models, is presented in [WBK09]. Figure 6 diplays the results for the classification schemes regarding component models. In information retrieval contexts, precision and recall are defined in terms of a set of retrieved objects and a set of relevant objects. The precision score depicts the relevant result retrieved by a search (but says nothing about whether all relevant objects were retrieved) whereas the recall score shows the relevant objects retrieved by the search (but says nothing about how many irrelevant objects were also retrieved). The precision recall performance according to form is slightly better than that of the function categories. The overall performance is quite low due to the fact that not all categories currently contain objects and have to be balanced in a better way (they contain e.g. very many chairs and tables compared to other kind of furniture). Note that the performance is better on other benchmarks, for instance the princeton shape benchmark [SMKF04], regard [WBK09].

For upgrading the construction of the RCGs and automatic enrichment by attributes, we established an admin interface whereby RCGs for a training set of building models are generated manually (see Figure 8). Once this ground truth data is existent, RCGs can be genereated more precisely, eventually allowing training according to human’s perception of building topology. Automatically extracted RCGs serve as a starting point for query-by-example related searches. The PROBADO3D service provides a graphical interface, see [BBWS09], enabling the user to easily draw room topologies he intends to search for. Extracted RCGs provide a rich amount of metadata that is important to architects and can be used for textual search. For example, we store the number of building stories, room areas, gross floor area, window areas per room and per floor, number of rooms per floor etc. in the PROBADO3D metadata database. 4. CONCLUSION AND FUTURE WORK Most parts of the depicted query engines have already been implemented. It is planned that the search with RCGs can additionally be refined by constraining the results to those in which attributes of rooms or connections fullfill certain constraints (e.g. only include results in which a room’s area lies within a certain range). Defining graph similarity with respect to the depicted attributes is still a challenge. Discrete features (e.g. room type) can be inserted more easy into the query engine’s matching algorithm than continuous features (e.g. square meters). The importance of different attributes still remains an open question. It is not clear if square meters or other continuous attributes are more important to the current user. In addition similar features can make two buildings


Figure 7: Automatically generated Room Connectivity Graph: rooms are represented by nodes, edges depict connections like doors or vertical connections like staircases.

Figure 8: Two examples from our interface for manual RCG assignation to achieve groundtruth data that is crucial for enhancing automatic RCG generation.

similar though the topology incorporated by the pure RCG is quite different. In other words: When are graphs considered to be similar and when not, regarding A) pure graph geometry and B) the attributes? To alleviate we plan slider as part of the RCG query interface for adjusting the ranking of graph geometry and the different discrete and continuous features. Future work will further concentrate on improving the automatic classification and processing of the 3D models. Similar to the categorization of components described in Section 3.2, we will additionally examine how building models can be automatically classified according to their RCG, regarding the concepts of style covering and structural core, see [Blü05], plus the concepts of form and function. We

have precision recall performance results for the categories of component models so far, and it has to be proved if the findings also apply to building models, and if, how the classifications can be improved to enhance the values for functional categories. Users will be able to browse building models by ground plan (core, form and function), form characteristic (cover and core, form), form typology/ building type (cover, form) and building function (cover, function). For developing we again use approved classifications as a starting point, which are the Getty Art&Architecture Thesaurus [Pet94], Dewey Decimal Classification [OCL03] and Neufert Bauentwurfslehre [Neu05]. One additional content-based indexer using Semantic enrichment methods based on procedural shape representations


[USF08] is currently implemented and integrated into the PROBADO3D system. By fitting a procedural description to the target model the semantic information carried with the generative description can then also by applied to the target model (e.g. number of columns, stairs, etc). 4.1. Acknowledgements PROBADO is a joint research project supported by the German Research Foundation DFG under the LIS program. PROBADO started in February 2006 with a tentative duration of five years. Partners are the University of Bonn, Technische Universitaet Darmstadt, Graz University of Technology, the German National Library of Science and Technology in Hannover, and the Bavarian State Library in Munich. The work presented in this paper was partially supported under grants INST 9055/1-1, 1647/14-1, and 3299/11. For further information, please visit the project website at http://www.probado.de/. References [BBW10] B ERNDT R., B LÜMEL I., W ESSEL R.: Probado3d towards an automatic multimedia indexing workflow for architectural 3d models. In Proceedings of 14th European Conference on Electronic Publishing (ELPUB) (2010), pp. 79–88. 2 [BBWS09] B ERNDT R., B LÜMEL I., W ESSEL R., S CHRECK T.: Demonstration of user interfaces for querying in 3d architectural content in probado3d. In ECDL (Sept. 2009), vol. 5714 of Lecture Notes in Computer Science, Springer. 3, 5 [BHF09] B ERNDT R., H AVEMANN S., F ELLNER D.: 3D modeling in a web browser to formulate content-based 3D queries. In Web3D ’09: Proceedings of the 14th International Conference on 3D Web Technology (New York, NY, USA, 2009), ACM, pp. 111–118. 3 [Blü05] B LÜMEL I.: Modeling techniques in architecture and information technologies: Building cover and core. In Proccedings of 3rd International Conference on Innovation in Architecture, Engineering an Construction (2005). 6 [BS09] B LÜMEL I., S ENS I.: Das PROBADO-Projekt: Integration von nichttextuellen Dokumenten am Beispiel von 3D Objekten in das Dienstleistungsangebot von Bibliotheken. Zeitschrift für Bibliothekswesen und Bibliographie 2 (2009), 79–87. 2 [BWK08] B LÜMEL I., W ESSEL R., K ROTTMAIER H.: The PROBADO Framework: A Repository for Architectural 3DModels. In Proceedings of International Conference on Online Repositories in Architecture (2008), Fraunhofer irb Verlag, pp. 250–259. 2 [CR92] C ROSS N., ROOZENBURGH N.: Modelling the design process in engineering and in architecture. Journal of Engineering Design 3, 4 (1992), 325–337. 1 [Fal04] FALCIDIENO B.: Aim@shape project presentation. In Proceedings of Shape Modeling International (SMI) (Washington, DC, USA, 2004), IEEE Computer Society, p. 329. 2 [Neu05] N EUFERT E.: September 2005. 6

Bauentwurfslehre, 38th ed.

Vieweg,

[NPWK05] N OVOTNI M., PARK G. J., W ESSEL R., K LEIN R.: Evaluation of kernel based methods for relevance feedback in 3d shape retrieval. In The Fourth International Workshop on Content-Based Multimedia Indexing (CBMI’05) (June 2005). 4

[OCL03] OCLC: Dewey decimal classification, 2003. OCLC Online Computer Library Center, Inc. 6 [Pet94] P ETERSEN T.: Art & Architecture Thesaurus. ACM Press, 1994. 6 [Sch93] S CHMITT G.: Architectura et Machina: Computer Aided Architectural Design und Virtuelle Architektur. Vieweg+Teubner, 1993. 1 [SMKF04] S HILANE P., M IN P., K AZHDAN M., F UNKHOUSER T.: The princeton shape benchmark. In Shape Modeling International (June 2004). 2, 5 [TV08] TANGELDER J., V ELTKMAMP R.: A survey of content based 3D shape retrieval methods. Multimedia Tools and Applications 39, 3 (2008), 441–471. 2 [USF08] U LLRICH T., S ETTGAST V., F ELLNER D. W.: Semantic fitting and reconstruction. J. Comput. Cult. Herit. 1, 2 (2008), 1–20. 7 [WBK08a] W ESSEL R., BARANOWSKI R., K LEIN R.: Learning distinctive local object characteristics for 3d shape retrieval. In VMV (2008), pp. 167–178. 4 [WBK08b] W ESSEL R., B LÜMEL I., K LEIN R.: The room connectivity graph: Shape retrieval in the architectural domain. In The 16-th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision’2008 (Feb. 2008), Skala V., (Ed.), UNION Agency-Science Press. 5 [WBK09] W ESSEL R., B LÜMEL I., K LEIN R.: A 3d shape benchmark for retrieval and automatic classification of architectural data. In 3DOR (Mar. 2009), pp. 53–56. 4, 5 [WK10] W ESSEL R., K LEIN R.: Learning the compositional structure of man-made objects for 3d shape retrieval. In EUROGRAPHICS 2010 Workshop on 3D Object Retrieval (May 2010), pp. 39–46. 4