Aims and Objectives
Aims
- Conduct interdisciplinary, internationally excellent research that involves the integration of knowledge and methodologies from computer science, architecture and archaeology, to improve upon existing work in digital cultural heritage.
- Advance the state of the art in the aforementioned areas of research through the introduction of an artificial neural network (ANN) framework for analysing historical monuments. Specifically, use concepts from computer graphics such as 3D reconstruction, geometric feature extraction and procedural modeling, as well as from machine learning such as convolutional neural networks (CNN), along with theoretical background from architecture and archaeology, to develop a framework for understanding historical monuments.
- Form a network of collaboration between the members of the consortium, University of Cyprus and Cyprus Institute, two local and internationally-respected institutes with complementary expertise, both recognized leaders in computer graphics and digital cultural heritage, as well as the University of Girona, acting as international partner.
- Generate interest in the research direction of employing ANNs for understanding architectural structure and style, allowing the members of the consortium to be established as experts in a novel and timely research direction, while expanding their technical skills in this area during the course of the project.
- Offer prospects for the members of the consortium to exploit the results of this research through potential out-licensing of the innovative software package for building analysis that will be developed during the project, and that is applicable beyond cultural heritage in the fields or architecture and engineering.
Objectives
Dataset
Collect a dataset of annotated monuments in 2D/3D form. This will be used to train the ANNs as well as test their accuracy in learning monument structure and style.
Collection of user requirements:
A user study will be conducted with the participation of 10-15 cultural heritage experts from The Cyprus Institute, the results of which will be used to precisely define their requirements regarding the documentation and analysis of cultural heritage monuments. The user study results will be used as a guide regarding the list of monuments that are of interest, as well as guide the design of the monument analysis software package.
Collection of the historical monuments dataset:
First, a list of candidate historic monuments will be compiled. The list will include buildings from different time periods, for example Ancient Greek, Roman, Byzantine, Venetian, Ottoman and British. The most promising monuments will be collected in 2D and 3D, either from pre-existing image and mesh repositories, or from laser scanning, or via structure-from-motion.
Annotation of the collected dataset:
While compiling the list of candidate monuments, a website will be created, that will be used by architecture students to label the collected monuments in 2D or 3D. The website will illustrate 2D (images) or 3D (meshes) representations of the monuments, allowing the expert to paint the surface of the monument with different colors. Each color will represent a different architectural component or style, and the expert will be able to name (label) the painted regions.
Architectural Neural Networks
Formulate, code, train and test a neural network model (structure ANN) that can be used to infer the architectural component label (e.g. pillar, door, window, roof) for each point on the surface of a monument with high accuracy, effectively segmenting it into its component parts. Create another neural network model (style ANN) that can be used to classify parts on the surface of a monument to their architectural style (e.g. Gothic, Roman, Ancient Greek, Byzantine) with high accuracy.
Theoretical formulation and coding of the structure ANN model:
The literature on ANNs for segmentation will be examined first. The volumetric and the multi-view rendering representations for feeding 3D shapes to ANNs will be investigated and the most promising will be chosen. A set of feature maps (e.g. convexity and curvature) depending on the representation will be extracted and fed to the ANN. A formulation based on a Convolutional Neural Network (CNN) will be investigated. A software tool for converting 3D monuments to the appropriate representation, extracting the feature maps and representing the ANN will be coded.
Training and testing of the structure ANN:
The annotated dataset will be used to train the network to learn and fine-tune its parameters.
Theoretical formulation and coding of the style ANN model:
The literature on ANNs for style identification will be examined first. The input representation and feature maps will be studied, as above. The architecture of the ANN will be formulated based on a CNN, in cooperation with the University of Amherst. A software tool for converting 3D monuments to the appropriate representation, extracting the feature maps and representing the ANN will be coded.
Training and testing of the style ANN:
The annotated dataset will be used to train the network in order to learn and fine-tune its parameters. The performance of the ANN will be evaluated through standard classification accuracy metrics.
Monument design
Formulate, code, and test a method for extracting a shape grammar from a segmented and style-labelled monument which will be output from the structure ANN and the style ANN.
Literature review and key method identification:
The goal is to find among the possibilities presented in the literature, the ones that will allow a direct extraction of the rules based on the output obtained from the ANN, and taking into account that many of the features of modern architecture that the current works take into account (e.g. a high degree of symmetry) are not available for historic monuments.
Development of the ruleset extraction method:
First, the requirements will be set for the method. The remaining time will be spent on formulating, coding and testing the software tool for extracting the ruleset and its parameters. User testing will be performed after the development. The result of the task will be a method for extracting the shape grammar from a segmented and style-labelled building given as input. The method will be evaluated by comparing the rulesets it produces with ground-truth rulesets based on their similarity.
Development of the ruleset comparison method:
The same methodology and software development tools as previous will be used again. User testing will be performed last. The result of the task will be a method for comparing two procedural rulesets for two distinct models in 3D, that will be able to highlight their structural differences. Since the ruleset for each monument is created by the same method, they will be directly comparable. The method will be evaluated by visual inspection of the resulting comparisons.
Monument analysis
Design, code and evaluate a software package for analysing historical monuments. The software will assist cultural heritage experts as described in the three scenarios described above. It will therefore be composed of three main parts:
a. A tool for organising collections of monuments and identifying their construction period according to their architectural style. This will be enabled by the style ANN.
b. A tool for identifying the architectural components and illustrating the various stylistic influences on the form of a monument. This will be enabled by both the structure and the style ANNs, using the segmentation provided by the structure ANN, as well as the fine-grained style-labelling provided by the style ANN.
c. A tool for identifying the design rules of monuments, generating novel buildings that follow these rules, and comparing the structure of monuments of the same period. This will be enabled by the structure and style ANNs as well as the method for extracting building grammars.
Development of the style-based organisation tool:
The requirements for the tool will be set according to the user requirements collected at the beginning of the project. The end product will be a GUI enabled by the style ANN. It will allow an expert to browse through monuments organised according to construction period and style, search for monuments of a specific period or style, select a monument to investigate or add new monuments of unknown style which will be classified into their style using the style ANN.
Development of the style-and-structure-based observation tool:
The end product will be a GUI enabled by the Structure ANN. After selecting a specific monument using the interface of the tool (like above), this tool will allow an expert to observe its architectural parts, highlight the different styles on the monument as well as their suggested influences.
Development of the shape-grammar-based design tool:
The end product will be a GUI enabled by the both the style ANN and the structure ANN as well as the ruleset extraction and ruleset comparison method. After selecting and observing a monument in the observation tool of the previous tool, this tool will reveal how the building is put together in the form of a shape grammar with detailed construction rules. Additionally, it will be an interactive modeling tool that will allow an expert to change the constructive parameters of a given monument, being able to switch sizes, proportions, construction period and style, as well as editing other style-defining elements, like windows, entrances, columns or balconies. This will let the expert to synthesize novel monuments to solidify his understanding of buildings similar to the one selected. Finally, it will allow the expert to compare two procedural rulesets for two distinct monuments in 3D, analyse their architectural structure, and study their design rules in a separate view as a shape grammar of components and relationships. During this comparison, the expert will be able to view the monuments side-by-side while the tool will highlight their structural differences.