RESHAPING THE FUTURE OF PORTUGUESE AZULEJO PATTERNS

: This paper introduces a new approach to the inventory and catalogue of azulejo patterns found in Portuguese buildings. It uses computer-vision based software tools for automatic search and matching of azulejo patterns, thereby improving the scalability and speed of existing cataloguing methodologies. The online catalogue of azulejo patterns is called Az Infinitum (Azulejo Referencing and Indexation System), a publicly accessible online portal suitable for both researchers and the general public who are interested in exploring and understanding this cultural heritage of Portugal. The effectiveness of this catalogue as a research support tool is demonstrated using a case study based on the Marvila pattern (i.e. P-17-00999). The online catalogue has inspired the development of an engaging application, called Azulejar , which allows one to create new patterns or understand the mathematical process behind existing azulejos patterns. This application has a potential to become an effective educational tool for inspiring everyone to explore and understand the science behind the beauty of azulejo patterns.


INTRODUCTION
The azulejo (1) is one of Portugal's most original, creative and distinctive art forms and has been in constant transformation since the late 15th century. Applied across the whole country, the azulejo has reinvented itself throughout the ages -with regard to its use, its style, the messages it conveys and even its conceptual meaning. It is thus an important part of Portugal's cultural heritage and collective memory, shaping the views of the past, but also of the present and the future.
For more than five centuries, decorative patterns were one of the most common azulejo types in the country, alternating or coexisting with figurative ones. Created using a standardized production technique, these azulejos only revealed their full decorative effect once they were applied, covering vast walls combining different patterns and frames. In other words, and unlike their figurative counterparts, patterned azulejos were not designed for a specific location. It was the combination of the different patterns and frames and their articulation with the architecture that made each covering unique, endowing a given space with a specific rhythm, colour and light -among other features that would significantly alter the way it was perceived.
The inventory, cataloguing and documentation process of existing azulejos still applied in situ, that is, still adorning the interiors and exteriors of Portuguese architectural works, remains a considerable challenge, both for researchers and for the institutions that own or manage buildings with this kind of * Corresponding author. 1 The Portuguese word Azulejo does not mean merely tile, but rather a concept and a tradition of laying ceramic wall coverings in Portugal. decoration, mostly due to the scale of the task (the number of buildings, their geographical distribution, etc.). This paper introduces the Az Infinitum -Azulejo Referencing and Indexation System (2) which is an umbrella project for various initiatives related to the collection, processing, visualization and dissemination of information related to azulejos from all centuries, including patterned azulejos (Carvalho, 2019) (3). A notable innovation of the Az Infinitum project is the use of the VGG Image Search Engine (VISE) software (Dutta et al., 2021a) -a computer-vision based toolfor the automatic matching of azulejos and the improvement of the cataloguing process. Based on Az Infinitum project, another the team has also developed an application, called Azulejar (4), which allows users to create new patterns or understand existing azulejo patterns.
This paper is organised as follows. The Az Infinitum project is described in more detail in Section 2. The computer-vision based approach for matching azulejo patterns is described in Section 3. Then, Section 4 describes a case study of the 17thcentury Marvila pattern (or P-17-00999) and Section 5 describes the Azulejar project, both of which are based on the research support tools developed in the Az Infinitum project. Finally, Section 6 points to possible directions for future research and application developments in this area.

CATALOGUING AZULEJO PATTERNS
Azulejos are spread all over Portugal, still serving their original goal of adorning churches, palaces, private houses, public buildings, façades and so forth. Such geographical spread and continued practical use bring challenges to the process of systematically cataloguing these historically important tiles. In other words, one needs to "open all doors" and systematically photograph all the walls and ceilings containing azulejos. Even after one has acquired a photographic representation of the azulejo patterns spread all over the country, it is difficult to manually catalogue them based on visual inspection -the cataloguer would want to know if a newly discovered pattern is already in the catalogue in an efficient way. So far, this process had been achieved by visual inspection of the patterns by experts. For example, (Simões, 1971) developed a methodology to catalogue azulejo patterns and applied it to create a comprehensive inventory of 17th-century azulejo patterns. This methodology was based on a set of manual processes which was time consuming, error prone and difficult to scale for other centuries. These limitations have contributed to the current state of the art, where we lack a comprehensive and systematic catalogue of azulejo patterns from other centuries. This has constrained scientific studies on azulejo patterns and therefore most research in this area is focused on specific examples.
The Azulejo Referencing and Indexation System (i.e., Az Infinitum), introduced in this paper, is an online portal for cataloguing and studying azulejos. Figure 2 shows a screenshot of the online portal. It uses a computer-vision based software tool, described in more detail in Section 3, to uniquely represent each azulejo pattern such that the azulejos can be automatically matched solely based on their patterns. This visual matching capability is unaffected by changes in lighting conditions, partial occlusion and camera perspective contained in a photo-graph showing an azulejo. The Az Infinitum also includes textual metadata associated with each azulejo which identifies its geographic location, designation, painter, factory, production date and details about possible sources of inspiration for the patterns. These metadata are often populated by researchers based on scientific evidence and research.
The azulejo's photographs and its metadata catalogued in the Az Infinitum portal are open to the public for exploration, search and study. This allows a user to search the catalogue in two ways. First, using text through the metadata associated with 4 https://azulejar.art each azulejo record. The text search feature, for example, can be used to find all the tiles produced in a certain century, with a specific motif, or by a particular factory. A second way is visual search, in which a user uploads photographs (e.g., taken using a mobile phone) to the portal that can be used as a search query. The search result shows a list of all azulejos in the catalogue that match the visual patterns contained in the uploaded photograph, as shown in Figure 5. This type of search is useful in the cataloguing process of newly found azulejos because a visual match with existing tiles in the catalogue indicates that some metadata (e.g., factory, production date, etc.) are mostly likely common between them.

Figure 2.
Az Infinitum website at https://redeazulejo.letras.ulisboa.pt From the start of the project, the Az Infinitum platform was designed and developed to allow researchers to address research questions such as the followings (Carvalho, 2019): § Establish precise chronologies of manufacture and application of patterned azulejos; § Recognise workshops/factories/authorship and evaluate the manufacture of the azulejos; § Distinguish families of motifs, analysing why certain patterns were favoured over time; § Determine how colours can be used for helping to establish chronologies; § Identify recurring combinations of patterns and frames; § Identify external influences and inspirational sources from different parts of the world; § Determine how matters of scale, design and application of patterns have been understood over time, highlighting their close interaction with architecture.
However, to reach some of these conclusions, researchers need access to all the patterns and associated information, which is a very challenging task. For example, the tools developed by the Az Infinitum project have allowed us to refine and extend the The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy inventory created by Santos Simões (Simões, 1971) for the 17th-century. Simões identified around 1800 places in Portugal with 17th-century azulejos in which he catalogued around 300 patterns and 190 frames (a total amount of around 500), as shown in Table 1. Az Infinitum has already a total of 232 patterns and frames, of which 133 are new, that is, they were not catalogued by Simões. Regarding the other centuries, the Az Infinitum catalogue contains 517 patterns and 360 frames, as shown in Table 2. However, the number of azulejo patterns scattered all over the country, including the archipelagos, is very large.

Type
Simões ,  Frames 360 Table 2. Chronological distribution of patterns catalogued in the Az Infinitum.
The task of processing, analysing and cataloguing a huge number of geographically distributed azulejo patterns is a challenging task for an individual researcher or even for a team of art historians relying solely on manual comparison and observations. Therefore, the search tools and catalogue introduced in this paper are a significant research support tool in realising a catalogue of all azulejo patterns found in Portugal.

WHEN PATTERNED AZULEJOS MEET COMPUTER VISION
The Az Infinitum project uses the VGG Image Search Engine (VISE) (Dutta et al., 2021a) tool to automatically search and match azulejo patterns, thereby easing and speeding up the cataloguing process. VISE is an open-source software application based on computer vision that allows for an instantaneous visual search of large image collections of, for example, azulejo patterns, using an image as a search query. VISE has already been successfully used for the matching and analysis of early printed book illustrations (Bergel et al., 2013, Dondi et al., 2020, Dutta et al., 2021b, buildings (5) and recurring objects in TV programmes (6). To our knowledge, this is the first time that this research application has been applied to patterned architectural surfaces for a real-world task and at scale.
VISE operates by first applying a region detector to images containing azulejo patterns. This process helps identify a set of image regions (or patterns) that can be consistently detected even when the azulejos get rotated, translated or are photographed from a different distance (i.e., changes in scale). These image regions are called scale and affine invariant regions and are depicted using white coloured ellipses as shown in Figure 6. Next, VISE extracts a 128 dimensional SIFT feature vector (i.e., a sequence of 128 numbers) for each such elliptical region and uses the K-means clustering technique to learn a set of cluster points (referred to as 'visual words') that can be used to represent visual content (i.e., patterns) efficiently and more robustly in the detected elliptical regions of an image. All the detected regions in a photograph containing an azulejo are represented using these visual words. The visual search operation now simplifies the task of efficiently searching and matching between the visual words assigned to the query image region and visual words corresponding to all the images in the catalogue. The correspondences are illustrated in Figure 6 using black coloured lines between elliptical regions in two photographs showing the same azulejo pattern. VISE uses an efficient data structure to precompute and store the visual words associated with all the images in the Az Infinitum catalogue which enables the efficient storage and nearly instantaneous lookup (i.e., search) of all the catalogued images that contain a particular visual pattern. A more detailed explanation of the visual search process is present in Section 2.1.1.2 of (Arandjelovic, 2013).
The VISE software assists the cataloguing in two ways. First, it allows researchers to take a photograph of a Portuguese architectural structure containing azulejos and use this image as a search query within the catalogued collection of azulejos, as shown in Figure 5. Matching search results allow the researcher to quickly identify and catalogue the new-found azulejos. Second, the software allows researchers to group azulejo images based on their patterns, thereby helping identify the reuse of certain motifs.
The results shown in Figure 5 help clarify the evolution of the "corncob" pattern. The first (P-17-00101) was catalogued by Santos Simões but the second is a new pattern (P-17-01068) and the third, in blue and white colours, corresponds to a new trend started in the last decades of the 17th-century. Finally, the fourth (P-20-00102) is a pattern from the 20th-century. Here, although the motif is the same, the scale is different -the corncob occupies an entire tile, while in the 17th century it occupied four. This change is justified by the application, because since the middle of the 19th-century patterns were applied on building façades and the available surface area was much smaller, which had the consequence of reducing the scale of motifs in order to be seen in their entirety.

A CASE STUDY BASED ON THE MARVILA PATTERN (P-17-00999)
In this section, we describe a case study based on the most impressive pattern produced in Lisbon's workshops, usually called Marvila pattern (or P-17-00999), because it was found in the Church of Marvila, Santarém (Figure 3). To form the module, 144 azulejos are needed. In comparison, other smaller tile patterns only require 4 azulejos. In spite of the known exceptions to this practice, small patterns were usually applied on the bottom walls, while the bigger patterns were applied on the upper walls, given that the largest drawing resists better to a visual reading and to a bottom-to-top viewing, or in perspective.
Santos Simões (Simões, 1971) established a chronology for the Marvila production from 1620s/1630s to the 1670s and provided a list of around 30 places where the P-17-00999 pattern was identified. The Az Infinitum project has already catalogued three places with this pattern. During the process of analysing visual data using the computer-vision based tools developed in this project, it was possible to distinguish at least three different versions of the same pattern, already catalogued, as shown in Figure 7. We expect more versions to appear in the future, as was the case of the ones found in the Church of the Monastery of São Salvador de Grijó (Vila Nova de Gaia) ( Figure 1) and the Church of Chagas (Lamego) (Figure 4). The search and matches of the Marvilla pattern illustrate the accuracy/capability and benefit of computer vision tools applied to azulejo pattern matching. Recent research has been corroborating and updating Santos Simões' work, adding new Marvila patterns to the catalogue and more precise chronologies, as is the case of the Church of Marvila (Santarém) documented by Vitor Serrão from 1634-1643 (Serrão, 1995), the Church of Santa Marta (Lisbon) from 1638-1639 (Serrão, 1977) (Léon, 1993) (Carvalho et al., 2012), or the Church of the Old Convent of Salvador, in Braga, with documentation placing this pattern after 1632 (Carvalho and Silva, 2016). Moreover, in Santarém, the documentation discloses the name of the tile layer (7), Gaspar Gomes, linking him to other patterns and places (8). Thus, research has been providing more and more precise data, advancing the existing knowledge, albeit slowly and on a single pattern basis. In the near future, this case study will be developed with the catalogue of all Marvila patterns identified.

ENGAGING APPLICATIONS BUILT AROUND AZULEJO ATTERNS
The Azulejar (9) project aims to inspire students and the public in general to explore and understand the science behind the beauty of azulejo patterns. The Azulejar application has the potential to be used as an educational tool and is useful for informing the younger generation about this cultural heritage.
The Azulejar also allows users to create new patterns or simply understand the mathematical processes involved in the repetition of patterns by rotation, translation, reflection or sliding re-flection. For example, 144 tiles are used to create the 7 The one who applies the tiles. The tile layers were an autonomous profession, separated from the masons since the beginning of the 17th-century and gained distinct importance throughout the centuries. Due to his responsibility in the planning and execution of the work, the tile layer can play the role of the general organizer, establishing contact with the commissioner, surveying the space (measurements), distributing the commission through the potters and the painters, and directing the final application. 8 https://redeazulejo.letras.ulisboa.pt/pesquisaaz/ficha.aspx?id=330&ns=402000&Lang=PO&c=autorias&I PR=%20442. 9 https://azulejar.art The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy full Marvila pattern. If these tiles are arranged in a different way, we get a different composition. This highlights the creativity and artistic capabilities of the people who laid the tiles (i.e., tile layers). With the Azulejar application, users can experience the process of creating new patterns and experiment with all the possible ways of combining tiles.
The Azulejar application relies on the Az Infinitum catalogue and database to allow users to search and browse the collection by applying various filters and search criteria such as pattern type, colour, chronology and location. This enables researchers to understand novel research questions about azulejo patterns. As the goal of Az Infinitum is to improve the size and scope of the catalogue in order to build a comprehensive database of azulejo patterns, we are also exploring the possibility of using citizen science to invite the general public to contribute. For example, a photograph captured by mobile and containing azulejo patterns can be uploaded to the Azulejar application and be used as a search query to check if the pattern has already been catalogued. If the pattern exists in the Az Infinitum catalogue, more details about the azulejo pattern can be shown to the user. If the pattern has not been catalogued, the information will be used to create a new entry and, above all, to map the tile's location. Such contributions will be essential to gather evidence from all over the country.

CONCLUSIONS
A comprehensive and easily searchable catalogue is essential for a systematic and scientific study of azulejo patterns which are geographically spread all over Portugal and are still fulfilling their original purpose of adorning buildings. This paper described a novel approach to building such a catalogue (Az Infinitum) by using computer-vision tools to scale and speed up the process of searching and automatically matching of azulejo patterns. A case study based on the Marvila pattern (P-17-00999) has shown the effectiveness of such an electronically and visually searchable catalogue by revealing that at least three more versions of this pattern exist in the catalogue. We have also shown the potential of the Azulejar application as an educational tool which not only allows for the study of existing azulejo patterns but also enables the creation of new patterns. The azulejo patterns catalogue is publicly available at https://redeazulejo.letras.ulisboa.pt. Figure 5. Visual search of the Az Infinitum catalogue using a photograph of a church wall in the Chapel of Nossa Senhora do Rosário (Escudeiros, Braga). The search results show matching azulejo images along with their catalogue number. In this example, the search results indexed as 1, 2 and 4 are from the 17th-century (i.e., P-17) and the result indexed as 3 is from the 20th-century (i.e. P-20).

Figure 6.
The VISE software can also highlight the matching regions (white ellipses) between the search query (left) and one of the matching results (right). VISE uses the visual pattern in these regions to find a match in the catalogue of azulejos images. The lines (shown in black) denote the correspondence between the region in the search query and that in the match image. For the match result shown above, there are 44 such regions but only 3 are shown here for clarity.