AR AND CHAMBER MUSIC: MUSICAL KALEIDOSCOPE

: DIGI-ORCH is a research project with objectives, on one hand, to design and develop "smart" brochures for concerts and educational programs of the State Conservatory of Thessaloniki (Greece), and, on the other, to develop an innovative system for the visualization of information on smart mobile devices (smartphones and tablets). The application of Augmented Reality (AR) provides free access to the information and content of the above concerts and educational programs, combining the information of a robust data server and an easy-to-use user interface of the smart device in real-time. The pilot implementation of the system in real conditions took place at the beginning of December 2022, in an event held at the facilities of the State Conservatory of Thessaloniki. The event was an evening of Chamber Music entitled Musical Kaleidoscope, with the internationally acclaimed artists Simos Papanas (violin), Dimos Goundaroulis (violoncello), and Vassilis Varvaresos (piano). They performed the Piano Trio no. 3 in G minor, Robert Schumann's opus 110, and the Piano Trio no. 2 in E flat major, Op. 100 D. 929 by Franz Schubert. The concert was attended by over 150 attendees. The paper will present the chaîne-operatoire of the development of the system, from the acquisition of raw data (text, video, image, and sound) to the methodology used to produce the "smart" event brochure and the AR application for the above concert. Essentially, this large amount of information that cannot be assessed by a regular printed concert brochure is included in the smart system and displayed on the mobile device when the user locates the appropriate AR patterns on the surface of the analog concert poster and/or on the pages of the brochure.


Augmented Reality in Industry Applications
Augmented Reality (AR) is a technology that is used to enhance our real-world-environment experience with the incorporation of additional digital or digitized information and content (Ibáñez and Delgado-Kloos, 2018). This digital information is projected over physical entities through the use of smart devices (e.g tablets, smartphones, etc) (Wu et al., 2013, Riegler et al., 2019. AR has gained momentum over the past decade with extensive use in retail commerce (Cruz et al., 2018, Lavoye et al., 2022 healthcare (Eckert et al., 2019, Laverdiere et al., 2023, education (Yakubova et al., 2021, Mystakidis et al., 2022, industrial operations regarding product design and assembly (Li et al., 2022) and of course cultural heritage (Choudary et al., 2009, tom Dieck andJung, 2017). In particular, the use of AR in cultural heritage applications includes an enhanced experience in visiting archaeological sites (Perdicca et al., 2015, Kasapakis et al., 2016 or museums (Franz 1 * Corresponding author et al., 2019, Serravalle et al., 2019, thus contributing in the creation of the so-called digital heritage. UNESCO has provided the following definition of digital heritage: "Digital Heritage is made up of computer-based materials of enduring value that should be kept for future generations. Digital heritage emanates from different communities, industries, sectors, and regions." (UNESCO, 2023). AR can be used in cultural heritage locations to make the user more conscious about his visit (Capuano et al., 2016), allow him to view an artificial reconstruction of historic places or artifacts (Boboc et al., 2017, Abate et al., 2018 and, overall, enhance his perception and knowledge of old and static artifacts. An additional benefit of applying AR technologies in cultural heritage is the use of the so-called "Marker" to enable digital content reproduction instead of a traditional QR code. A marker can be any 1D or 2D image that can be identified or captured by a mobile device's camera without having to place an artificial mark (e.g a QR code) on a museum's wall or artifact. Once the marker is identified by the camera the application can overlay on it different content including video, audio, images, 3D models, etc. This content can be updated at regular intervals to The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy keep the user interested in the exhibition, the artifacts, or the artists even after he has departed from the physical premise of the cultural heritage event.

The DIGI-ORCH Research Project
Classical music events and performances are usually accompanied by printed material which -in most cases-includes a poster with essential information on the event and a booklet containing the actual concert program. The limited printed space and the need to avoid cognitive overload inevitably lead to information reduction. The content of the printed material is static and due to the limited time span of each concert is never updated. As a result, each member of the audience is required to carry online research on the composer and his work, the soloists, the orchestra, etc. In the case of classical music, which is considered by some to be more elitist than other genres, this research can be more time-consuming since the available resources are more scattered over the internet. Indeed the providers and aficionados of classical music usually have an educational background and are not active content creators. Furthermore, one has to bear in mind the uniqueness of each performance -as in any form of art-which is attributed to the conductor or the soloists, so a general analysis of a composer's work that can be found online may only provide a small insight on the specific concert. DIGI-ORCH is a research project that aims at designing and developing "smart" and interactive printed concert materials and also developing an innovative system that will allow for the visualization of this information through smart mobile devices (i.e smartphones & tablets). More specifically, its research objectives include the designing, development, and implementation of a system for mobile devices that will provide an extension of the experience of visiting classical music concerts or relevant educational programs. This system will use database and augmented reality functionalities to allow access to more detailed information on the music events through an intuitive and user-friendly mobile application. The application will also be able to record comments on the event by the audience in the form of an electronic guestbook. DIGI-ORCH brings together an Academic Organisation (Laboratory of Photogrammetry and Remote Sensing, School of Rural and Surveying Engineering, AUTH), a Cultural and Educational Foundation (State Conservatory of Thessaloniki, Greece-TSC), and two private companies in the field of visual communication/advertising (Beetroot) and consulting (VBC). The project is organized into three phases and seven work units to which each partner contributes according to his expertise. Musical Kaleidoscope, which was a chamber music concert for piano, violin, and cello, organized by the State Conservatory of Thessaloniki at its premises in early December 2022, was the pilot project for the implementation of the system under development. In this event, the internationally acclaimed artists Simos Papanas (violin), Dimos Goundaroulis (violoncello), and Vassilis Varvaresos (piano) performed the Piano Trio no. 3 in G minor, Robert Schumann's opus 110, and the Piano Trio no. 2 in E flat major, Op. 100 D. 929 by Franz Schubert.

User Requirements Analysis and System Design
The first step in developing the "smart" concert material and application involved a detailed analysis of the end-users' needs, the definition of the technical specifications of the system and the data to be used in it, and the design of the system architecture. The initial conceptualization of the system is presented in Figure   1. The data that is expected to be included in the application can refer to the concert itself, other educational material, or online resources. It can take the form of video, audio, text, animation, and 360 or 3D data as well. The implementation of the application includes the development of a content management system that will include functionalities for data storage, management, and serving of dynamic information. In addition, smart communication materials/documents will be developed that along with a mobile application will allow for the projection, and superimposition of 3D objects on the printed documents, the visualization of the additional content in a user-friendly interface, and the promotion of the events on social media.

Figure 1. DIGI-ORCH System Conceptualisation
The implementation of the aforementioned conceptual schema requires the development of three (3) distinct functional units, namely the Mobile User Interface (UI) Component, the Content Management UI Component, and the Back End Components Infrastructure. The development of each component is preceded by the analysis of the users' needs in terms of functionalities and data. A detailed survey was drafted by AUTH and filled out by the State Conservatory of Thessaloniki (TSC) and members of the audience of various concerts to provide the necessary information on specific user requirements. Based on the results of this survey, TSC appears to organize or participate in an average of 60 music concerts of symphonic music per year. It also holds multiple seminars and educational programs. As a result of its intense activity and participation in multiple cultural events, TSC has established a significant archive of more than 30,000 records, which could be further exploited and presented to the general public in a visually enhancing and interactive way. This existing material (e.g posters of former events, brochures, etc) was analyzed by the project's partners to understand the semantic design that is followed by the institution, as music -and culture in general-does not follow the communication patterns of other commercial products. Furthermore, it appears that TSC has a loyal audience that participates in approximately 70% of the institution's events per year. This audience is primarily informed about the concerts through social media platforms and TSC's web page. In addition, the audience is strongly positive about maintaining an analog printed communication material (e.g concert program) preferably containing information such as soloists' curriculum vitae, information on the composers and their work, musical analysis, or even rare/exclusive footage. User requirements analysis was followed by the actual system design. Two main subsystems had to be developed. The first one The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy regards the content management infrastructure and the second regards the development of the mobile application that the users will install on their devices. The content management system is implemented through a web interface that allows users with administrative privileges, to create new events, upload information and files that are to be included and served through the mobile application and connect this data to each communication material (Figure 2). The mobile application is organized in several frames as presented in the wireframe of Figure 3. The initial screen presents the various events (upcoming and previous) and newsletters that are available from the State Conservatory of Thessaloniki.. Once the user selects an event, he is redirected to the following screen where he can receive information on the geographic location of the event, a 3D virtual reality tour of the concert premises, and, of course, the smart communication materials that are available for the occasion. Each communication material has its content markers designed on it, to facilitate the user's browsing experience. If the user, enables the camera of the smart device he can enable the smart content visualization option.

Database Development
The database development started with its conceptual design based on the material provided by the State Conservatory for previous events and the analyzed system, data, and user requirements. The conceptual design was implemented by drafting an entity-relationship model on an initial set of database tables. These tables included the "Events" table where all the information on each concert/event will be stored, the "Venue" table which will contain information on the event's location, the "Communication Material" table where all information on the specific communication material drafted for each event will be stored, and, finally, the "Content" table which will contain all the information that will be superimposed to the analog content material using AR techniques. The entity-relationship diagram created during the conceptual design can be seen in Figure 4.

Figure 4. Entity-Relationship Diagram
The logical database design included the implementation of the common database normalization rules to our data (i.e 1NF, 2NF, 3NF, and Boyce-Codd). As a result, a total of 8 tables and 7 Look-Up Tables (LUT) were created in the database. The physical database development was implemented in PosgreSQL. A database server and instance were developed to record information on each smart communication material as recorded by the authorized users through the web interface presented in Figure 2.
The recorded data can be retrieved from the database and the CMS using two custom Python scripts that return all the content in a JSON format ( Figure 5).

Drafting of the Communication Material
The drafting of the communication material is an essential early step as it is inextricably linked to the data that has to be collected or created for the application. In the case of the pilot project "Musical Kaleidoscope", the stakeholders agreed in drafting two distinct communication materials, namely the concert poster (printed size A3) to be distributed and posted a couple of weeks before the event at various locations and online to inform the city's audience about the forthcoming concert, and the concert's program brochure that would be distributed at the audience in the day of the event (printed size A5). The drafting of the communication material required immediate and continuous interaction between the stakeholders in a chain format as follows.
The State Conservatory of Thessaloniki, being the organizer of the event, had to decide on the content to be included in each material either in the form of text or images. The content was organized in thematic units regarding the event itself, the composers, the soloists, the sponsors, etc. Beetroot, which is the visual communication expert, had to design the material bearing in mind the technical requirements and limitations of the AR module which demands clear patterns on each document to enable the activation of content delivery. The Laboratory of Photogrammetry and Remote Sensing had to test the design of the documents with regard to pattern recognition and propose corrective actions to the designers. The result of this interaction can be viewed in Figure 6 containing the original and the final event's poster design. The soloists' images had to be re-arranged and modified so that critical patterns could be identified by the mobile device's camera. Other images and text patterns also had to be re-arranged and re-sized. This task was particularly challenging in the limited space of an A3 printed document.

Data Collection and Organisation
The fourth step of the process involved the collection of the necessary data to be included in the application and served to its end-users. At this stage, the dissemination actions were also set in motion, including the development of a web page for the project (https://digiorch.eu/) and the creation of contact and information points on several social media platforms including Facebook and Twitter. Data collection is organized on a per-event basis. The main organizer of the event, i.e TSC, collects all the necessary material to be included in the AR module. The material can be either in text, audio, video, or hyperlink format. This data has to be subsequently recorded in the project's database using the web interface presented in Figure 2. The data is then related to each marker of the individual communication materials. The relevant accompanying files are uploaded through the web interface to the content management system that has been developed. In the case of the "Music Kaleidoscope" chamber music concert, the State Conservatory of Thessaloniki provided the bulk of the content to be superimposed using AR technology on the smart communication material designed by Beetroot. Furthermore, a 3D Virtual Tour of the concert hall was created by VBC as well as exclusive footage of the concert's general rehearsal with the three soloists providing information on the work of Schuman and Schubert. All the material was recorded in DIGI-ORCH's The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy database. Seventy-three (73) additional content features were added to the static communication material content, following the distribution presented in Figure 9.

Application Deployment and Testing
The smart application was developed using the Vuforia Software. It was initially deployed on a private server and will soon be available on Google Playstore and Apple Store. The user, once the printed communication material is ready, can browse through the application as follows: The initial welcoming screen allows the user to select forthcoming or previous events organized by the State Conservatory of Thessaloniki. Once an event is selected, the user can view its venue's location as a google map pin and get directions to it. An additional button provides a 360 tour of the concert premises ( Figure 10).  Once the user enables his mobile device's camera he can scan the printed communication material at the locations where each marker is designated. Once the camera identifies the marker, a 3D polygon appears on the screen. The user can select the type of content he wishes to view (i.e video, audio, text, images) by making the appropriate selection on the panel located at the bottom of his screen. If a certain type of content is not available in a particular marker its option is greyed out. If more than one contents are available per type, the user can browse through them using the back-and-forth arrows located under the window screen ( Figure 12).

Presenting the Application at the MUSIC KALEIDOSCOPE event
The chamber music concert "Music Kaleidoscope" took place on the premises of the State Conservatory of Thessaloniki on the 10 th of December, 2022 with approximately 150 people attending. Before the start of the concert, the audience was requested by a representative of DIGI-ORCH' s partners to simply download the application using a QR code that could be found on the concert's program back page. Later in the concert, during the break between the two music acts, the application and its use were fully presented to the audience using video instructions and a Q&A section. The audience was encouraged to use and evaluate the application onsite or even later on. Several people participated in the process and were keen in asking questions regarding the application and its content. Figure 13. Presentation of the smart-app's tools and functionalities during the concert's break

Building & Filling the Evaluation Survey
A short evaluation survey was developed to assess the design, the user interface (UI), and user experience (UX) as well as the overall effectiveness of the DIGI-ORCH mobile application and the "smart" communication material. The questions followed a LIKERT scale format and can be found in Table 1.
Questions of the Evaluation Survey 1. What is the cost of the mobile device with which you are accessing and evaluating the application? 2. Rate on a scale of 1-5 your browsing and ease-of-use experience of the application 3. Rate on a scale of 1-5 the way information is presented. 4. Rate on a scale of 1-5 the responsiveness -in terms of timeof the application 5. Rate on a scale of 1-5 the design of the user interface 6. Rate on a scale of 1-5 the design of the smart printed document 7. Rate on a scale of 1-5 the importance/benefits of the mobile application Table 1. Evaluation Survey Questions Fifteen (15) members of the audience participated in the survey while an overall total of sixty-two (62) people accessed the survey platform. The vast majority (approximately 87%) accessed the application with an average-cost smart mobile device ranging from 100-500 euros. The overall browsing experience was rated between "average" (26.67%) and "excellent" (46.67%). The information presentation experience of the app was generally considered to be very good (73.34%). The responsiveness of the application was also very well-rated (80% considered it to be "very good" or "excellent"). The design of the user interface and the "smart" printed material were well received (86% considered it to be "very good" or "excellent"). Finally, approximately 90% of the survey participants rated the mobile app as very useful. Although the survey results remain to be verified by the second pilot eventexpected in June 2023-it can be said that the mobile application was generally well received by the audience of the concert and was considered a novelty in the area of classical music events. The fact that the application was responsive even in average-cost smartphones, using the 4G mobile network connection assisted significantly in its level of approval.

DISCUSSION
The use of AR technologies can contribute significantly to enhancing user experience in several fields with culture proving to be a major beneficiary. An AR-enabled application can turn our mobile devices into sources of knowledge with exclusive content being prepared specifically for each event, concert, museum, archaeological site, etc. For the AR technologies to be enabled through the device's camera we need an adequate amount of markers to be identified in which we can add content to be superimposed on via our devices. Unfortunately, in the case of museums or archaeological sites, the marker is usually a static artifact. In this sense, the user can only use the app during his presence at the specific venue. DIGI-ORCH is a research project that brings together partners of different backgrounds with one common goal: to create smart printed communication materials (e.g posters, concert program booklets) and an accompanying AR application to provide this enhanced audience experience to classical music lovers. In this case, the user can take the material with him and revisit the application and concert experience at his convenience. The pilot project named "Music Kaleidoscope" proved to be a success, but several issues remain. For example, the majority of the audience were older people who did not seem very comfortable with downloading and using mobile applications. In addition, they appeared keener on the actual concert than the additional content that the application offers (e.g rehearsal footage, soloists bios, etc). In that regard, and looking forward to the main that is to be organized in June, two improvements are suggested and discussed by DIGI-ORCH's partners. The first one has to do with the application's User-Interface and User-Experience which has to be more intuitive and better designed. The second has to do with the actual content which needs to be of various types (e.g audio, video, images) and also needs to be exclusively made for the concert so that the enduser is more intrigued to view it. Additional capabilities could be added to the application, such as an online booking system, but the key remains in the content itself and the user interface.