iBLUECULTURE: A NOVEL SYSTEM OF REAL-TIME UNDERWATER IMAGE TRANSMISSION IN A VIRTUAL REALITY ENVIRONMENT, AS A NEW MANAGERIAL APPROACH FOR UNDERWATER CULTURAL HERITAGE

: The Mediterranean is rich in ancient and modern shipwrecks and submerged sites of great archaeological, historical, and cultural value. Most of those sites tend to be relatively unknown, even to people living in the area, mostly due to the inherent limitation in physical approach. As a result, these sites are left unknown and unprotected. The aim of the iblueCulture project was to create a system that can make Underwater Cultural Heritage sites broadly accessible and raise public awareness on their location, content, and value. This approach is also meant to provide an effective management strategy for the constant, real-time monitoring and, therefore, protection of such sites, while in accordance with the sustainable tourism development goals for islands and coastal areas. This project, as implemented, can offer an immersive dry dive experience to the public, while providing management bodies and stakeholders with the capability of remotely monitoring the UCH sites in real-time


INTRODUCTION
Beneath the seas, but especially the Mediterranean, lies a rich treasure trove, one of ancient and modern shipwrecks and sites, with enormous archaeological, historical, and cultural value.One has only to consider the Antikythera wreck, and how much it enhanced the development of Maritime Archaeology as a discipline, since its discovery in 1900 by sponge divers.Near the Turkish coast, the Uluburun shipwreck, contained a treasure trove of cultural artifacts, approximately 18000 in number, coming from various cultures and dated to 1300 BC (Delgado, 2000).The Kyrenia shipwreck, which was remarkably wellpreserved, is on display in the Crusader Castle in Kyrenia, Cyprus, while a replica, the Kyrenia Liberty, sailed from Cyprus to Athens for the 2004 Olympic Games (Harpster, 2019).With such considerations in mind, the value of Underwater Cultural Heritage (UCH) sites becomes easily evident.
Apart from shipwrecks, Maritime Archaeology also involves and studies all evidence of maritime activity, such as harbours or even areas such as settlements, that are now submerged due to the changes in sea level (Bailey, 2016).Such sites, that are on land or partially submerged, are relatively famous or, at least, known to people in their vicinity.UCH sites, however, tend to be relatively unknown to the public compared to surface ones, even to the populace living in nearby areas.In the case of UCH sites in significant depth, very few would know of their location or even existence.The main factor behind this lack of knowledge is both the scarcity of multimodal information and the overall difficulty and challenges involved in a physical approach, as they are currently accessible only to divers.
In some cases, UCH site stakeholders have tried to make such information more readily accessible to the public, through websites or exhibitions of the salvaged ships, as is the case with LaSalle's La Belle or the travelling exhibition of the Julia Ann.This method offers the possibility of seeing the ship, or parts of it, as it is now recovered and on the surface.While these recovery and exhibition efforts have been excellent, this approach cannot be applied to all UCH sites, as a multitude of submerged wrecks remain.Most of these will probably not be retrieved in the near future, if ever, seeing how costly that endeavour can be.This leaves many UCH sites in obscurity, unknown to the public, while the sites themselves are still in danger, be it from natural disasters, natural decay or even theft.
The iblueCulture research project developed a 3D virtual diving exploration system, with an innovative live-streaming and texturing method, that captures real time video underwater and incorporates it in the virtual environment.This enhances user experience, by offering much greater immersion than a video or a Virtual Reality (VR) reconstruction, as the user can see the UCH site and its immediate environs in real time.The system incorporates various enhancements, for example changes in lighting conditions in accordance with the actual time of the day, thus effectively offering users on dry land a "dry dive" experience.
The iblueCulture system can be installed in almost any UCH site, depending mainly on depth and the immediate environs.It can be used in various contexts, from museum exhibitions dedicated to underwater sites, to portable devices that mirror user movements on sea level above or nearby the underwater site, to edutainment applications and school visits in suitably prepared kiosks.The system has been designed as location agnostic, meaning that, after being installed and modified for each case and UCH site, these 3D virtual explorations can be experienced in-situ, while on dry land nearby, or at sea level, on board a vessel, directly over the actual site.It can also be used remotely, in a museum or awareness kiosk, in research labs and educational institutions, with the appropriate VR equipment and installation.There is also the possibility to experience the system in a multitude of portable devices, such as smartphones, tablets or even VR headsets, while Augmented Reality (AR) devices could also work.It can also serve as a security method for UCH sites, offering stakeholders the ability to monitor the area in real time.
The pilot system has been implemented to allow non-divers to dry dive and navigate in select UCH sites in the Greek archipelago, including three Byzantine shipwrecks as concentrations of amphoras at Peristera islet near Alonissos island, as well as the modern wreck of the cargo ship "Christophoros" at Skopelos island.

STATE OF THE ART
There have been many AR/VR Cultural Heritage applications, though most of them focus on surface sites.Similar projects that aim at UCH sites are, however, being implemented and becoming more numerous in recent years.One such project proposed and developed an AR guide for divers, using as an example a 3D reconstruction of an ancient villa at the underwater archaeological site of ancient Baiae, near Naples, Italy (Cejka et al. 2019) (Bruno et al. 2019).DOPLYHN is a proposed AR underwater solution, tested in swimming pools and aimed at divers (Bellarbi et al. 2013).The VISAS project developed a virtual diving system in 3D reconstructions of two UCH sites, in Capo Collona and Cala Minnola, Italy (Bruno et al. 2018).The iMARECULTURE project aimed to bring UCH sites to light and to increase public awareness through virtual visits and edutainment solutions for surface visitors, and multimodal information for divers (Scarlatos et al. 2016).The VIRTUALDiver platform aims to create a complete set of tools that encompasses the creation, management, and implementation of tours in UCH sites, to enhance both surface (via dry dive) and underwater tourism (Pehlivanides et al. 2020).The Lab4Dive Mobile Smart Lab was designed to help underwater archaeologists by offering a complete solution in collecting, collating, and reconstructing data from a dive, to make survey planning easier and more accurate (Scaradozzi et al. 2018).The Virtual Museum -Underwater Malta has an ample collection of 3D models, from shipwrecks in the area (Gambin et al. 2021).The Mazotos shipwreck can be experienced in an immersive virtual dive, with a static 3D model and procedurally generated environs (Liarokapis et al. 2017).Most of the projects that focus on UCH sites use data taken in previous visits, in order to recreate the shipwreck and the environment around it.This, of course, offers a highly detailed view of the site, via a 3D model that can be improved after later visits.The iblueCulture project's innovation is that it combines the pre-created 3D model with the live, real-time view and it can offer it to the users, as explained below.

THE PILOT SITES AT PERISTERA
Two pilot sites were selected for this project.The Byzantine shipwrecks A, B and C are in the bay of Vasiliko at the islet of Peristera in Alonissos and are located at a depth of 50-57 meters, all in the form of piles of amphoras of the same type preserved in a very good condition, estimated at 3,000-4,000 at each shipwreck.The shipwrecks date to the 10/13th century A.D. The Christophoros contemporary shipwreck (Figure 1) dates back to 1983 and is much larger in size than all the Peristera shipwrecks.Before moving on to these pilot sites, however, we opted to test our system in the ancient shipwreck (5 th century B.C.) at Peristera islet in Alonissos (Figure 1), sitting separately from the Byzantine ones.It is this site we will focus on, in the rest of this paper.

THE IBLUECULTURE SYSTEM
The iblueCulture system consists of three linked subsystems: a camera setup, a relay station and a processing station (Figure 4).It is location agnostic, meaning that the only physical requirement is having the camera setup in situ, and the supporting hub on dry land nearby.The small footprint of the supporting hub also means it can be placed at even a small rock islet, as in the case of the ancient shipwreck of Peristera (Figure 3).Other than these two limitations, the rest of the system, including the end user's device, can be located anywhere.

Camera Setup
The 8 cameras used for the pilot project were Arduino Cams, each attached to a Raspberry Pi4, that handles the video streaming.Each camera and Raspberry are encased in a specially crafted waterproof housing that can withstand pressures of up to 80m depth.The housing has an automated wiper, that works once a day to clear any debris that might have been collected on the housing window and could obstruct the camera's vision.The cameras connect to two underwater hubs, as mentioned below, and rely on them for power and data transmission, via CAT5 Ethernet cables in appropriate housing.All cameras connect to a nearby underwater hub (Figures 5, 6).Cameras 1 to 4 connect to the main hub, while cameras 5 to 8 to a secondary one.The hubs are also enclosed in similar waterproof housing and contain an ethernet switch to collect and transmit data.The secondary hub connects only to cameras 5 to 8 and to the main hub.This one, in turn, connects to it's assigned cameras and to the surface relay station with a 300m long PVC cable housing that includes a Fiber optic cable and a 6 mm2 copper cable.The main reason for using two hubs instead of one, is the number of cameras, as any of the hubs can currently support a maximum of 4.

Relay Station
As already mentioned, the relay station can sit in any nearby dry land location.It consists of four solar panels and batteries for power, while it also includes a wind turbine as a secondary power generator and backup solution.A small weather station and surveillance camera can be remotely accessed to monitor the hub and nearby weather conditions.The recording and some pre-processing of the camera streams occurs here, in a small form factor computer that both records the video streams, as they come from beneath the sea, in a network attached storage solution and, simultaneously, downscales and transmits them to the remote processing station.
A long-range Wi-Fi antenna is included in the system, that can be used with a nearby network.In case that is not possible, the data transmissions can also occur via a 5G modem router that is responsible for networking the whole station.

Processing Station
The remote processing station consists of two computers.One receives the data streams from the relay station, performs the necessary pre-processing and forwards them.The second is the one where the Unity application, that incorporates a pre-built 3D model of the shipwreck, is running.
The first, a small form factor Intel NUC, receives the data streams from the relay station.It records some of the data provided, such as lighting changes due to the time of day, and any changes in texture or form.It, then, forwards the streams and this data on to the second.
The second computer has the Unity application running.This incorporates a small pre-dive sequence and a pre-built 3D model of the shipwreck and its immediate environment.The cameras are placed far enough apart that there is little overlap in viewpoints.

User Devices
With the iblueCulture application installed onto his device, the user can see a pre-dive environment as soon as he starts the application.This 3D environment simulates the area just above the shipwreck, making it seem just as if the user was standing on a boat.One can look around or just move on to the appropriate marker, dive into the water and descend to the level of the UCH site.User navigation is achieved via an on-screen joystick for movement, while user orientation is registered as the device's orientation.Users can move around freely in the underwater environment, up to a distance of approximately from the actual shipwreck.
The user's movements are transmitted into the application, as location and orientation, and are processed into the Unity environment.A Unity camera matches the user's position and view and begins an outgoing video stream, which arrives directly to the user's device.This camera follows the user's movement until he has been stationary for 5 minutes, at which point it stops and the user is disconnected from the system.Currently, the system is designed with a hard limit of 10 users.Depending on bandwidth and processing power, it can support more, as a case of multiple users on a local Wi-Fi would have much different requirements than those in a vessel right above the shipwreck.

Data Processing
Thanks to the reliance on open-source applications and libraries, the iblueCulture system can be used with a variety of devices.
Most smartphones with gyroscopes can effectively work with the application.The fact that the end-user application essentially only streams video in and user movement and orientation out, makes it very easy to modify it for any number of devices, from VR headsets to AR glasses.
The real innovation, however, is in the way the application streams video from the UCH site and uses it to enhance the user experience.This is done in 2 ways.
Initially, the video stream is pre-processed and a texture and hue for the wreck are determined in the first processing station computer.Due to the depth of the shipwreck, lighting changes are not that common, as they only occur only when a very large vessel passes above.The day-night cycle, however, has a significant impact.At night, there is nearly zero visibility at that depth without any artificial light sources.That means, at the time when the sun is setting or rising, the view and colours at the bottom of the sea are different than those at noon.For this reason, the current hue of the shipwreck's texture is copied from the camera streams and replaced, live, onto the pre-built 3D model.Therefore, lighting changes in the underwater view are reflected directly onto the user's device.While this could also have been achieved somewhat with a premade day-night system in Unity, that option would also significantly hamper the user's ability to navigate, as we use no artificial lights in the Unity environment, and the lighting changes in early evening or late morning could potentially make the scene very dark.The second way that the live video stream enhances the user experience, is the 3D model.While the shipwreck is a pre-made 3D model that was created with photogrammetry, by using data from previous visits, we must accept that it can, possibly, change.For example, in the case of Peristera, an amphora could fall off the pile and land further away from the rest.In the case of an actual ship, like the Christophoros, a mast or other piece could be dislocated and end up in a different place than the one at the time it was photographed.In that case, the model is no longer 100% accurate.Therefore, the iblueCulture system is designed to perform a comparison between photographs previously taken from each camera location and the actual camera view at the time.If any significant difference is determined, such as the examples mentioned above, the system remakes the 3D model using the older photos for the other camera views and the live stream frames for the camera view where the variation was detected.This new 3D model is then inserted directly in the Unity environment.
The screenshots in Figures 7, 8 and 9 are samples of the application in its current form.We purposefully removed any additional background and environment to make the images clearer.We are allowing user movement in a 50x50 area with the model at the centre.Beyond that point, any movement is prohibited and a static background, created from pictures during the dive, is displayed.

CURRENT STATUS
One of the issues we have encountered so far, is hardware.The 3D model, as created in medium-high settings, is approximately 84 million polygons.That is a staggering amount for even a high-end PC to process and manipulate.The data transfer from the processing PC to the user application completely took over all possible bandwidth, while our FPS count dropped from 15 to 0.5.
We reduced the polygon count significantly to 1.7 million polygons and are currently working to slowly increase it.
Reducing it even further effectively made the model and its elements unrecognisable as amphoras.We are attempting to further optimize the model, in order to have as much detail as possible.We are also improving the software pipeline, to handle an amount of data large enough, so that the model is realistic, but not so large that the PC processing power and bandwidth are completely used while serving a single user.This seems to be, in our opinion, a purely hardware issue.The current PC we use to run the application is an 11 th Gen Intel i9-11900KF with 64 GB of RAM and a Nvidia 3090 GPU with 24GB of VRAM.We believe the real bottleneck occurs at the GPU and, in part, the processor.A PC created for this task would be better served by a newer processor, with more available threads, or even a workstation with dual Xeons, and possibly a GPU with even higher core count and VRAM or perhaps more than 1 GPU in an SLI configuration.We are still testing on other hardware configurations and trying to determine specifically what can be improved, apart from the model itself.
In Figure 10 below, one can see how the model has lost detail, as the amphorae look like they have melted and seem to merge.This is a direct result of reducing polygon count and, therefore, losing significantly in detail, in our attempts to lighten the load on the PC.We are improving our model as much as possible, to eliminate this degradation.We are also considering adding various extras on the application, mostly aesthetic additions like 3D models of the marine environment around the actual shipwreck.With pictures of the environs taken during the dive, we are currently creating a detailed surrounding area to add to the main model.This, however, will be completed after the main model is finalized, so that we have determined the extend of any bandwidth and FPS issues.

DISCUSSION
As mentioned previously, the real time video offers many benefits.It allows a more immersive, realistic experience for the end user, thus enhancing interest in the area and, by extension, tourism development.It can also provide stakeholders a method to monitor and safeguard UCH sites.The detection system currently in place, determines changes in the shipwreck and can be modified to notify the user of the presence of large objects in camera view, such as divers, or perhaps even detect changes due to natural causes that can adversely affect the site.The system could be used for educational purposes and in scientific research, especially for sites at significant depth.
Since all the heavy processing, the real-time texturing and the 3D modelling that takes place, are performed in the processing station, thus making the application lightweight.This enables even more users to access it, since most users would have a middle tier smartphone or tablet.It can also work on most, if not all, VR headsets currently in the market.
Users can be in any area, even in a vessel above the shipwreck, when they use the application.The only consideration in that case is data bandwidth, as 4G might not be enough, limiting the user devices to newer ones.In a museum or kiosk environment, there is no such issue, with any network of Wi-Fi 5 specification and above.
The small footprint of all the in-situ parts of the system and the sustainability inherent in it, make it ideal for remote locations, where traffic and human activity is minimal.It requires no upkeep or maintenance and most of the elements can be accessed remotely.
There are some inherent technological limitations in this implementation.As explained above, for every 4 cameras the system requires 1 hub.For very large shipwrecks, like the Christophoros, we could potentially need 16, if not more, cameras to completely capture it.This adds up to 8 added cameras and 2 hubs, which would significantly increase power and data transmission requirements.Adding even more cameras and hubs would probably require a second relay station, and perhaps some upgrades in the processing station.Overall, the current 8 camera system is sufficient for smaller UCH sites and parts of larger ones, such as part of the Christophoros, rather than the whole ship.

FUTURE RESEARCH
There are many ways the iblueCulture system could potentially be modified and enhanced.The model and hardware bottleneck we are currently experiencing is already being investigated, tested and improved.As mentioned previously, one option is to modify iblueCulture into a surveillance option for UCH site security, something that most current sites lack.The camera resolution could also be improved, with any downscale performed on the relay station removed, as technological advances in networking might allow for faster, more stable data streams.Finally, a larger shipwreck should be the subject of a future study, to determine the actual technological limitations with the current configuration, before adding any extra relay stations.A very large wreck with more than 12-16 cameras would possibly require significant changes in power and data flow.

Figure 7 .
Figure 7. Initial 3D model of the ancient shipwreck at Peristera.

Figure 8 .
Figure 8.View of the shipwreck from within the application.

Figure 9 .
Figure 9. Another view of the shipwreck from within the application.

Figure 10 .
Figure 10.Close-up view of the 3D model.