Deep (Ocean) Learning

ii Editorial Board

iii On the Cover

iv Publishing Schedule and Advertiser ’s Index

v Editor’s Note

vi Soundings … Jennifer LaPlante

Essays

1 Sea Ice Visual Perception and Risk Management Using Deep Neural Networks: A System Overview

Ravindu G. Thalagala, Dan Oldford

ABS Harsh Environment Technology Centre

Oscar De Silva, David Molyneux

Memorial University

10 AI and Autonomy: How the Aquaculture Industry is Evolving for the Future

Chad Gillen

Deep Trekker Inc.

19 AI for Corrosion Detection on Marine Structures

Shahrizan Jamaludin, Md Mahadi Hasan Imran

Universiti Malaysia Terengganu

27 AI and Improving the Blue Economy

Mark Spalding

The Ocean Foundation

38 Blowing Away Limits: The Cutting-edge Integration of Machine Learning in Offshore Wind Farm Development and Management

Masoud Masoumi

Cooper Union

46 A Bibliometric Review of Research Publications on Digital Twin Predictive Maintenance Systems in the Maritime Industry

Abdelmoneim Soliman, Mervin A. Marshall

Fisheries and Marine Institute

Md Safiqur Rahaman

King Fahd University of Petroleum and Minerals

Mohamed A. Ouf, Ahmed El-Sayed

Arab Academy for Science, Technology, and Maritime Transport

77 Lodestar … Benjamin Misiuk, Yan Liang Tan, Zahra Jafari, Comfort Eboigbe

81 Improving Detection and Localization of Green Sea Urchin by Adding Attention Mechanisms in a Convolutional Network

M. Israk Ahmed, Lourdes Peña-Castillo, Andrew Vardy, Patrick Gagnon

Memorial University

Spindrift

100 Q&A with Zoheir Sabeur

102 Trade Winds … Transformative AI for Unprecedented Subsea Survey Solutions

Chetan Chitnis, Unique Group

104 Inside Out … Object Tracking for Uncrewed Surface Vehicles

Oliver S. White, Sean Daniel, Oliver S. Kirsebom, Fritz Stahr, Open Ocean Robotics

108 Turnings

111 Perspective … Zelim

112 Reverberations … Optimizing Fish Farm Management with Deep Learning Analysis of Underwater Drone Data

Mira Nagle, Oceanbotics Inc.

114 Homeward Bound … Marine Animals: What do They Have to Say?

Lawrence Taylor, IntegraSEE

116 Parting Notes … AI Generated Artwork

PUBLISHER (ACTING)

Kelley Santos info@thejot.net

MANAGING EDITOR

Dawn Roche

Tel. +001 (709) 778-0763 info@thejot.net

ASSISTANT EDITOR

Bethany Randell

Tel. +001 (709) 778-0769 bethany.randell@mi.mun.ca

TECHNICAL CO-EDITORS

Dr. David Molyneux Director, Ocean Engineering Research Centre Faculty of Engineering and Applied Science Memorial University of Newfoundland

WEBSITE AND DATABASE Scott Bruce

Dr. Keith Alverson University of Massachusetts USA

Dr. Randy Billard Virtual Marine Canada

Dr. Safak Nur Ertürk Bozkurtoglu Ocean Engineering Department Istanbul Technical University Turkey

Dr. Daniel F. Carlson Institute of Coastal Research Helmholtz-Zentrum Geesthacht Germany

Dr. Dimitrios Dalaklis World Maritime University Sweden

Randy Gillespie Windover Group Canada

Dr. Sebnem Helvacioglu Dept. Naval Architecture and Marine Engineering Istanbul Technical University Turkey

A publication of

GRAPHIC DESIGN/SOCIAL MEDIA

Danielle Percy Tel. +001 (709) 778-0561 danielle.percy@mi.mun.ca

Dr. Katleen Robert Canada Research Chair, Ocean Mapping School of Ocean Technology Fisheries and Marine Institute

FINANCIAL ADMINISTRATION Michelle Whelan

EDITORIAL BOARD

S.M. Asif Hossain National Parliament Secretariat Bangladesh

Dr. John Jamieson Dept. Earth Sciences Memorial University Canada

Paula Keener Global Ocean Visions USA

Richard Kelly Centre for Applied Ocean Technology Marine Institute Canada

Peter King University of Tasmania Australia

Dr. Sue Molloy Glas Ocean Engineering Canada

Dr. Kate Moran Ocean Networks Canada Canada

EDITORIAL ASSISTANCE

Paula Keener, Randy Gillespie

Kelly Moret Hampidjan Canada Ltd. Canada

Dr. Glenn Nolan Marine Institute Ireland

Dr. Emilio Notti Institute of Marine Sciences Italian National Research Council Italy

Nicolai von OppelnBronikowski Memorial University Canada

Dr. Malte Pedersen Aalborg University Denmark

Bethany Randell Centre for Applied Ocean Technology Marine Institute Canada

Prof. Fiona Regan School of Chemical Sciences Dublin City University Ireland

Dr. Mike Smit School of Information Management Dalhousie University Canada

Dr. Timothy Sullivan School of Biological, Earth, and Environmental Studies University College Cork Ireland

Dr. Jim Wyse Maridia Research Associates Canada

Jill Zande MATE, Marine Technology Society USA

research activities taking place around the proceedings must be properly cited, including JOT volume, number and page(s). info@thejot.net

Cover

Merriam-Webster defines artificial intelligence as “the capability of computer systems or algorithms to imitate human behaviour,” while deep learning is defined as “a form of machine learning in which the computer network rapidly teaches itself to understand a concept without human intervention by performing a large number of iterative calculations on an extremely large dataset.” Cover design by Carla Myrick.

CHAIWUT SRIDARA / VECTEEZY.COM

Publishing Schedule at a Glance

The JOT production team invites the submission of technical papers, essays, and short articles based on upcoming themes. Technical papers describe cutting edge research and present the results of new research in ocean technology or engineering, and are no more than 7,500 words in length. Student papers are welcome. All papers are subjected to a rigorous peer-review process. Essays present well-informed observations and conclusions, and identify key issues for the ocean community in a concise manner. They are written at a level that would be understandable by a non-specialist. As essays are less formal than a technical paper, they do not include abstracts, listing of references, etc. Typical essay lengths are up to 3,000 words. Short articles are between 400 and 800 words and focus on how a technology works, evolution or advancement of a technology as well as viewpoint/commentary pieces. All content in the JOT is published in open access format, making each issue accessible to anyone, anywhere in the world. Submissions and inquiries should be forwarded to info@thejot.net.

Upcoming Themes

All themes are approached from a Blue Economy perspective.

Fall 2024 Sensing the ocean: lights, camera, sensors

Winter 2024 Safety first: humans at sea

Spring 2025 Marine tourism

Summer 2025 Ocean monitoring

Fall 2025 Maritime security

Winter 2025 Indigenous use of technology

Stay informed

Each issue of the JOT provides a window into important issues and corresponding innovation taking place in a range of ocean sectors –all in an easy-to-read format with full colour, high-resolution graphics and photography. CIOOS 36,37

Systems 18

Index The Journal of Ocean Technology c/o Marine Institute P.O. Box 4920 155 Ridge Road St. John's, NL A1C 5R3 Canada

+001 (709) 778-0763 info@thejot.net www.thejot.net

Editor's Note

Welcome to our summer issue where we focus on how artificial intelligence (AI), machine learning, deep learning, and neural networks are used in the ocean technology sector.

AI is not an area that I know a lot about. Sure, there is plenty of information in the news and online about AI – its benefits and risks. But I was not aware of the specific ways AI is being used in the ocean technology sector. As contributors responded to our call for content and submitted their materials, what struck me is how many different ocean industries use AI to enhance and improve operations. The uses for AI in the ocean sector are vast and varied. In the realm of science and data collection, AI is being used in the areas of subsea surveys, visual perception and risk management of sea ice, classifying subsea image datasets, and benthic habitat mapping. Many marine industries, from aquaculture to energy, use AI for monitoring infrastructure, detecting problems, and predicting when and where they could occur. These same industries are also using AI to optimize their operation to reduce risk and cost. Further to reducing risk, AI is assisting search and rescue operations and being used to classify objects passing oil and gas structures. It is even being used to help ports and ocean transport become more efficient.

A good starting point for this issue is the Soundings column by Jennifer LaPlante with Canada’s Ocean Supercluster, who discusses how to advance the use of AI for industry growth. We close the issue with Parting Notes, this time featuring a piece of artwork generated by AI. Check it out to see what happens when you ask AI to create an image of “deep ocean learning.ˮ On the pages in between, you can read about how AI is revolutionizing the ocean technology sector across varied industries around the globe.

I hope you enjoy reading this issue as much as I enjoyed learning more about AI and its myriad uses in the ocean!

Dawn Roche

RANDY GILLESPIE

Dawn Roche is the JOT’s managing editor.

SoundingS

Building Canada’s Ocean AI Ecosystem through Collaborative Innovation

With the world’s longest coastlines, a robust history of ocean research and innovation, and leadership in artificial intelligence (AI), Canada is uniquely positioned to lead in ocean AI. To achieve this, we must collaborate to identify opportunities, share data, and focus on initiatives that will significantly impact our economy.

For the ocean sector to fully embrace AI, clear economic benefits must be demonstrated. While academia thrives on novel solutions, businesses require tangible value before making substantial investments in AI. Effective management of data access and ensuring mutual benefits are crucial. Although there are inherent risks, with the right balance, these risks can transform into significant opportunities.

Companies investing in AI are making substantial commitments, dedicating their subject matter experts’ time, potentially hiring new talent, and allocating valuable data to these projects. These efforts involve significant costs without guaranteed financial returns. The AI market can be challenging to navigate, with many promises of partnerships. Therefore, it is essential for potential collaborators to understand the investments private companies are making and to focus on developing solutions that address real-world problems.

Industry and academia can struggle with different goals because businesses prioritize immediate, practical solutions that drive economic returns, while academic institutions focus on exploring innovative ideas and advancing theoretical knowledge. Adding to this risk, the potential savings and returns from AI are not always immediately apparent for private sector companies. Early AI projects may take time to yield returns, making it challenging to justify the investment without clear economic benefits. Demonstrating the tangible economic value and potential savings from AI is crucial to encouraging more companies to invest in this technology.

To advance AI growth, we must also bridge the gap between company expectations and the understanding of regulators and governments. It is vital to educate all stakeholders about the potential of AI solutions and work together to align their goals and expectations.

A Blueprint for Success: The Collaborative Model

This entails coming together as a broad ecosystem to support the exploration and trial of AI solutions. A successful collaboration has already united various government departments, ecosystem enablers, funders, and companies from across Canada to develop testing sites and opportunities for marine autonomous surface systems. This partnership has demonstrated the value of aligning individual strengths and appreciating each party’s contributions. By sharing data and combining opportunities, they are working to create the infrastructure and support necessary to develop usable autonomous systems.

To be successful, we should explore replicating this collaborative approach across the ocean and maritime sectors to achieve greater economies of scale and maximize the return on our efforts. By working together, Canada can become the global leader in ocean AI.

Unlocking AI’s Potential with Data Collaboration

All of this will only be possible with data. Data is the backbone of AI, but collecting enough can be challenging. A major hurdle in developing AI is obtaining labelled data, which is essential for training effective models. In certain environments, acquiring high-quality, labelled data is difficult, slowing down progress. To overcome this, we need to foster a culture of open data sharing. Often, data owners do not realize the value of their data. Data collectors, such as researchers or companies, are often hesitant to share their data, fearing potential missed opportunities if others use it successfully. Additionally, there is concern that sharing data may reveal sensitive information that some would prefer to keep confidential.

Canada’s Ocean Supercluster has recently invested in collaborative AI innovation projects and partnered with leaders nationwide to develop an Ocean AI Strategy for Canada. This strategy aims to identify effective ways to advance AI awareness and adoption. Over the coming year, efforts will focus on enhancing AI awareness and exploration. However, this program alone is not a complete solution for Canada. It requires a collective willingness to learn, engage, and collaborate. By sharing ideas and data, we can unlock the full potential of AI and drive significant advancements in the ocean and maritime sectors. Together, we can position Canada as a global leader in ocean AI innovation.

Jennifer LaPlante is the chief growth and investment officer for Canada’s Ocean Supercluster (OSC) supporting innovation and growth in Canada’s ocean economy. As Canada’s national ocean cluster, the OSC is a convenor of members, partners, and networks, and a catalyst for transformative growth that helps build the robust ecosystem needed to help realize Ambition 2035 – a 5X growth potential in ocean in Canada by 2035.

This image was created with the assistance of DALL·E.

ALUMNI MARINE INSTITUTE

• Special events

• Reunions

• Degree frames

• Alumni and campus news

• Group rates on insurance services

• Alumni credit card

• Retail discounts

• University library access

CAREER SERVICES

• Resume/CV critique and evaluation

• Mock interviews

• Job search skills

– personal branding, networking, job search resources

• Resume referral services

• Job postings

• Access to career events

Stay connected and up to date on new alumni services, events, latest news and how you can become involved.

www.mi.mun.ca/alumni/reconnectwithmi alumni@mi.mun.ca 1 800 563 5799, ext. 0367

linkedin.com/edu/ FisheriesandMarineInstituteofMemorialUniversity

facebook.com/ MarineInstituteAlumni twitter.com/ MarineInstitute

youtube.com/marineinstitutepr

instagram.com/marine.institute SERVICES AND

SEA

A SYSTEM OVERVIEW

by Ravindu G. Thalagala, Oscar De Silva, Dan Oldford, and David Molyneux

Introduction

Navigating through ice covered waters safely poses challenges for vessels for any polar voyage. Mitigating the risks associated with ice navigation is crucial for the safety of the onboard crew and the vessel. Ice risk quantification is typically carried out incorporating well established synthetic aperture radar satellite image-based strategies and onboard radar-based strategies. In contrast, camera image-based methods offer continuous observation and collection of real-time sea ice information. These captured images can be used to extract essential information, such as ice concentration, size, and trend change, for risk quantification. With the aid of image processing algorithms, ice parameters can be extracted and computed from high-resolution image data. Due to the harsh conditions that prevail in the Polar Regions, incorporating classical image processing techniques might fail when the lighting conditions, image clarity, and noise levels change.

Artificial intelligence (AI) is at the forefront in the domains of image-based navigation and multi-sensory navigation as an efficient alternative to classical methods, marking significant advancements in how visual data is interpreted and utilized for operational guidance. Initially developed using traditional feature-based methods where specific image features were manually programmed, AIbased image understanding has transitioned towards self-learning systems known as AI discriminatory modules. In maritime navigation, such discriminatory modules can differentiate between various objects in a visual scene, such as sea ice conditions, ships, or landmasses, by effectively learning from vast amounts of image data. This capability is similar to the technology used in selfdriving cars, where AI discriminates between pedestrians, other vehicles, and road signs to navigate safely. Beyond discriminatory capabilities, AI in navigation also extends to generative methods. These AI systems are similar to AI chatbots with image understanding abilities that can generate new

image data or simulate visual environments based on learned information. This feature is especially useful in training simulations or planning maritime routes where actual sensor data may be unavailable.

Ice navigation currently utilizes discriminative supervised AI due to the challenges of limited data and onboard computational constraints. Discriminative supervised AI involves models that learn to classify different types of input, such as distinguishing sea ice from open water, using labelled datasets where outcomes are predefined. The complexity and resource demands of more advanced generative AI models, which could theoretically generate new data scenarios, exceed the current computing infrastructure on most maritime navigation systems on vessels. A review of the literature indicates that a discriminative AI system would be most suitable to undertake multiple tasks for effective ice risk management. The tasks can be organized in a system architecture as shown in Figure 1. This general ice risk mitigation architecture includes the following modules:

• [1], [2] Image Classification Module(s): These modules were used to classify images into different classes considering the image as a whole, based on the scene of the image. This kind of classification technique is commonly used in various real-world applications such as satellite imagery analysis for sea ice monitoring, where it helps distinguish between different landforms and sea areas. Another example includes self-driving vehicles classifying images captured by onboard cameras to identify road signs, lane markings, pedestrian crossings, and other vehicles. In the proposed architecture, Image Classification Module – I evaluates whether the captured image is relevant, specifically if it contains ice, and if the image quality, such as lighting conditions, is adequate for ice detection. It classifies images into five categories based on the scene: forward-looking, side-looking,

stern-looking, lighting condition, and irrelevant. Serving as a preprocessing step, this module forwards only the images containing ice and those with sufficient lighting to the next stage for further analysis. The input for this module is the image feed from the onboard camera, and the output is the images that fall into the forward-looking category. Image Classification Module – II receives forward-looking images as input and primarily differentiates among images of open water (i.e., no ice is present), images containing ice, and images featuring distinct objects like individual icebergs, ships, and boats. This module outputs images that contain ice, specifically those showing pack ice, and it also outputs images that include distinct objects separately.

• [3] Semantic Segmentation: Semantic segmentation is an advanced technique in machine vision that goes beyond simple object classification. It not only identifies objects within an image but also labels each pixel with a class identifier specific to that object type. This method allows

for detailed understanding and analysis of complex scenes, assigning distinct classes such as people, buildings, and vehicles, which are crucial for applications like urban scene recognition. In the proposed system architecture, semantic segmentation is used to detect ice types and ice concentration. The output of this module is ice type, and ice concentration supports the Risk Index Outcome calculations using Polar Operational Limitations Assessment Risk Indexing System guidelines.

• [4] Instance Segmentation Module: Instance segmentation is another advanced machine vision technique widely used in applications such as self-driving vehicles. Unlike semantic segmentation, which classifies each pixel under a broad category, instance segmentation identifies and categorizes each instance of multiple object classes independently. For instance, in self-driving systems, this allows the system to distinguish between individual vehicles on the road, not just recognizing them as vehicles but also identifying each one separately assigning a unique label to it. Similarly, in sea ice classification,

Figure 1: Discriminatory AI-based ice detection and tracking architecture. The numbered boxes indicate the AI-submodules that are designed to the complete system.

instance segmentation can differentiate between various ice forms, marking them individually even if they belong to the same class. For example, one image may contain multiple small floes and instance segmentation can assign distinct labels to all the small floes that were detected. This module takes pack ice images as input and outputs individually segmented ice floes enabling multiple ice floe tracking in the preceding modules.

• [5] Object Detection Module: Object detection modules are commonly used in autonomous navigation of vehicles and drones to avoid obstacles by detecting objects such as pedestrians, vehicles, trees, and buildings. In the proposed architecture, the object detection module is used to identify distinct objects such as icebergs, ships, and boats that are present in the input ice image. Detected objects can be tracked in the preceding modules to effectively plan safe navigation.

• [6], [7] Region Tracking Module(s):

This module provides the system with the ability to detect and continuously monitor the position of specific regions or objects across the image feed. For instance, in autonomous driving, this might involve tracking vehicles or pedestrians to predict their future positions and plan a safe navigating path. In the context of sea ice, tracking regions helps in understanding ice floe and iceberg dynamics, which are crucial for safe polar water navigation.The input for the module is the segmented individual ice floes/objects and the output would be the tracked velocity of ice floes/icebergs.

• [8] Ice Load Prediction Module: Ice load prediction involves forecasting the pressures and forces that ice exerts on a vessel, which can vary based on several environmental parameters including ice thickness, density, and floe size. AI models leverage both historical (time-averaged) and real-time (time-dependent) data to simulate and predict the dynamic and static forces acting on ships. Input for the ice load prediction module is the tracked ice

floes from the tracking modules and the output would be ice load prediction based on the floe characteristics.

The deployment of these modules has become increasingly accessible, allowing engineers with minimal coding experience to engage in such designs using open-source, easy to use tools. This essay illustrates how the first module of the described architecture can be prototyped and implemented utilizing AI tools.

Image Classification Module – I

Development of an AI-based image classifier involves creating a training dataset with class balance, i.e., having the same number of images in each class. Typically, using approximately 200 images per class helps to enhance the accuracy of the classifier to a satisfactory level. The next steps involve labelling the images, augmenting the dataset, and then conducting training, validation, and testing to assess the performance of the algorithm. The image dataset utilized for developing this classifier was compiled using images sourced from open-source platforms.

There are different image classification models available in the literature including PSPNet, ICENET, Deeplabv3+, and YOLOv8. From these classifiers YOLOv8 is the highest accuracy, real-time capable image classifier to date that is being widely used in self-driving vehicles and aerial surveillance applications. The subtasks that were completed to implement the Image Classification Module – I are described below.

• Labelling: This process entails manually categorizing each image into its appropriate class by assigning a label to it. This is typically carried out using platforms like Roboflow or Supervisely, which provide graphical tools for labelling. In this study, the Roboflow platform was used to label the images and create the image dataset. For instance, as shown in Figure 2, the image on the top would be labelled as part of the forward-looking

image class, while the image on the bottom would be categorized under the sternlooking image class. Similarly, all the 1,000 images in this dataset were labelled into five classes (as mentioned in [1]), each class consisting of 200 images.

• Augmentation: This is a method used to improve a classifier’s ability to recognize images by artificially increasing and varying the training data. This is achieved by making modified copies of existing images through techniques like rotating, resizing, and adjusting colours.

These changes help the AI model learn to recognize objects under different conditions and from various perspectives, making it more accurate and versatile in real-world tasks. In this study, Roboflow automated tool is used to augment the dataset. The dataset underwent augmentation from 1,000 to 2,400 images by applying greyscale to 15% of the images, blurring by 2.5 pixels, and adding noise up to 1.5% of pixels.

• Training/Validation: Training and validation are essential phases in the

Figure 2: Sample images from dataset. Forwardlooking image (top). Sternlooking image (bottom).

development of any AI model. During the training phase, the classifier learns to recognize patterns within and across different classes, adjusting its model parameters based on the training data provided. Following training, the validation phase involves testing the classifier using a distinct set of data that it has not previously encountered to evaluate its accuracy. One pass of the entire dataset through training is called an epoch. The number of epochs, which is determined by the user, influences how thoroughly the model learns from the data, aiming to optimize the learning without overfitting.

For training and validation, the augmented dataset was split into 2,100 training images (Figure 3), 200 validation images, and 100 testing images in the Roboflow platform. The created dataset is then imported to Google Colab notebook. Inside the notebook, classifier model YOLOv8 has been trained using the T4 GPUs provided by Google. In YOLOv8, model type is set to classify, number of epochs is set to 1,000, and image size is set to 128x128 pixels. After the training, the model achieved an accuracy of 95.06% for the 100-testing image set.

• Classifier Testing: Classifier testing is a critical step in evaluating an AI model, where the model is put through various tests using new data it has not

encountered during training. This helps verify that the classifier can accurately identify and categorize data in realworld conditions, ensuring its reliability and effectiveness outside of the training environment. The sample images that are used to test the model are shown in Figure 4. The images used for testing are not taken from the test set of the training data.

The confusion matrix for the classifier is shown in Figure 5. The confusion matrix indicates only 5% false positive rate, meaning that the classification is highly accurate. After the model training, the Top-1 class accuracy graph is shown in Figure 6. Top-1 class accuracy refers to the proportion of times the model correctly predicts the most likely class as the actual class out of all predictions made. In simpler terms, it measures how often the model’s highest confidence prediction is exactly correct. The classification model developed in the proposed module has a Top1 class accuracy of 95.06%.

Conclusion

The YOLOv8 model achieved a high (95.06%) accuracy on a 100-image test set, highlighting its effectiveness in sea ice classification through supervised discriminatory AI tools. This performance exemplifies the potential of using deep learning-based classifiers like YOLOv8 for advancing systems equipped with AI modules. The same YOLOv8 model used

Figure 3: Sample set of images used for training.

classification results of the developed classifier. Each image shows the classification probability, per class accuracy, number of images per class, number of training images, and number of testing images.

Figure 4: Example

Figure 5: The confusion matrix for Image Classification Module – I.

in the first module, as shown in Figure 1, can also be applied to implement the second image classification module. Furthermore, the semantic segmentation module is currently operational within a mobile app. Similarly, the instance segmentation module, essential for the precise tracking of different ice floes, is planned to be developed. This AI system facilitates the accurate prediction of ice loads from collected images. This integrated approach establishes a comprehensive framework for AI-based ice navigation on board ships, enabling real-time, safe navigation through ice-infested waters. u

Balasooriya, N.; Dowden, B.; Chen, J. et al. [2021]. In-situ sea ice detection using DeepLabv3 Semantic Segmentation. DOI: 10.23919/OCEANS441 45.2021.9705801.

Cao, Y.; Liang, S.; Sun, L.; et al. [2022]. Trans-Arctic shipping routes expanding faster than the model projections. DOI: 10.1016/J.GLOENVCHA.2022.102 488.

Dowden, B.; De Silva, O.; Huang, W.; et al. [2021]. Sea ice via deep neural network semantic segmentation DOI: 10.1109/JSEN.2020.3031475.

Zhang, C.; Chen, X.; and Ji, S. [2022]. Semantic image segmentation for sea ice parameters recognition using deep convolutional neural networks. DOI: 10.1016/J.JAG.2022.102885.

Zhao, X.; Gao, L.; Chen, Z.; et al. [2018]. CNN-based large scale Landsat image classification. DOI: 10.23919/APSIPA.2018.8659654.

Figure 6: Top-1 class accuracy graph for Image Classification Module – 1.

Dr. Ravindu G. Thalagala received a B.Sc.(Hons.) in mechanical engineering from the University of Moratuwa, Sri Lanka, in 2016, and a master’s degree in mechanical engineering from Memorial University of Newfoundland, Canada, in 2019. He is currently a postdoctoral fellow with the ABS Harsh Environment Technology Centre, St. John’s, Newfoundland. His main research interests include state estimation of autonomous vehicles and localization.

Dr. Oscar De Silva received a B.Sc. degree in engineering from the University of Moratuwa, Sri Lanka, in 2009, and a PhD from Memorial University of Newfoundland (MUN), St. John’s, NL, Canada, in 2015. Following post-doctoral work with the ABS Harsh Environment Technology Centre, St. John’s, he joined MUN as a faculty member in 2016. He is currently an associate professor with the Faculty of Engineering and Applied Science. His main research areas are autonomous robotics, navigation system design, and machine learning.

Dan Oldford is the ABS technology manager in the ABS Harsh Environment Technology Centre (HETC). He has a bachelor’s degree in ocean and naval architectural engineering, as well as an engineering master’s degree in ice mechanics. Both degrees are from Memorial University in St. John’s, NL. As of June 2024, he is in his 20th year with ABS – working as a surveyor in Canada for the first nine years where he witnessed ships getting damaged and struggling with low temperature environments. In 2012 he took his knowledge and experience to the HETC where he specializes in winterization, ice classed machinery, and the Polar Code. Mr. Oldford supported many of the discussions that happened during the Polar Code’s development, including the temperature analysis that resulted in the Polar Service Temperature definition. In 2017 he authored the original ABS Process instructions for the Polar Code implementation. As manager in the HETC, he is involved in many projects that utilize his unique skill set and experiences. This includes managing projects to develop new guidance for winterization, further development of the ice class requirements, helping shipping companies comply with the Polar Code, and establishing critical scenarios for icebreaker design specifications. He also serves on

technical committees for development of international standards, such as the current draft ISO/DIS 24452, of which he was one of the main authors.

Dr. David Molyneux is director of the Ocean Engineering Research Centre and an associate professor of naval architecture at Memorial University of Newfoundland. He joined Memorial in 2015 after a career in private and public sector research organizations. He has worked on a wide range of projects related to physical modelling and numerical simulation of ships and offshore structures, specializing in harsh environments. His research at Memorial has focused on many approaches to improving safety for ships and their crews. He obtained his PhD from Memorial, M.A.Sc. in mechanical engineering from University of British Columbia, and his B.Sc. in naval architecture from University of Newcastle-Upon-Tyne.

Regular inspections of aquaculture farms address environmental integrity, animal welfare, food safety, and sustainable practices. This image shows a topside aquaculture pen in Norway.

DEEP TREKKER

The Importance of Aquaculture Farm Inspections

Regular inspections of aquaculture farms play a vital role in ensuring the industry’s long-term viability. These inspections serve as vanguards of environmental integrity, animal welfare, and food safety, while also nurturing ethical and sustainable practices. By proactively identifying potential challenges and ensuring compliance with regulations, inspections become catalysts for the balanced and responsible growth of aquaculture.

Through systematic assessments, fish farmers can preemptively address maintenance needs before they escalate into significant issues. By routinely evaluating factors such as net integrity, fish health, water quality, and environmental conditions, farmers can mitigate risks and uphold the integrity of their operations.

The High Cost of Diver Inspections

The exorbitant expenses associated with diver inspections in maritime operations stem from various factors, including the need for specialized labour, stringent safety protocols, extensive training, equipment procurement, logistical considerations, insurance coverage, and the inherently time-consuming nature of underwater assessments. In response to these challenges, the industry is actively exploring innovative solutions to ensure cost-effective yet comprehensive inspection outcomes, with a notable focus on integrating remotely operated vehicles (ROVs) into inspection practices.

The investment required to train a team member for diver inspections often exceeds $30,000 CAD, not including additional expenses related to equipment acquisition and insurance coverage, which can escalate significantly. Moreover, the cost of hiring a diver for underwater inspections can surpass $6,500 per day for routine assignments, with expenses mounting based on the complexity of the task at hand.

In addition to financial burdens, farm operators face logistical constraints imposed

by diver schedules, prolonging the time needed to complete inspections. Furthermore, safety risks are also a major concern in underwater environments, amplifying the appeal of ROVs as alternatives or complements to diver-led inspections. By leveraging ROV technology, maritime operations can mitigate safety risks while optimizing inspection efficiency.

How Emerging Technology is Transforming the Maritime Industries

The maritime sector is experiencing an extensive transformation driven by the rapid evolution of emerging technologies and robotics innovations. These advancements are reshaping various facets of maritime operations, ushering in enhanced efficiency, safety, sustainability, and connectivity. Several pivotal technologies are spearheading this transformation:

• Underwater Robotics: Autonomous underwater vehicles and ROVs are revolutionizing maritime exploration, inspection, and maintenance operations in challenging underwater environments. By reducing human involvement in these situations, they enhance safety while significantly improving efficiency.

• Digital Twins: Digital twin technology generates 3D models of physical assets, enabling operators to monitor and simulate real-world scenarios. This facilitates performance optimization, risk assessment, and the exploration of new strategies before their actual implementation, thereby enhancing operational efficiency and minimizing risks.

• Remote Monitoring and Control: Internet of Things devices and remote monitoring systems provide real-time insights into vessel and cargo conditions. Operators can remotely manage various aspects of ship operations, resulting in improved safety levels and operational efficiency.

• Data Analytics and Predictive Maintenance: Data-driven insights enable predictive maintenance practices, thereby

allow users to station hold against currents and pilot the vehicle precisely and accurately through varying water conditions.

boosting the efficiency and reliability of ships and port facilities. By analyzing data from sensors and systems, maritime operators can anticipate maintenance requirements before potential breakdowns occur, thus minimizing downtime and optimizing operational performance.

Furthermore, the integration of artificial intelligence (AI) and autonomy features within ROVs is further augmenting the accuracy and efficiency of inspections. Technologies such as sonar and environment samplers are enhancing the capabilities of robotic systems, rendering ROVs indispensable for a myriad of underwater applications.

Utilizing ROVs to Conduct Regular Fish Farm Inspections

Integrating ROVs into fish farm inspection protocols yields better operational efficiency, enriched data acquisition, and improves the overall sustainability and prosperity of aquaculture operations (Figure 1).

ROVs, uncrewed underwater vehicles outfitted with cameras, sensors, environment samplers, and manipulators, facilitate

meticulous assessments of aquaculture farms without the need for human inspectors to be underwater physically. Particularly advantageous for remote or challenging-toaccess farm locations, this capability allows for detailed inspections to be conducted effectively and remotely.

Equipped with state-of-the-art cameras and sensors, ROVs deliver crisp and comprehensive imagery of underwater infrastructure, fish populations, and equipment. This heightened visibility and data collection proficiency empower inspectors to discern issues and monitor fish health and behaviour with precision.

Moreover, ROV-led inspections bolster safety measures by removing the necessity for human divers to endanger themselves in water. Alternatively, they can complement divers by supplying real-time monitoring while submerged. Consequently, potential hazards associated with underwater operations, such as adverse weather conditions and strong currents, are significantly mitigated.

Figure 1: Remotely operated vehicles (ROVs) facilitate meticulous assessments of aquaculture farms. Deep Trekker’s REVOLUTION, for example, provides advanced stabilization to

While initial investment costs are associated with procuring ROVs, the long-term costeffectiveness is substantial. By circumventing diver training, equipment procurement, insurance, and operational expenditures, significant savings accrue over time. For instance, the acquisition cost of a single ROV is comparable to hiring a diver for just a couple inspections. Furthermore, ROVs enable daily inspections without the constraints of diver schedules, training, or certifications.

Autonomous remote inspections further elevate aquaculture operations. Deep Trekker’s Mission Planner feature facilitates the programming of waypoints, enabling ROVs to autonomously traverse predefined paths or routes guided by operators. This ensures consistent coverage of inspection areas, upholding standardized inspection and data collection practices, and allows operators a greater focus on the inspections rather than piloting the ROV.

Leveraging ROVs in aquaculture operations translates to more frequent and comprehensive inspections. This proactive approach minimizes the likelihood of significant issues by preemptively identifying maintenance needs before they escalate. Additionally, the digital records generated during inspections facilitate historical data analysis, enabling the tracking of changes, identification of trends, and informed decision-making for farm management and enhancements.

What is Project Sentry?

Project Sentry, a collaborative effort between Deep Trekker and Visual Defence Inc., aims to pioneer a cutting-edge remote aquaculture monitoring solution by leveraging in-cage remotely operated and autonomous vehicle technology (Figure 2).

Central to Project Sentry is the integration of an underwater garage housing a Deep Trekker ROV, primed for deployment to execute inspections and report anomalies based on a preprogrammed autonomous routine. In

collaboration with Visual Defence, Deep Trekker’s inspection system will harness the power of artificial intelligence and machine learning, thereby alleviating the burden on human operators in flaw detection. The project’s primary objective is to enable ROVs to autonomously and continuously survey the cage in circular patterns across various trophic levels, removing the necessity for human intervention.

Leveraging high-resolution imagery, machine learning algorithms, and AI capabilities, resident ROVs will possess the capability to recognize instances of net breaches within the cage. Upon detection of a breach, the system will capture photographic evidence while registering the precise location and depth of the incident.

These vehicles can also be equipped with sondes capable of accommodating up to four simultaneous environmental sensors, facilitating the tracking of potentially hazardous compounds. The implementation of a continuous monitoring system ensures swift responses to any issues, ranging from net integrity breaches and accumulation of deceased fish to parasite infestations or concerning environmental variables.

How Project Sentry Will Impact the Canadian and Global Economy

This initiative is positioned to sustain 448 Canadian jobs and is forecasted to create an additional 59 positions by December 31, 2027. Furthermore, it will significantly bolster Canada’s digital ocean workforce by fostering employment opportunities in AI, machine learning, mechatronics, software engineering, and cloud-based data processing.

The development of a specialized AI engine tailored specifically for caged aquaculture, in conjunction with autonomous underwater vehicle technology, represents a fundamental milestone towards advancing other oceancentric inspection projects. This initiative sets a precedent for both aquaculture and

broader ocean inspection reporting standards. Additionally, the project will notably enhance the digital capabilities of participating entities by establishing a scalable solution applicable to a diverse range of underwater inspection operations.

These sectors encompass a wide array of industries, including environmental monitoring, shipping, ship maintenance, search and recovery, and traditional and renewable energy, such as nuclear and offshore wind industries, to name a few. The system also holds potential utility within Canada’s defence sector, facilitating tasks such as harbour surveillance, security operations, damage control, and mine countermeasure applications.

What Does the Future of Aquaculture Look Like?

The primary objective of Project Sentry is to redefine the parameters of ROV automation. Breakthroughs in ROV mechatronics will facilitate effortless automatic monitoring of

netpens and the surrounding environment, thereby alleviating the workload for both present and future farm personnel. Additionally, the introduction of in-cage ROVs endowed with the capability to analyze and report complex scenarios such as breaches or mortalities diminishes the decision-making burden on operators.

Project Sentry will introduce a revolutionary digital system for asset inspection and reporting, offering unparalleled flexibility in terms of geographical location. With a remotecontrolled configuration, operators can oversee inspection feeds and make informed decisions from any corner of the globe. This represents a departure from the previous necessity of having on-site ROV operators to manually manoeuvre the vehicle.

The ROV’s focal point will be to address issues associated with diving, often arising from net handling operations like installation, treatments, or harvesting. The incorporation of a resident

Figure 2: Project Sentry will reduce reliance on physical human presence in cages. The use of video, sensors, AI, and machine learning in these in-cage vehicles will reduce the level of decision-making for the human operator.

DEEP TREKKER

ROV ensures consistent inspections of net pens without necessitating manual intervention. This not only curtails direct diving expenditures but also eliminates the potential for human errors, alongside risks associated with bounce diving and prolonged bottom times.

In summary:

• Enhanced safety during inspections

• Reduced operational costs

• Improved accuracy and consistency in environmental monitoring

What is Deep Trekker’s Role in This Space?

Deep Trekker plays a pivotal role in shaping the trajectory of aquaculture by offering the essential tools required to drive both the sector and underwater inspection methodologies forward.

With an unwavering dedication to innovation, Deep Trekker delivers groundbreaking solutions tailored to the integration of cuttingedge technologies such as AI, machine learning, and autonomous robotics.

By bridging the gap between conventional aquaculture methodologies and the potential afforded by these state-of-the-art tools, Deep Trekker empowers the industry to attain unprecedented levels of efficiency, accuracy, and sustainability.

The Future Possibilities of AI in Maritime Operations

Integrating AI into aquaculture practices and the broader spectrum of maritime operations holds immense potential for advancing sustainability, enhancing production efficiency, and fostering responsible industry growth while minimizing environmental impact. AI algorithms possess the capability to process and analyze vast datasets derived from sensors, cameras, and various other sources. This data-driven approach empowers operators to make informed decisions concerning feed management, water quality, and disease detection in real time, thereby optimizing production outcomes.

AI-powered sensors enable the continuous monitoring of water quality parameters such as temperature, pH, dissolved oxygen, and nutrient levels. Deviations from optimal conditions trigger timely alerts, facilitating swift interventions to uphold a healthy aquatic environment.

Furthermore, AI facilitates the analysis of fish behaviour, feeding patterns, and growth rates to ascertain the optimal feeding schedule and quantity. This minimizes feed wastage, reduces costs, and ensures the well-being of the fish population.

AI also enables the detection of anomalies and unusual behaviours in fish populations, allowing the monitoring of fish health, stress levels, and interactions. Early identification of disease signs based on visual cues from fish behaviour and appearance allows for prompt diagnosis and targeted treatment, thereby mitigating the impact of disease outbreaks.

AI-driven ROVs automate various tasks such as feed distribution, water quality monitoring, and underwater inspections, reducing the reliance on manual labour and enhancing operational efficiency.

By leveraging historical data and current conditions, AI facilitates the prediction of trends and potential issues, meaning maritime operations can proactively mitigate risks and optimize resource allocation.

Additionally, AI streamlines compliance efforts by automating data collection, recordkeeping, and reporting processes, ensuring adherence to regulatory standards. It also accelerates research endeavours by simulating environmental conditions, genetic traits, and growth patterns, expediting the development of new breeds and farming techniques.

What Autonomy Means for Aquaculture Farms

Autonomy within aquaculture farms represents a paradigm shift towards self-

directed operations, empowering farms to deploy automated systems and technologies that oversee, manage, and optimize diverse processes with minimal human intervention.

This autonomy fosters informed decisionmaking, facilitating continuous data collection that can be remotely reviewed and analyzed from any location. It also enables more frequent inspections and streamlines operations, significantly reducing time consumption. Automation mitigates the need for human personnel to undertake tasks in hazardous or challenging conditions, such as underwater inspections or maintenance duties, reducing the risk of accidents and enhancing overall safety.

From precise feeding schedules to roundthe-clock monitoring, autonomy offers the prospect of bolstering safety, sustainability, and productivity in aquaculture and a broad range of maritime operations. It contributes to safer

and more responsible practices by reducing risks and optimizing resource efficiency.

More frequent, or even daily, inspections allow farmers to substantially enhance fish health, uphold water quality, and preemptively address issues before they escalate. Consequently, this leads to more cost-effective operations and a heightened positive environmental impact.

In essence, the integration of autonomy into aquaculture signifies not only a safer environment but also safer inspections and operations, ultimately driving towards a more sustainable and efficient industry.

Charting a Course for the Future: Navigating Towards Sustainable Maritime Operations

As the maritime industry continues to embrace technological innovations, the adoption of autonomous remote inspections emerges as a cornerstone of its evolution (Figure 3).

Figure 3: Deep Trekker’s PHOTON ROV is designed for remote inspections to help elevate aquaculture operations.

Through initiatives like Project Sentry and the pioneering efforts of companies like Deep Trekker, the future of aquaculture appears promising, characterized by efficiency, sustainability, and responsible stewardship of marine resources.

By leveraging AI and autonomy, aquaculture stakeholders can navigate towards a brighter, more resilient future, where innovation and environmental stewardship go hand in hand. u

Chad Gillen has been the content strategist for Deep Trekker Inc. for over a year. Since joining the company in 2023, he has studied and authored a variety of research and technical papers, case studies, and long-form editorials surrounding the use of ROVs and remote pipe crawlers across industries such as aquaculture, offshore energy, infrastructure, maritime, ocean science, police, and defence. In his first year, he has seen his works published in Unmanned Systems Technology, WorkBoat Magazine, Journal of Ocean Technology, Marine Technology Reporter, Wevolver, and Ocean News & Technology Magazine, to name a few.

Artificial Intelligence for Corrosion Detection on Marine Structures

by Shahrizan Jamaludin and Md Mahadi Hasan Imran

Introduction

The maritime industry stands as a vital pillar of global trade, transportation, and resource extraction, relying extensively on robust infrastructure to support its operations. However, this infrastructure faces an enduring threat: corrosion. Marine corrosion, driven by the relentless assault of seawater and its corrosive agents, undermines the integrity and longevity of vessels, offshore platforms, and underwater pipelines. Traditional methods of corrosion detection and mitigation, reliant on periodic inspections and reactive maintenance, struggle to keep pace with the dynamic nature of marine environments. In this context, the integration of artificial intelligence (AI) with predictive maintenance offers a transformative opportunity to enhance the efficiency, accuracy, and safety of corrosion detection in marine settings.

Understanding Marine Corrosion

Before delving into the applications of AI in corrosion detection, it is essential to grasp the mechanisms and implications of marine corrosion. Seawater, with its high chloride content and dissolved oxygen, serves as a potent corrosive medium, initiating electrochemical reactions that corrode metal surfaces. The resulting corrosion products compromise the structural integrity of marine infrastructure, posing risks to human safety, environmental sustainability, and economic viability. Image noise can also affect the detection of corrosion on marine structures (Figure 1). Understanding the complex interplay of environmental factors, material properties, and corrosion processes is paramount for devising effective detection and prevention strategies.

Figure 1: Corrosion images with the presence of blurry effect, Gaussian noise, and periodic noise.

The Role of Artificial Intelligence

Artificial intelligence, encompassing machine learning, deep learning, and neural networks, offers a paradigm shift in corrosion detection methodologies. Unlike traditional approaches that rely on predefined rules or human expertise, AI algorithms can analyze vast amounts of data, identify intricate patterns, and make informed decisions autonomously. By learning from historical corrosion data and continuously adapting to evolving conditions, AI-powered systems can enhance the accuracy, efficiency, and reliability of corrosion detection in marine environments.

Applications of AI in Marine Corrosion Detection

The applications of AI in marine corrosion detection are multifaceted and encompass

various techniques and technologies. Image recognition algorithms, trained on extensive datasets of corroded metal surfaces, can automatically detect and classify corrosionrelated defects such as pitting, cracking, and rust accumulation in underwater inspection images. These AI-based systems not only streamline the inspection process but also improve the accuracy and consistency of defect identification compared to manual inspection methods (Figure 2).

Sensor Data Analysis

In addition to visual inspection, AI algorithms can analyze real-time sensor data collected from corrosion monitoring systems installed on marine structures. These sensors measure parameters such as temperature, humidity, salinity, and pH levels, which influence

Figure 2: Automated corrosion detection with AI on marine structures.

corrosion rates. By integrating sensor data with AI models, it is possible to assess corrosion risk, predict future degradation trends, and recommend proactive maintenance strategies. This data-driven approach enables asset owners and operators to prioritize resources effectively and minimize the risk of unexpected failures.

Predictive Modelling

One of the key strengths of AI lies in its ability to develop predictive models based on historical data and environmental factors. By leveraging machine learning algorithms, it is possible to forecast corrosion rates, estimate remaining useful life, and identify critical failure points in marine structures. These predictive insights empower decision-makers to optimize maintenance schedules, allocate resources efficiently, and mitigate the longterm effects of corrosion on asset integrity and operational reliability. These parameters are needed for predictive maintenance of corrosion on marine structures.

Automation and Robotics

AI-enabled automation and robotics offer transformative solutions for underwater corrosion detection and maintenance.

Uncrewed underwater vehicles equipped with AI algorithms and advanced sensors can navigate complex underwater environments, inspecting submerged structures for signs of corrosion without the need for human intervention. These autonomous inspection systems not only reduce operational costs and downtime but also enhance safety by eliminating the risks associated with human divers working in hazardous underwater conditions. A drone can also be used inside of a ship by installing corrosion detection algorithm into the drone.

Benefits of AI in Marine Corrosion Detection

The adoption of AI for marine corrosion detection offers a wide range of benefits across different aspects of asset management and maintenance:

• Improved Accuracy: AI algorithms

can detect corrosion defects with greater accuracy and consistency compared to manual inspection methods, reducing the risk of missed or misclassified defects.

• Enhanced Efficiency: By automating inspection tasks and prioritizing maintenance activities based on predictive insights, AI streamlines the corrosion detection process, saving time and resources for asset owners and operators.

• Cost Savings: Proactive maintenance strategies enabled by AI predictive modelling help minimize downtime, repair costs, inspection costs, and operational disruptions associated with corrosionrelated failures.

• Enhanced Safety: Automation of underwater inspection tasks reduces the reliance on human divers, mitigating the risks of accidents, injuries, and fatalities in hazardous marine environments.

Challenges and Limitations

Despite its considerable potential, the widespread adoption of AI in marine corrosion detection faces several challenges and limitations:

• Data Quality and Availability: The effectiveness of AI algorithms depends on the quality, quantity, and diversity of training data available for model development. In the marine domain, accessing high-quality corrosion data from diverse environments and operating conditions can be challenging.

• Model Interpretability: The blackbox nature of some AI models poses challenges in understanding and interpreting their decision-making processes as in Figure 3 and Figure 4. In safety-critical applications such as corrosion detection, model transparency and interpretability are essential for gaining stakeholders’ trust and confidence.

• Environmental Variability: The dynamic and unpredictable nature of the marine environment introduces uncertainties that can affect the performance of AI

algorithms. Variations in water chemistry, temperature, and flow dynamics can impact corrosion rates and complicate the development of accurate predictive models.

• Integration with Existing Systems: Integrating AI-based corrosion detection systems with existing infrastructure and workflows requires careful planning, coordination, and investment in staff training and technology adoption. Resistance to change and organizational inertia may pose additional barriers to the adoption of AI in marine corrosion detection.

Future Directions

To address these challenges and unlock the full potential of AI in marine corrosion detection, several research directions and technological advancements are worth considering:

• Data Sharing and Collaboration: Collaborative efforts among industry stakeholders, research institutions, and government agencies can facilitate data sharing, standardization, and benchmarking initiatives, enabling the development of more robust and reliable AI models for corrosion detection.

• Explainable AI: Research into explainable AI techniques aims to enhance the transparency and interpretability of AI models, enabling stakeholders to understand the underlying rationale behind model predictions and recommendations.

• Multimodal Sensing and Fusion: Integrating multiple sensing modalities, including visual, acoustic, and chemical sensors, can provide a more comprehensive and accurate assessment of corrosion-related phenomena in marine environments. Fusion of data from diverse sources enhances the resilience and reliability of AI-based corrosion detection systems.

• Edge Computing and IoT Integration: Leveraging edge computing and Internet of Things (IoT) technology enables realtime processing and analysis of sensor data at the source, reducing latency and bandwidth requirements for AI-based corrosion detection systems deployed in remote or bandwidth-constrained environments.

• Human-Machine Collaboration: Augmenting human expertise with AIpowered decision support tools enables collaborative decision-making processes, where AI algorithms provide actionable

Figure 3: Interpretation of corrosion predictive modelling with greyscale values.

Figure 4: Interpretation of corrosion predictive modelling with binary values.

insights and recommendations, while human operators retain control and oversight over critical decisions.

Conclusion

In conclusion, artificial intelligence offers a transformative approach to corrosion detection for marine structures, offering enhanced accuracy, efficiency, and safety compared to traditional methods. By leveraging AI algorithms for data analysis, predictive modelling, and automation, stakeholders can proactively manage corrosion risks, optimize maintenance strategies, and ensure the integrity and longevity of marine structures in a sustainable manner. While challenges remain in terms of data availability, model interpretability, and environmental variability, ongoing research, collaboration, and technological advancements hold the key to realizing the full potential of AI in safeguarding our marine infrastructure against corrosion. u

Dr. Shahrizan Jamaludin, P.Tech., is a senior lecturer in the Maritime Technology and Naval Architecture program, Faculty of Ocean Engineering Technology, Universiti Malaysia Terengganu, Malaysia. In this role, he is responsible for the development of computer vision and image processing algorithm for marine structures maintenance. He brings 16 years’ experience as project engineer and research officer

to the role. He is a graduate of the Electronic and Computer Engineering program from Universiti Teknikal Malaysia Melaka. His research interests include image processing, computer vision, pattern recognition, computer engineering, electronics, industrial automation, robotic and remote sensing of marine applications, among others.

Md Mahadi Hasan Imran is currently pursuing a master’s degree in maritime engineering at Universiti Malaysia Terengganu (UMT). With a solid foundational education in maritime technology, he completed his bachelor’s degree at the same institution, where he developed a strong base in both theoretical and practical aspects of maritime engineering. Before his studies at UMT, he earned a diploma in marine technology from IIMAT College in Kuala Lumpur, which kickstarted his journey in the maritime field. His academic and research interests are diverse, focusing on critical areas such as marine corrosion, which poses significant challenges in maritime engineering. Additionally, he is deeply involved in the cutting-edge fields of image processing, autonomous ships, and artificial intelligence in naval architecture. His work aims to integrate modern technological advances with traditional maritime engineering practices to innovate and enhance the efficiency and safety of maritime operations. His research could potentially lead to significant advancements in the design and operation of autonomous ships, offering solutions to some of the contemporary maritime industry’s most pressing issues.

Blue

AI and Improving the Economy

by Mark J. Spalding

RYAN LOUGHLIN ON UNSPLASH

Introduction

The ocean covers more than 70% of the Earth. It regulates the climate, absorbs greenhouse gas emissions, and generates the oxygen on which all life depends. The ocean economy has always existed for transport and trade, resource extraction, and waste disposal (intentional and unintentional) – the known maritime trade dates to 2000 BCE in the Indian Ocean. International trade expanded steadily as advances in shipbuilding and navigation enabled seafaring across greater distances and further from land. As transocean trade grew, governments carried most of the financial risk until investing in shares of merchant ship cargoes became an attractive private gamble. The fortunes of investors, speculators, and the nascent insurance industry were built from participation in the international ocean trade in tobacco, alcohol, textiles, chinaware, spices, whale oil, precious metals, and enslaved peoples.

Previously, ocean economy activities were largely shortsighted and unsustainable. It was assumed there was no limit to what we could dump into or extract from the ocean. We are now seeing simultaneous quantity and rate of change in the ocean and its adverse effects on the ocean economy. Human activities have caused perturbations in the relatively stable climate and weather patterns that had allowed shipping and fishing to move along relatively predictable schedules for centuries. Likewise, excess greenhouse gas emissions are changing ocean temperature, depth, and chemistry in ways that will affect all marine life and coastal and in-ocean infrastructure. As a result, we now need to focus on reducing the risk to our water, food, oxygen, and energy security.

The Blue Economy

For the purposes of this essay, the blue economy is defined as the subset of the ocean economy that takes the long view, as does the “green business” sector, where broad sustainability principles are applied and promoted with an eye towards global social

and economic well-being. The call for a new blue economy is to focus on sustainable subsets within the global ocean economy and push other subsets to become more ocean friendly. The “old” ocean economy includes offshore oil and gas extraction, recreational and commercial fishing, open-pen aquaculture, shipping, coastal development, and telecommunications, all of which need to be brought up to 21st-century standards that acknowledge our global dependence on restoring ocean health.

For example, technological advances have enabled better monitoring and understanding of the flawed economics of heavily subsidizing high-seas fishing, which takes place in the two-thirds of the ocean that lies outside of any national jurisdiction. Ending all government fleet subsidies would limit fishing to profitable activities that rely less on forced labour and wasteful methods and might increase abundance simultaneously.

Shipping is the best way to move goods from the perspective of their carbon footprint by weight and distance. Still, transport activities need to be conducted in ways that eliminate the costs of harm, such as noise pollution, air pollution, accidental dumping, and transport of alien species in ballast water. The industry must embrace clean fuel, clean ballast, clean air, and noise reduction technologies.

The world’s coasts and ocean are a valuable and delicate part of our natural capital. The business-as-usual model threatens marine ecosystems, coastal communities, and the functions of our planet. The new blue economy aims to address these issues and create a more sustainable and ocean-friendly world.

New industrial sectors reflecting changes in the ocean economy include renewable ocean energy, seabed mining, nature-based solutions, blue technology and biotechnology, and nutrition and nutraceuticals. These emerging sectors need to be assessed for their value to

the new blue economy, tapping into existing science and technology to support positive activities and limit negative consequences. Thus, the blue economy is a crucial approach to sustainable development, focusing on sustainably using ocean resources for economic growth, improved livelihoods, and job creation while preserving the marine ecosystem. It decouples socio-economic development from environmental degradation, promoting marine ecosystem protection, equity in access, and improved well-being. The challenge lies in generating knowledge, supporting innovation, and developing scalable solutions for an equitable, resilient, and sustainable ocean economy (a blue economy) to be developed in parallel to and in keeping with rapidly changing environmental, social, political, and climate conditions.

The blue economy must be based on accessible knowledge, sustainable environmental practices, and effective governance. It must be managed equitably for current and future generations, with clear distinctions between rights holders and stakeholders. Effective management is closely linked to knowledge and action, and exhibits resilience in changing conditions. Empowering local communities, particularly Indigenous groups, is essential for a holistic understanding of the marine environment and as inputs for policy-making. Technology and innovation can empower the transition to a blue economy, but only with context-appropriate technology and free, prior, and informed consent for such new technologies and innovations.

Artificial Intelligence

Suppose we commit to fully restoring abundance and managing the human relationship with the ocean for the good. In that case, the scale of the challenges is such that we need tools that can collect, correlate, and analyze vast quantities of data. Artificial intelligence (AI) is the simulation of human intelligence processes by computer systems and other machines. It encompasses various technologies, including machine

learning, natural language processing, speech recognition, and machine vision. As such, AI can play a pivotal role in advancing the goals of the blue economy by enabling data-driven decision-making, promoting sustainability, and fostering responsible use of marine resources.

AI uses a set of algorithms that classify, process, and manipulate data and can learn by statistically improving its accuracy and results. In development since the 1950s, AI can perform tasks that traditionally require humans, such as grammar and spell checking, translation, driving cars, and chess-playing. AI can perform legal case research, trade stocks, translate languages, and be a personal trainer or assistant.

Generative AI, like ChatGPT, generates writing or images based on instructions. AI processes and products depend on what is out there to “read,” which is limited to digitized data. Progress in AI depends on people working together to discover new knowledge and find better ways of doing things, which requires motivation, self-awareness, creativity, imagination, strategic thinking, and emotional intelligence.

By leveraging AI, the blue economy can move humankind closer to the long-sought balance between economic prosperity and ecological resilience. AI-enabled solutions can address the ocean health effects of the blue economy by monitoring marine ecosystems, promoting efficient logistics operations, and mitigating water pollution. AI data collection and analysis technologies allow real-time monitoring of ocean conditions, fish behaviour, and shipping activities. Thus, AI can provide valuable information and facilitate informed decision-making, ideally contributing to both the sustainable use of marine resources and improving the ocean’s overall health simultaneously.

AI churns through data in volumes and speeds that seemed impossible just a few years

ago. It can provide insights that consider every imaginable factor, helping in decisionmaking and intervention to prevent accidents, anticipate coastal storm events, and promote sustainable use of ocean resources. At the very least, this allows us to monitor the marine ecosystem and the changes it is undergoing. For example, adaptive fisheries enforcement and coastal planning will continue to expand their dependence on AI technologies, such as remote sensing and computer analysis of satellite data. The businesses that make this technology are part of the blue economy.

Using AI to Foster the Blue Economy

The mitigation of risks to the security of food,

water, oxygen, and energy is the focus of the blue economy subset of the ocean economy. Here are some examples of how the adoption and application of AI and machine learning can help strengthen the blue economy and reduce such risks:

• Ports and Port Operations: AI can significantly enhance port operations and contribute to a more sustainable future by reducing greenhouse gas emissions, improving operational efficiency, and improving working conditions (Figure 1). AI can help ports achieve net-zero operations and contribute to green shipping. Through automation and optimization, AI

Figure 1: AI can help ports achieve net-zero operations and contribute to green shipping through automation of port operations; mitigation of port congestion; and improved operational efficiency.

BERND DITTRICH ON UNSPLASH

can help ports achieve net-zero operations and contribute to green shipping. Wait time and carbon emissions can be reduced by using AI to provide real-time data and analytics to mitigate port congestion, including managing vessel movements, and optimizing berthing schedules. These benefits are shared with improving operational efficiency by using AI to automate the buying and selling of port services.

• Ocean Transport: More than 90% of the world’s goods are moved via ocean transport. It has the lowest carbon footprint per kilometre of any transportation method. The sector is under pressure from governments and consumers to reduce emissions and increase operational sustainability, even as the risk from intense storms and unpredictable weather patterns is growing. AI and autonomous systems can optimize ship operations, reducing emissions (including those related to ocean noise, waste streams, and CO2 emissions). AI can also improve

fuel efficiency, aid decision-making, promote sustainable practices, and enhance operations’ transparency and efficiency. Using AI for route optimization, predictive maintenance, and real-time performance monitoring, ships can operate more efficiently, cutting fuel costs and lowering emissions. Building transparent AI systems that provide visibility to all stakeholders in the shipping sector will allow more informed decision-making and, hopefully, achieve critical improvements in vessel and worker safety.

• Ocean-sourced Food: There are real blue economy investment opportunities in food production – sustainable aquaculture and coastal and oceanic fisheries. These sectors have significant embedded inequity, environmental harm, and related issues that must be addressed. Food sector companies should explore opportunities for returns from emissions reductions from fisheries operations, including aquaculture (Figure 2), wild capture, and processing (e.g., low-carbon or zero-emission vessels),

MARK J. SPALDING

Figure 2: AI can play a role in maximizing the potential of ocean resources for food production in an environmentally and socially sustainable manner. The sustainable Bakkafrost Applecross aquaculture facility (shown here) is a purpose built, state-of-the-art recirculation aquaculture system on the remote peninsula of Wester Ross in the Scottish Highlands.

energy efficiency in postharvest production (e.g., cold storage and ice production), and alternative aquaculture feeds (algal, microbial, fungal, and insect). The social and environmental goals of the blue economy should also apply to new emerging sub-sectors, including new cellularly manufactured seafood (such as BlueNalu), kelp farming, and byproduct transformation in fisheries. Governments concerned about ocean health can make regulatory changes that eliminate fuel subsidies for fishing fleets and call for vessel and gear improvements that increase efficiency while strongly constraining catch to sustainable levels.

AI can help make ocean-sourced food more environmentally and socially sustainable by enabling more accurate fishery stock assessments, facilitating more sustainable fisheries management, and providing valuable data for decisionmaking for all seafood industry sectors. By processing quantitative and qualitative data, AI-driven models open new opportunities for a sustainable food system. Additionally, AI-based tools can automatically recognize the size and species of each fish, facilitating the handling of fish, recording of catches, and reducing the problem of seafood fraud, which distorts the management of crucial consumer species. Ocean food security experts have also highlighted the potential for the ocean to produce more food while driving sustainable economic growth. This indicates that AI can play a role in maximizing the potential of ocean resources for food production in an environmentally and socially sustainable manner.

AI technologies such as computer vision and remote sensing can aid in decision-making by making it easier to gather relevant data and encouraging the sustainable use of marine resources. AI-enabled technology will likely promote efficient logistics and monitor marine

ecosystems to identify trends that catalyze actions to reduce threats to ocean health.

Thus, artificial intelligence may play a significant role in promoting the growth of the blue economy by enabling data-driven decision-making, promoting sustainability, and promoting the prudent use of marine resources. Using AI, growth in the blue economy can bring humankind closer to achieving a balance between ecological resilience and economic prosperity.

What are the Downsides?

AI is not flawless, nor is it without risks. All technology, all tools, can be used for good or evil. A bow and arrow can be used to obtain food or for murder; chemistry for lifesaving medicine or poison. From machinery to weapons to power tools, we review and redesign these technologies to be safer and to improve their effectiveness. And so shall it be for artificial intelligence. We must acknowledge that some applications of AI may move us in the opposite direction of sustainable development or a blue economy:

• False Modelling Assumptions: Machine learning, by definition, is used to predict the future by looking at the past. Thus, it is the wrong tool for many applications, especially when designing interventions and mechanisms to improve the blue economy. Predictions should not be an end in themselves. We should remember that a “digital twin” model of ocean systems is not the ocean. It is not an exact copy. At best, it is informative.

Similarly, modelling the ocean economy “as it is” can be helpful. Nevertheless, there is a false assumption underlying many machine-learning models that the model itself will not change the reality it is modelling. However, as soon as we begin using that model to make decisions that hopefully move us toward a blue economy, we are changing the ocean in ways large and small.

• Equity Concerns: Many optimistic AI scientists and worried AI policy-makers have noted that issues related to AI will affect everyone around the world in some manner. Some of the most valuable elements of an authentically blue economy depend on local management, local empowerment, and low-tech solutions that generate local economic benefits. Many AI solutions are about global data analysis and application – from weather prediction and shipping routes to predicting fish stock shifts in a warming ocean. Adopting AI in the blue economy could widen the gap between countries and regions with advanced technological and scientific capabilities and those without. In other words, adopting AI could exacerbate global inequality by deepening the uneven distribution of wealth. And, unfortunately, decisions made by AI systems could favour short-term financial gains over long-term environmental sustainability if they are not adequately managed.

In addition, it is well established that AI systems (and findings) are as biased as the datasets they explore. Race, gender, class, or religion may be among the variables an algorithm uses for decision-making (or inferred from hidden correlations in the data). Risk assessments must value communities equally – aligning the blue economy with environmental and economic justice. For example, we should ask how AI could help end waste dumping in poorer communities or prevent predatory fishing practices by more affluent nations. Such pursuit of equity may decrease distant shareholder profits but can and should improve local and regional willingness to manage ocean resources more sustainably.

• Losing Jobs to Automation: AI can automate various tasks in fishing and related industries, such as monitoring fish stock levels, catch tracking, processing, and even the operation of fishing vessels, including the newest automated sailing

ship technology. While efficiency may increase, it could also lead to job losses in sectors where human labour is currently essential. Such losses will harm local economies and profoundly affect the cultural identity of artisanal fishing families and their communities. Where AI can support community management of fisheries and empower communities to improve the quality and thus the price of their catch, the local economy can better absorb some of the losses if automation replaces human labour and concentrates income. In other words, we need to anticipate displacement or job losses that AI will cause, ensuring that displaced people can take advantage of the new opportunities it can create.

• Environmental Concerns: Using AI in the blue economy could have unintended environmental consequences. For example, AI-powered monitoring systems might inadvertently contribute to overfishing if they do not accurately reflect fish stock levels or changing ocean conditions’ effects on fishery reproduction (the systems are only as good as the data available). Increased fishing efficiency alone may exacerbate illegal, unreported, and unregulated fishing. Autonomous vehicles and drones can threaten wildlife and natural habitats, just like the well-documented risks posed by deep seabed mining vehicles operating autonomously on the seafloor.

We must also recognize the increased demand for energy. AI model training and operation require significant energy, outpacing efforts to reduce consumption and increase the use of renewables, thus contributing to increased greenhouse gas emissions. For example, training the MegatronLM language model (similar to GPT-3) consumed almost as much energy as three average U.S. households use in a year. The training used 512 of Nvidia’s V100 GPUs running nine days, consuming an estimated 27,648 kilowatt

hours (kWh) of electricity. Finally, there is a shortage of recycling and proper disposal options. At the end of life, AI technology hardware becomes electronic waste that contains hazardous chemicals such as lead, mercury, and cadmium. Improper disposal contaminates soil and water supplies, putting human communities and ocean life at risk.

• Regulatory Challenges: AI in the blue economy will collect and analyze large amounts of data, presenting privacy and security concerns. While collecting ocean data and automating processes, different AI technologies will be deployed in areas with

overlapping jurisdiction and those without jurisdiction. Integrating AI into the blue economy will require new regulations to ensure the technology is used responsibly and does not negatively affect the environment or burden local communities. In addition, the data of fishing fleets in national waters (e.g., location, catch, etc.) has long been protected by law, creating gaps that AI will need help to fill. Ensuring the privacy of sensitive data and working to use data to inform the management of public resources in the public’s best interests while protecting against cyber threats will be critical issues for the blue economy as it embraces AI. Lastly, some companies

Figure 3: Mitsui O.S.K. Lines, Ltd. uses a modern commercial sail-powered cargo ship, Shofu Maru, equipped with Wind Challenger – a hard sail wind power propulsion system – to reduce its greenhouse gas emissions.

MITSUI O.S.K. LINES, LTD.

prioritize financial gain over environmental, social, or governance concerns, and the need for more transparency and complexity of AI systems can make it challenging for users to understand their footprint within any of those concerns.

Conclusion

At scale, the blue economy moves us away from destructive extraction-focused business sectors that deplete our ocean resources, damage the underpinnings of the global economy, and cost communities jobs and vital natural resources. The blue economy moves us toward more ocean-positive economic activities such as greener shipping (Figure 3), sustainable tourism, sustainable aquaculture, and new environmentally friendly technologies. This includes engagement to improve practices in the “old” ocean economy and embraces emerging sectors such as renewable energy, blue carbon, and blue biotechnology. A sustainable blue economy is crucial for planetary and human health and makes good business sense.

While AI has the potential to contribute significantly to growing the blue economy and addressing the negative impact of elements of the broader ocean economy, AI also poses challenges that need to be addressed through careful policy-making, investment in digital skills, and international cooperation to ensure that the benefits of AI are distributed equitably, and the environment is protected. To mitigate these potential harms, a multifaceted strategy is needed. This includes developing energy-efficient hardware and AI algorithms, promoting ethical AI design standards, and fostering a culture of openness and responsibility. Governments and regulatory agencies should adopt standards and restrictions to ensure the ethical creation, use, and disposal of AI technologies, and collaboration between businesses, academics, and policy-makers is essential. u

Christian, B. [2020]. The alignment problem: machine learning and human values. Norton & Co. Gasser, U. [2024]. Guardrails: guiding human decisions in the age of AI. Princeton University Press.

Khanna, R. [2022]. Progressive capitalism: how to make tech work for all of us. Simon & Shuster.

Mitchell, M. [2019]. Artificial intelligence: a guide for thinking humans. Picador.

Schaich Borg, J. [2024]. Moral AI: and how we get there. Pelican.

Schellmann, H. [2024]. Algorithm: how AI decides who gets hired, monitored, promoted, and fired and why we need to fight back now. Hachette Books.

Sengupta, S. [2023]. Harnessing the power of AI against climate change. Seventeen goals magazine.

Spalding, M. [2019]. The new blue economy: taking care of the ocean to sustain us. The Ocean Foundation.

Mark J. Spalding, president of The Ocean Foundation, also serves on the Sargasso Sea Commission. He is a senior fellow at the Center for the Blue Economy at the Middlebury Institute of International Studies and an advisor to the High-Level Panel for a Sustainable Ocean Economy. In addition, he serves as the advisor to the Rockefeller Climate Solutions Fund, the Rockefeller Global Innovation Strategy, and the UBS Rockefeller and Kraneshares Rockefeller Ocean Engagement Funds (unprecedented ocean-centric investment funds). Mr. Spalding is a member of the United Nations Environmental Programme Guidance Working Group for its Sustainable Blue Economy Finance Initiative. He coauthored the Transatlantic Blue Economy Initiative, a joint project of the Wilson Center and Konrad Adenauer Stiftung. Mr. Spalding designed the first-ever blue carbon offset program, SeaGrass Grow. From 2018 to 2023, he served as a member of the Ocean Studies Board and the U.S. National Committee for the Decade of Ocean Science for Sustainable Development, both of the National Academies of Sciences, Engineering, and Medicine (USA). He is an expert on international ocean policy and law, blue economy finance and investment, and coastal and marine philanthropy.

The Cutting-edge Integration of Machine Learning in Offshore Wind Farm Development and Management

by Masoud Masoumi

Introduction

As humanity races against time to combat climate change, offshore wind farms have emerged as one of our beacons of hope in our quest for sustainable energy. However, offshore wind energy has faced major challenges, such as high initial costs, technological limitations, grid connection, environmental and regulatory considerations, as well as operations and maintenance. Machine learning (ML) techniques have the potential to revolutionize the field of wind farm development and monitoring, particularly in the realms of structural health monitoring, maintenance, and layout optimization. These advanced

methodologies have significant potential to enhance the accuracy of failure identification, paving the way for precision maintenance strategies. Moreover, they can play a pivotal role in optimizing wind farm layouts, refining power production forecasting, and mitigating wake effects, consequently, boosting energy generation efficiency.

The integration of ML-driven control systems can also generate promising results by enhancing the operational strategies of offshore wind farms. This integration has already demonstrated notable improvements in overall performance and energy output in

Figure 1: Two technicians stand on a transfer vessel dock looking at an offshore wind farm in the North Sea near Germany. Predictive maintenance using machine learning can help reduce downtime, optimize maintenance schedules, and prolong equipment lifespans.

some published research studies. Additionally, ML algorithms have been implemented in climatic data prediction and environmental studies, offering predictive capabilities that optimize power generation, while facilitating comprehensive assessments of environmental impacts. These initial promising results mark a significant leap forward in the sustainable and efficient management of wind energy resources.

Yet, the integration of ML in wind farm development and management is not without its complexities and challenges. In this essay, I will delve into the transformative potential of ML in three key areas of offshore wind

farm development and monitoring: system optimization and operational maintenance, environmental impact assessments, and socioeconomic impact analysis. Additionally, I will briefly discuss some of the associated risks in deploying ML in these areas, which must be clearly addressed before large-scale implementation.

Optimization and Predictive Maintenance

The transition from onshore to offshore wind farms is inevitable, but it comes with a host of challenges. A primary concern is the difficulty of performing maintenance operations in offshore environments (Figure 1), which

often face harsh conditions, resulting in extended periods of downtime. The promise of predictive maintenance powered by ML is tantalizing: reduced downtime, optimized maintenance schedules, and prolonged equipment lifespans. The integration of real-time data and dynamic modelling for adaptive optimization within the context of offshore wind farm operations can result in maintenance cost reduction. While research studies in the last few years have extensively focused on machine learning techniques for tasks, such as optimizing wind farm layouts, estimating power losses due to wake effects, and refining control strategies, there is a conspicuous lack of emphasis on integrating real-time data and dynamic modelling into these optimization processes.

Real-time data, representing variables such as weather conditions, power demand, and turbine health, holds potential to significantly influence offshore wind farm performance. Research that focuses on developing adaptive optimization strategies capable of continuously adjusting system configurations based on real-time data inputs may usher in a new era of efficiency and reliability in offshore wind farm operations. Furthermore, dynamic modelling techniques that factor in evolving environmental conditions and equipment degradation over time could substantially bolster the accuracy of performance predictions and optimization endeavours.

Granted all the revolutionary changes that ML can bring to the field of predictive maintenance for offshore wind farms, the reliance on ML algorithms for critical decision-making raises concerns about algorithmic bias and interpretability. The question of “can we trust these algorithms to prioritize safety and reliability over shortterm gains?” may need to be clearly discussed and appropriately addressed. Moreover, the transition from reactive to proactive maintenance strategies necessitates robust data infrastructure and continuous model validation. Failure to address these challenges

could lead to unforeseen disruptions and safety hazards, undermining the very goals predictive maintenance seeks to achieve. ML models often operate as black boxes, leaving stakeholders grappling with opaque decision-making processes. How do we strike a balance between model complexity and interpretability without compromising performance? Moreover, the quest for interpretability must not overshadow the need for ethical considerations and algorithmic accountability. Blind trust in ML outputs can lead to unintended consequences and reinforce existing biases, highlighting the imperative to challenge assumptions and biases embedded in ML systems.

Environmental Impact Assessment

Understanding and mitigating environmental impacts associated with offshore wind farms is yet another compelling application for harnessing ML techniques. Some research studies have focused on analyzing underwater sources of sound near offshore wind farm sites. By employing sophisticated unsupervised learning models, researchers uncovered temporal, spatial, and spectral patterns in underwater recordings, shedding light on potential impacts on marine life. Similarly, studies assessing species distribution models and collision risk maps for birds near offshore wind turbines underscore the meticulous efforts to balance renewable energy production with conservation concerns.

One notable area ripe for exploration lies in the integration of diverse ML techniques and ecological data sources to achieve a holistic understanding of wind farm projects’ environmental footprint. This involves leveraging ML algorithms for tasks like monitoring bird and marine species, forecasting environmental changes, and analyzing habitat dynamics. The potential advantages of such an integrated approach are manifold. By combining these ML capabilities into a cohesive framework, researchers and industry stakeholders can

gain deeper insights into the complex interactions between offshore wind farms and marine and avian ecosystems. This holistic perspective not only facilitates more accurate environmental impact assessments but also enables the optimization of wind energy systems, while minimizing adverse effects on biodiversity.

While ML-driven environmental impact assessments offer a nuanced understanding of the complex interactions between offshore wind farms and ecosystems, the reliance on historical data and predictive models introduces uncertainties and potential blind spots. How do we account for emergent ecological dynamics and long-term impacts that may elude conventional modelling approaches? Furthermore, there is a need to democratize processes, ensuring diverse perspectives and local knowledge are integrated into decision-making frameworks.

Failure to do so risks overlooking community concerns and undervaluing the intrinsic value of biodiversity.

Socioeconomic Impact Analysis

ML-enabled socioeconomic impact analysis tools promise to quantify the farreaching implications of offshore wind farm development. ML algorithms can analyze diverse datasets, such as demographic information, employment records, economic indicators, and infrastructure data, to provide predictions regarding the socioeconomic impacts of offshore wind farm projects. Some of the impacts of offshore energy systems that can be gauged using ML include job creation, income distribution, economic growth, and community development in areas surrounding wind farms (Figure 2). By leveraging historical and real-time data, ML models can provide insights into how wind farm initiatives may influence local economies and societies. While

Figure 2: A view of San Juan coast from the Castillo San Felipe del Morro on a windy day. Machine learning can help analyze socioeconomic trends of wind farms on a community such as employment, property values, tourism, among others.

MASOUD MASOUMI

historical data points may be limited due to the relatively small number of large-scale offshore wind farm projects across diverse communities, a hybrid approach that combines data from similar projects with expert opinions can pave the way forward. This strategy is essential for initial implementations until a more comprehensive understanding of these projects is developed over time.

Once more historical data is provided, ML can facilitate the assessment of long-term socioeconomic sustainability of wind farms by analyzing trends, patterns, and feedback loops in socioeconomic data over time. This analysis can help identify the evolving impacts of wind farms on employment trends, local businesses, property values, tourism, environmental conservation efforts, and community wellbeing. By identifying potential challenges and opportunities early on, decision-makers can implement targeted interventions and policies to enhance the positive socioeconomic outcomes of wind farm projects.

ML techniques, such as natural language processing and sentiment analysis, can be used to monitor and analyze public sentiment, concerns, and attitudes towards offshore wind farm projects. This analysis may include social media discussions, news articles, public surveys, and community forums. By understanding community perceptions and engagement levels, stakeholders can tailor communication strategies, address concerns effectively, and foster greater acceptance and support for wind farm developments, while addressing as many stakeholders’ concerns as possible.

I would be remiss to mention that the reliance on quantitative metrics may increase the risk of overlooking qualitative dimensions, such as cultural heritage, community resilience, and social cohesion. This challenge might rely on appropriately addressing important questions, such as “how do we capture and integrate these intangible factors into our frameworks?” Beyond addressing this need, it is crucial to gauge distributional impacts

and ensure equitable access to the benefits of these renewable energy projects. Ignoring socioeconomic disparities could exacerbate existing inequalities and hinder the transition toward a more just and sustainable future.

Final Comments

ML techniques provide valuable tools for developing optimization procedures and maintenance strategies, conducting environmental impact assessments, and exploring innovative approaches to comprehensive socioeconomic studies for offshore wind farms. These tools enable stakeholders to make data-driven decisions, optimize project planning, and promote sustainable socioeconomic development in wind farm regions. However, in navigating the associated complexities of using ML-based approaches, the offshore wind industry must embrace a culture of continuous learning, adaptation, and critical inquiry. Relying solely on technological solutions without addressing underlying systemic challenges is akin to building castles on shifting sands. As we harness the winds of change, let us not forget the enduring power of human ingenuity, empathy, and collective action in shaping a resilient and equitable energy landscape for future generations. u

Dr. Masoud Masoumi holds a PhD in mechanical engineering from Stony Brook University in New York. He also earned an M.Sc. and B.Sc. in mechanical engineering from Semnan University in Iran. With a strong background in offshore renewable energy and machine learning, Dr. Masoumi has made significant contributions to the field through numerous peerreviewed journal publications, conference presentations, and industry collaborations. He is also recognized for his research and work with undergraduate students in offshore energy research and advocacy. Currently, Dr. Masoumi is a data scientist and an adjunct professor at Cooper Union where he continues to explore the potential implementations of deep learning and artificial intelligence for the offshore energy industry. masoud.masoumi@cooperunion.edu

A Bibliometric Review of Research Publications on Digital Twin Predictive Maintenance Systems in the Maritime Industry

Abdelmoneim Soliman, Mervin A. Marshall, Md Safiqur Rahaman, Mohamed A. Ouf, and Ahmed El-Sayed

Improving Detection and Localization of Green Sea Urchin by Adding Attention Mechanisms in a Convolutional Network

M. Israk Ahmed, Lourdes Peña-Castillo, Andrew Vardy, and Patrick Gagnon

Digital Twin Predictive Maintenance Systems

Researchers examine existing research to identify trends and potential avenues for future exploration.

Who should read this paper?

Anyone who is interested in the current state of Digital Twins in the marine industry will be interested in this study. A Digital Twin may be viewed as a virtual model (influenced by real-world data) to address unforeseen scenarios. It can be used to replicate what is happening to an actual product in the real world and offers real-time feedback. In general, any simulation that is representative of a real-world operation is an instance of a Digital Twin; for example, a ship navigation simulator.

Why is it important?

This study draws insights from twelve data clusters (resulting in 1,074 papers). It not only uncovers significant growth in interest from 2016 to 2023, but also synthesizes key findings from the evolution of Digital Twins. By employing bibliometric techniques, the study maps country collaborations, illustrating international research networks in this field. Moreover, it highlights the most cited papers, underlining influential contributions and their impact.

This comprehensive review offers a unique perspective on the development, collaborations, and key research themes in the context of Digital Twins Predictive Maintenance Systems in the maritime industry. It offers researchers and decision-makers comprehensive and up-to-date knowledge. Findings can be used to assist in establishing prospective research directions for Digital Twin Predictive Maintenance Systems in the maritime industry advancement.

About the authors

Abdelmoneim Soliman obtained his B.Eng. in mechanical engineering in 2016 from the Arab Academy for Science, Technology, and Maritime Transport (AASTMT). He also explored a mechatronics major through courses. Mr. Soliman is presently completing his B.Tech. in engineering and applied science from the Marine Institute of Memorial University, NL. His research interest is in integrating advanced technologies (e.g., artificial intelligence, machine learning, and Digital Twins) in the operational phase of marine vessels.

Dr. Mervin A. Marshall, P.Eng., is a faculty member at the Marine Institute of Memorial University, NL. His areas of expertise are structural integrity monitoring using stochastic processes, finite element analysis, hydro-elastic modelling, and applied mechanics. He has also been a major contributor in a five-year national power utility research project, where they implemented and used Digital Twin technology.

Abdelmoneim Soliman

Dr. Mervin A. Marshall

Dr. Md Safiqur Rahaman

Dr. Md Safiqur Rahaman, deanship of library affairs, King Fahd University of Petroleum and Minerals, Dhahran, is a highly experienced professional in the field, having more than 17 years of experience as a librarian. He has a PhD in library and information science. He provides research support services to library users such as literature search, specialized training, reference and information services, research guidance, and consultation. His field of expertise is bibliometric and scientometric studies.

Mohamed Ashraf Ouf obtained his B.Sc. in computer engineering from AASTMT in July 2023. His passion is in integrating innovative technologies to redefine industrial standards. He focuses on leveraging AI to unlock new possibilities across various sectors, while constantly seeking innovative ways to merge machine learning to solve complex challenges and enhance operational efficiency.

Ahmed El-Sayed will receive his B.Sc. in computer engineering from AASTMT in July 2024. He has been actively engaged in multiple research projects as an undergraduate research assistant at the Intelligent Systems Laboratory in AASTMT. His interests span diverse fields such as computer vision, natural language processing, machine learning, and Digital Twins.

Mohamed Ashraf Ouf

Ahmed El-Sayed

A BIBLIOMETRIC REVIEW OF RESEARCH PUBLICATIONS ON DIGITAL TWIN

PREDICTIVE

MAINTENANCE SYSTEMS IN THE MARITIME INDUSTRY

Abdelmoneim Soliman1, Mervin A. Marshall1, Md Safiqur Rahaman2, Mohamed A. Ouf3, and Ahmed El-Sayed3

1Fisheries and Marine Institute of Memorial University of Newfoundland, St. John’s, NL, Canada

2King Fahd University of Petroleum and Minerals, Dhahran, Kingdom of Saudi Arabia

3Arab Academy for Science, Technology, and Maritime Transport, Cairo, Egypt

ABSTRACT

This bibliometric review delves into the topic of “Digital Twin Predictive Maintenance System in the Maritime Industry,” examining existing research to identify trends and potential avenues for future exploration. Through analysis of 12 data clusters (consisting of 1,074 publications) from maritime sources, this study uncovers significant growth in interest from 2016 onwards and synthesizes key findings from the historical evolution of Digital Twins. The review highlights various research clusters, including advancements in Digital Twin technology, Smart Manufacturing applications, and the integration of Blockchain. By using bibliometric techniques, the study maps country collaborations and illustrates international research networks in this field. It also highlights the most cited papers, underlining influential contributions and their impact. This comprehensive review offers a unique perspective on the development, collaborations, and key research themes in the context of Digital Twin Predictive Maintenance Systems within the maritime industry.

Keywords: Digital Twin (DT); Predictive maintenance system; Maritime industry; Industry 4.0; Bibliometric; Digital Twin Predictive Maintenance System (DTPMS); Artificial intelligence (AI); Internet of Things (IoT); Total publications (TP); Total citations (TC)

1. INTRODUCTION

In recent years, technological advances on the Internet of Things (IoT), data analytics, artificial intelligence (AI), and data acquisition techniques have paved the way for the full realization of Industry 4.0, and for its many applications to emerge, which totally changed how some industries managed their business [Attaran et al., 2023; IBM, n.d.].

Digital Twin (DT) is one of those technologies that has utilized the recent changes to the extent that it has become the centre of interest for both industry and academia alike [Fuller et al., 2020]. A Digital Twin may be conceptualized as a virtual model (influenced by real-world data) to address unforeseen scenarios. It can be used to replicate what is happening to an actual product in the real world and offers real-time feedback. In general, any simulation that is representative of a real-world operation is an instance of a DT; for example, a ship navigation simulator.

The global market size for DT was valued at USD 3.1 billion in 2020 [Report Linker, 2020]. It has been projected to grow from USD 10.1 billion in 2023 to 110.1 billion by 2028 at a compound annual growth rate of 61.3% [Report Linker, 2024]. (With such anticipated growth, one can only expect DT applications to become more popular in the future.) By perusing those industries which contributed the most to the world DT economy alluded to earlier, the automotive and transport sector were at the top of the list; they contributed more than 20%. There were other industries with large contributions also; for example, the agriculture, aerospace, and energy industries [Report Linker, 2024].

Seven key applications of DT in a variety of industries are: in design or planning or both, optimization, maintenance, safety, decisionmaking, remote access, and training. It may be an invaluable resource for businesses to boost their productivity, efficiency, and competitiveness [Singh et al., 2021].

In this section, we will briefly discuss the concept behind DT, its potential in the maritime industry, and present an overview of the increasing adoption of DT technology in predictive maintenance.

1.1 Reasons Behind the Increasing Adoption of Digital Twin

At a first glance, it could be assumed that DT is not fully implemented yet. However, one sees the increasing adoption of DT being made possible because of the decreasing costs of technologies that enhance both IoT and the DT, and the recent advances in IT infrastructures and Data Analytics. Moreover, the recent disruptions that the world has gone through (especially the COVID-19 pandemic and the war in Ukraine) may have also contributed immensely to the increasing adoption of DT because researchers were able to devise transformative solutions to vulnerabilities that were shown in the previously adapted methods [Attaran et al., 2023].

DT’s potential is vast and, despite it being relatively new, one can find its adaptations in many technologies and domains ranging from healthcare to smart cities and manufacturing [Bilberg and Malik, 2019]. In the following subsection, we will briefly discuss Digital Twin’s potential adoption

in the maritime industry and its already established applications

1.2 Digital Twin and the Marine Industry

The marine industry is a broad and multifarious industry with a diverse economy ranging from the reservation and utilization of marine resource to seawater distillation to the enhancement of the coastal tourism, and to the more industrial based applications like the shipbuilding and offshore marine industries. So, it is apparent that DT has found its way to enhance the above listed fields [Lv et al., 2023A].

Predictive maintenance is an important application of DT, and it could improve the marine industry immensely. Many algorithms and applications were proposed concerning this matter. For instance, VanDerHorn and colleagues proposed a model that could monitor and estimate the remaining service time of an active ship. These models could aid in the operational and planning side (of an active ship) to monitor and detect fatigue failures [VanDerHorn et al., 2022]. It should also be noted that the recent adoption of machine learning with DT has leveraged the applications of DT in predictive maintenance as several scholars have proposed models in that aspect; for example, Ren et al. proposed comprehensive equipment-centric DTs through the combination of DT and several machine learning algorithms [Ren et al., 2022]. It could be said that DT’s usage in ship maintenance focuses on fault prediction and failure prevention through predictive maintenance [Lv et al., 2023A], which is the aim of this paper.

The operational cost of maintaining aging marine vessels can get expensive if

manufacturers stop fabricating the spare part(s) necessary for refurbishing. Furthermore, the health and safety of the crew on board these aging vessels could be impaired because technicians – who are responsible for performing the periodic inspection of machines as part of a planned maintenance program – make errors. For instance, these technicians may be unable to predict where the excessive wearing of the mechanical parts is located or when the failure has occurred. Consequently, the maintenance processes will be reactive; so, it will not be started until after the failure has occurred. This could increase the possibility of injuries on board a marine vessel. Correspondingly, this impediment will reflect on the business side of marine companies because that means the shutdown of the mechanical systems of ships until every suspected piece of equipment is maintained, which further amounts to increasing the maintenance cycle of these vessels. Here, the application of DT as a predictive maintenance tool would play a significant role.

How is this predictive maintenance and monitoring process achieved? To monitor the performance of critical components on a marine vessel or offshore platform, sensors are installed. The data from the sensors are then acquired using specialized data acquisition procedures. This process is epitomized in Figure 1. These acquired data are then transported to the appropriate on-land sites. Figure 2 depicts a synopsis of this intricate data transfer mechanism. Referring to the illustration, at location D, the data acquisition system acquires information from the sensors installed on the object(s). These data are then transmitted to the appropriate satellite, designated S. From there,

1.3 Future Trends

Several trends are expected to shape the future of DTs in the maritime industry [Fuller et al., 2020]:

• Technological Advancements. The evolution of IoT, AI, and edge computing will enhance DT capabilities, enabling comprehensive data collection, real-time analysis, and informed decision-making.

the data are then routed through an onshore radar antenna (R) to a cloud storage, designated CS. Data stored there can then be accessed by the on-land information processing centre (IPC). At the IPC, a digital model, also called a DT, runs simulations or analytical models to assess the state of the object. Here, potential improvements are generated and relayed back to the object on the vessel.

• International Collaboration. Collaborative initiatives among countries and stakeholders will be critical for successful DT integration in maritime operations. Engineers, data scientists, experts, policy-makers, and governments must collaborate across borders to develop DT solutions for maritime challenges.

• Research Gap. A significant research gap exists in the Digital Twin Predictive Maintenance System (DTPMS) in the maritime industry. As DTs are adopted in maritime contexts, there will be a need for predictive maintenance strategies tailored to the complexities of maritime equipment and changing environmental conditions.

Figure 1: An overview of the data acquisition process.

Figure 2: An overview of the data transfer mechanism.

1.4 An Overview of this Paper

This paper is a bibliometric review that aims at answering questions about the contemporary trends in DT’s research and to provide an in-depth overview of the current state of the academia, the latest trends in DTs, and the proposed future directions concerning DT’s adaptation in the marine industry.

2. LITERATURE REVIEW

2.1 Historical Background

The concept of DTs finds its origins in the early instances of complex simulations and modelling of real-world systems. One of the earliest recorded instances of utilizing complex simulations to address real-world challenges can be traced back to the Apollo 13 mission in 1970. During this mission, NASA faced a critical situation when an unexpected explosion damaged the spacecraft, and the spacecraft deviated from its intended course. Swift action was needed to ensure the safe return of the astronauts to Earth.

NASA employed high-fidelity simulators and modified them in real time to mirror the damaged conditions of the spacecraft. This practical application, though not explicitly called a Digital Twin, exhibited the fundamental characteristics of a DT; that is, a virtual model influenced by real-world data to address unforeseen scenarios [Boy, 2020].

2.2 Evolution of Digital Twins

The evolution of the DTs concept has been marked by key milestones that span multiple industries and applications. This timeline highlights examples of pivotal moments in its development [Singh et al., 2021]:

• 1991 Imagining Mirror Worlds. David Gelernter introduced the concept of “Mirror Worlds.” In this vision, software models replicate reality by using information from the physical world as input. Although not explicitly called DTs, this idea set the stage for virtual representations of physical entities [Gelernter, 1993].

• 2002 Emergence of the DTs. The concept of DTs emerged in relation to product lifecycle management (PLM) at the University of Michigan. Michael Grieves introduced the initial model known as the “Mirrored Spaces Model.” This model emphasizes real space, virtual space, and a linking mechanism for data exchange between the two [Grieves and Vickers, 2016].

• 2003 Agent-Based Architecture for PLM Kary Främling and colleagues proposed an agent-based architecture featuring virtual counterparts or agents associated with product items. This approach addressed the inefficiency of transferring production information via paper in PLM, which laid the groundwork for digital representations of physical entities [Främling et al., 2003].

• 2006 Evolution to Information Mirroring Model. Grieves’ conceptual model evolved from the “Mirrored Spaces Model” to the “Information Mirroring Model.” This evolution places greater emphasis on bidirectional linking mechanisms and introduces the concept of multiple virtual spaces for a single real space, thus setting the stage for more complex DT structures [Schleich et al., 2017].

• 2010 DT Defined. NASA introduced the term “Digital Twin” in its technological roadmap. DT is described as an integrated

multi-physics, multi-scale simulation of a vehicle or system. It leverages physical models, sensor data, fleet history, and more to mirror its real-world counterpart [Shafto et al., 2010].

• 2010’s Application Diversification. Building on NASA’s lead, the US Air Force adopted DTs technology for aircraft design, maintenance, and predictive capabilities. This expansion demonstrates the versatility of DTs beyond aerospace [Gockel et al., 2012]. The term “Digital Twins” gains popularity and broadens its scope, reaching industries beyond aerospace. Manufacturing, energy, healthcare, and more explore the potential of DTs for enhancing operations and decision-making.

• Present and Beyond. The concept of DTs continues to evolve and expand, encompassing a wide array of applications. This includes machines, products, processes, and even complex biological systems. Interest and research in DTs technology persist, with ongoing efforts to refine and adapt its applications for an increasingly digital world [Singh et al., 2021].

2.3 Components of Digital Twins

DTs consist of interconnected components that drive dynamic and data-driven capabilities, resulting in virtual replicas for real-time insights, predictive analytics, and optimization [Grieves, 2005].

Digital Representation and Data

Integration: DTs rely on accurate digital models mirroring physical entities [Tuegel, 2012; Gockel et al., 2012], achieved through data integration from sensors, IoT

devices, and historical records. This sustains synchronization, empowering DTs to adapt and make informed decisions.

At various levels of scale, DTs are generally structured hierarchically [Tao et al., 2019]:

• Unit level. Smallest manufacturing unit, e.g., equipment or material. Unit-level DT mirrors physical twin’s aspects.

• System level. Combines unit-level DTs in a production system. Enhances data flow and resource allocation. Complex items like aircraft can be system-level DTs.

• System of Systems level. Links systemlevel DTs. Facilitates collaboration across enterprise areas, integrating product life cycle phases.

Real-Time Monitoring and Simulation:

DTs excel in real-time monitoring, allowing stakeholders to analyze physical entity behaviour virtually. Continuous data flow from sensors offer immediate feedback. Real-time simulation permits risk-free testing and prediction, aiding issue identification and prompt decisionmaking. (Refer to Figures 1 and 2.)

In most Digital Twins, three interaction levels can exist [Kritzinger et al., 2018]:

• Digital Model. Manual data exchange –changes not mirrored between physical and digital objects.

• Digital Shadow. Automatic data flow from physical to digital format – changes reflected one-way.

• Digital Twin. Automatic bidirectional data exchange – changes in one object affect the other.

IoT and Sensor Integration: IoT and sensors are fundamental in collecting real-world data for DTs. Sensors provide crucial information and facilitate updating of the digital model in real time. This ensures accuracy, enhancing real-time simulations, analytics, and efficiency optimization [Tao et al., 2017].

Analytics and Machine Learning:

Analytics and machine learning elevate DTs from descriptive models to predictive and prescriptive tools. Analytics process data, unveiling patterns. Machine learning enables predictive and prescriptive analytics, predicting behaviour and suggesting actions. Continuous data streams improve accuracy, enabling predictive maintenance and operational optimization [Tao et al., 2017].

2.4 Applications of Digital Twins in Marine Industries

DTs have revolutionized industries by offering a wide array of transformative applications. From manufacturing to healthcare, DTs have reshaped how processes are managed, monitored, and optimized. Within the maritime sector, DTs bring their own set of innovative possibilities [Smogeli, 2017]. Examples of these include:

• Predictive Maintenance Strategies. DTs revolutionize maintenance by employing predictive analytics through continuous sensor-based monitoring. Potential failures are predicted, enabling initiative-taking maintenance scheduling, asset longevity, and minimized operational disruptions [Ibrion et al., 2019].

• Vessel Performance Optimization. DTs optimize vessel performance by simulating

real-world conditions. Virtual replicas allow for continuous monitoring and analysis, enhancing engine efficiency, propulsion, fuel usage, and navigation strategies. Marine operators can identify efficient operational configurations, energy-saving tactics, and route optimization [Smogeli, 2017].

• Smart Fleet Management. DTs centralize data for fleet-wide insights. Real-time integration of vessel positions, fuel consumption, weather, and scheduling empower data-driven decisions, elevating fleet performance, route planning, and fuel management [Rudrusamy et al., 2023].

• Offshore Asset Management. DTs replicate offshore assets, enabling realtime monitoring of structural integrity, equipment health, and safety. Data from sensors informs asset management through early issue detection and optimized asset utilization [Golestani et al., 2023].

• Maritime Training and Simulation. DTs offer a risk-free training platform for maritime professionals. Navigation, manoeuvring, and emergency scenarios can be practiced, enhancing skills and decision-making capabilities [Smogeli, 2017].

• Autonomous Vessel Development. Autonomous vessels benefit from DTs for testing navigation algorithms and decision-making processes. Real-world scenario simulations expedite autonomous integration into maritime transportation [Lv et al., 2023A; Mauro and Kana, 2023].

• Maritime Incident Analysis. DTs aid postincident analysis by recreating events in a virtual environment. Insights into incident factors facilitate preventive measures and enhance safety practices [Lv et al., 2023B].

2.5 Challenges and Barriers

The integration of DTs into industries is not without its challenges and barriers. In the maritime sector, these obstacles manifest in many ways [Singh et al., 2018]:

• Novelty. The adoption of DTs faces hurdles due to uncertain value perceptions, limited successful cases, and evolving technologies like 3D simulations, IoT, and AI. Infrastructure and software enhancements are necessary [Bulygina, 2017].

• Time and Cost. Developing DTs is resource-intensive, requiring time, expert personnel, and substantial funds. High expenses are incurred for detailed modelling, simulations, computational power, sensor integration, and IT infrastructure [Gabor et al., 2016]. (But this is also a benefit compared to using the in-situ equipment being simulated.)

• Lack of Standards. The lack of industrywide frameworks and standards impedes the adoption of DT. Standardizing interfaces, data flows, models, and data itself are crucial. The absence of clear regulations for innovative technologies adds to the challenge [Wagner et al., 2019]. It should be noted, however, that, recently, ISO issued a series of standards (ISO 23247) that deals with a Digital Twin framework for manufacturing. It is a generic framework [Shao et al., 2023].

• Data Challenges. Challenges involve data privacy, ownership, transparency, and sharing due to company policies and societal views. Data silos, interoperability issues, and cybersecurity risks affect DT performance and data handling [Singh et al., 2018].

• Lifecycle Mismatch. DTs face compatibility issues with long-lifecycle products, risking software obsolescence and vendor lockin. Balancing product and technology life cycles is essential [Goasduff, 2018].

3. RESEARCH QUESTIONS

This research embarked on an expedition to address the following questions:

• How has the research output and citation impact on the DTPMS in the maritime industry evolved over time?

• Which countries, authors, and institutions are leading in the research and publication output on DTPMSs in the maritime industry?

• What are the countries’ collaboration patterns among researchers working on DTPMSs in the maritime industry?

• What is the pattern of authorship in the field of study?

• What are the most influential sources and research areas on the maritime industry’s DTPMS?

• What are the emerging research trends and future directions?

• What are the leading author keywords on the maritime industry’s DTPMS?

• What are the most cited publications in the field?

4. RESEARCH METHODS AND TOOLS

This in-depth mapping study included the following methods and tools.

4.1 Methodology

Bibliometrics is a quantitative study method

used to analyze and rate scientific literature. It uses statistical and mathematical methods to measure and evaluate various parts of scholarly writings, such as the number of publications, the number of citations, authorship patterns, the impact of a journal, and networks of collaboration [Ball, 2018]. In the subject of library and information science, bibliometric methods are used extensively. Scientometrics is a sub-discipline of bibliometrics that deals with examining scientific publications.

Figure 3: The PRISMA flow diagram used to identify, screen, and include research publications on Digital Twin Predictive Maintenance Systems in the maritime industry between 2016 and 2023 for the bibliometric analysis.

This study employed the bibliometric method to assess the research productivity on DTPMSs in the maritime industry research. This technique, also known as science mapping, represents the relationship between disciplines, domains, specialties, documents, and authors. The present study focused on bibliometric indices such as yearly growth of literature, productive country and organizations, prolific authors, significant sources, author keywords, collaborative country, most cited research papers, exploring research themes, etc. A synopsis of the process is illustrated with a PRISMA flow diagram in Figure 3.

4.2 Search Query

The following search query was framed

in the advanced search box of the Web of Science database to retrieve the bibliographic data [Clarivate Analytics, 2023]. TS = (“Digital Twin”) AND TS = (“Maritime 4.0” OR “Predictive Maintenance” OR “Industry 4.0” OR “InLive Engine Performance” OR “Engine Monitoring System” OR “Engine Fuel Optimization” OR “Real-Time Operational Sensory Data” OR “Condition Based Maintenance” OR “Predictive Maintenance” OR “Wear Fault Diagnosis”).

4.3 Date of Data Extraction

The search was started on May 14, 2023, at the King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia. Using the mentioned search query, 1,098 research papers were found.

4.4 Inclusion and Exclusion Criteria

Exclusion and inclusion criteria were applied in the initial search results of 1,098 documents. The analysis eliminated a single research paper from the category of type of publications (Letter). The study only included English papers and omitted 20 non-English language publications (Chinese, Korean, Japanese, German, French, Italian, Turkish, etc.). Furthermore, the authors removed three duplicate publications from the analysis. Finally, 1,074 research papers were chosen for final analysis.

4.5 Data Analysis

All the selected 1,074 research papers have been downloaded in different file formats and analyzed with bibliometric analysis tools such as VOSviewer [van Eck and Waltman, 2010], Biblioshiny [Aria and Cuccurullo, 2017],

HistCite, BibExcel [Persson, 2016], and Microsoft Excel.

5. RESULTS AND DISCUSSION

Table 1 presents the quantitative information about the various aspects of the DTPMS in the maritime industry research publications between 2016 and 2023. The dataset includes 1,074 papers published in 508 different sources such as books, reviews, book chapters, journals, and others. These publications received a total of 21,006 citations, indicating their impact and influence within the scholarly community. Each publication, on average, received 19.56 citations. These indicate the impact and visibility of the publications within the scholarly community.

The analysis shows an annual growth rate of 62.04% and average publication age of 2.18 years. The analyzed 1,074 publications comprise 39,561 references. This metric indicates the number of external sources cited within the analyzed publications. The analysis identified 997 keywords plus and 2,843 author keywords. These keywords are specific terms or phrases chosen by the authors to represent their research themes or concepts. A total of 3,432 authors participated in producing 1,074 publications. This includes first authors, co-author, and corresponding authors. Among the total number of authors, there are 38 authors who are responsible for single-authored publications. The analysis also reveals that there are 40 single-authored publications in total. On average, each publication had 4.03 co-authors, and 27.09% of the publications involved international collaboration.

Table 1: Main information about the data – time span 2016-2023.

Table 2: Yearly growth of publications and citation trends on Digital Twin Predictive Maintenance Systems in the maritime industry between 2016 and 2023. TP = total publications. TC = total citations.

5.1 Yearly Growth of Publications and Citations Trends

Table 2 and Figure 4 show the yearly growth of publications and citation trends on DTPMS in the maritime industry between 2016 and 2023. The analysis reveals that 2016 published the first research papers in the field with three publications and 433 citations. From 2019 to 2022, the number

of publications was more than 100. There was a noticeable increase in output in 2017 from 2016 to 2017 (i.e., from three to 17 articles) and in citations (433 to 2,430). The total citations/total publications (TC/ TP) ratio remained high, suggesting a high mean citation count for each publication. The h-index rose from 3 to 13, indicating the presence of highly cited works. The number

of articles climbed to 43 in 2018. Conversely, the number of citations declined to 2,289. From 2019 to 2021, publications increased steadily, while citations fell. The analysis shows that the year 2022 had the most research publications with 300 publications and 1,292 citations, 2021 with 299 publications and 4,447 citations, and 2020 with 196 publications and 4,609 citations. Articles and citations dropped dramatically in 2023 to 83 and 47, respectively. As a result, the TC/TP ratio and h-index fell to their lowest levels in the whole era. The year 2023 is not a complete year, as data were taken in May 2023 – expecting more publications

and citations by the end of the year. The analysis recorded that 2019 received the highest number of total citations with 5,459, followed by 2020 with 4,609, and 2021 with 4,447 citations. The year 2020 has the highest h-index with 38, meaning there were at least 38 publications with more than 38 or more citations, followed by 2019 with 36 h-index and 2021 with h-index of 35.

5.2 Types of Publications

Table 3 illustrates the types of publications on DTPMS in the maritime industry between 2016 and 2023. The following types of papers

Figure 4: Yearly growth of publications and citation trends on Digital Twin Predictive Maintenance Systems in the maritime Industry between 2016 and 2023.

Table 3: Types of publications on Digital Twin Predictive Maintenance Systems in the maritime industry between 2016 and 2023. TP = total publications. TC = total citations.

Table 4: Top 10 most productive sources in Digital Twin Predictive Maintenance Systems in maritime industry publications between 2016 and 2023. TP = total publications. TC = total citations. JIF = journal index factor.

are considered in the current analysis: Article, Proceedings Paper, Review, Editorial Material, and Book Chapter. The analysis reveals that many of the researchers in the field preferred to publish their work as “article” with 519 publications and 13,123 citations, followed by “proceedings papers” with 436 publications and 4,275 citations, “review” with 111 publications and 3,471 citations, “editorial material” with six publications and 101 citations, and “book chapter” with two publications and 36 citations. The analysis shows that most of the citations were received by articles with 13,123, followed by proceedings paper with 4,275, and review with 3,471. Regarding the highest average citation per publication (TC/TP), the review received the highest TC/TP with 31.27, followed by the article with 25.29, and the book chapter with 18.

5.3 Productive Sources

Table 4 depicts the top 10 most productive sources of DTPMSs in the maritime industry between 2016 and 2023. The table shows

that five sources contributed more than 20 publications each. The journal Applied Sciences-Basel (journal impact factor (JIF) = 2.83) from Switzerland, published by MDPI, contributed the highest number of research papers with 42 publications and 531 citations; followed by IFAC Papers online (JIF = 0.2) from the UK, published by Elsevier, with 32 publications and 590 citations; IEEE Access (JIF = 3.47) from USA, published by IEEE, and Sensors (JIF = 3.84) from Switzerland, published by MDPI, with 31 publications each and 2,714 and 327 citations respectively; and Journal of Manufacturing Systems (JIF = 9.49) from the UK, published by Elsevier, with 24 publications and 862 citations. Springer-published Journal of Intelligent Manufacturing (JIF = 7.13) from the Netherlands was the least productive source among the top 10 with 11 publications and 579 citations.

Regarding the most citations, IEEE Access received the highest number of citations with

2,714, followed by Journal of Manufacturing Systems with 862 citations, and IFAC Papers online with 590 citations. Among the top 10 most productive sources are four each from the UK and Switzerland, and one each from the USA and Netherlands. The analysis also reveals that among the top 10 sources, MDPI published four; two each were from Springer and Elsevier; and one each was from IEEE and Taylor & Francis. A similar top three productive sources in the research of Digital Twin and health management were reported by De Oliveira Ribeiro et al. [2022].

5.4 Productive Affiliations

Table 5 depicts the top 10 most productive affiliations on DTPMS in maritime industry publications between 2016 and 2023. Among the top 10 most productive affiliations, researchers from Research Libraries of the UK were the most productive affiliation with 41 publications and 1,001 citations, followed by Polytechnic University of Milan, Italy, with 24 publications and 1,146 citations, RWTH Aachen University, Germany, with 22 publications and 489 citations, Centre

National De La Recherche Scientifique, France, with 20 publications and 801 citations, and N8 Research Partnership, UK, with 16 publications and 217 citations.

Among the top 10 productive affiliations, the University of Auckland, New Zealand, was the least prolific, with 12 publications and 912 citations. The analysis reveals that Beihang University, China, was the most cited affiliation, with 2,400 for 13 publications, followed by Polytechnic University of Milan, Italy, with 1,146 citations for 24 publications. Fraunhofer Gesellschaft was the least cited affiliation and the least average citation per publication, with 125 citations and 8.33 TC/ TP, respectively. Beihang University also has the highest average citation per publication (TC/TP) with 184.62. The analysis reveals that among the productive affiliation, three are from the UK, two are from Germany, and one each is from Italy, France, Norway, China, and New Zealand.

5.5 Productive Country

Table 6 illustrates the top 10 most productive

Table 5: Top 10 most productive affiliations on Digital Twin Predictive Maintenance Systems in maritime industry publications between 2016 and 2023. TP = total publications. TC = total citations.

countries on DTPMS systems in maritime industry publications between 2016 and 2023. The analysis shows that among the top 10 most prolific countries, 60% (n = 6) of countries are in Europe, followed by two countries from Asia, and one each from North America and South America. Among the top 10 leading countries, three contributed over 100 publications each. Germany was identified as the most productive country on DTPMS in maritime industry publications, with 173

publications and 3,294 citations, followed by China with 121 publications and 5,791 citations, Italy with 118 publications and 2,780 citations, the USA with 94 publications and 1,426 citations, and England with 71 publications and 1,600 citations. India was the least productive country in the list, with 33 publications and 291 citations.

Regarding the most cited countries, China also received the highest citations with 5,791,

Table 6: Top 10 most productive countries on Digital Twin Predictive Maintenance Systems in the maritime industry between 2016 and 2023. TP = total publications. TC = total citations.

Table 7: Top 10 most productive research areas on Digital Twin Predictive Maintenance Systems in the maritime industry between 2016 and 2023. TP = total publications. TC = total citations.

followed by Germany with 3,294 citations and Italy with 2,780 citations. India was the least cited country in the list, with 291 citations and 8.82 TC/TP. University-affiliated institutions in China received the highest average citation per publication, with 47.86, followed by universities in Italy, with 23.56.

5.6 Productive Research Areas

Table 7 represents the top 10 most productive research areas on DTPMS in maritime industry publications between 2016 and 2023. The analysis shows that Engineering ranks first, with 725 articles and 16,913 citations, for a TC/TP ratio of 23.33. Computer Science ranks second with 398 articles, 9,289 citations, and a TC/TP ratio of 23.34. With 173 publications, 3,855 citations, and a TC/TP ratio of 22.28, Automation Control Systems takes third place. Despite having fewer publications (94), Chemistry receives 923 citations, resulting in a lower TC/TP ratio of 9.82. Operations Research Management Science ranks fifth with 93 articles, 2,420 citations, and a TC/TP ratio of 26.02. Telecommunications, Materials Science, Physics, Science Technology Other Topics, and Instruments Instrumentation contributed with

91, 88, 65, 59, and 53 publications, respectively.

Regarding citation trends on productive research areas on DTPMS in maritime industry publications, Engineering, Computer Science, and Automation Control Systems were the topcited research areas with 16,913, 9,289, and 3,855 citations, respectively. The bibliometric study sheds light on the comparative performance of various research disciplines based on publication and citation metrics, demonstrating these subjects’ relative impact and influence within the scholarly landscape.

5.7 Prolific Authors

Table 8 optimizes the results from the evaluated of the top 10 prolific authors on DTPMS in maritime industry publications between 2016 and 2023. This bibliometric table provides a summary of these top 10 productive authors’ publications and citation impacts, highlighting their research output and importance within their respective fields of study.

Xu X from the University of Auckland in New Zealand is the top-ranked author, with 10 publications, 944 citations, and an h-index of

Table 8: Top 10 most prolific authors on Digital Twin Predictive Maintenance Systems in maritime industry publications between 2016 and 2023. TP = total publications. TC = total citations.

8. Lu Y, also from the University of Auckland, comes close behind with nine publications, 832 citations, and an h-index of 6. Zhang C from China’s Xi’an Jiaotong University is third, with nine publications, 254 citations, and an h-index of 5. Zhang Y from the Rochester Institute of Technology in the United States comes in fourth place, with eight publications, 372 citations, and an h-index of 7. Fumagalli L from Italy’s Politecnico di Milano takes fifth place with seven publications, 958 citations, and an h-index of 6. Liu Y and Dolgui A have seven publications each, and Leng J, Liu C, and Liu Q have six publications each. Fumagalli L from Politecnico di Milano in Italy received the highest citation with 958, followed by Xu X from the University of Auckland in New Zealand with 944 citations and Liu C from Xi’an Jiaotong University in China with 832 citations.

5.8

Pattern of Authorship

Figure 5 portrays the authorship pattern of the DTPMS in maritime industry publications between 2016 and 2023. This figure illustrates

that authorship patterns ranged from 1 to 16. The analysis reveals 96% (TP = 1,034) of the publications were contributed by more than one author, and only 4% (TP = 40) of the publications were contributed by a single author. This indicates that authors preferred collaborative works in the fields. Furthermore, the figure shows Pattern 3 is the most common authorship pattern, with 325 publications and 5,655 citations. Alade et al. [2022] also reported a similar pattern of authorship in their Journal of Superconductivity and Novel Magnetism publication. Pattern 4 is close behind, with 247 publications and 5,518 citations. Pattern 2 has the third highest frequency, with 133 publications and 3,552 citations.

The authorship patterns, numbered 5 to 17, show a decrease in publications and citations, indicating fewer common authorship patterns in the given context. It is worth noting that Pattern 12 has only one publication and no citations. This bibliometric diagram sheds light on the collaborative nature and productivity within the investigated research field or dataset

Figure 5: Authorship pattern.

by revealing the distribution and frequency of various authorship patterns.

5.9 Analyzing the Word Cloud from Title

Keywords

A title keyword Word Cloud is a visual depiction of the most common or relevant keywords taken from the titles of a group of papers or publications. The Word Cloud indicates the frequency or relevance of specific words used in the titles in this context.

This visualization technique offers a quick and effortless way to identify the major themes or topics in the document titles. As can be seen in Figure 6, the most used terms are “design” (159), “digital twin” (153), and “industry 4.0” (151), emphasizing their importance and ubiquity in the subject. “Framework” (98), “systems” (96), “model” (85), “big data” (81), “internet” (79), and “future” (76) are all noteworthy terms. These terms denote the most important research themes in the field, including design methodologies, DT technology, advancements in Industry 4.0, system modelling and management, Big Data Analytics, internet applications, cyber-physical systems,

optimization techniques, and simulation methodologies.

Based on the Word Cloud, potential research areas for the future include supply chain management advancements, Industry 4.0 challenges, optimization methods, Big Data Analytics, IoT integration, and AI advancements. Research gaps should be prioritized, and these areas should be investigated to progress in these fields.

Figure 6 captures the co-occurrence of author keywords in the field of DTPMS in maritime industry publications, using the VOSviewer software. The frequency and patterns of authorassigned keywords in research articles and publications are called co-occurrence. It entails finding correlations and associations between terms that appear in the same publications. Author keyword co-occurrence can reveal the interconnection of subjects, themes, and concepts in a research domain. This study might reveal common research areas, emerging trends, interdisciplinary connections, or clusters of associated concerns, offering a holistic view of the research environment and guiding future research. A minimum

Figure 6: Word Cloud of title keywords from Biblioshiny software 5.9. Mapping and visualizing the co-occurrence of author keywords.

of 10 author keywords were considered for the present analysis. From the 2,675 author keywords, 62 keywords met the criteria. The selected 62 keywords were grouped in five clusters/themes and represented in assorted colours, as illustrated in the figure:

• Cluster 1 (Red). This cluster comprises 20 author keywords. The main research theme of this cluster is Smart Manufacturing and its Integration with Advanced Technologies. The leading topics in this cluster include smart manufacturing, IoT, AI, cyberphysical systems, blockchain, big data, sustainability, deep learning, smart factory, cloud computing, digital transformation, intelligent manufacturing, additive manufacturing, data analytics, literature review, robotics, condition monitoring, and cyber-physical production systems.

• Cluster 2 (Green). This cluster includes 16 author keywords. The main theme of this cluster is Digital Transformation. The leading research topics of this cluster comprise Digital Twin, Industry 4.0, asset administration shell, IoT, digitalization, automation, interoperability, cyberphysical production system, cyber-physical systems (CPS), virtual commissioning, OPC UA, ontology, and virtualization.

• Cluster 3 (Blue) This cluster consists of 12 author keywords, and Industrial Processes, Digitalization, and Advanced Technologies are the major research themes in this cluster. This cluster’s leading study topics are predictive maintenance, Digital Twins, machine learning, DT, monitoring, anomaly detection, cloud manufacturing, edge computing, sensors, and industrial IoT.

• Cluster 4 (Yellow). This cluster comprises 10 author keywords. The main research theme of this cluster is Advancing Manufacturing Processes. The leading topics include simulation, manufacturing, digital manufacturing, production, optimization, manufacturing systems, review, framework, modelling, and systematic literature review.

• Cluster 5 (Purple). The main themes of this cluster are Digital Technologies and Simulation in Manufacturing and the leading topics in this cluster include Industry 4.0, virtual reality, augmented reality, discrete event simulation, and maintenance.

Figure 7 displays the leading occurred author keywords such as Digital Twin (n = 463), Industry 4.0 (n = 183), Industry 4.0 (n = 170), smart manufacturing (n = 80), internet of things (n = 72), predictive maintenance (n = 70), Digital Twins (n = 65), artificial intelligence (n = 61), simulation (n = 59), machine learning (n = 47), cyber-physical systems, manufacturing, blockchain, asset administration shell, IoT, big data, industrial internet of things, and virtual reality. N.B.: The weight of an item determines the size of its circle, with heavier items having larger circles.

5.10 Bibliographic Coupling of Documents

Bibliographic coupling is a technique employed in bibliometrics and information science to analyze the relationships between scientific documents based on their shared citations. It is a type of co-citation analysis focused on the bibliographic references cited by various documents, instead of the documents themselves. By analyzing the

bibliographic coupling network, researchers can gain insight into various scientific thematic similarities and interrelationships of the documents. It can aid in identifying clusters or groups of related research and disclose patterns of knowledge diffusion or research collaboration within a field. Figure

8 (bibliographic coupling) was generated using the VOSviewer software. Minimum (0) number of citations of a document were considered for analysis. Out of the documents, 1,074 meet the thresholds. The documents with the greatest total link strength will be selected. A total of 1,000

Figure 7: Mapping and visualizing the co-occurrence of author keywords using the VOSviewer software.

Figure 8: Bibliographic Coupling of documents using the VOSviewer software.

Table 9: Significant themes or clusters or both on Digital Twin Predictive Maintenance Systems in maritime industry publications between 2016 and 2023. TP = total publications. TC = total citations.

publications were selected, and all the selected publications were grouped into 12 clusters/themes, which are discussed below and shown in Table 9:

• Cluster 1 – DT in Industry 4.0. This is the largest cluster in terms of total publication (TP = 291) and total citations (TC = 7,893). The theme of this cluster

is DT in Industry 4.0. The most cited publication in this cluster is entitled A review of the roles of Digital Twin in CPS-based production system by Negri et al. [2017] with 536 citations, followed by The future of manufacturing industry: a strategic roadmap toward industry 4.0 by Ghobakhloo [2018] with 501 citations.

• Cluster 2 – Advancements in DT

Technology. This is the second largest cluster in total publications with 142 publications, while the fifth largest in total citations with 1,290. The most cited paper in this cluster is Review of Digital Twin applications in manufacturing by Cimino et al. [2019] with 242 citations, followed by Digital Twin: origin to future by Singh et al. [2021] with 91 citations.

• Cluster 3 – DT Technology for Smart Manufacturing and Optimization in Industry. This is the third largest cluster in total publications with 112 publications and second highest cluster in total citations with 4,539. The primary theme of this cluster is Digital Twin Technology for Smart Manufacturing and Optimization in Industry. The most cited publication in this

cluster is Digital twin in industry: stateof-the-art by Tao et al. [2019] with 819 citations, followed by Digital Twin and big data towards smart manufacturing and industry 4.0: 360-degree comparison by Qi and Tao [2018] with 571 citations.

• Cluster 4 – DT Applications for Smart Manufacturing and Predictive Maintenance. This is the fourth largest cluster in total publications with 105 publications, and third largest in terms of total citations with 2,029. This theme deals with DT Applications for Smart Manufacturing and Predictive Maintenance. The most cited paper in this cluster is Digital Twin shop-floor: a new shop-floor paradigm towards smart manufacturing by Tao and Zhang [2017], with 479 citations, followed by A digitaltwin-assisted fault diagnosis using deep transfer learning by Xu et al. [2019] with 145 citations.

• Cluster 5 – Leveraging Blockchain and DT Technologies for Advanced Industrial IoT and Manufacturing. This is the fifth largest cluster with 83 publications and 652 citations.

• Cluster 6 – Exploring DT Applications and Benefits in Construction Industry. This cluster also comprises 83 publications and 818 citations.

• Cluster 7 – Digital Transformation and Blockchain-enabled DTs for Efficient Management in Energy and Construction Sectors. Includes 69 publications and 519 citations.

• Cluster 8 – DT: Technologies, Challenges, and Integration for Industry 4.0 Comprises 54 publications and 1,309 citations.

• Cluster 9 – Encompass the Utilization of Blockchain and Digital Twin Technologies in Industry 4.0. This cluster includes 40 publications and 756 citations.

• Cluster 10 – DT, Cyber-physical Systems, and Smart Manufacturing in Industry 4.0. This cluster comprises 15 publications and 525 citations.

• Cluster 11 – DT in Product Lifecycle Management and Business Innovation, Machine Learning Techniques within Cloud Computing Paradigms. This cluster includes only four publications and 201 citations.

• Cluster 12 – Nonlinear Observer Design for Railway Vehicle Guidance and Traction, and Predictive Maintenance in Railway Systems. This is the smallest cluster with two publications and a single citation.

5.11 Examining Most Cited Papers

Table 10 presents the top 10 most cited research papers on DTPMS in maritime industry publications between 2016 and 2023. Among the top 10 most cited papers, the citation ranged from 348 to 819. The analysis reveals that four research papers received more than 500 citations:

1. Digital Twin in Industry: State-of-the-Art authored by Tao et al. [2019] in IEEE Transactions on Industrial Information received the highest total citations with 819.

2. Digital Twin and Big Data Towards Smart Manufacturing and Industry 4.0: 360 Degree Comparison authored by Qi and Tao [2018] in IEEE Access, which received 571 citations.

3. A Review of the Roles of Digital Twin in CPS-based Production Systems authored

by Negri et al. [2017] in Procedia Manufacturing, which has 536 citations.

4. The Future of Manufacturing Industry: A Strategic Roadmap Toward Industry 4.0 authored by Ghobakhloo [2018] in Journal of Manufacturing Technology Management with 501 citations.

Tao and Zhang [2017] published Digital Twin Shop-floor: A New Shop-Floor Paradigm Towards Smart Manufacturing in IEEE Access with 479 citations. The

article Digital Twin: Enabling Technologies, Challenges, and Open Research by Fuller et al. [2020], published in IEEE Access, was the least cited paper in the list, which received 348 citations.

N.B.: These publications have received much attention and citations in the scientific community, suggesting their importance and influence in the fields of Digital Twins, smart manufacturing, Industry 4.0, and other related areas.

Table 10: Top 10 most cited research papers on Digital Twin Predictive Maintenance Systems in maritime industry publications between 2016 and 2023. TC = total citations.

5.12 Country Collaboration

In the bibliometric study, the country collaboration map (i.e., Figure 9) depicts collaboration trends among different countries, based on the number of copublications (TP) between any two countries. The analysis shows that Germany and Spain have collaborated on 11 co-publications, suggesting that the two countries have a good research collaboration, followed by China and the United Kingdom which have collaborated on 10 co-publications, indicating a substantial research collaboration. Italy and the United Kingdom have worked together on nine co-publications, showing a fruitful scientific partnership; China and the United States have collaborated on eight copublications, indicating a significant research partnership between these two countries; and Finland and Sweden have worked on eight co-publications, reflecting the two countries’ strong research connection. Italy and Spain have the least collaboration in the top 10 list, with six co-publications.

Regarding the number of co-publications across countries, the rankings show the most prominent cooperation. It sheds light on international research collaborations and networks in the topic studied by the bibliometric study.

6. CONCLUDING REMARKS

This study employed a bibliometric performance analysis of the Digital Twin Predictive Maintenance System research in maritime industry publications between 2016 and 2023. Based on the 1,074 publications from the Web of Science, this research explored the yearly growth of publications, productive authors, affiliations, countries, most relevant journal, most cited papers, and evaluated author keywords and leading research themes in the field of DTPMS in maritime industry publications.

The finding shows that the DTPMS (in maritime industry publications) has grown

Figure 9: Country collaboration map generated from Biblioshiny software.

exponentially in terms of productivity and citations since 2016. The year 2022 contributed the highest number of research papers with 300 publications, while 2019 received most of the citations with 5,459. In terms of the most relevant sources, Applied Sciences-Basel is the most productive with 531 publications in the field. At the same time, IEEE Access is the most impactful source on DTPMS in maritime industry publications.

Based upon the reported results and evaluations, three authors, Xu X, Lu Y, and Zhang C, were the most prolific authors, while Fumagalli L is the most impactful author in the field. Germany (TP = 173) is identified as the leader in producing the highest research in the field, while China is the most impactful country in total citations with 5,791. Research Libraries of the UK (TP = 41) is the most prolific affiliation, while Beihang University, China, is the most cited affiliation with a TC of 2,400.

Engineering contributed the most publications and citations in terms of research area. Threeauthorship is the most common authorship pattern, with 325 publications and 5,655 citations. The analysis shows that Germany and Spain have collaborated on 11 copublications, suggesting that the two countries have a good research collaboration.

The analysis also explored 12 research themes on DTPMS in maritime industry publications, namely:

1. Cluster 1: DT in Industry 4.0

2. Cluster 2: Advancements in Digital Twin Technology

3. Cluster 3: DT Technology for Smart Manufacturing and Optimization in Industry

4. Cluster 4: DT Applications for Smart Manufacturing and Predictive Maintenance

5. Cluster 5: Leveraging Blockchain and DT Technologies for Advanced Industrial IoT and Manufacturing

6. Cluster 6: Exploring DT Applications and Benefits in Construction Industry

7. Cluster 7: Digital Transformation and Blockchain-enabled DTs for Efficient Management in Energy and Construction Sectors

8. Cluster 8: DT Technologies, Challenges, and Integration for Industry 4.0

9. Cluster 9: Encompass the Utilization of Blockchain and DT Technologies in Industry 4.0

10. Cluster 10: DTs, Cyber-physical Systems, and Smart Manufacturing in Industry 4.0

11. Cluster 11: DT in Product Lifecycle Management and Business Innovation, Machine Learning Techniques within Cloud Computing Paradigms

12. Cluster 12: Nonlinear Observer Design for Railway Vehicle Guidance and Traction, and Predictive Maintenance in Railway Systems

Furthermore, the concept of DTs, with its origins dating back to the Apollo 13 mission in 1970, has resurged in recent years due to technological advancements like 3D simulations and IoT. Its historical importance and ability to bridge the physical and digital realms make it a crucial tool in addressing contemporary challenges across industries, including the maritime sector. This study is an early attempt to investigate the conceptual framework, driving forces, and theoretical

underpinnings of the DTPMS in the maritime industry, which offers current researchers and decision-makers comprehensive and up-to-date knowledge. Researchers can use the findings of this study in developing prospective future research directions for Digital Twin Predictive Maintenance Systems in the maritime industry advancement.

7. ACKNOWLEDGMENTS

The authors would like to thank the Marine Institute (MI) of Memorial University, NL, Canada, and the King Fahd University of Petroleum and Minerals, Kingdom of Saudi Arabia, for the use of their facilities. The funding for this research from Mitacs (under a grant through Lab2Market) is gratefully acknowledged. A special word of thanks is also extended to Capt. Fabian Lambert, Capt. Fred Anstey, and Joe Singleton (assistant head, former head of the School of Maritime Studies, and head of the School of Ocean Technology, respectively, at MI) for their continued support and encouragement. Gratitude is also extended to Arthur Anastasiadi (marine engineer and faculty at MI) for his many helpful suggestions at the initial stage of this research project, and to Capt. Philip Bulman (faculty at MI) for his many helpful suggestions pertaining to the formatting of this document.

8. REFERENCES

Alade, I.O.; Rahaman, M.S.; and Qahtan, T. F. [2022]. A comprehensive review of superconductivity research productivity. Journal of Superconductivity and Novel Magnetism, Vol. 35, No. 10, pp. 2621-2637. https:// doi.org/10.1007/s10948-022-06326-1.

Aria, M. and Cuccurullo, C. [2017]. Bibliometrix: an R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11 Vol. 4, pp. 959-975. https://doi.org/10.1016/j.joi.2017.08.007.

Attaran, M.; Attaran, S.; and Celik, B.G. [2023]. The impact of Digital Twins on the evolution of intelligent manufacturing and Industry 4.0. Advances in Computational Intelligence, Vol. 3, No. 3. https://doi. org/10.1007/s43674-023-00058-y.

Ball, R. [2018]. An introduction to bibliometrics. Elsevier. https://linkinghub.elsevier.com/retrieve/pii/ C20160036951.

Bilberg, A. and Malik, A.A. [2019]. Digital Twin driven human-robot collaborative assembly. CIRP Annals, Vol. 68, No. 1, pp. 499-502. https://doi.org/10.1016/j. cirp.2019.04.011.

Boy, G.A. [2020]. Human-systems integration. CRC Press. https://doi.org/10.1201/9780429351686.

Bulygina, I. [2017]. Product owner talks: 20 Digital Twins solution providers. Retrieved from: https:// www.dashdevs.com/blog/product-owner-talks-20digital-twins-service-companies.

Cimino, C.; Negri, E.; and Fumagalli, L. [2019]. Review of Digital Twin applications in manufacturing. Computers in Industry, Vol. 113: 103130. https://www.sciencedirect.com/science/article/pii/ S0166361519304385.

Clarivate Analytics [2023]. Web of Science. Retrieved from: https://clarivate.com/webofsciencegroup/ solutions/web-of-science.

De Oliveira Ribeiro, A.; Costa, K.A.; Costa, K.A.; and De Souza Cruz, B.S. [2022]. A bibliometric analysis of the relationship between Digital Twins and health management: based on the Web of Science (WoS) platform. Research, Society and Development, Vol. 11. No. 12, e152111234270. https://doi.org/10.33448/ rsd-v11i12.34270.

Främling, K.; Holmström, J.; Ala-Risku, T.; and Kärkkäinen, M. [2003]. Product agents for handling information about physical objects. Report of Laboratory of Information Processing Science Series B, TKO-B, Vol. 153, No. 03. https://www.cs.hut.fi/ Publications/Reports/B153.pdf.

Fuller, A.; Fan, Z.; Day, C.; and Barlow, C. [2020]. Digital Twin: enabling technologies, challenges, and open research. IEEE Access, Vol. 8, pp. 1089528971. https://doi.org/10.1109/access.2020.2998358.

Gabor, T.; Belzner, L.; Kiermeier, M.; Beck, M.T.; and Neitz, A. [2016]. A simulation-based architecture for smart cyber-physical systems. IEEE International Conference on Autonomic Computing (ICAC). https://doi.org/10.1109/icac.2016.29.

Gelernter, D. [1991]. Mirror worlds. Oxford University. Ghobakhloo, M. [2018]. The future of manufacturing industry: a strategic roadmap toward Industry 4.0. Journal of Manufacturing Technology Management, Vol. 29, No. 6, pp. 910-936. https://doi.org/10.1108/ jmtm-02-2018-0057.

Goasduff, L. [2018]. Confront key challenges to boost Digital Twin success. Retrieved from: https://www. gartner.com/smarterwithgartner/confront-key-

challenges-to-boost-digital-twin-success.

Gockel, B.; Tudor, A.; Brandyberry, M.; Penmetsa, R.; and Tuegel, E. [2012]. Challenges with structural life forecasting using realistic mission profiles. In: 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference and 20th AIAA/ASME/AHS Adaptive Structures Conference 14th AIAA (p. 1813). http://dx.doi. org/10.2514/6.2012-1813.

Golestani, N.; Arzaghi, E.; Abbassi, R.; Garaniya, V.; and Meng, H. [2023]. A system dynamics model of offshore wind farm degradation: enabling operation and maintenance planning under foreseen asset management impacts. Applied Ocean Research, Vol. 138, 103685. https://doi.org/10.1016/j.apor.2023. 103685.

Grieves, M.W. [2005]. Product lifecycle management: the new paradigm for enterprises. International Journal of Product Development, Vol. 2, No. 1/2, 71. https://doi.org/10.1504/ijpd.2005.006669.

Grieves, M. and Vickers, T. [2016]. Origins of the Digital Twin concept. 10.13140/RG.2.2.26367.61609. https:// www.researchgate.net/publication/307509727_ Origins_of_the_Digital_Twin_Concept/citation/ download.

IBM [n.d.]. How Industry 4.0 technologies are changing manufacturing. Retrieved from: https://www.ibm. com/topics/industry-4-0#+How+Industry+4.0+techno logies+are+changing+manufacturing.

Ibrion, M.; Paltrinieri, N.; and Nejad, A.R. (2019). On risk of Digital Twin implementation in marine industry: learning from aviation industry. Journal of Physics: Conference Series, Vol. 1357, No. 1, 012009. https://doi.org/10.1088/1742-6596/1357/1/012009.

Kritzinger, W.; Karner, M.; Traar, G.; Henjes, J.; and Sihn, W. [2018]. Digital Twin in manufacturing: a categorical literature review and classification. IFAC-Papers Online, Vol. 51, No. 11, pp. 1016-1022. https://doi.org/10.1016/j.ifacol.2018.08.474.

Lv, Z; Lv, H.; and Fridenfalk, M. [2023A]. Digital Twins in the marine industry. Electronics, Vol. 12, No. 9, 2025. https://doi.org/10.3390/electronics12092025. Lv, J.; Li, X.; Sun, Y.; Zheng, Y.; and Bao, J. [2023B]. A bio-inspired LIDA cognitive-based Digital Twin architecture for unmanned maintenance of machine tools. Robotics and Computer-integrated Manufacturing, Vol. 80, 102489. https://doi.org/10.10 16/j.rcim.2022.102489.

Mauro, F. and Kana, A.A. [2023]. Digital Twin for ship life cycle: a critical systematic review. Ocean Engineering, Vol. 269, 113479. https://doi.org/10. 1016/j.oceaneng.2022.113479.

Negri, E.; Fumagalli, L.; and Macchi, M. [2017]. A review of the roles of Digital Twin CPS-based production systems. Procedia Manufacturing, Vol. 11, pp. 939-48. https://www.sciencedirect.com/science/ article/pii/S2351978917304067.

Persson, O. [2016]. BibExcel is a versatile bibliometric toolbox developed. Inflibnet, Vol. 23, No. 4. https:// ir.inflibnet.ac.in/bitstream/1944/2132/1/INFLIBNET %20NEWSLETTER%20Vol.23%20No.%204%20 (October-%20December%202016).pdf.

Qi, Q. and Tao, F. [2018]. Digital Twin and big data towards smart manufacturing and industry 4.0: 360-degree comparison. IEEE Access, Vol. 6, pp. 3585-35. https://doi.org/10.1109access.2018.2793265.

Ren, Z.; Wan, J.; and Pan, D. [2022]. Machine-learningdriven Digital Twin for lifecycle management of complex equipment. IEEE Transactions on Emerging Topics in Computing, Vol. 10, No. 1, pp. 9-22. https:// doi.org/10.1109/tetc.2022.3143346.

Report Linker [2020]. The global Digital Twin market size was valued at USD 3.1 billion in 2020 and is projected to reach USD 48.2 billion by 2026. https://finance.yahoo.com/news/global-digital-twinmarket-size-101800677.html#:~:text=The%20 global%20digital%20twin%20market%20size%20 was%20valued,projected%20to%20reach%20 USD%2048.2%20billion%20by%202026.

Report Linker [2024]. Digital Twin market by application, industry, enterprise and geography – global forecast to 2028. Retrieved from: https://www.reportlinker. com/p05092748/Digital-Twin-Market-by-End-UserAnd-Geography-Forecast-to.html?utm_source=(GNW. Rudrusamy, B.; Teoh, H.C.; Pang, J.Y.; Lee, T.H.; and Chai, S.C. [2023]. IoT-based vehicle monitoring and driver assistance system framework for safety and smart fleet management. International Journal of Integrated Engineering, Vol. 15, No. 1, pp. 391-403. https://doi.org/10.30880/ijie.2023.15.01.035.

Schleich, B.; Anwer, N.; Mathieu, L.; and Wartzack, S. [2017]. Shaping the Digital Twin for design and production engineering. CIRP Annals, Vol. 66 No. 1, pp. 141-144.

Shafto, M.; Conroy, M.; Doyle, R.; Glaessgen, E.; Kemp, C.; LeMoigne, J.; and Wang, L. [2010]. Draft modeling, simulation, information technology & processing roadmap. NASA Technology Area 11. Shao, G.; Frechette, S.; and Srinivasan, V. [2023]. An analysis of the new ISO 23247 series of standards on Digital Twin framework for manufacturing. MSEC Manufacturing Science & Engineering Conference, New Brunswick, NJ, US. Retrieved from: https:// tsapps.nist.gov/publication/get_pdf.cfm?pub_ id=935765.

Singh, S.K.; Shehab, E.; Higgins, N.; Fowler, K.; Tomiyama, T.; and Fowler, C.G. [2018]. Challenges of Digital Twin in high value manufacturing. SAE Technical Paper Series. https://doi.org/10.4271/201801-1928.

Singh, M.; Fuenmayor, E.; Hinchy, E.; Qiao, Y.; Murray, N.; and Devine, D.M. [2021]. Digital Twin: origin to future. Applied System Innovation, Vol. 4, No. 2, 36. https://doi.org/10.3390/asi4020036.

Smogeli, O. [2017]. Digital Twins at work in maritime and energy. DNV-GL Feature, Vol. 1, No. 7. https:// www.scribd.com/document/459983273/DIGITALTWINS-AT-WORK-IN-MARITIME-ANDENERGY.

Tao, F. and Zhang, M. [2017]. Digital Twin shop-floor: a new shop-floor paradigm towards smart manufacturing. IEEE Access, Vol. 5, pp. 20418-20427. https://doi. org/10.1109/access.2017.2756069.

Tao, F.; Cheng, J.; Qi, Q.; Zhang, M.; Zhang, H.; and Sui, F. [2017]. Digital twin-driven product design, manufacturing and service with big data. International Journal of Advanced Manufacturing Technology, Vol. 94, No. 9-12, pp. 3563-3576. https://doi.org/10.1007/ s00170-017-0233-1.

Tao, F.; Zhang, H.; Liu, A.; and Nee, A.Y.C. [2019]. Digital Twin in industry: state-of-the-art. IEEE Transactions on Industrial Informatics, Vol. 15, No. 4, pp. 2405-2415. https://doi.org/10.1109/tii.2018.2873 186.

Tuegel, E. [2012]. The airframe Digital Twin: some challenges to realization. In: 53rd AIAA/ASME/ ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference and 20th AIAA/ASME/ AHS Adaptive Structures Conference 14th AIAA (p. 1812). http://dx.doi.org/10.2514/6.2012-1812.

van Eck, N.J. and Waltman, L. [2010]. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, Vol. 84, No. 2, pp. 523-538. doi: 10.1007/s11192-009-0146-3.

VanDerHorn, E.; Wang, Z.; and Mahadevan, S. [2022]. Towards a Digital Twin approach for vessel-specific damage monitoring and prognosis. Journal of Reliability Engineering and System Safety Systems, Vol. 219, 108222. https://www.sciencedirect.com/ science/article/pii/S0951832021007006.

Wagner, R.; Schleich, B.; Haefner, B.; Kuhnle, A.; Wartzack, S.; and Lanza, G. [2019]. Challenge and potentials of Digital Twins and Industry 4.0 in product design and production for high performance products. Procedia CIRP, Vol. 84, pp. 88-93. https:// doi.org/10.1016/j.procir.2019.04.219.

Xu, J.; Sun, Y.; Liu, X.; and Zheng, Y. [2019]. A Digital-Twin-assisted fault diagnosis using deep transfer learning. IEEE Access, Vol. 7, pp. 1999019999. https://doi.org/10.1109/access.2018.2890566.

DR. BENJAMIN MISIUK

ASSISTANT PROFESSOR

DEPARTMENTS OF GEOGRAPHY AND EARTH SCIENCES

MEMORIAL UNIVERSITY OF NEWFOUNDLAND ST. JOHn'S, NL, CANADA

Dr. Ben Misiuk leads a project to establish a global dataset of seafloor images to support the training of large deep learning models. These models will be used for the automatic annotation, labelling, and classification of underwater images with a goal of creating a compilation of images that represents a global diversity of seafloor habitats.

Training large deep learning models requires vast training datasets. Currently, adequate volumes of data exist on which to train these models, but they are dispersed across many different research groups around the world. Dr. Misiuk and his team aim to compile and curate these seafloor image datasets to produce a comprehensive dataset as a resource to develop artificial intelligence image processing tools that may substantially improve capacity to ingest and manage large volumes of data.

Underwater images can be collected much faster than they can be analyzed. New tools must be developed to assist with these analyses, yet the development itself must be supported by comprehensive datasets. It is critically important to meet this data need to support the monitoring of marine environments and facilitate conservation efforts during this current period of rapid change.

So far, the team has compiled a collection of over 11 million seafloor images. This has been

pared down to a subset of 1.3 million that they believe are representative of the breadth of environments in the dataset. A large selfsupervised deep learning model has been developed and trained on this image compilation. This is a general model that may be used for a range of more specific image processing tasks through a technique called transfer learning. Preliminary results suggest the model may increase the accuracy with which underwater images can be automatically annotated and labelled compared to the current state-of-the-art.

www.mun.ca

https://doi.org/10.20383/103.0614 bmisiuk@mun.ca

EMILIE NOVACZEK

YAN LIANG TAN (JAKE)

PHD CANDIDATE IN OCEANOGRAPHY

DALHOUSIE UNIVERSITY

HALIFAX, NS, CANADA

Yan Liang Tan (Jake) is developing new deep learning tools tailored for more effective benthic habitat mapping. He seeks to reduce the level of technical machine learning skills required by benthic habitat researchers to utilize deep learning to support their research and better equip them to extract richer information from their data. This will enable them to make more nuanced analyses that inform decision-making in policy and industry, and ultimately influence how ocean resources are managed sustainably.

His PhD research centres around the theme “applications of deep learning to benthic habitat mapping for marine conservation.” His current project examines the automated feature learning capabilities of deep neural networks applied to acoustic seabed data such as bathymetry and backscatter for ecological and habitat modelling. The embeddings in his models are extracted and their utility for mapping habitat classes or species distributions are examined and compared against conventional data features used in mapping.

Another aspect of Mr. Tan’s research involves processing high-resolution synthetic aperture sonar (SAS) data using deep neural networks and investigating their value for benthic habitat mapping. As part of a Mitacs Accelerate program, he is interning with Kraken Robotics Systems Inc. and learning from its expertise in SAS data acquisition and analysis.

Early experiments have shown that even simple deep neural networks can significantly

automate the tedious manual feature selection process commonly used in benthic habitat modelling. This will save researchers time and effort from the modelling process and enable them to dedicate more resources towards interpreting and analyzing the modelling results. Data features are currently extracted mostly from only one mode of data, such as either bathymetry or backscatter. Deep neural networks can extract features from combining multiple modes of data.

After a decade working in the IT sector, Mr. Tan is grateful to apply his expertise to interdisciplinary research that bridges the ocean mapping and computer science disciplines, and that has potential to benefit ecological and environmental research.

www.dal.ca/faculty/science/oceanography. html

www.seafloormapping.ca

https://krakenrobotics.com/products/ synthetic-aperture-sonar jake.yltan@dal.ca

ZAHRA JAFARI

PHD CANDIDATE

FACULTY OF ENGINEERING AND APPLIED SCIENCE

C-CORE AND MEMORIAL UNIVERSITY OF NEWFOUNDLAND

ST. JOHn'S, NL, CANADA

Zahra Jafari is driven by a passion for utilizing advanced technologies to address real-world challenges, particularly those related to environmental protection and safety in marine environments.

As a PhD student funded by Equinor and completing an internship at C-CORE, her current research focuses on developing machine learning/deep learning models for the detection and classification of icebergs and ships in maritime environments, with a specific emphasis on mitigating risks to oil and gas structures. Additionally, she is working on implementing tracking algorithms to monitor iceberg movement and prevent potential hazards to offshore installations.

This research will help enhance safety and security in maritime operations, particularly along the East Coast of Canada where monitoring ships and icebergs is challenging due to harsh weather conditions. Specifically, it aims to address areas where oil and gas structures are vulnerable to iceberg collisions to minimize the risk of damage and environmental impact.

By improving detection and classification of icebergs and ships, this research has the

potential to prevent accidents and environmental disasters, thereby safeguarding marine ecosystems and preserving human lives. It also supports sustainable practices by reducing the environmental footprint of maritime activities.

Her interest in this area stemmed from her background in computer engineering and a desire to apply machine learning techniques to solve practical problems. As she delved deeper into the field of remote sensing and oceanography, she became fascinated by the complexity of the challenges and the potential impact of innovative technologies. Ms. Jafari is grateful for the opportunity to contribute to the advancement of knowledge in this important field and is committed to continuing her research efforts to address the multifaceted challenges of maritime safety and environmental conservation.

www.mun.ca https://c-core.ca zjafari@mun.ca

COMFORT EBOIGBE

OCEAN DATA SPECIALIST

CENTRE FOR APPLIED OCEAN TECHNOLOGY FISHERIES AND MARINE INSTITUTE

ST. JOHN'S, NL, CANADA

Comfort Eboigbe is drawn to data. After completing her undergraduate degree in chemical engineering in Nigeria and her master’s degree in environmental systems engineering at Memorial University of Newfoundland, she took additional courses in data management and data processing.

Today, she works as an ocean data specialist at the Centre of Applied Ocean Technology. In this role, Ms. Eboigbe is responsible for developing data products that can be used by scientific, academic, and industrial users as well as the general public.

One of her current projects is to create an open-source artificial intelligence (AI) model for the preliminary processing of underwater videos to identify certain kinds of species by their common/generic names and annotating the videos with time stamp information. Species being identified include crab, sculpin, wolffish, hake, Atlantic cod, and others. She uses data from both the Holyrood Subsea Observatory in Conception Bay and the Marine Conservation Area project – a joint initiative of Fisheries and Oceans Canada and the Centre for Fisheries Ecosystems Research at the Marine Institute. Although at the early stages of this project, already Ms. Eboigbe has seen promising results.

Understanding and identifying species in particular marine areas is critical for researchers. However, species identification

can be challenging, time consuming, and costly. That is where AI can play a valuable role. By using a model to analyze and annotate hours and hours of video data, it reduces the need for humans to sit in front of a computer. Although AI can play an important role in processing large amounts of data quickly and identifying patterns and trends that might be missed by humans, videos still need to be vetted and species identified by a human expert.

For Ms. Eboigbe, she will continue her work on this and other ocean-based data projects, helping to further disseminate knowledge and information needed to protect and monitor our ocean.

www.mi.mun.ca/ctec comfort.eboigbe@mi.mun.ca

DANIELLE PERCY

Finding Green Sea Urchins

M. Israk Ahmed

Memorial University researchers present the first step in a research program on the development of a model for the recognition of green sea urchins in natural habitats.

Who should read this paper?

Researchers, scientists, and professionals working in the domains of robotics, computer vision, and marine benthic habitat mapping and ecology would find this study interesting.

Why is it important?

By offering a computational framework for object detection in underwater settings, this work is a stepping stone for future automated sea urchin detection and harvesting procedures that can be used for aquaculture and marine resources monitoring, management, and conservation applications. Underwater robots, equipped with proper object recognition algorithms, can aid in the development of a large bank of green sea urchin images for this ecologically and economically important marine invertebrate.

This study has the potential to aid in the development of robots capable of harvesting sea urchins in an ecologically responsible manner that can help regulate populations and avoid overfishing. The proposed technology may be used for underwater benthic community surveys, with a focus on green sea urchin populations and their influence on ecological succession. The development of similar approaches creates prime opportunities for multidisciplinary collaborations between computer scientists and marine scientists to address key ocean-related questions of ecological and societal relevance.

About the authors

M. Israk Ahmed is a master’s student in computer science at Memorial University of Newfoundland, Canada. He holds a bachelor of science degree in computer science and engineering from the Daffodil International University in Dhaka, Bangladesh. His research interests lie in the fields of deep learning and computer vision, particularly focusing on leveraging cutting-edge methods to address contemporary challenges.

Dr. Lourdes Peña-Castillo is a professor in the Departments of Computer Science and Biology (jointly appointed) in the Faculty of Science at Memorial University of Newfoundland (MUN). She obtained her PhD from the Otto-von-Guericke University in Germany and did a three-year postdoc in the Banting and Best Department of Medical Research at the University of Toronto. Throughout her academic career, she has developed and/or applied artificial intelligence (mostly machine learning) in various areas such as bioinformatics, games, and augmented virtuality. She leads the Bioinformatics lab at MUN, which is focused on the application of machine learning-based methods to solve biological problems.

Dr. Lourdes Peña-Castillo

Dr. Andrew Vardy

Dr. Patrick Gagnon

Dr. Andrew Vardy is a professor jointly appointed to the Department of Computer Science and the Department of Electrical and Computer Engineering at Memorial University (MUN) in St. John’s, Canada. He completed degrees in electrical engineering (B.Eng., MUN, 1999), evolutionary and adaptive systems (M.Sc., University of Sussex, 2000), and computer science (PhD Carleton University, 2005). His main research area is swarm robotics, but he has also developed new techniques in visual robot navigation. He leads the Bio-Inspired Robotics group, which is focused on developing swarms of robots that can actively organize their environments, for example by sorting objects or cleaning a space.

Dr. Patrick Gagnon is professor of oceanography and marine ecology in the Department of Ocean Sciences of Memorial University of Newfoundland. Research in the Gagnon lab investigates impacts of environmental variability on marine species interactions and the stability of key benthic ecosystems, including kelp forests and rhodolith beds. Major foci include development of multiscale methods to marine habitat mapping, and of environmentally responsible approaches to sustainable harvesting and commercialization of marine resources, including green sea urchin.

IMPROVING DETECTION AND LOCALIZATION OF GREEN SEA URCHIN BY ADDING ATTENTION MECHANISMS IN A CONVOLUTIONAL NETWORK

M. Israk Ahmed1, Lourdes Peña-Castillo1,2, Andrew Vardy1,3, and Patrick Gagnon4

1Department of Computer Science, Memorial University of Newfoundland, St. John’s, NL, Canada

2Department of Biology, Memorial University of Newfoundland, St. John’s, NL, Canada

3Department of Electrical and Computer Engineering, Memorial University of Newfoundland, St. John’s, Canada

4Department of Ocean Sciences, Memorial University of Newfoundland, St. John’s, Canada

ABSTRACT

Green sea urchin, Strongylocentrotus droebachiensis, exerts considerable influence on the structure and function of marine benthic habitats in Arctic and sub-Arctic regions, including highly biodiverse kelp forests. The species’ gonads (roe) also is a highly prized delicacy on Asian markets. Autonomous detection and quantification of the species’ biomass and ecological footprint are desirable to address questions at relevant management scales. Underwater robots, equipped with proper object recognition algorithms, may be used for this purpose. The present study represents the first step in a research program on the development of a model for the recognition of green sea urchin in natural habitats. Several factors affect the detection of target objects in underwater environments (in the present case, green sea urchins), including lighting, background, and size and shape of the objects. We try to overcome some of these challenges by developing a multi-step process that includes colour enhancement and augmentation for accurate recognition of green sea urchins in natural habitats. We also proposed an improved model for better interpretation of target characteristics based on the state-of-the-art YOLOv7 object detector. To investigate the possible advantages on extracting the feature information of target objects, this study utilizes the spatial and channel attention mechanism.

Keywords: Green sea urchin; Object detection; Attention mechanism; YOLOv7; Channel attention; Spatial attention

INTRODUCTION

Strongylocentrotus droebachiensis, known as the green sea urchin, is a widespread Arcticboreal benthic echinoderm. While the green sea urchin is omnivorous, its diet primarily consists of macroalgae, with kelps being the main component [Gagnon et al., 2004]. Green sea urchins (Figure 1) are edible and a lucrative target for the commercial fishing industry because of their nutrient-filled gonadal tissue. Manual labour is extensively required in the monitoring and fishing of such valuable marine products. To support the sea urchin fishery, it is crucial to develop a robust underwater recognition system for green sea urchins to monitor ecological conditions and find a balance between controlling overgrazing due to an excessive number of sea urchins and overfishing them.

Underwater robots are often viewed as a viable alternative for monitoring marine resources, given the challenging

characteristics of undersea environments, which include poor visibility, high pressure, and low temperature [Teng and Zhao, 2020]. An efficient object detection model that will help the robot to recognize the intended marine organisms is one of the key parts of developing such a robotic system. Recent advancements in object detection have been useful [Padilla et al., 2020], but it is still difficult to detect target objects in marine environments at a fast pace and classify them precisely. The varying sizes, orientations, and distribution of green sea urchins in different underwater scenes (i.e., urchin barrens, kelpbarren interfaces, rocky floor, etc.), along with the limited computation power of the underwater system, make the detection task quite challenging.

The average size of a green sea urchin is 50-60 mm [DFO, 2022]. Low-resolution photographs of small objects often lack the visual information needed for effective detection and localization, making it difficult

Figure 1: Green sea urchins (Strongylocentrotus droebachiensis) at a depth of ~5 m in Flat Rock Cove, southeastern Newfoundland (Canada).

P. GAGNON

to recognize the distinctive features that are required for such tasks.

Green sea urchin, Strongylocentrotus droebachiensis, is the only species of regular sea urchin in shallow rocky subtidal habitats in eastern Canada. The species’ roughly spherical test (shell) can reach up to about 8 cm in diameter and is covered in tens of relatively long (a few centimetres) needlelike spines. These morphological characteristics provide the species with a unique appearance, which considerably reduces the probability of confusing it visually with co-occurring marine species or objects in its natural habitats. However, similarity across species and backgrounds is a significant challenge in underwater target recognition since it makes it hard to extract differentiating characteristics from a distance. Figure 2 depicts green sea urchin habitats with environmental complexity and colour shifting. The computer vision community is aware of this difficulty and has developed solutions to address it, including the use of attention mechanisms to train networks to distinguish between overlapping classes [Bayoudh et al., 2021].

This study is a first step towards building an underwater robot that will help in monitoring green sea urchins and their habitats. For that, it is necessary that the robot have the ability

to recognize green sea urchins. Hence, this study proposed a deep learning model based on YOLOv7 (you only look once) [Wang et al., 2023], which is able to identify small targets at a fast pace. Moreover, a colour correction and an image enhancement method were incorporated, which are used in the preprocessing step to minimize visibility and colour shifting issues in underwater imagery.

There is a benthic data repository available with an emphasis on kelp and rhodolith beds to provide a significant amount of new georeferenced data products including green sea urchin in the east and west coasts of Newfoundland [Borealis Data, 2023]. As there is no available dataset of underwater green sea urchins that meets our study requirements, such as distance measurements, we have collected 2,400 images of green sea urchins that include urchins of different sizes, in different habitats, etc. The images were taken from locations off the coast of Newfoundland near the communities of Bay Bulls and Flatrock. The test result shows that the proposed model is able to localize and detect green sea urchins correctly over 83% of the time.

RELATED LITERATURE

There are two primary methods for identifying underwater targets that can directly observe

Figure 2: Sample images from green sea urchin images dataset.

or sense the physical characteristics (i.e., shape, appearance, location) of the objects in real time, namely optical vision and acoustic vision [Ge et al., 2022]. Acoustic sensors such as sonar operate by propagating sound waves. The sound wave-based imaging technique is good for long-range object detection. However, due to the noise and long wavelength of the hydroacoustic signal, it is not optimal for shortrange underwater object detection [Zhao et al., 2022]. On the other hand, optical equipment like cameras depends on light to illuminate and capture images. As a result, optical imaging can suffer from absorption and scattering of light in underwater. Still, images obtained with the visible light band can provide higher resolution and more detailed information about the object in short range than acoustic images [Ge et al., 2022]. Also, with the advancement of the convolutional neural network (CNN), optical data-based detection models are getting more popular as they do not require any extra hardware like sonar, LiDAR, etc. [Li et al., 2020; Chen et al., 2023].

CNNs are undoubtedly the principal deep learning method in computer vision; hence, it has widespread applications for monitoring underwater marine environments [Saleh et al., 2020]. A simple CNN consists of a convolution operation and a pooling layer. To learn more complex insights or to extract high-level information from the input data, it requires more parallel and sequential convolution operations, which points to the term “deep” convolutional neural network or large-scale network such as Residual Network [He et al., 2015], Visual Geometry Group [Simonyan et al., 2014], Cross Stage Partial Network [Wang et al., 2019], etc. However, implementing

large-scale networks onto underwater equipment (e.g., autonomous underwater vehicle, remotely operated underwater vehicle, and underwater robotics and manipulators) might present significant challenges [Zhang et al., 2021] depending on the capabilities of the underwater vehicle used. These challenges are generally associated with the low computer power and energy constraints of some underwater vehicles. As a result, there is a dearth of efficient and lightweight algorithms for finding underwater targets in real time. Hence, it has become a critical field of study to find ways to minimize network complexity and boost detection speed without compromising detection accuracy. The development of deeplearning networks has allowed multi-layer neural networks to completely extract target visual data and learn complex insights, leading to improved identification and detection [Ge et al., 2022]. Some of the most popular CNNbased object detection algorithms are Faster R-CNN [Ren et al., 2017], Mask R-CNN [He et al., 2020], Single Shot Detector (SSD) [Liu et al., 2016], YOLOv3 [Redmon and Farhadi, 2018], YOLOv4 [Bochkovskiy et al., 2020], and YOLOv7 [Wang et al., 2023]. Both Faster R-CNN and Mask R-CNN use a two-stage approach that requires extensive processing time to extract potential regions from an image, making them unsuitable for certain real-time uses.

On the other hand, YOLO and SSD are single-stage target recognition methods since they need no prior generation of candidate regions. As a result, they are able to swiftly and accurately pinpoint the target’s position and classify it, which speeds up the detection process significantly.

SSD was first developed in 2016 by Liu et al. [2017], based on YOLOv3 [Redmon and Farhadi, 2018]. To improve the recognition accuracy of small targets, the revised SSD algorithm for echinus identification described by Hu et al. [2020] improves the system’s ability to recognize small objects. The technique proposed in this work achieved 81% average precision, which is a 10% improvement over the standard SSD algorithm under identical training and testing settings. Nevertheless, it can only identify a single marine organism and the identification speed is slow. So far, this is the only work related to the identification of a sea urchin genus. Using transfer learning, Sun et al. [2018] effectively evaluated low-quality underwater recordings, achieving 99.68% average classification accuracy on a dataset consisting of 23 fish species while maintaining a detection speed of 23 frames per second (FPS). Using the Fish4Knowledge dataset, Pan et al. [2021] investigated the multi-scale challenge of underwater item identification and achieved 92.3% mAP (mean average precision) with a speed of 23 FPS.

In pursuit of real-time underwater object identification with fast pace and accuracy, a number of researchers have utilized the YOLO series as a foundation for future study [Saleh et al., 2020]. An upgraded version of the YOLOv3 method was utilized by Han et al. [2020] to recognize marine species in enhanced photos. This algorithm was subsequently deployed to an underwater robot to enable real-time detection. The system, however, had a problem with missed detections and was unable to identify marine species with blurry edges. Mao et al.

[2021] proposed a technique for identifying marine species in shallow waters by using a modified version of the YOLOv4 network. In order to achieve more precision in the detection process, they modified the tail end of the YOLOv4 network. Both the detection accuracy and the quantity of computation required have been significantly improved.

Deep learning is a potent technique for acquiring feature representations from massive amounts of data. With the introduction of YOLOv7 and YOLOv8, the YOLO series is now more robust and faster than the SSD for detection in real time. Consequently, this work builds on prior work in the field by using YOLOv7 techniques for green sea urchin identification.

MATERIALS AND METHODS

Dataset

All source images used in the present study were acquired with a submersible video camera system (GitUp/Git2P camera within a GroupBinc.com/GPH2-1750M housing) attached to a stainless-steel frame towed by a boat at a distance of <3 m above the seabed. The relatively high weight (~20 kg) of the steel frame and low speed (<3 km hr-1) of the boat simulated the types of motion and stability (e.g., roll, pitch, yaw) that a remotely operated vehicle equipped with a similar video camera system would have experienced. Because of system limitations, we could not determine the exact distance between the camera and the sea urchins. However, we selected only video segments in which the distance appeared less than 1 m based on image clarity and size of sea urchins relative

to a segment of the steel frame of known dimension that was intentionally placed (as a scale bar) in the camera’s field of view. We generated still images from these video segments to create the dataset. The dataset features a wide range of conditions, such as green sea urchins in different positions, different lighting, distorted colours, and blurred images. As natural light was used to take the pictures, there is a great deal of contrast between different areas. The dataset contains a total of 2,400 images where green sea urchins are seen in a variety of camouflaged poses, resting on the ground, feeding on kelp, and scaling rocks against a backdrop of kelp rock in a group, hidden by vegetation or rocks and sand, etc. In addition, we have created two subsets from the original dataset based on whether the distance between the camera and the urchin was less than or greater than 1 metre. The first subset of green sea urchin images where the distance of the urchin is less than 1 metre from the camera has 1,205 photos and the second subset contains 1,195 images. Sample images from our dataset that were taken underwater are shown in Figure 2. For annotating the images, we have used an open-source image labelling tool named LabelImg to draw bounding box and manually label the object with corresponding class labels. To maintain the quality and reliability of annotation, annotators responsible for labelling were supervised by an expert in marine biology and ecology.

Data Augmentation and Enhancement

Data augmentation techniques that use multiple random samples to create new samples are called multi-sample data augmentation. Mosaic data augmentation is

one such technique [Neugent et al., 2020]; it involves combining four different photos to form a single new image. It helps to augment data through a diverse range of composited scenes. Figure 3 shows an example of mosaic augmentation. So, we utilized the mosaic augmentation technique along with other common techniques such as scaling, translation, and shear as a data augmentation process. Scaling refers to the technique of making the image data larger or smaller by a scaling factor. To reduce the positional bias in the data, translation is a useful augmentation technique, as it can shift the image up, down, left, or right by a translation factor. Shearing helps the detection network to perceive the object from different angles. It creates a parallelogram by shifting the vertical or horizontal edge of the input along the vertical axis or horizontal axis by a certain amount.

Our datasets include images of green sea urchins captured at varying depths with a moving camera. Due to non-uniform illumination, colour attenuation, blur, scattering effects, and poor contrast, underwater images can lose critical information. At first, we applied Gamma Correction in our dataset to correct the colour shift and improve visibility [Rahman et al., 2016]. Then, we used Contrast Limited Adaptive Histogram Equalization (CLAHE) [Reza, 2004], which is a popular adaptive histogram equalization method on the gamma corrected images. It is used to boost contrast in certain areas of a picture. By imposing limits on the local contrast of the picture, CLAHE prevents the amplification of image noise during the contrast enhancement process. The approach does this by clipping

the histogram at a fixed value, named clip limit. How much histogram noise is reduced and contrast is boosted depends on the clipping level. As a result, CLAHE is useful for highlighting edge characteristics throughout an image and increasing local contrast. Figure 4 illustrates the images with gamma correction and CLAHE

Attention Mechanism

Underwater images often suffer due to light scattering, colour distortion, and less visibility. In addition to that, the size of green sea urchins, their habitats, their orientation, and their nature of camouflage make the detection task more difficult. Due to these characteristics,

state-of-the-art deep models struggle to extract related features, which often leads to poor performance and localization in underwater environments. Attention is a useful solution to make the model focus on relevant areas for better learning of information extraction by assigning different weights to different channels or areas in the data while training [Zhao et al., 2022].

There are different kinds of attention mechanisms that are widely used in computer vision, such as channel attention, spatial attention, hybrid attention, etc. Channel attention structure is designed for channel-wise transformation and to retain the input data of

Figure 3: Mosaic data augmentation.

Figure 4: Colour corrected and enhanced images after applying gamma correction and Contrast Limited Adaptive Histogram Equalization (CLAHE).

each channel [Hu et al., 2018]. Spatial attention uses two convolutional layers to efficiently combine spatial features, allowing the network to focus on important contextual areas in the input [Jaderberg et al., 2015]. In order to make our proposed model better at focusing on relevant information, we have utilized channel and spatial attention in a sequential manner.

SCS Attention Mechanism

Attention mechanism is a technique used to enhance the information extraction in complicated scenarios by giving different weights to different input components in a neural network. Figure 5 illustrates how the Sequential Channel-Spatial (SCS) attention mechanism processes input features via channel and spatial attention methods. This mechanism allows the model to focus on key information while ignoring unnecessary features, which leads to improved interpretability. A summary of the SCS attention mechanism’s entire equation is given by Equation (1) and Equation (2), where Im, Rc, and Rs stand for the input feature from the previous layer, resultant feature map from channel attention, and resultant feature map from spatial attention, respectively.

Figure 5: Sequential Channel-Spatial (SCS) attention module.

In order for the SCS to function, first, the input feature layer I m needs to be passed through the channel attention mechanism, producing R c. The output, shown as I'm in Equation (1), is produced by multiplying this R c with the input I m to amplify specific channels. I'm is then used as the spatial attention mechanism’s input, resulting in R s. To obtain the final output R s is multiplied by I'm. Output I''m, shown in Equation (2), which indicates amplification of spatial position and channel content.

Fusion of Attention Mechanism in YOLO

We employed YOLOv7 architecture for detecting green sea urchins. Figure 6 illustrates the original YOLOv7 architecture. Furthermore, the SCS attention mechanism was fused with the original architecture in order to increase the model’s interpretability. Additionally, we have implemented the SPPFCSPC (Faster Spatial Pyramid Pooling, Cross-Stage Partial Channel) module instead of the original SPPCSPC (Spatial Pyramid Pooling, CrossStage Partial Channel) module in an effort to reduce the computational burden on the model and increase the speed of inference without changing the perceptual field.

Proposed Model Architecture for Green Sea

Urchin Detection

To feed the images into the network, we resized the images to 640 x 640. Figure 7 depicts the whole network structure with proposed modules highlighted in red. The first stage is to pass the input data via the backbone network to extract features. The

backbone is responsible for extracting the low-level features from the input, which are important elements for constructing highlevel feature maps later in the deep layer of the model architecture. The fusion of SCS mechanism happens in the backbone network to help and improve the capacity of the backbone module to concentrate on the relevant location and information of the input, hence boosting the extracted features by the model from the beginning. Then, the head network receives multi-scale feature maps from the backbone network that are sent at different scales, in order to maintain the information in various scales. Head module distributes the learning effort of the

model at different scales across different prediction heads of varying sizes. Finally, the feature data from the backbone and head is combined and turned into an output for detection predictions.

The backbone network extracts visual properties such as colour and texture. There are three key components that make up the YOLOv7 backbone network. Firstly, the CBS (Convolution-BatchNorm-SiLU) submodule is a combination of convolution followed by a batch normalization layer along with a SiLU activation function at the end. Secondly, E-ELAN preserves the basic ELAN structure and enhances the learning

Figure 6: YOLOv7 architecture (adapted from Figure 2, Shi et al. [2023]).

Figure 7: Proposed architecture of YOLOv7 with Sequential Channel-Spatial (SCS) attention mechanism and SPPFCSPC.

potential of the network by enabling various blocks for feature computation without changing the gradient route. We have added the SCS attention mechanism right after each of the E-ELAN architectures in the backbone to improve the extraction of lowlevel features such as edges, corners, colour gradients, etc. in the backbone. Lastly, MP1 is a two-branched structure where in the top branch a down-sampling technique named max pooling (MaxPool) and a CBS is used to decrease the height, length, and channel of the image by half. The other branch has two CBS of different kernels and stride and does the same operation as the top branch. Then, all the features derived from upper and lower branches are fused by a concatenation operation. Merging MaxPool with CBS, the network’s capacity to collect features from small local regions is enhanced by obtaining both maximum and all value information.

The head of YOLOv7 utilizes the architecture of the Feature Pyramid Network [Lin et al., 2017]. Along with multiple CBS blocks and MP2, it has ELAN-H. Multiple E-ELAN based feature layers are fused by ELAN-H, which improves the feature extraction even further.

MP2 and MP1 are structurally identical, only having differences in the output channel numbers. Finally, features extracted by the backbone and head network are combined using a 1x1 convolution to estimate the category and confidence.

Fusing SPPFCSPC

Spatial Pyramid Pooling (SPP) [He et al., 2014] is more effective than using only the maximum pooling [Zafar et al., 2022]. In order to successfully separate important contextual characteristics and extracting multiscale features, the head network of YOLOv7 utilizes Convolutional Spatial Pyramid Pooling (CSP) inside the Spatial Pyramid Pooling (SPPCSPC). Figure 8 shows the structure of SPPCSPC and the SPPFCSPC. We incorporated the SPPFCSPC in the head network to fasten the model’s computation speed. SPPCSPC structure utilizes three distinct pooling layers of kernel size 5, 9, and 13 to form a SPP. In SPPCSPC, inputs of all three pooling are the same. But in SPPFCSPC, it has three pooling with constant kernel size. To minimize the computation effort, the output of the previous pooling is passed through the next pooling while maintaining the module’s perceptual field.

RESULTS AND DISCUSSION

Our main goal is to create a lightweight model in terms of computational cost so that it can fit in an underwater robot. Keeping that in mind, we utilized the most popular singlestage object detection model, YOLOv7. In addition to that, we incorporated the attention mechanism to increase the localization ability of the model.

Figure 8: SPPCSPC architecture in YOLOv7 and adopted SPPFCSPC architecture in proposed model.

The dataset of green sea urchins consists of 2,400 images that include green sea urchins from diverse underwater habitats. The dataset contains images of green sea urchins from varying distances between the camera and the urchin. In order to check the model’s capabilities, we partitioned the dataset into two subsets based on whether the distance between the camera and the urchin was less than or greater than 1 metre. The subset of green sea urchin images where the distance of urchin is less than 1 metre from the camera has 1,205 photos and the second subset contains 1,195 images.

Precision-recall analysis and average precision (AP) are used to evaluate green sea urchin identification. Green sea urchins, denoted by true positive (TP), are those that were successfully identified, whereas false positives, denoted by false positive (FP), are other items or background features that mimic green sea urchins. When an object is not recognized by the system, it is marked as a false negative (FN).

Recall (Equation (3)) is the proportion of green sea urchins successfully recognized, whereas precision (Equation (4)) is the ratio of TP to the total number of detections. Intersection over union (IoU) measures the overlap or similarity between the ground truth and predicted bounding box. It is determined by dividing the intersection area of two bounding

boxes by their union area, while the threshold is a preset value used to distinguish between TP and FP. By integrating the area under the precision-recall curve that is generated by varying the detection threshold, we can determine the AP. By providing a single measure to evaluate a system’s detection capacity, this integration provides an overview of the model’s performance across a range of confidence levels. Equation (5) is the formula for calculating AP. A higher AP score suggests a more effective classifier. (5)

mAP computes the average precision across different classes at a certain IoU threshold.

0.5 is the commonly used IoU threshold to assess object detection accuracy. mAP at 0.5 IoU threshold (mAP@.5) estimates the average precision at 0.5 IoU threshold across all the classes.

Table 1: Relevant parameter configuration.

The associated hyper-parameters for training were the same for the proposed YOLOv7 with attention mechanism and state-of-the-art YOLOv7 model and are shown in Table 1. To balance between computational efficiency and model convergence we used a batch size of 16 with a learning rate of 0.001. The other

parameters were set based on commonly recommended values for training a YOLOv7 network. We trained both the subset and the whole dataset on each model to explore the model’s performance based on the distance. For a better understanding of the learning outcomes of each model and to ensure that it generalizes well to unseen data, all of them were trained through five-fold cross-validation techniques. As we have only one class in our dataset that needs to be classified, we divided each of the selected dataset partition into five roughly equal-sized folds after a random shuffle. For total of five training iterations each fold served as the test set exactly once and after completing all the iterations evaluation results were aggregated. Table 2 shows the

test results of the original YOLOv7 and the proposed model. From the analysis, we can see that the proposed model with the attention technique outperformed the original YOLOv7 in all three cases. When the urchins are closer to the camera, we got the maximum precision of 87.8%. From the numerical results of both models, we can see an improvement in terms of precision, recall, and mAP@.5 in the proposed model’s performance. To visualize and assess the SCS attention compatibility with YOLOv7, we conducted further studies comparing it to state-of-the-art YOLOv7.

During the assessments, qualitative analysis was carried out using Gradient Weighted Class Activation Mapping (Grad-CAM) [Zhou et al., 2016]. Grad-CAM visibly depicts changes

Table 2: Performance of proposed model. In all three cases proposed model recognized green sea urchin more successfully than original model.

Figure 9: Visualizing the model’s interpretability.

in areas of interest impacted by the attention or other mechanisms of the model. So, it can be helpful to determine whether the attention mechanism effectively guides the model to focus on relevant features more accurately than the original YOLOv7 architecture or not. Figure 9 shows the Grad-CAM visualization of both the original and proposed model, where the change from blue to red light indicates the feature’s increasing prominence. The coverage area of the heat map around an urchin produced by the SCS mechanism is wider. This implies that in order to ensure more thorough information extraction of the green sea urchins, the model concentrates on a bigger region at places that include urchins. This eventually helps the model to be better at capturing true positives. Recall achieved by the proposed model (Table 2) supports this interpretation in Figure 9. In the first row (distance more than 1 metre) from Figure 9, when the urchins are at a distance more than 1 metre from the camera, the original model is focusing in regions inside the actual urchins; while the proposed model is more accurate in localizing the urchin’s position in the same scene. Additionally, the red area enlargement indicates an increase in effective target details, highlighting the model’s attention to relevant characteristics of green sea urchins. The findings of the experiment suggest that the model prioritizes the target’s feature information for recognition when the SCS attention mechanism is included.

CONCLUSION AND FUTURE WORK

The use of deep convolutional neural networks has shown promising results in computer vision, with excellent classification accuracy rates being attained. One major challenge

is the calculation of neural network models since embedded and mobile systems have limited capacities in terms of computational power. Secondly, when it comes to underwater target detection, especially small targets such as green sea urchins, the process gets more difficult due to lack of data, quality of data, complex environment, etc. Keeping that in mind, to make a robust recognition model, we have chosen the YOLOv7 model. We fused channel and spatial attention mechanisms in a sequential manner in YOLOv7 to explore the advantages of the attention technique. At the same time, we implemented SPPFCSPC in the head of the proposed model for faster computation. Results show that the performance of the proposed model is better than the original YOLOv7, supported by the qualitative result that shows improvement in the ability to focus on relevant and targetspecific features. However, we plan to conduct further studies to analyze the proposed model’s real-time capability and conduct a comparative analysis with other state-of-the-art models and related dataset. Also, in future studies, there is scope to validate the impact of data enhancement on the model’s performance to detect green sea urchins in underwater habitat.

REFERENCES

Bayoudh, K.; Knani, R.; Hamdaoui, F.; and Mtibaa, A. [2021]. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 38:8, 2939-2970. Bochkovskiy, A.; Wang, C.Y.; and Liao, H.Y.M. [2020]. YOLOv4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934. Borealis Data [2023]. Distribution and abundance of kelp, rhodoliths, and co-occurring benthic assemblages in coastal Newfoundland and Labrador. Retrieved from: https://borealisdata.ca/dataverse/rhodoliths. Chen, R.; Hei, L.; and Lai, Y. [2023]. Object detection in optical imaging of the Internet of Things based on

deep learning. PeerJ Computer Science, 9, e1718.

DFO [2022]. Green sea urchin. Retrieved from: https:// www.dfo-mpo.gc.ca/species-especes/profiles-profils/ green-sea-urchin-oursin-vert-eng.html.

Gagnon, P.; Himmelman, J.H.; and Johnson, L.E. [2004]. Temporal variation in community interfaces: kelp-bed boundary dynamics adjacent to persistent urchin barrens. Marine Biology, 144.

Ge, H.; Dai, Y.; Zhu, Z.; and Liu, R. [2022]. A deep learning model applied to optical image target detection and recognition for the identification of underwater biostructures. Machines, 10.

Han, F.; Yao, J.; Zhu, H.; and Wang, C. [2020]. Underwater image processing and object detection based on deep CNN method. Journal of Sensors.

He, K.; Zhang, X.; Ren, S.; and Sun, J. [2014]. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1904-1916.

He, K.; Zhang, X.; Ren, S.; and Sun, J. [2015]. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778.

He, K.; Gkioxari, G.; Dollár, P.; and Girshick, R. [2020]. Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42.

Hu, J.; Shen, L.; and Sun, G. [2018]. Squeeze-andexcitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7132-7141.

Hu, K.; Lu, F.; Lu, M.; Deng, Z.; and Liu, Y. [2020]. A marine object detection algorithm based on SSD and feature enhancement. Complexity.

Jaderberg, M.; Simonyan, K.; and Zisserman, A. [2015]. Spatial transformer networks. Advances in Neural Information Processing Systems, 28.

Li, K.; Wan, G.; Cheng, G.; Meng, L.; and Han, J. [2020]. Object detection in optical remote sensing images: a survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 159, 296-307.

Lin, T.Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. [2017]. Feature pyramid networks for object detection. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 936–944.

Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C. Y.; and Berg, A.C. [2016]. SSD: single shot multibox detector. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 9905.

Mao, G.; Weng, W.; Zhu, J.; Zhang, Y.; Wu, F.; and Mao, Y. [2021]. Model for marine organism detection in shallow sea using the improved YOLO-V4 network. Nongye Gongcheng Xuebao/Transactions of the Chinese Society of Agricultural Engineering, 37. Neugent, K.F.; Massey, P.; Lambert, R.A.; Rhode, K.L.;

Vesperini, E.; Hao, W.; and Zhili, S. [2020]. Improved mosaic: algorithms for more complex images substructure in the globular cluster populations of the Virgo Cluster Elliptical Galaxies M84 and M86 Journal of Physics: Conference Series, 1684, 12094.

Padilla, R.; Netto, S.L.; and Da Silva, E.A.B. [2020]. A survey on performance metrics for object-detection algorithms. International Conference on Systems, Signals, and Image Processing, 237-242.

Pan, T.S.; Huang, H.C.; Lee, J.C.; and Chen, C.H. [2021]. Multi-scale ResNet for real-time underwater object detection. Signal, Image and Video Processing, 15. Rahman, S.; Rahman, M.M.; Abdullah-Al-Wadud, M.; Al-Quaderi, G.D.; and Shoyaib, M. [2016]. An adaptive gamma correction for image enhancement Eurasip Journal on Image and Video Processing, 1-13. Redmon, J. and Farhadi, A. [2018]. YOLOv3: an incremental improvement. arXiv preprint arXiv: 1804.02767.

Ren, S.; He, K.; Girshick, R.; and Sun, J. [2017]. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39. Reza, A.M. [2004]. Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement. Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology, 38, 35-44.

Saleh, A.; Laradji, I.H.; Konovalov, D.A.; Bradley, M.; Vazquez, D.; and Sheaves, M. [2020]. A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis. Scientific Reports, 10(1), 14671.

Shi, H.; Yang, W.; Chen, D.; and Wang, M. [2023]. CPA-YOLOv7: contextual and pyramid attentionbased improvement of YOLOv7 for drones scene target detection. Journal of Visual Communication and Image Representation, 97, 103965.

Simonyan, K. and Zisserman, A. [2014]. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556.

Sun, X.; Shi, J.; Liu, L.; Dong, J.; Plant, C.; Wang, X.; and Zhou, H. [2018]. Transferring deep knowledge for object recognition in low-quality underwater videos. Neurocomputing, 275.

Teng, B. and Zhao, H. [2020]. Underwater target recognition methods based on the framework of deep learning: a survey. International Journal of Advanced Robotic Systems.

Wang, C.; Liao, H.M.; Yeh, I.; Wu, Y.; Chen, P.; and Hsieh, J. [2019]. CSPNet: a new backbone that can enhance learning capability of CNN. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1571-1580.

Wang, C.Y.; Bochkovskiy, A.; and Liao, H.Y.M. [2023]. YOLOv7: trainable bag-of-freebies sets new stateof-the-art for real-time object detectors. https://arxiv

org/abs/2207.02696.

Zafar, A.; Aamir, M.; Mohd Nawi, N.; Arshad, A.; Riaz, S.; Alruban, A.; Dutta, A.K.; and Almotairi, S. [2022]. A comparison of pooling methods for convolutional neural networks. Applied Sciences, 12, 8643.

Zhang, M.; Xu, S.; Song, W.; He, Q.; and Wei, Q. [2021]. Lightweight underwater object detection based on YOLO v4 and multi-scale attentional feature fusion. Remote Sensing, 13.

Zhao, S.; Zheng, J.; Sun, S.; and Zhang, L. [2022]. An improved YOLO algorithm for fast and accurate underwater object detection. Symmetry, 14(8), 1669.

Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. [2016]. Learning deep features for discriminative localization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2921-2929.

In the past, Dr. Zoheir Sabeur has fulfilled various roles including science director, School of Electronics and Computer Science, IT Innovation Centre, University of Southampton; director and head of research in marine informatics at BMT Group Limited; senior research fellow in computational fluid dynamics at Oxford Brookes University; SERC research fellow in computational molecular physics at University of Leeds and triatomic molecular laser physics at the University of Strathclyde; research scientist in the Intensive Computing Lattice QCD Theoretical Physics Group at the University of Wuppertal, Germany. He has published over 150 papers in scientific journals, conference proceedings, and books; and is a peer reviewer of several European research programs as well as a fellow of the British Computer Society and of IMaREST. Currently, Dr. Sabeur is professor of data science and artificial intelligence at Bournemouth University and head of the Processes and Behaviour Understanding Research Group; program leader for M.Sc. data science and AI and M.Sc. digital health and AI at Bournemouth University; and visiting professor of data science at Colorado School of Mines, US.

What is your occupation?

Professor of data science and artificial intelligence (AI).

Why did you choose this occupation?

Research in data science and AI has been my lifetime specialty for real-world problem solving.

Where has your career taken you?

All over the world for field experimentation and concepts validation as well as meeting the most interesting thinkers in solving real-world problems and challenges, including marine pollution, climate change, respiratory diseases, security in urban environments, and more.

What is your personal motto?

Nowadays it is “AI for science discovery.” When I was younger, as a student, it was: “la science est le bien-être de l’humanité” (in French) meaning “science is humanity’s well-being.”

What hobbies do you enjoy?

Gardening, particularly grafting fruit trees and rose plants. I listen to classical music but also 1960’s rock and roll music.

Where do you like to vacation?

France during winter, Spain during summer, Scotland during spring, and Cornwall during autumn … when I can, of course.

Who inspires you?

First and foremost, my parents, who allowed me to be what I am. Professor Richard Feynman (Nobel laureate), California Institute of Technology – my hero in theoretical physics and his teaching methods – and one of his representatives, Dr. Steven Wolfram, for introducing me to AI in 1987 at St-Andrews University, “The future for accelerating science discovery.” Professor Chris Harris (retired emeritus professor, my godfather in data fusion methods), University of Southampton, School of Electronics and Computer Science. Dr. Ian Barbour (my late PhD supervisor), Glasgow University.

What has been the highlight of your career so far?

I was heavily involved in platform decommissioning programs for the offshore oil and gas industry in the North Sea by providing the first-

of-its-kind mathematical model of the long-term fate of underwater industrial drilling wastes and their impact on the marine environment. I established sensor data fusion grid-less approaches and AI methods for predicting environmental water quality; introduced AI for understanding adaptive behaviour of marine mammals due to climate change; and used AI for understanding human behaviour for security of critical infrastructure and for understanding respiratory diseases, including the effects of environmental factors, lifestyles, and genomics.

What do you like most about working in this field?

Accelerating our understanding of natural phenomena using AI.

What technological advancements have you witnessed?

Big data tools for faster parallel data processing and storage as well as wireless technologies.

What does the future hold for this industry?

The future looks great as we are now able to miniaturize sensing with efficiencies for exploring all sorts of environments, including harsh ones. As for intensive computing tools for processing big data, we are now able to process them faster than ever before. Nonetheless, the excitement is on the mathematical approaches for machine learning that are pushing the barriers to emulate new horizons of intelligence for science discoveries. This is the exciting part for me, since this will help us develop new theories for understanding natural processes at unprecedented spatial/temporal scales.

What new technologies would you like to see?

Accelerated adoption of AI to critically tackle climate change issues … as time is running out. This is in the context of reducing our continuous pressures on natural ecosystems. We need to adopt fuel efficient and green shipping technologies, as well as protect animal cycles versus food consumption trends and security. We should also seriously take care of the impact environmental pollution has on human health, including respiratory diseases and other related ones.

What advice do you have for those just starting their careers?

May you live in interesting data science times.

Trade Winds

Transformative AI for Unprecedented Subsea Survey Solutions

Unique Group

In the challenging world of survey, artificial intelligence (AI) is heralding a new era of efficiency and precision in subsea survey solutions. AI is widely used in the energy sector for real-time data analysis, predictive maintenance, navigation, and environmental impact assessment. Autonomous and uncrewed survey vessels have also applied it for obstacle avoidance and situational awareness. Advanced survey data processing software has also been developed backed by AI which eliminates the need for additional desktop hardware to process the survey data and reduces the ping to chart time. At Unique Group, we are proud to introduce Aquila Subsea, a groundbreaking generative AI chat solution set to revolutionize the subsea survey industry.

Aquila Subsea is not just another tool; it is a transformative solution designed to empower survey engineers and researchers with instant, accurate responses to an extensive number of queries. Developed in-house by Unique Group’s Research and Development Team, this state-of-the-art technology represents our unwavering commitment to pushing the boundaries of technological advancement in the marine industry.

The technology’s strength lies in its comprehensive repository of over 20,000 survey equipment documents. These documents include manuals, specification sheets, data sheets, original equipment manufacturer service reports, and troubleshooting documents. This vast knowledge base enables Aquila Subsea to provide insightful responses, particularly focusing on the diverse aspects covered in the manuals. Whether it is troubleshooting a

technical issue or accessing critical information on-the-go, Aquila Subsea ensures that users have the answers they need at their fingertips.

More than just a database, Aquila Subsea is a dynamic AI solution capable of understanding complex queries and generating contextually relevant responses. By leveraging natural language processing algorithms, the AIbased solution can decipher user queries and provide tailored solutions in real time. This capability not only saves valuable time but also enhances the overall user experience, enabling prompt understanding of manuals and troubleshooting procedures.

Furthermore, when Aquila responds to user queries, it not only provides answers but also suggests related questions concerning equipment functionality, service, maintenance procedures, fault diagnosis, and solutions. This feature enhances users’ understanding of the equipment operations and fosters comprehensive knowledge acquisition.

The benefits of AI technologies extend beyond providing quick answers. By streamlining operations and facilitating efficient problemsolving, technologies like Aquila Subsea promise to boost productivity and elevate industry standards. It is unnecessary to sift through countless manuals or wait for technical support; now survey professionals can tackle challenges head-on with confidence and efficiency. Aquila can also be used by sales teams to gather comprehensive information about different survey equipment with the click of a button, which can reduce their dependency on technical personnel to get the required information.

The integration of AI technologies in subsea surveys goes beyond information retrieval. AI is driving significant changes across the industry by automating data analysis, enabling predictive maintenance, and enhancing imaging techniques. By harnessing the power of machine learning algorithms, subsea surveyors

can extract valuable insights from vast datasets, allowing for more informed decision-making and proactive maintenance strategies.

For instance, AI-powered predictive maintenance algorithms can analyze equipment performance data in real time, identifying potential issues before they escalate into costly downtime or safety hazards. Similarly, AI-enhanced imaging techniques, such as computer vision and deep learning, can enhance the quality and resolution of subsea images, enabling clearer visualization of underwater structures and ecosystems.

The adoption of AI in subsea surveys represents a paradigm shift in how we explore and understand the ocean environment. By leveraging AI technologies, we can overcome traditional limitations and unlock new possibilities for discovery and innovation. From improving operational efficiency to advancing environmental monitoring efforts, the potential applications of AI in subsea surveys are vast and far-reaching.

At Unique Group, we recognize the transformative power of AI, and we are committed to driving innovation in the subsea

survey industry. Aquila Subsea is just the beginning of our journey towards harnessing the full potential of AI for subsea exploration and research. As we continue to push the boundaries of technological innovation, we are keen to embrace AI as a catalyst for positive change in the marine environment.

The integration of AI technologies, shown by Aquila Subsea, is positioned to revolutionize the subsea survey industry. By providing instant access to critical information, streamlining operations, and enabling proactive maintenance strategies, AI is reshaping the way we explore and understand the ocean environment. As we embark on this journey of innovation, it is important that we embrace the transformative potential of AI and work together to create a more efficient, sustainable, and technologically advanced future for subsea surveys. https://aquilasubsea.ai/

Chetan Chitnis, head of research and development, Unique Group, is an accomplished professional with over two decades of experience in technical sales, project operations, engineering, and R&D management, specializing in offshore and subsea industries.

Object Tracking for Uncrewed Surface Vehicles

by Oliver S. White, Sean Daniel, Oliver S. Kirsebom, and Fritz Stahr

Small uncrewed surface vehicles (USVs) are increasingly used for ocean environmental monitoring, seafloor mapping, and surveillance. Typically, USVs are equipped with cameras providing situational awareness, safe navigation, and capacity for solving a wide range of other tasks. However, having a human operator watch video streams continuously is expensive and inefficient, and only possible if the video can be transmitted ashore in real time – rarely an option on the ocean where data must be transmitted via satellites. Therefore, it is imperative for the USV to extract actionable insights from the video streams on its own through computer vision algorithms. At the most basic level, these algorithms must detect, locate, and classify objects appearing in the camera’s field of view, track objects in time, and relay observations to shore in a condensed form. Moreover, since USVs often operate on tight power budgets, the algorithms must be sufficiently “lightweight” to run on a small, onboard computer with limited computational power.

In the last decade, artificial intelligence (AI) has revolutionized computer vision. Given sufficient training data, AI models can detect and classify objects in images more accurately than humans and without tiring. Numerous AI models trained on large, public datasets are openly available under permissive licenses. While typically trained on images captured in terrestrial environments, these models can readily be re-purposed for maritime environments by re-training on custom datasets. They learn to recognize ships, buoys, or even breaching whales, instead of cars, bicycles, or street signs.

However, these AI models typically detect and classify objects one video frame at a time

without considering preceding frames. (This is particularly the case for the lightweight models suitable for small computers.) Thus, such models have no concept of time and do not provide information on object identity and motion. For this, a tracking algorithm capable of correlating objects between frames is needed.

In this short article, we highlight unique challenges encountered in applying tracking algorithms to the maritime environment and discuss solutions being developed at Open Ocean Robotics (OOR), a Canadian company producing and operating rugged, solar-powered USVs for a variety of applications.

OOR’s DataXplorer™ is a small USV equipped with visual and thermal video cameras (Figure 1). Mounted on fixed supports about 1 m above the water surface, the cameras provide a 360° view of the USV’s surroundings. The onboard computer allows in-situ, real-time processing of the video streams. Owing to the USV’s small size, waves on the sea surface induce rapid and large camera motion which must be corrected to correlate objects between frames – a task that is further complicated by the low videoframe processing rates possible on a powerconstrained computer.

In terrestrial settings, edge-detection algorithms are often used to correct for fast, small-amplitude camera motion. However, on the ocean, there are few, if any, fixed features to allow for such corrections. Instead, we use an inertial measurement unit (IMU) sensor to correct for large and small-amplitude motion. As illustrated in Figure 2, this approach allows us to stabilize the horizon and determine the azimuth of detected objects. Accurate

Figure 2: Computer vision algorithm: inertial measurement unit (IMU) sensor data are used to stabilize the horizon and determine the camera azimuth. An AI model detects and classifies objects on a frame-by-frame basis. Finally, object motion is modelled using a linear Kalman Filter, allowing objects to be tracked across frames and helping reject spurious detections.

OPEN OCEAN ROBOTICS

Figure 1: The DataXplorer uncrewed surface vehicle.

OPEN OCEAN ROBOTICS

synchronization of video streams and IMU data at a 10 ms timescale, or better, is essential to yield satisfactory results.

To detect and classify objects, we use an off-theshelf you only look once (YOLO) model, pretrained on the public common objects in context (COCO) image dataset and re-trained on our own custom dataset which contains over 7,000 images acquired during DataXplorer missions under varying conditions of light, sea state, etc. For every processed frame, the YOLO model draws bounding boxes around detected objects and assigns them to one of several categories.

Once camera motion is corrected and objects are detected, we use a simple, linear Kalman filter to model the change in object position and apparent size between frames. Specifically, we consider the object’s azimuth and height of its bounding box as we expect both quantities to change gradually with time. As illustrated in Figure 2, the Kalman filter allows us to correlate objects between frames and helps reject spurious detections thereby reducing the false alarm rate. The Kalman filter has a small number of adjustable parameters which must be carefully chosen to ensure optimal performance in a maritime environment.

The tracking algorithm described above is currently being tested in the field with promising early results. Meanwhile, we are also exploring more advanced methods for correlating objects between frames, e.g., using features implicitly learned by the AI model to quantify the similarity in “appearance” of two detected objects.

Computer vision for small USVs is a nascent and quickly growing field, now with a dedicated conference series, The Workshop on Maritime Computer Vision (MaCVi). At Open Ocean Robotics, we are uniquely positioned to contribute to the advancement of this field. As detailed in this short article, our latest efforts are focused on object tracking, a key step to achieving autonomous navigation.

Oliver S. White is a recent M.Sc. graduate in electrical engineering from the University of Calgary, now an ECO Canada inter n at Open Ocean Robotics. Sean Daniel is the director of engineering at Open Ocean Robotics. Oliver Kirsebom is the lead data scientist at Open Ocean Robotics and an adjunct professor at Dalhousie University. Fritz Stahr is the CTO at Open Ocean Robotics, an MTS fellow, and an affiliate professor at the University of Washington.

Marine Institute Award Winners, 2024 Regional Science Fair

Nutrient Rich

Kelp and Fish Waste Vs. Commercial Fertilizer

Jonathan Anstey (left in photo) and Charlotte Burry (right in photo) are high-school students at Pearson Academy, New-Wes-Valley, Newfoundland and Labrador, Canada; and winners of the Science and Technology Marine Institute Award during the 2024 Regional Science Fair. Congratulations, Jonathan and Charlotte!

“Our winning project to compare the effects of kelp and fish waste to that of commercial fertilizer was inspired by local culture and the ingenuity of the traditional way of life as an outport community in rural Newfoundland. Tapping into natural resources available in our coastal waters, such as the abundance of sea kelp or the unused byproducts of our fishing industry, to cultivate gardens is a practice praised by many of our community members. After diving into the literature surrounding the use of each type of fertilizer, we found that numerous studies pointed to both kelp and fish waste being excellent sustainable alternatives to commercial fertilizer. With this information, we conducted a comparative analysis of the biomass of radishes grown using each fertilizer. Our results showed that radish plants grown in locally sourced fertilizers produced higher biomass than that of commercial fertilizer.”

The US Coast Guard Research and Development Center and Zelim, a start-up based in Edinburgh, Scotland, are jointly exploring the potential application and effectiveness of AI-enabled detection and tracking technology in search and rescue. Over the last century this task has been undertaken by search and rescue units, their pilots and, more recently, drone pilots who often scan the sea surface for hours looking for an object no larger than a football, as much of the human body remains hidden below the surface.

Over the last three years, Zelim has been developing ZOE, a solution that employs AI to detect and track multiple people, boats, or target objects in the water by day or night, storm, or fog. Like the driving aids that reduce driver fatigue and provide hazard alerts and timely information in cars, ZOE aids the search operator by consistently scanning the search area looking for anomalies and providing visual and audible alerts.

https://www.zelim.co/

Optimizing Fish Farm Management with Deep Learning Analysis of Underwater Drone Data

by Mira Nagle

In the search for sustainable food production techniques, aquaculture has emerged as a crucial player. With the world’s population growing, the need for innovative solutions in food production is greater than ever before. However, ensuring the quality of our food sources relies heavily on overcoming the environmental obstacles that hinder traditional farming techniques, including a lack of fresh water, animal disease, and decreasing farmland.

Enter deep learning analysis of underwater drones, or remotely operated vehicles (ROVs), in aquaculture practices (Figure 1) – harnessing the power of algorithms to gather critical data about the health of fish, disease patterns, food quality, and the effects of feeding techniques with speed and accuracy, compared to conventional methods of data collection. The utilization of underwater drones equipped with deep learning capabilities, such as autonomous scheduled surveillance, water quality testing, and protection tactics against disease or predators, marks a significant advancement in our ability to monitor and understand the intricacies of aquaculture systems. These drones have the capacity to collect vast

amounts of data in real time, providing farmers and scientists with valuable insights into the factors influencing fish health and productivity. From monitoring water quality parameters to tracking the behaviour of fish populations, the data gathered by these drones offer a comprehensive view of aquaculture operations.

One of the key advantages of employing deep learning analysis in aquaculture is its ability to facilitate proactive management strategies. By continuously learning from the data they collect, underwater drones can identify trends and patterns that may indicate potential issues before they escalate. This proactive approach empowers farmers and scientists to intervene early, implementing corrective measures to mitigate risks and optimize fish farm productivity.

The insights from deep learning analysis can enable farmers to make informed choices regarding feeding regimes, stocking densities, and environmental management practices. Underwater drones can visualize fish activity in real time, analyzing practices using auxiliary cameras and imaging sonar. The data are reviewed by the operator via a closed-source software and can be saved and reviewed at a later date for further observation. Once the data are reviewed, farmers and scientists can log the information for comparison. By tailoring farming practices to the specific needs of the fish and the conditions of their environment, farmers can optimize growth rates, minimize disease outbreaks, and enhance overall farm efficiency.

The continuous learning capabilities of underwater drones hold the promise of driving innovation in aquaculture practices. As these drones gather more data and refine their algorithms, our understanding of the complex interactions within aquaculture systems will deepen. This ongoing process of learning and adaptation paves the way for the development

of sustainable techniques that will further enhance fish farming operations.

By predicting and preempting potential challenges, farmers can avoid the costly consequences of fish health issues and environmental degradation. This proactive approach not only safeguards the welfare of the fish but also ensures the long-term viability of aquaculture as a sustainable food production method.

In conclusion, the integration of deep learning analysis with underwater drones represents a shift in fish farm management. By harnessing the power of data-driven

insights, we can optimize aquaculture practices, enhance fish health and welfare, and mitigate the environmental impacts of fish farming. With continued innovation and investment in this transformative technology, we can pave the way for a more sustainable and resilient future for aquaculture.

Mira Nagle is the marketing administrator at Oceanbotics Inc., overseeing all digital marketing endeavours. With a strong background in marine science and aquaculture, she brings a unique understanding to her work. She enjoys fishing and camping near the ocean in her free time.

Figure 1: By continuously learning from the data they collect, underwater drones can identify trends and patterns that may indicate potential issues before they escalate.

Marine What do They Have to Say? Animals

by Lawrence Taylor

Bivalves have been used as marine sentinels since the 1960s by analyzing their tissues for pollutants, but non-invasive tools and artificial intelligence (AI) can now provide novel real-time insights into their environment and their biology.

Talking to the animals, crazy – yes? Well, maybe not so much. In Thomas Friedman’s book, Thank You for Being Late, data scientists, who may have never stepped on a farm, gained huge insights into dairy cow breeding when they were armed with pedometer data. Cows not only walking-thewalk but talking-the-talk.

To paraphrase, the new breed of data scientists does not need to understand a process to find patterns or weak signals. As cows approach estrus, their step counts go up, so much so that AI predicted with 95% accuracy the onset of estrus. Dairy farmers have an 18-hour window to artificially inseminate their cows, and the data showed that if insemination occurred within the first four hours, the chances of a female offspring were 75% versus the higher probability of a bull-calf during the second four hours. The data also provided the early detection of eight different diseases. Who knew that a device many of us have strapped to our wrists would give dairy cattle a voice regarding their reproductive and health status. The question is, can we do the same with marine animals as they face ever-changing seas?

Bivalves are very sensitive to changes in their environment and a few water treatment

plants have taken advantage of this. By attaching magnets to the animals’ shells, water treatment managers have found that a handful of bivalves can provide a “more relatable estimation of the water’s overall toxicity, [because they can] account for a broad range of factors simultaneously.ˮ In a marine setting, however, where there can be a lot of good and bad “stuff” in the water, a broader context of what is going on around the sentinels is needed and that is why IntegraSEE is focusing on vision technology.

This is a great example of a picture being worth a thousand words. Early on while collecting video training data to teach the AI model to recognize open and closed mussels, the top row of mussels was closing up each night when they were illuminated with underwater flood lights (Figure 1). The reason why? The lights were attracting schools of Mysid shrimp which caught the attention of a local juvenile sculpin. Our mussels made the perfect perch for the hungry sculpin to sit and dine on the passing shrimp.

Like a canary in a coal mine, bivalves have a huge early warning potential, but we still have a lot to learn about their body language. They will clam up when challenged by warming seas, drops in salinity, or low dissolved oxygen levels. But when faced with hazardous algal blooms, they perform micro-closures. Imagine then, the value of bivalve biosensors in, and around, and upstream of finfish pens, warning farmers of toxic algae and animal health and welfare. A whole new window of opportunity also opens up for shellfish growers, providing insights never seen before.

As one grower once muttered, “there is not enough beer to make hand grading oysters a fun job.ˮ Even at one oyster per second, a vision grader can do in an hour what a team can do in a day. By analyzing size and length,

Homeward Bound commentary

oysters are quickly classified into a size group and dropped into the desired bin. But what about those measures?

When grader manufacturers and growers were asked what they did with those measurements, the answer was – nothing. This is a valuable lesson learned. Grading and biological data gathering do not need to be mutually exclusive events; nor does water quality assessment and bivalve biology.

At this time, IntegraSEE’s goal is to work with shellfish growers and researchers to better understand what bivalves are saying about their environment, maximize production yield, and predict spawning and spat settlement

events – to generate real or proxy data about the “stuff” surrounding the animals. IntegraSEE is working with Clean Valley which is converting land-based finfish waste into algae to feed oysters. Versus open sea experimentation, IntegraSEE is fine tuning its technology by analyzing wastewater, algae, and oysters separately and collectively. In the future, the goal is to address finfish animal welfare and social licence issues, as well as providing early warning services against killers like hazardous algal blooms and low oxygen events.

Giving marine life a voice: exciting times and exactly where AI and vision technology should be directed.

In 2016, biologist, underwater photographer, and entrepreneur Lawrence Taylor, M.Sc., realized that visionbased seafood grading and biological data gathering did not have to be mutually exclusive. As IntegraSEE’s CEO and cofounder, he is excited about automating image processing to give marine life a voice and give us a better understanding of our changing seas.

Figure 1: IntegraSEE’s AI model captures and classifies images of bivalves as they react to changes in their natural environments –changes such as warming seas, low dissolved oxygen levels, and the presence of other species.

INTEGRASEE

INTEGRASEE

Parting Notes

Given this issue is on deep (ocean) learning, we thought we would create some ocean technology related artwork using Meta AI, an artificial intelligence laboratory owned by Meta Platforms (formerly known as Facebook Inc.) Our designer Danielle Percy typed in the search words "deep ocean learning" and this is what we got!