NATO Chief Scientist Report - AI

NATO SCIENCE AND TECHNOLOGY ORGANIZATION (STO) RESEARCH ON ARTIFICIAL INTELLIGENCE (2010 – 2021)

NATO Chief Scientist Research Report

DISCLAIMER

The research and analysis underlying this report and its conclusions were conducted by the NATO Science & Technology Organization (STO). This report does not represent the official opinion or position of NATO or individual governments.

This report has been optimised for reading digitally, including internal and external links.

NATO Chief Scientist Research Report

T. Granak

D.F. Reding

A. De Lucia

L. Lim

NATO Science & Technology Organization

Office of the Chief Scientist

NATO Headquarters B‑1110 Brussels

Belgium

http:\www sto.nato.int

NATO Chief Scientist Research Reports provide evidence‑based advice or policy insights based on research and analysis activities conducted across the NATO Science & Technology Organization.

Activity findings relevant to this Report are already published or will be published on the NATO Science & Technology Organization website: <http:\www sto.nato.int>.

This report is distributed free of charge for informational purposes. The sale and reproduction of this report for commercial purposes is prohibited. Extracts may be used for bona fide educational and informational purposes subject to attribution to the NATO Science & Technology Organization.

Unless otherwise credited, all non‑original graphics are used under Creative Commons licensing.

For graphics from DVIDS, please note: The appearance of U.S. Department of Defense (DoD) visual information does not imply or constitute DoD endorsement.

For graphics used from DARPA, please note: DARPA and DoD do not endorse or sponsor any product, service, or entity.

Copyright © NATO Science & Technology Organization, 2022.

First published, August 2022.

FOREWORD

FOREWORD

Of all the Emerging and Disruptive Technologies (EDTs) considered by NATO, Artificial Intelligence (AI) is the development area that attracts the most interest and attention, both in the defence and security community and civilian domains. Unlike the other EDTs, AI is a general‑purpose technology with many potential applications, and the scope of research for NATO’s Science & Technology Organization (STO) is therefore considerable. From the augmentation of intelligence to networked platforms for multi‑domain operations to the potential development of lethal autonomous systems, AI is predicted to have a transformative effect on the shape of the future battlefield and on NATO’s ability to deter and defend. The publication of NATO’s AI Strategy further underscores the importance the Alliance has assigned to AI and its centrality to our understanding of the EDT portfolio more broadly.

In support of this effort, the Office of the Chief Scientist has produced this report, the fifth in a series of NATO Chief Scientist Research Reports. This report shares the findings from STO research with the wider NATO community. It consolidates the STO’s research activities in this area over the past decade, spanning questions of Human Machine Symbiosis, logistics and predictive maintenance, advanced algorithms, training and simulation, and command, control, communication, and computing. In sharing this considerable body of research, we hope to guide NATO and its Allies as AI is integrated into our strategic thinking and warfighting capabilities.

We are immensely grateful to the national experts who make up the STO and whose efforts are reflected in this report. Their collective spirit of inquiry and unwavering dedication to tackling the most pressing scientific and technological problems underpin the findings presented here.

Dr. Bryan Wells – NATO Chief Scientist

Dr. Bryan Wells

EXECUTIVE SUMMARY

Artificial Intelligence (AI) is changing the global defence and security environment. As a foundational general‑purpose technology, it will affect the full spectrum of activities undertaken by the Alliance to support its three core tasks: collective defence, crisis management, and cooperative security. AI capabilities underpin the use of some automated and autonomous technologies deployed in theatre and generate advantages in intelligence collection, command‑and‑control systems, and data management. These technological developments span all domains of operation. In the future, it is also expected that AI will increase the rate of innovation itself. Enhanced analytical capabilities will result in shorter innovation cycles, transforming our understanding of invention and progress.

This report addresses the body of NATO Science & Technology Organization (STO) research conducted between 2010 and 2021 on AI in various guises for a general audience. The report identifies seven central areas:

• Advanced Algorithms (AA);

• Command, Control, Communications and Computers (C4);

• Human‑Machine Symbiosis (HMS);

• Intelligence, Surveillance and Reconnaissance (ISR);

• Predictive Maintenance and Logistics (PML);

• Training, Modelling and Simulation (TMS); and

• Unmanned Vehicles (UxV).

The research described in this report provides a robust evidence‑based framework for ensuring informed decisions are made in the development and adoption of AI.

The completed body of work is substantial, with a considerable volume of research activities focused on applying AI to ISR. In the maritime domain, the STO’s Centre for Maritime Research and Experimentation (CMRE) has conducted significant world‑leading research, which further contributes to our comprehensive understanding of the challenges and opportunities presented by AI. In particular, the Centre has considered using AI in target recognition, image classification, and data management. This report also provides details of ongoing research activities that will further advance our collective knowledge in this field.

Key findings drawn from across the research undertaken by the STO include:

• Advanced Algorithms (AA): STO research has recognised that Machine Learning (ML) and Deep Learning (DL) advances improve heterogeneous data‑driven decision‑making and intelligence analysis applications.

• Command, Control, Communications and Computers (C4): Current operational processes lack automatic data curation and sanitation, credibility assessment, detection of circular reporting, adjustments for cognitive biases, and natural language processing to detect current and evolving narratives. Multi‑source information fusion delivers better situational awareness and indications of a threat than a lone source. AI tools can also support the identification of abnormalities in data and lead to automated authentication of suspicious events.

• Human‑Machine Symbiosis (HMS): The research facilitated by the STO recommends that AI applications in machine systems are explainable, and human machine teams should improve their performance by co learning. They have also noted that AI’s inability to infer context derives from a lack of annotated ground truth data for military applications of autonomy. Therefore, the design of future human machine systems must adapt to evolving circumstances, verify and validate the human machine team.

• Intelligence, Surveillance and Reconnaissance (ISR): Automation is required to find weak signals and interconnections in data, and soft and hard data must be fused to deliver actionable intelligence to support decision‑making and planning. ML and DL techniques are promising methods capable of processing conflicting multisensory information and enabling easily scalable solutions. In the future, STO scientists engaged in the Collaborative Programme of Work (CPoW) envisage automated scene understanding for battlefield awareness based on the growth of AI techniques and increasingly large sets of military relevant annotated data.

• Predictive Maintenance and Logistics (PML): STO research has explored Bayesian methods for evaluating fleet replacement decisions and ML techniques to deliver descriptive analysis for various logistics processes, predictive maintenance, planning, reporting and decision support. It is expected that, in the future, more advanced analytics products based on AI will be developed and integrated into Enterprise Resource Planning (ERP) systems for real‑time decision support related to logistics.

• Training, Modelling and Simulation (TMS): Activities in the STO CPoW have concluded that embedded training allows skills to be rapidly developed, maintained, and adapted close to the battlefield. The STO AI scientific community has focused on Human Behaviour Modelling (HBM) in military training, including integrating DL and ML techniques to assist in modelling teams, organisations, government actions, societies, cultures, and military contributions.

• Unmanned Vehicles (UxV): AI‑enhanced UxVs can significantly improve a force’s ability to project combat power and adapt to more complex and uncertain future environments. Furthermore, removing humans from the cockpit enables a more efficient and cost‑effective design for vehicles and platforms. However, centralised management of UxV systems will not deal with the dynamic challenges in information‑poor environments. Still, AI techniques and Bayesian approaches promise to foster real‑time dynamic planning mechanisms and autonomous decision‑making.

Supported by the fast accelerating strategic and policy landscape exploring AI in NATO, the STO has made extraordinary progress in advancing the Alliance’s collective knowledge about the opportunities and challenges presented by AI for the future battlefield.

INTRODUCTION

The present NATO Office of the Chief Scientist (OCS) report represents an aggregation of the research conducted through the NATO Science & Technology Organization (STO) Programme of Work (PoW) over ten years examining Artificial Intelligence’s (AI) impact on military operations, defence capabilities and decision‑making. Its primary purpose is to share with the wider NATO community the findings of STO research on AI and support NATO’s understanding of this highly synergistic and interconnected S&T area, considered a major strategic disruptor.

The OCS is the STO’s executive body closest to political and military leaders at NATO HQ. The OCS supports the NATO Chief Scientist’s two essential functions: first as the Chairperson of the Science and Technology Board (STB) and second as the senior scientific advisor to NATO leadership. Beyond providing executive support to the STB and its chartered responsibilities, the OCS acts as a focal point for the STO PoW and its users at NATO HQ. The STO’s PoW is carried out through two primary components:

1. The STO Collaborative Programme of Work (CPoW) follows a collaborative business model where scientists, engineers, and analysts are resourced by their Nations or organisations; and

2. The Centre for Maritime Research and Experimentation (CMRE) programme of work follows an in‑house delivery business model where research is customer‑funded.

To that end, the OCS builds on the S&T results generated through these components and promotes their use in political and military context. In addition, the OCS aims to bring forward the most relevant and up‑to‑date S&T results to inform senior NATO decision‑making by engaging the committees and staff at NATO HQ and beyond.

Note that this document situates itself within NATO’s long‑established engagement with AI as well as other related Emerging and Disruptive Technologies (EDTs), such as Big Data and Autonomy. For example, the Science & Technology Trends 2020 – 20401 published by OCS explored the larger S&T context driving the development of EDTs. Equally, adopting the Artificial Intelligence Strategy for NATO in 2021 has accelerated AI adoption by enhancing key AI enablers and adapting relevant policy.

1 Reding, D.F., and Eaton, J. (2020). Science and Technology Trends 2020 – 2040: Exploring the S&T Edge. NATO S&T Organization.

HOW TO READ THIS REPORT

This extensive report provides summaries and analyses of the STO’s relevant activities since 2010.

Given its length, readers are encouraged to use the Table of Contents to identify areas of interest.

Since several EDTs converge synergistically, the research on AI is inherently crosscutting. Despite the proliferation of research on AI techniques, they are not necessarily always the research goal, but more often, they represent the means to reach other diverse military goals.

The research summarised in this report comes from the STO CPoW Scientific and Technical Committees, composed of six panels and one group:

• Applied Vehicle Technology (AVT)

• Human Factors and Medicine (HFM)

• Information Systems Technology (IST)

• System Analysis and Studies (SAS)

• Systems Concepts and Integration (SCI)

• Sensors and Electronics Technology (SET)

• NATO Modelling and Simulation Group (NMSG)

The research focused on the maritime domain comes from three programmes managed by the CMRE:

• Autonomous Naval Mine Counter Measures (ANMCM)

• Environmental Knowledge and Operational Effectiveness (EKOE)

• Cooperative Antisubmarine Warfare (ASW)

Research activities are each given a reference code linking to the panel that conducted them, e.g., SET‑216 Readers may note some activity codes consist of two reference codes. This double coding identifies a collaborative activity shared by two panels. For example, activities managed by the Centre for Maritime Research and Experimentation are given a reference code “CMRE.”

This report is divided into seven broad chapters, providing overviews of seven central themes:

1. Advanced Algorithms

2. Command, Control, Communications, and Computers (C4)

3. Human‑Machine Symbiosis (HMS)

4. Intelligence, Surveillance and Reconnaissance (ISR)

5. Predictive Maintenance & Logistics (PML)

6. Training, Modelling & Simulation (TMS)

7. Unmanned Vehicle (UxV)

These themes align with the NATO emerging and disruptive S&T agenda and previous OCS publications, framed around increasing the understanding within the Alliance of the potential for S&T developments to enhance or threaten Alliance military operations.

While individual STO reports often cover more than one of these seven central themes, the same activity is featured only once in the context of the most relevant theme.

STO research activities represent differing degrees of effort, investment, and time. The following activities are relevant to this report:

• Exploratory Team (ET) – a feasibility study to establish whether it is worth starting a more extensive activity, usually one year in duration

• Research Task Group (RTG) – a study group, three years in duration unless delayed.

• Research Symposium (RSY) – over one hundred attendees, 3 – 4 days in duration.

• Specialist Team (ST) – AI report quick reaction.

• Research Specialists’ Meeting (RSM) – over one hundred attendees, 2 – 3 days in duration.

• Research Workshop (RWS) – selected participation, 2 – 3 days in duration.

• Research Lecture Series (RLS) – junior and mid‑level scientists.

• Research conducted by the Centre for Maritime Research and Experimentation (CMRE) – sea‑proven maritime innovation and interoperability solutions

This report consists of a review and annotated bibliography. The review provides a programmatic overview of similar activities, where each research activity summary lists the type of activity conducted and central conclusions. Chapters refer to “precursor activities,” which represent early‑stage activities that have served as the basis for additional research activities. They then describe related activities that often develop thinking and consider a subject in greater depth. In this way, readers can follow the evolution of a particular research trajectory.

The annotated bibliography is the main annex and serves as a comprehensive listing of all relevant completed and ongoing activities. In addition, ongoing activities with their respective expected completion dates are provided for readers who wish to follow these activities and release the final reports.

TERMINOLOGY

These activities are built upon national efforts and do not rely on approved STO‑wide language. As such, they may demonstrate inconsistencies in their use of language. This inconsistency reflects variations among the scientific community in using Artificial Intelligence‑related terms and their definitions.

ACCESSING ACTIVITY REPORTS

It is a welcome reflection of the fullness of the STO’s work in this area that this report provides only a snapshot of the detailed research.

To access completed reports, readers should be mindful of the restrictions imposed by different classification levels. Readers must first locate the activity code or title allocated to the activity of interest (e.g., ‘SAS‑097’ or ‘Robotics Underpinning Future NATO Operations’). Once found, this code or title can be searched in the ‘Publications’ search function of the STO website. Access based on classification is divided as follows:

1. Open Access: Some papers are open access and can be accessed in full using the ‘Publications’ search function on the STO website by searching for the activity codes or titles you are interested in: < https://www.sto.nato.int >.

2. NATO Unclassified: This report is released at NATO Unclassified, reflecting most research activities detailed within the report. These papers can be accessed using the ‘Publications’ search function on the STO website as detailed in Step 1; however, access can only be gained by logging into the STO website. Allied nations, STO Partner nations, and in‑house staff may gain full access to these reports by gaining individual login credentials for the STO website. These can be obtained by contacting the STO. Contact details can be found at: < https://www.sto.nato.int/Pages/contactus.aspx >.

3. NATO Restricted (or above): A minority of activities are NATO Restricted or above; the metadata for these reports is listed on the STO website, but not the actual report. Access to these reports is given on a need‑to‑know basis. Please contact your national S&T points of contact. Contact details can be found at: < https://www.sto.nato.int/Pages/contactus.aspx >.

ADVANCED ALGORITHMS (AA)

“By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.”2

2 Eliezer Yudkowsky.

ADVANCED ALGORITHMS

OVERVIEW

Advanced Algorithms (AA) are integral to the latest Artificial Intelligence (AI) developments. The computational approaches allow machines to mimic human cognition, such as recognising patterns, learning from experience, drawing conclusions, and making predictions. AA research focuses on several statistical and algorithmic techniques, including bio‑inspired learning approaches.

AA cover a broad field of research consisting of Machine Learning (ML), Deep Learning (DL), Neural Networks and Bayesian Networks that form the computational foundation on which AI works. The key to developing AA is training them by repetitively evaluating the output of each algorithm against the desired result, enabling the machine program to learn by making connections within the available data.

Different algorithms have advantages and disadvantages regarding accuracy, performance, and processing time. Therefore, algorithms are chosen based on the application and the nature of the data points available.

AA are becoming progressively more sophisticated as the pool of potentially meaningful variables within the Big Data universe continues to mushroom. Consequently, AA development and proper implementation directly correlate with improvements in the effectiveness and generalisation of AI. In practice, continued R&D into new and more general‑purpose algorithms will increase embedded AI’s efficiency in military applications. It will also be critical in maintaining the current momentum behind AI research and moving AI beyond its current practical limitations.

Research Conclusions

• AI‑AA enable real‑time exploitation of data from multiple sources for sense‑making, decision support and knowledge generation. This includes the contextual understanding of events and event prediction.

• Joint exploitation of multiple media requires effort at several distinct levels of data management. Namely, identification, extraction, and fusion.

• ML/DL systems are a crucial element in decision support systems and autonomous systems in NATO operations.

Research Challenges

• Automatic detection and classification of real‑world events by ML/DL tools.

• Research focused on systems robustness of ML/DL tools.

• Research focused on adjusting training, testing, validation, and product phases of ML/DL tools to make them resilient to input manipulation.

Figure 1: Advanced Algorithms (Credit: iStock)

COMPLETED RESEARCH

RTG SAS‑IST‑102 conducted by the SAS Panel was the first to identify the lack of methodologies and advanced tools for exploiting unstructured data. The IST Panel manages the bulk of scientific research activities related to Advanced Algorithms. These activities include IST‑ET‑086, RTG IST‑144, RSM IST‑158, RTG IST‑177 and RTG IST‑169, all of which tackle exploiting multimedia data sources. The STO’s ongoing research on using OSINT derived from social media has further used the outcomes from this activity cluster.

Intelligence Exploitation of Social Media (RTG SAS‑IST‑102), 2013 – 2016

The RTG builds on activities in the cyber domain carried out during the Arab Spring and seeks to determine ways to exploit social media sources for Actionable Intelligence in the military domain. The RTG identifies the lack of knowledge, methodologies, and tools for exploiting unstructured data and the need for further research on AI and related tools.

Since 2011/2012, the RTG SAS‑IST‑102 introduced Social Media (SM) as part of the battle‑scape in which military operations are conducted during conflict and peacetime establishment. Identifying and spotting weak signals and interconnections represent the main challenge in today’s digitally connected world. Moreover, the volume of data from heterogeneous sources grows exponentially, thus increasing excessive noise. Accordingly, the activity identified the need:

• For knowledge, methodologies, and tool development to exploit unstructured data while enabling Actionable Intelligence.

• To develop new methodologies and tools in AI, sentiment analysis and deep content analysis.

Complex Event Processing for Content‑Based Text, Image, and Video Retrieval (IST‑ET‑086), 2015 – 2016

This ET draws from knowledge produced by SAS‑IST‑102. Its main objective is the development of theoretical and algorithmic tools supporting the joint exploitation of multimedia data sources (text, image, video, voice).

As adversary nations and non‑state actors threaten NATO Allies with hybrid‑warfare operations, it is critical to advance parallel and integrated processing of text, image, video, and voice information. At the same time, this requires developing methods to cross‑pollinate developments across these domains. Analysis methods for extracting information to support

content cannot be done in isolation. Hence, it is necessary to identify NATO’s need for interoperable tools that cross‑cue knowledge obtained from one method to generate tasking in another. Suggested activities include:

• Data collection and ground truth labelling.

• Developing ML tools to detect and classify events from combinations of data classes.

Content‑Based Multi‑media Analytics (CBMA) (RTG IST‑144), 2016 – 2020

This RTG is the outcome of the state‑of‑the‑art review from the exploratory team IST‑ET‑86. This activity showed the possibilities for exploitation of multimedia data through concept and advanced demonstrations, using methods including ML and DL approaches. These advances improve heterogeneous data‑driven decision‑making and intelligence analysis applications. Thus, boosting the agility of military operations and increasing the ability to understand adversary perspectives.

Following suggestions from IST‑ET‑86, it recommends specific architectural design features for real‑time analytics of heterogeneous multi‑media streams3 in distributed coalition environments. Enhancements enable this activity in the contextual understanding of complex events through advances in computational/human processing techniques. Developments focus on four key technical aspects shown through concept demonstrations in a realistic military scenario:

• Intelligent Capture and indexing of motion imagery.

• Expansion of ML/DL approach to semantic video analytics.

• Cross‑cueing from text analytics to drive video and imagery indexing and retrieval.

• Explore architectures and frameworks that could optimise the implementation of multi‑media analytics in distributed coalition environments.

As we consider AA applications, further research and performance measurements are needed due to the high false alarm rates with large data sets particular to defence and security. In addition, system co‑design for algorithms and hardware is necessary to engineer and integrate system architectures optimised for CBMA and DL. Finally, an overall challenge is integrating with and exploiting the current, future commodity distributed systems (e.g., servers, cloud) and

3 Image, video, text, and speech.

distributed services (e.g., computation, storage) expected to be available to an enterprise. Challenging technical areas include but are not limited to the following: hardware for DL, including neuromorphic chips, low power, high‑performance field‑programmable gate arrays, optimisation methods across the convolutional layers and back propagation techniques, training with fewer examples, and distributed learning across systems.

IST‑144 claims that to achieve trust in military systems using ML, the military must prove robustness and accountability. Robustness is essential for the availability and integrity of any military system, with or without sensors and effectors. Accountability is a future requirement for such systems, and the more complex a system becomes, the more the documentation of accountability will grow towards “non‑human” complexity. Decision‑makers must be able to document how the decision‑making systems operate to show why their system recommends specific actions based on the input from sensors. This approach is the rationale for ongoing RTG IST‑169, which will further determine the foundational elements of how computational algorithms can be understood and can accurately exploit context in human scenarios.

Content‑Based Real‑time Analytics of Multi‑media Streams (RSM IST‑158), 2017 – 2018

This RSM was a necessary adjunct to IST‑144 and provided constructive challenge and critique to the foundational work. The objective of this meeting was to debate and explore the problem spaces, solution spaces and the utility of approaches in the exploitation of multimedia data sources. It gathered experts from NATO member military agencies, industry leaders, and academic visionaries to present the following specific capabilities integrated within a relevant military scenario:

• Image classification and indexing capability allowed images to be searched for video frames, including target objects.

• Image classification capability allowed rapid scanning of video frames for location, object, and behavioural activities of interest in a large video repository.

• Text analytics capability extracts entities and relationships from various source documents.

• Text analytics capabilities classify social media users and followers as likely to belong to a terrorist group.

COMMAND, CONTROL, COMMUNICATIONS AND COMPUTERS (C4)

“[…] our ability to respond faster through cleverer decision‑making, which is enabled by the flow of information, is actually frankly as important if not even more important than whether our tanks out‑range an anti‑tank missile.”4

4 General Sir Gordon Messenger, UK MOD, Vice Chief Defence Staff.

COMMAND, CONTROL, COMMUNICATIONS AND COMPUTERS (C4)

OVERVIEW

Data from a broad spectrum of sources continue to increase and requires analysis to present the necessary and sufficient information to the operators and decision‑makers. Users need relevant information in a timely fashion. However, defining what is appropriate can be difficult and varies with the environment and context. The goal is to retain information superiority over an adversary. AI‑enhanced methods can help achieve this goal. Especially when coupled with visual analytics, AI methods can enhance understanding of the information landscape. Extensive training data is critical for the development of AI tools. Unfortunately, some data can be sparse or sensitive in military operations. Synthetic data derived from other models or data farming can supplement incomplete training data. Using such data can lead to conflict with the exacting standards of trust and security demanded by the military. Confidence in the applications is essential but can be challenging to achieve. The desire to implement an automatic response can arise, especially when the time to react is less than normal human reaction times. The goal of the research focused on AI applications in the C4 domain is to “Ensure that the Alliance has AI and Big Data supremacy for decision support by exploiting data and technology to its full potential across NATO and Nations to enhance both operational effectiveness and cost effectiveness.”5

Research Conclusions

• Humans need AI assistance to analyse the vast quantity of available data. First, however, users must understand the limitations of such AI.

• Confidence in the presented information derived from AI and Big Data is essential with operators exercising a high level of training, critical judgement, and awareness.

• AI methods lack automatic curation and data sanitation, assessing the credibility of information, detecting circular reporting, countering analysts’ cognitive biases and the ability to process natural language to identify narratives.

• AI techniques are in commercial use, with some appropriate to form the basis for developing robust military tools.

• Valuable data, computing resources and their sharing amongst military HQ, NATO entities, or non‑NATO entities is critical for training and establishing a meaningful output in a timely fashion.

• Development of structured learning may assist where training data is not readily available, and synthetic data can substitute missing data in specific areas.

• AI remains a heuristic process with no current means to verify the output. An improved understanding of human‑machine interactions can alleviate this issue.

5 NATO summit 2018.

Figure 2: Command, control, communications and computers (C4) (Credit: iStock)

Research Challenges

• Seek a sound understanding of AI risk, benefits, and develop guidelines for its appropriate use in decision‑making.

• Contribute to improving AI robustness, trustfulness, explainability, verification and validation in its underpinning and deployment in a military context.

• Identify relevant commercial applications of AI in advanced stages of implementation and identify necessary modifications for military applications.

• Identify military requirements not currently researched in the commercial field and fill the research gaps.

• Increase transferability of AI between military applications.

• Develop AI training methods and tools with the appropriate raw and synthetic data.

• Determine the amount of labelled data required to train an AI to solve a given problem.

• Ensure wide‑scale availability of data repositories and use existing data management infrastructure.

• Develop personnel training methods for various forms of AI, including analysing their advantages and risks, benefits, and costs of using them.

• Develop adequate interfaces to match the user and the environment. Leverage sensors to determine the users’ psychological state.

• Improve the understanding of AI, Big Data and Data Science and inform political leaders and military decision‑makers about potential and drawbacks.

• Work with PDD to explain the use of AI in military operations to the public.

COMPLETED RESEARCH

The IST TP conducted most of the activities in this C4 cluster. The activities include IST‑106, IST‑ET‑084, IST‑160, IST‑173 and IST‑178. Initially, the portfolio of research activities examines existing tools for multi‑source information fusion. Then, the research proceeds to map challenges in the planning processes and identify synergies between different TPs and activities to establish a common roadmap for utilising AI in decision‑making. Lastly, the research addresses the latest developments in the growing information domain. RTG SAS‑111, managed by SAS TP, is the only outlier activity focusing on enhancing deployed NATO headquarters decision‑making capabilities. SAS‑ET‑EG examines utilising AI to support and speed up the intelligence cycle. This cluster contains a wide range of activities evenly spread across the investigated timeframe. AI is the focal point of research for all listed activities. This portfolio of research has strong synergy with all TP and slight overlap with the research on data fusion in ISR conducted by SET TP.

Information Filtering and Multi‑source Information Fusion (RTG IST‑106), 2011 – 2014

This RTG examines AI decision support tools and processes for synchronising and aligning heterogeneous and differing information from various sources for decision support.

RTG IST‑106 took inspiration from the SET TP, specifically RTG SET‑189 and identified gaps in the research on C4. Concerning AI, SET TP focuses on the wide range of ISR pertinent topics that feed into decision‑making. RTG IST‑106 highlighted the importance of exploiting, aligning, and synchronising a broad spectrum of sources to attain information superiority but also identified challenges and suggested methodological approaches and processes that lead to faster, more accurate and agile decision‑making. Simply put, this activity laid the foundation for further research on data fusion in C4. The RTG suggested a generalised three‑stage process for data exploitation.

• Firstly, data must be discovered, collated, filtered, matched, and assigned to the suitable process.

• Secondly, the data must be exploited, validated, merged, aligned, inferred, fused, and augmented.

• Thirdly, the data must be presented and stored to be retrieved and shared easily to foster institutional Information Knowledge Management.

This RTG concluded that the AI tools, supporting data infrastructure and associated processes must be set up to enable real‑time decision‑making. Multi‑source information fusion delivers better indications of a threat than a single source. It is necessary to have concepts for normality, authentication, and verification to detect abnormalities in multi‑source data. AI tools can support the identification of these patterns and lead to the development of a methodology for automated authentication of suspicious events. This RTG was reorganised in 2014, and the research continued under the oversight of different activities.

Continuous Planning Process and Decision Support at Tactical Levels (RSM IST‑ET‑084), 2015

This ET explores the challenge of making planning and execution processes more dynamic to the extent of being able to plan and execute continuously in changing situations.

IST‑ET‑084 examined the role of planning and decision support tools in NATO’s ability to increase its operational tempo. Specifically, these included the use of synthetic agents for dynamic planning and the use of automated reasoning to foster execution processes. The ET argues that such tools would enable the military to deliver effects at an optimum tempo, thus staying inside an adversary’s decision‑action cycle.

Collection and Management of Data for Analysis Support to Operations (RTG SAS‑111), 2015 – 2018

This RTG identifies data issues affecting NATO operations and recommends means by which NATO deployed HQs and forces could enhance their decision‑making.

RTG SAS‑111 is the only activity not led by IST in the field of C2. It provided recommendations on improving deployed NATO headquarters’ ability to collect and manage the data required for analytical support to operations. It stressed the importance of accurate geospatial and environmental data with fine granularity for training and conditioning autonomous systems with AI. The RTG demonstrated the usefulness of ML in data collection and filtering for strategic electronic warfare. Like other activities, the RTG concluded that sharing raw and curated data among various branches of a military HQ, among NATO entities, or with non‑NATO entities is critical for trust and verification.

Big Data and Artificial Intelligence for Military Decision Making (RSM IST‑160), 2017 – 2018

This RSM identifies areas of common interest associated with AI and Big Data within NATO decision‑making and strategy to align various research endeavours.

The meeting provided a forum for operational experts and scientists to identify areas of interest and concerns associated with utilising Big Data and Artificial Intelligence for information superiority within NATO and establish a common road map. In summary, this RSM identified elements of information analysis, architectures, training and visualisation and information warfare necessary for optimal decision support in information‑dense environments. Furthermore, the specialists presented, discussed future technologies, and developed TAPs for future panel‑overarching research activities. The general conclusions regarding C4 are following.

• The current state‑of‑the‑art AI can provide high‑quality support only if entities of interest lie within a narrow scope. Multi‑modal cases can be addressed by treating each aspect separately. That is because AI development lacks an overreaching theoretical basis and is explored ad hoc. Militaries should explore how to exploit AI tools used in the commercial sector.

• The quantity of code and complexity has exceeded any formal analysis capability; thus, verifying and validating statistically adaptive AI is challenging. The intervention of the operator or decision‑maker is desirable, but the penalty is that the involvement may influence the reaction time. Advanced interfaces are needed to mitigate this weakness.

• The data’s volume and quality for training and establishing the information is paramount. Data collection capability needs to be fully engaged to keep abreast of adversaries. In some instances, data for training will be rare. Data derived from modelling can be employed to fill the void. Data Farming was proposed as an alternative. Nevertheless, uncertainty in the results needs to be recognised, and third parties injecting false and corrupted data will influence a measure of confidence expressed as the output of AI tools.

• Researchers must address the ethical and legal aspects of data sharing. This will enable learning‑based AI to obtain intelligence associated with establishing nodal connections by training the tools on known data. In principle, the more data, the higher the reliability. Adversaries with fewer restrictions on access to data could readily achieve information superiority.

Mission‑Oriented Research for AI and Big Data for Military Decision Making (RSM IST‑173), 2018 –2020

This RSM provides the first instantiation of an S&T roadmap for AI and Big Data for military decision‑making to address the challenges of NATO. Unlike its predecessors, this RSM organises mission‑oriented research.

The activity builds upon RSM IST‑160 to align various activities seeking solutions for legal, operational, technical, implementation and project management questions using AI and Big Data for military decision‑making. The RSM IST‑173 is the first to employ a mission‑oriented research approach. Pursuing this approach, it mapped critical aspects of the AI infrastructure for NATO‑wide development, the evaluation of AI technology, and the scalability of AI‑driven systems. In addition, the RSM summarised the following to foster AI and Big Data‑based decision‑making:

• Improve the understanding of AI, Big Data and Data Science by educating military decision‑makers on the status of the potential of AI and its drawbacks for NATO.

• Increase trust in AI and raise awareness of AI’s legal, ethical, and philosophical challenges and Big Data for NATO.

• Facilitate embedding the roadmap in military practice via military exercises and training.

• Improve the availability and sharing of resources for AI training across NATO members. That is, copious amounts of annotated, labelled, and validated data, computing resources and evaluations.

• Conduct additional research into the Human‑Machine interaction and build trust in these technologies.

• Estimate the cost implications of introducing Artificial Intelligence.

• Increase machine understanding of operational environment metrics.

• Develop technical solutions to deal with trust in data sharing. For example, automatic curation and data sanitation, detection of circular reporting and data incest, assessment of credibility and reliability of information and sources, countering analysts’ cognitive biases and identifying narratives.

Big Data Challenges: Situation Awareness and Decision Support (RWS IST‑178), 2019 – 2020

This RWS provides a platform for scientists to discuss decision‑making challenges associated with the proliferation of data on social media coupled with the Internet of Things.

Provides fertile ground for different RTGs to present, discuss and share their work on AI and Big Data challenges in decision support. The RWS concluded that raw data is expanding rapidly, requiring AI assistance for decision‑makers to comprehend the information. Human involvement remains essential; thus, smart interfaces play a significant role in assisting the users. Trust in the information displayed is paramount but recognised as difficult to achieve. Defining the source would help, as would accepting data only from a known repository, but this may be too restrictive. Identifying corrupted data remains critical, particularly for training neural networks and DL algorithms. Social media, coupled with the internet of things, permits adversaries to influence the opinion by disseminating fake news with bots relaying the data as if it were a true actor. Another tactic is to drive a wedge through the community to create division. NATO needs to consider a proactive stance.

Autonomy to Accelerate the Intelligence Cycle (SAS‑ET‑EG), 2019 – 2020

This ET explores the potential of AI‑enhanced autonomy in supporting the intelligence cycle. The activity examines which level of autonomy is required in each military context.

SAS‑ET‑EG seeks to lay the foundation for integrating autonomy in the intelligence cycle. The ET will achieve this by analysing the potential of the different autonomy levels in the intelligence cycle. For example, this may consider collecting, processing and fusing data from multiple sources and crosschecking all data sources to reduce deception in data processing, including pattern recognition and classification of intelligence products, as well as secure dissemination and interoperable sharing of data within NATO.

HUMAN‑MACHINE SYMBIOSIS (HMS)

“Optimising human and machine capabilities in teams that maximise strengths and mitigate weaknesses is essential.”6

6 Human Machine Teaming, UK MoD Joint Concept Note 1/18. UK Ministry of Defence, Development, Doctrine and Concepts Centre.

May 2018

HUMAN‑MACHINE SYMBIOSIS (HMS)

OVERVIEW

New forms of Human Machine Symbiosis emerge at all levels and across all types of military activities. Aspects can range from human‑machine interfaces and human‑centric AI to augmented decision‑making abilities. Deep Learning further extends the range and depth of human‑machine interactions.7 It gave rise to a spectrum of innovative machine implementations, including robots, autonomous vehicles and smart advisors that act autonomously. These machines with cognitive planning and reasoning capabilities can manage uncertainty, act in unpredictable situations, and carry out creative tasks.

HMS connects intelligent autonomy with the decision‑making responsibilities of human operators to maximise mission effectiveness and increase situational awareness. Since the trend is towards increased use of autonomous systems, the time is ripe to address the critical human factors issues involved. Future battlefields will be rife with manned and unmanned platforms that will overwhelm the operators’ ability to conduct their assigned missions effectively unless technologies intervene to alleviate their multi‑tasking requirements. On top of that, AI‑based agents have the potential to function as personal advisors, efficiently fuse available information and enhance the ability of operators to make decisions and manage human‑machine teams.

Interactions between humans and automated processes are a multi‑faceted topic that requires collaborative and inter‑disciplinary research to counteract the risks like trust and responsibility in hybrid human‑machine ensembles.

7 http://www.horizon‑observatory.eu/radar‑en/index.php

Research Conclusions

• Advances in AI‑enabled control of multiple UxVs (swarms) are increasingly automated, and the operator’s role is becoming supervisory.

• AI will be inherent in human‑machine systems as the prerequisite for optimal situational awareness.

• UxVs distributed AI is essential for the stability of communication and coordination between automated platforms and the operator.

• Artificial Agents increase human factors related variables and improve the operator‑to‑vehicle ratio in a military environment.

• Hybrid Human‑AI teams’ strengths and weaknesses are supplemented and compensated when participants function as mutually adaptive (learning) agents.

• Modes of reasoning developed by AI systems and operators have a considerable risk of not being aligned and mutually understandable.

• AI’s inability to infer context derives from the lack of annotated ground truth data for military applications of autonomy.

• The military sector must remain knowledgeable and seize opportunities to influence the utility of commercial applications.

Figure 3: Human‑Machine Symbiosis (HMS)(Credit: iStock)

Research Challenges

• Interactions between humans and automated processes, such as task distribution in controlling multiple UxVs.

• Standardisation of mission management practices and control strategies for supervisory systems.

• Specialised training for the commanders and operators.

• Balancing the allocation of artificial cognition and human cognition in missions and phases.

• Better, compact Human‑Machine Interfaces that improve mission effectiveness and increase verification rates.

• Explainability and verifiability of AI applications in human‑machine systems.

• Co‑learning in hybrid Human‑Machine teams.

COMPLETED RESEARCH

The HFM TP conducted most of the activities in this HMS cluster. The activities include RTG HFM‑078, RTG HFM‑170, RSY HFM‑217, RTG HFM‑247, RSY HFM‑300 and HFM‑ET‑178 HFM’s activities focused on researching topics of supervisory control of UxVs and many aspects of meaningful human control. Three exceptions were managed by different TPs, which shifted the research focus. First, SAS‑ET‑BX conducted by SAS focused on autonomy/robotics. Second, RSM SCI‑296 conducted by SCI provides more details on the system‑level topics. Finally, HFM‑AVT‑ET‑185 was a cross‑panel activity that investigated pilot‑aircraft interactions.

Unmanned Military Vehicles: Human Factor Issues in Augmenting the Force (RTG HFM‑078), 2002 – 2006

This RTG was an early precursor activity aiming to increase NATO´s operational effectiveness through effective force augmentation with UxVs. It did so by assembling current research and considering key factors affecting HMS, such as military relevance, theoretical frameworks, systems‑of‑systems, artificial cognition and cooperative automation, controls interfaces, and human‑automation integration. While some of the findings might be considered outdated, this RTG provided a single point of focus for identifying, prioritising, and addressing human factors challenges associated with UxVs.

Supervisory Control of Multiple Uninhabited Systems: Methodology and Enabling Human‑Robot Interface (RTG HFM‑170), 2008 – 2011

This RTG is the foundation of the HMS cluster of activities. The activity focuses on enabling a single operator to control simultaneously multiple UxVs with varying degrees of autonomy. To do that, the RTG develops and demonstrates proper supervisory control system methodologies, interface design practices and concepts for such operations.

RTG HFM‑170 identified the paradigm in which the operator’s role is becoming supervisory in nature, overseeing the automated activation of planned events, and managing unexpected changes that influence the automated mission plans. Considering control of multiple UxVs simultaneously and their capability to make certain decisions independent of operator input and pre‑defined mission plans constitutes a new set of challenges. To address these challenges, the RTG:

• Demonstrated the self‑organisation and protection capabilities of multiple autonomous ground vehicles simulating air vehicles using computer simulations. Supervisory control of multiple autonomous vehicles can be pursued by studying aspects of human interfaces and the basics of self‑organising and protecting autonomous control. Distributed computing power is essential for self‑organisation, overall system stability and effective communication. AI can alleviate issues with increasing sensory demands requiring a complex vision system. Further research is needed relating to more robust control station interfaces.

• Proved that swarm intelligence is a promising approach for multiple UxVs control in terms of algorithmic performance and robustness. Human Factors and especially human‑machine communication and interaction were properly adapted. However, this RTG has shown a solid requirement for adapting command and system feedback representations to facilitate operators’ work while maintaining system capabilities. In short, support for adopting commands and meaningful interaction via better design and modality for man‑machine interfaces is needed.

• Presented a generic approach to develop a knowledge‑based assistant system enabling the deployment of multiple UAVs as remote sensor platforms in military helicopter missions. The rationale is that multi‑UAV operators’ control strategies are highly individual, resulting in too little training on the job or underdeveloped procedures.

• Examined distributed intelligence using swarm technologies and centralised intelligence using an intelligent agent as an intermediate supervisor. Hybrid supervisory systems have clear advantages when controlling multiple UxVs, and artificial cognition‑based agents can improve the operator‑to‑vehicle ratio in a military environment.

Robots Underpinning Future NATO Operations (Analysis of Human Machine Interfaces) (SAS‑ET‑BX), 2011 – 2012

This ET examines the broader role of robotics in military operations, focusing on the larger cognitive framework and the applicability of AI control and perception tools.

SAS‑ET‑BX also identified the need for further research on the role of robotics systems due to their growing use in military operations and asked questions such as how to reassess their advances in NATO operations. The ET outlined the following needs:

• To develop perception algorithms applicable in the defence and embedding them into the cognitive framework.

• To use perception, self‑localisation and navigation capabilities and building up a proper machine representation of the operating environment.

• To homogeneously team unmanned robots to boost the capabilities and reliability of a single robotic system.

• To incorporate humans as operators or ordinary team members in the system, sharing common knowledge with robots and assisting each other in a complementary way.

• Computational simulations are suggested to test design methods and technologies leading to scalable solutions.

Supervisory Control of Multiple Uninhabited Systems – Methodologies and Human‑Robot Interface Technologies (RSY HFM‑217), 2011 – 2013

This RSY disseminates the results of RTG HFM‑170 as well as addresses research gaps. These include task distribution in human‑machine systems and reciprocal interactions between humans and automated processes.

RSY HFM‑217 built upon RTG HFM‑170 and disseminated the results and lessons learned from respective technical demonstrations. This RSY brought together representatives of the research and operational communities to present technical demonstration results and review progress in

HMS. In addition, reciprocal interactions between humans and automated processes, such as task distribution in controlling multiple UxVs matching with a real‑world scenario and the importance of consistency in user interface design decisions around NATO, were brought to attention.

Human‑Autonomy Teaming: Supporting Dynamically Adjustable Collaboration (RTG HFM‑247), 2014 – 2018

This RTG shifts the research focus to Human‑Autonomy Teaming (HAT). The activity identifies and demonstrates teaming methodologies and interface design practices that increase the performance of human‑machine systems. RTG HFM‑247 highlights the importance of compensating strengths and weaknesses and identifies the potential risk of mutual comprehension in human‑machine interactions.

RTG HFM‑247 addressed the research gap identified in previous activities and shifted the research focus from supervising uninhabited systems to multi‑faceted, collaborative, and reciprocal interactions between human and automated processes. Human‑machine interactions are based on interdependency management, coordination, conflict resolution, team mutual and self‑monitoring and evaluation. To understand stress levels and accordingly adapt strategy, autonomous systems must be equipped with emotion detection and assistance with emotion coping. Mutual comprehension and a perfect balance of supervision and automation are mandatory to compensate for strengths and weaknesses in hybrid teams. To achieve this state, AI applications in machine systems should be explainable, and the hybrid human‑machine teams should be able to improve their performance by co‑learning.

Autonomy from a System Perspective (RSM SCI‑296), 2016

This RSM provided a perspective outside the HFM panel and, in doing so, identified areas where NATO should increase S&T focus. Due to the pervasive nature of autonomy, it pulled knowledge from all technical areas relevant to autonomy, including AI and ML.

RSM SCI‑296 tackled the challenge of overcoming the limitations imposed by the inability of AI to infer context. Hence, assessing, and discerning what information is essential and what is not. There is a lack of annotated ground truth data for military autonomy applications, as required, for example, to train and validate machine‑learning algorithms. AI will be inherent in many human‑machine systems, as the requirement

for shared situational awareness will require new paradigms for the design from a system’s perspective. The design must accommodate adaptation to evolving circumstances, the associated training and verification and validation of the human‑machine team, as well as appropriate consideration of security and risk.

Human Autonomy Teaming (RSY HFM‑300), 2018 – 2019

This RSY disseminates the results and lessons learned from RTG HFM‑247. In addition, the RSY explores AI‑based data fusion in situational awareness applications and the distribution of cognitive functions throughout mission phases.

RSY HFM‑300 addressed human‑autonomy teaming from the perspectives of the overall system, built upon RTG HFM‑247 and disseminated the results and lessons learned via presentations and demonstrations. Furthermore, it examined some aspects of HMS already mentioned in SAS‑ET‑BX, specifically potential applications of AI in increasing situational awareness and the potential of increasing efficiency by precisely allocating cognitive functions throughout the mission phases.

Meaningful Human Control Over AI‑Based Systems (HFM‑ET‑178), 2018 – 2019

This ET examines potential ways to expand on human‑autonomy teaming design patterns, specifically by identifying essential features of such systems.

HFM‑ET‑178 addressed suggestions mentioned during RSM SCI‑296 and HFM‑247. It clearly defined the task to develop a definition of features, scope and the key drivers that contribute to MHC for broader NATO use. Critical factors include human‑machine cooperation and interaction and how human‑machine teams will train. Most current human‑machine interface designs are for low‑grade autonomous systems. Specific designs for highly autonomous systems where the interface and communicative interactions will have a more significant impact on MHS than a system’s physical capabilities are needed.

Air Vehicles Crew’s Neuro‑Psychophysiological Based Real‑Time Stress Monitoring for Human Machine Interfaces Workload Evaluation Enforcing Mission Execution (HFM‑AVT‑ET‑185), 2019 – 2020

This ET focuses on examining Human‑Machine Interface interactions between pilot and aircraft.

HFM‑AVT‑ET‑185 examined the latest developments in the state‑of‑the‑art interactions between pilot and aircraft. This ET rethought human‑machine interaction based on the latest advancements in AI and neuroscience that opened the possibility of generating an objective methodology. According to the ET, well‑defined and validated standards, and auxiliary systems such as AI and DL that are easily installable and usable on military aircraft improve operational use and training effectiveness.

“You can have data without information, but you cannot have information without data.”

ISR – DATA FUSION

OVERVIEW

“Hard and Soft Data Fusion” combines incomplete and imperfect pieces of mutually complementary information from various sensors and non‑sensor sources to understand underlying real‑world phenomena or events. Typically, this insight is either unobtainable from a single source of information or results exceed that produced from a single information source in accuracy, reliability, or cost. Appropriate collection, registration, alignment, filtering, analysis, integration, exploitation of redundancies, quantitative evaluation, and proper visualisation are part of Data Fusion and the integration of related context information. Data Fusion is evolving rapidly and is present across the whole spectrum of defence and security systems. In practice, the Alliance’s operations rely on the fusion of copious amounts of sensor data and context knowledge for a network of disparate Intelligence, Surveillance and Reconnaissance (ISR) assets such as sensors, sensing platforms, human intelligence, and networking elements. Recent advances in Artificial Intelligence (AI) and Machine Learning (ML) make possible the development of powerful multisensory fusion engines that will constitute the core of NATO’s situational awareness capabilities.

Research Conclusions

• Enabling AI techniques into a sensing framework can lead to reduced costs, improved accuracy, and enhanced robustness.

• Natural language processing algorithms based on AI techniques allow helpful contextual information to be extracted from text.

• Machine Learning (ML) is applicable for surveying urban areas and tracking small, manoeuvrable targets.

• Bayesian approach is an effective solution to extended object and cluster tracking

Research Challenges

• Input/training data changes can compromise AI learning approaches. Therefore, their robustness is critical to safeguard them from adversarial attacks.

• Currently available AI methods are slow to train, thus slow to adapt to rapidly changing threats, making them unsuited to use in volatile military operations.

• As massive quantities of labelled data are needed to train AI, it is essential to record all available data and augment it with high‑fidelity simulated data.

• Better understanding of how to convert open source or commercially available algorithms for military purposes.

• Enabling open source or commercially available algorithms to train on data relevant to military applications.

• Simultaneous development of AI techniques with large sets of annotated data relevant to military operations.

• Establishment of a reference database containing data sets to support the training of future ML and DL algorithms for ISR.

• Enhancing interoperability and development of common standards, metrics, and best practice for joint NATO activities on data fusion for ISR.

• Increasing the explainability of algorithms using AI techniques to users across the chain of command.

Figure 4: ISR Data Fusion (Credit: iStock)

COMPLETED RESEARCH

The cluster of research focusing on Data Fusion for ISR contains a diverse range of activity types. The research portfolio started with precursor activity RLS SET‑157 in 2010, introducing the necessity of fusing data in modern distributed sensor networks. In late 2014, precursor activities RSY IST‑SET‑126 and RTG SET‑218 provided a much‑needed interdisciplinary forum that stimulated further research. RLS IST‑134, RLS IST‑155 and RSM SET‑262 are core activities that, between 2015 and 2018, examined an inventory of various data fusion techniques in the context of individual military application challenges. From 2018 to 2020, AVT‑ET‑204, SET‑ET‑107 and SET‑ET‑110 became the latest additions to this cluster. They outlined more nuanced, contemporary ISR challenges such as advanced computing, wide‑area surveillance, and system interoperability.

Multisensor Fusion: Advanced Methodology and Applications (RLS SET‑157), 2010 – 2012

The present RLS is a precursor activity that introduced distributed sensor networks and sophisticated data fusion methods.

RLS SET‑157 presented state‑of‑the‑art data fusion technology and its applications, thereby increasing awareness of its value to the NATO scientific communities. For example, the RLS concluded that Bayesian techniques could serve in tracking extended objects or collectively moving target clusters in many military applications. The RLS also argues that NATO’s decision‑makers have access to vast amounts of data. However, data fusion among streams is compulsory to effectively leverage the volume of information in real‑world applications so as not to overwhelm decision‑makers.

Information Fusion (Hard and Soft) for ISR (RSY IST‑SET‑126), 2014 – 2015

This RSY is a precursor activity, which provided an interdisciplinary forum to discuss research and highlighted challenges and opportunities in fusing soft and hard data into actionable intelligence.

Information Fusion (Hard and Soft) for ISR (RSY IST‑SET‑126) provided an interdisciplinary forum for research scientists, military experts, and system engineers to present state‑of‑the‑art research in various aspects of military data fusion. The vast amount of information available on social media dictates that automation is required to find weak signals and interconnections in a veritable haystack of background noise.

The RSY highlights the importance of fusing soft and hard data into actionable intelligence and

exploiting the results to support decision‑making and planning. Among many examples, the RSY presented machine‑learning tools to monitor moving vehicles and detect anomalies. As the inference drawn from the data is context‑sensitive, combining parametric data with subjective information was problematic. Furthermore, it is easy to confuse the algorithms and data collection by poisoning the data by creating false online content. A measure of confidence in the derived information should be expressed to assist the decision‑makers. The RSY echoes the need for further research on advanced algorithms emulating human cognitive processes.

Interoperability & Networking of Disparate Sensors and Platforms for ISR Applications (RTG SET‑218), 2014 – 2017

This RTG is a precursor activity which sought to achieve coalition interoperability of disparate ISR sensors from different nations and develop NATO STANAGs for integration and interoperability of the ISR assets to increase military effectiveness.

SET‑218, despite not addressing the topic of AI techniques directly, this activity considered challenges associated with their utilisation in ISR. As a result, the RTG focused on the research gap in interoperability frameworks, architectures, standards, metadata representation, and data and information sharing policies.

The RTG concludes that all ISR approaches need a recalibration on AI techniques, particularly the analytical processes conducted at the sensor level. This would relieve communication bandwidth to avoid overwhelming C2 networks and accelerate relevant information transmission to preserve better autonomy of the standalone sensors by alerting only when relevant situations append.

Advanced Algorithms for Effectively Fusing Hard and Soft Information (RLS IST‑134), 2015 and Advanced Algorithms for Effectively Fusing Hard and Soft Information (RLS IST‑155), 2016 – 2017 These RLSs disseminate knowledge about methodologies and algorithms for hard and soft fusion and discusses the integration of context information in data fusion systems. In addition, the RLSs highlighted challenges in defining the context for soft data.

Advanced Algorithms for Effectively Fusing Hard and Soft Information (RLS IST‑134) and Advanced Algorithms for Effectively Fusing Hard and Soft Information (RLS IST‑155) addressed a wide range of opportunities and challenges associated with advanced algorithms and data fusion processes in ISR. TLS IST‑134 and RLS IST‑155 concluded

that accessing the underpinning methodology and applying the inventory of various data fusion techniques to solving individual military application challenges is very challenging.

Contextual knowledge has been pointed out as a critical element in reasoning about the properties and relations of the entities of interest in several ISR domains, particularly computer vision and anomaly detection. Bayesian approaches emerged as potential solutions to extended object and cluster tracking among many presented applications. In addition, natural language processing algorithms enhanced by AI techniques and text analytics allow much useful contextual information gleaned from the text. The RLS also suggested developing a flexible data storage infrastructure to enable contextual knowledge fusion processes based on pruned and categorised data.

Artificial Intelligence for Military Multisensory Fusion Engines (RSM SET‑262), 2018

This RSM fosters further research and builds synergies in the research area of AI‑inspired sensor informatics in military applications. Critical is the role of collecting and managing Big Data as means of training for AI algorithms.

Artificial Intelligence for Military Multisensory Fusion Engines (RSM SET‑262) presented the latest research on advanced methodologies and algorithms of AI‑inspired sensor informatics. RSM SET‑262 concluded that Multisensor Fusion Engines are the backbones of NATO’s situational awareness capabilities, claiming that the technology will disrupt friendly and adversary forces.

Enormous quantities of real‑time sensor data present opportunities for new data fusion algorithms optimised for big data processing. While AI‑based data analytics are proficient in detecting anomalous behaviour and tracking extended and group objects, ML and DL techniques are promising methods capable of processing conflicting multisensor information and enabling easily scalable solutions. Furthermore, they can significantly improve the performance of hyperspectral imaging.

The RSM highlights that minor changes to the input data can compromise Deep Learning Neural Networks. Therefore, it is critical for any AI method to be robust against adversarial attacks. Furthermore, any output from algorithms using AI techniques, whether based on the ML or DL approaches, must be explainable to end‑users. Different explanations will be needed for other users across the chain of command based on the context. Military applications have less data than commercial counterparts do. Therefore, it is essential to record all available labelled data and augment it by injecting it with simulated data where lacking. Significant AI challenges are finding the optimised structure and architecture, reducing resource consumption, assessing confidence in the results, effectively using the available bandwidth, and minimising the need for labelled data for DL.

Data Fusion and Assimilation for Scientific Sensing and Computing (AVT‑ET‑204), 2019 – 2020

The ET provides the NATO community information on state‑of‑the‑art data fusion and assimilation methodologies for military applications. Further, the ET stressed the role of AI as enabling force behind low‑cost instrumentation of multi‑sensor data.

Data fusion and assimilation for scientific sensing and computing (AVT‑ET‑204) provided state‑of‑the‑art information on data fusion methodologies in military contexts and devised experiments focused on the optimised deployment of sensors. AVT‑ET‑204 concludes that a network of multiple inexpensive sensors integrated with advanced algorithms could bridge several gaps in traditional scientific instrumentation and facilitate AI techniques integration into a sensing framework with the various aims of reducing costs, improving accuracy, and enhancing robustness.

High‑fidelity computer simulations are increasingly influential in marine, ground, and air vehicle design decisions. As the sophistication of the models grows, so does the need for experimental data to develop new models and validate existing ones. Potential applications have a complementary nature. On the one hand, low‑cost multi‑sensory data is collected to validate existing computational models, which provide helpful information for identifying optimal sensor deployment. On the other hand, low‑cost sensors and multi‑sensor data are assimilated into computer simulations to develop new computational models such as building digital‑twin components for various systems or modelling environmental dynamics.

Automated Scene Understanding for Battlefield Awareness (SET‑ET‑107), 2018 – 2019

This ET aims to develop a strategic plan for standardising data and evaluating metrics for joint NATO activities in data fusion, image processing and sensing. According to the ET, the key is the simultaneous development of AI techniques with large sets of annotated data.

Automated Scene Understanding for Battlefield Awareness (SET‑ET‑107) aimed to develop common standards and metrics for joint activities in data fusion for ISR. SET‑ET‑107 also focused on identifying established algorithms from industry for potential use in military scenarios.

The ET concludes that the key to automated scene understanding for battlefield awareness lies in the simultaneous growth of AI techniques and the large sets of annotated data relevant to the military. The ET further suggests modelling and simulations augment collected data sets to improve the training of algorithms. In addition, participating nations should share collected and simulated data to advance the development of AI techniques with military relevance. This is important because military data do not generally exist in sufficient quantity. Other paths could include transitioning existing civil algorithms to military systems by making the military data available to selected private industry leaders.

Machine Learning for Wide Area Surveillance (SET‑ET‑110), 2018 – 2019

This ET aims to examine the application of ML approaches to improve the performance and versatility of wide‑area sensors to provide enhanced ISR capabilities. Establishing a reference database for ML development is suggested as the desired outcome of this activity.

Machine Learning for Wide Area Surveillance (SET‑ET‑110) focused on improving the provision of wide‑area surveillance through ML techniques. SET‑ET‑110 argues that despite developments in sensor technology, the continued evolution of the problem space raises challenges to achieving robust surveillance. For example, the migration of sensors to high‑altitude platforms leads to surface clutter interference resulting in the degradation of radar performance. As a result, surveillance of urban areas coupled with the desire to detect and track small manoeuvrable targets is exceptionally challenging.

Conventional detection and tracking approaches perform poorly under these new cluttered environments. However, the ability of ML techniques to accurately describe and model the statistical processes associated with clutter and target signatures are applicable in the surveillance of urban areas coupled with detecting and tracking small manoeuvrable targets in support of activity‑based intelligence.

“Computational

ISR – COMPUTATIONAL IMAGING AND COMPRESSIVE SENSING

OVERVIEW

Unlike conventional optical systems, where an image is captured by optically focusing the incident light at each of the millions of pixels in a focal plane array, computational imaging refers to image formation techniques that use digital computation to recover an image from an appropriately multiplexed or coded light intensity measurement of the scene. In this case, the desired aspects of the scene can be selected at the time of image reconstruction, allowing greater flexibility for Electro‑Optical and Infrared (EO/IR) systems.

Compressive sensing involves capturing a smaller number of specifically designed measurements from the scene to recover the image from specific scene information computationally. The demand for high‑quality, high‑performance radar and EO/ IR products is vital in military applications such as target acquisition at long ranges and over large search areas. Compressive Sensing (CS) has the potential to reduce dramatically the system cost and data volume required while delivering the same or greater performance as a conventional design. For defence applications, CS holds the promise to gather more information at greater ranges to allow the warfighter to react faster to threats and gain a tactical advantage.

Research Conclusions

• Approaches based on AI techniques provided remarkable results in big data analysis, especially in applications related to image analysis and computer vision.

• Approaches based on AI techniques show potential in terms of physics‑based signature modelling to target detection in an operational context.

• The activities demonstrated that AI techniques lead to faster event detection and automated target recognition in computer vision.

• Compressive Sensing combined with AI techniques shows promise in extracting representations from sparse data and could potentially be able to fill in the gaps through prediction.

Research Challenges

• To succeed in classification or segmentation problems, AI approaches need a large amount of annotated data.

• AI techniques, particularly Deep Learning, require a large amount of annotated training data to account for spatial and environmental complexity.

• Labelling large amounts of data is a considerable effort and might be infeasible for many military tasks.

• The future use of CS suffers from an unclear path towards verification and validation that hinders its uptake for operational applications.

• Enhancing interoperability via joint data collection and exploitation is required to advance computational imaging.

• Many publications with few practical systems imply that computational imaging may still be around the “peak of inflated expectations.”

COMPLETED RESEARCH

The research portfolio focusing on Computational Imaging (CI) and Compressive Sensing (CS) contains diverse types of activity. This cluster of activities extended the knowledge and understanding from the previous chapter Data Fusion by focusing on reconstructing information from sparse sources in scenarios relevant to military operations. The SET TP manages all the activities in this cluster. Despite the coherency and complementarity of all referenced activities, there are slight idiosyncrasies that differentiate them. RTG SET‑232 and RSM SET‑265 focused on utilising CI and CS techniques to enhance Electro‑Optical and Infrared (EO/IR) imaging systems. SET‑ET‑119, as an addition to the two previous activities, explored the performance

Figure 5: ISR Computational Imaging and Compressive Sensing (Credit: iStock)

of such systems in military‑relevant tasks. RTG SET‑240 and RWS SET‑277 provided an overview of a variety of hyperspectral technologies, including those targeting CBRN threats. Finally, SET‑ET‑111 integrates CI and CS capabilities in novel radar applications.

Computational Imaging and Compressive Sensing for EO/IR Systems (RTG SET‑232), 2016 – 2019

This RTG assessed Computational Imaging (CI) and Compressive Sensing (CS) techniques for Electro‑Optical and Infrared (EO/IR) imaging sensors, as well as developed design concepts on how to apply them. In addition, the RTG highlighted non‑line‑of‑sight imaging and comprehensive field‑of‑view sensing as an optimal military application for respective techniques.

Computational Imaging and Compressive Sensing for EO/IR Systems (RTG SET‑232) provided a means to assess CI and CS techniques, designed concepts for their applications and conducted joint research activities. RTG SET‑232 concluded that CI and CS techniques have significant military potential when used for non‑line‑of‑sight and multiplexed imaging to gain wide field‑of‑view situational awareness. This capability is further enhanced by employing AI optics and image processing techniques in EO/IR systems. Furthermore, CI and CS offer options to reduce multispectral sensing systems’ increasing complexity and thus solve the optimisation challenges by moderating computational costs. The RTG also demonstrated the usefulness of Deep Neural Networks in recovering sparse signals and reconstructing low‑resolution to high‑resolution images. The key is the multitude of layers in Neural Networks, enabling them to learn complex feature representations.

The combined benefits of CI and CS capabilities have the potential to revolutionise imaging sensor technologies across a wide variety of NATO applications. However, further research is required to assess their impacts on military systems and identify gaps where current technologies are inadequate. The main bottleneck for CI and CS applications in military systems is the severe lack of annotated data sets relevant to military purposes. These are required to train the AI algorithms that underpin the CI and CS. Fostering interoperability, and cooperative research is necessary to advance the development of AI techniques for ISR scenarios. Similarly, NATO members should jointly conduct data build up and exploitation.

Exploitation of Longwave Infrared Airborne Hyperspectral Data (RTG SET‑240), 2016 – 2020

This RTG established recommendations for hyperspectral remote sensing technology for CBRN threat detection. Unsupervised Machine learning and Deep Learning methods fostered by robust data sets showed promise in tackling this challenge.

Exploitation of Longwave Infrared Airborne Hyperspectral Data (RTG SET‑240) significantly advanced understanding of hyperspectral target detection and associated phenomenology. This understanding has helped develop novel exploitation methodologies adapted to specific contexts. As part of the RTG, Unsupervised Machine Learning strategies were successfully applied to anomaly detection in terrain textures. Deep Learning approaches showed promise in achieving state‑of‑the‑art classification performance in varying atmospheric conditions experienced in a real‑world detection scenario. However, to succeed in classification or segmentation problems, where specific material categories are the output, Deep Learning methods need a large amount of annotated data. Labelling copious amounts of data is a considerable effort and might be infeasible considering the range of potential military applications and environmental conditions.

Compressive Sensing Applications for Radar, ESM and EO/IR imaging (RSM SET‑265), 2019

This RSM provided a platform for specialists to present and share state‑of‑the‑art knowledge and identify challenges and opportunities of using Compressive Sensing (CS) systems in various applications. The RSM concluded that DL and ML profoundly affect CS efficacy.

SET‑265 concluded that significant benefits of CS emerge when used to achieve capabilities that are not presently possible instead of being a replacement for conventional processing. This is best exemplified in the research on computational imaging, in which completely new means of imaging are beginning to arise. This includes using Deep Learning in infrared and hyperspectral machine vision, leading to faster event detection, and Deep Learning approaches in spatially controlled coherent illumination that can acquire spatially resolved imaging of non‑line‑of‑sight objects. From a radar standpoint, current efforts have focused on using CS to compensate for the loss of resources due to spectral and radar timeline management to meet conflicting goals. However, whether new capabilities emerge on the radar front remains to be seen.

CS has excellent potential to enhance capabilities under resource‑constrained conditions and lead to new sensing modes if the physical characteristics of the sensing environment are appropriately incorporated into the problem formulation. There has been considerable research on the theoretical attributes of CS, followed by a significant amount of simulation results. Serious consideration needs to go into how it can best be utilised on experimental fronts. Like any other AI technique, the future use of CS suffers from an unclear path towards verification and validation that hinders its uptake for operational applications.

Workshop on Phenomenology and Exploitation of Hyperspectral Sensing Within NATO (RWS SET‑277), 2019

This RWS conducted experiments and theoretical investigations to illustrate the benefits and challenges of enabling cognition‑based capabilities in radar systems. The RWS demonstrated that even though Ml and DL are promising in ISR applications, the techniques require substantial amounts of data.

Workshop on Phenomenology and Exploitation of Hyperspectral Sensing within NATO (RWS SET‑277) illustrated the benefits and challenges of enabling cognition‑based capabilities of radar systems. There was a particular focus on the developments in ML for physics‑based signature modelling and target detection.

RWS SET‑277 recognised Machine Learning as having the potential to capture the complexity of physics‑based spectral characterisation of materials necessary to address the spectral variability encountered in an operational context to achieve high target detection/identification scores.

The RWS underlined the importance of developing rigorous metrology for hyperspectral signature measurements. Reported results of phenomenology studies demonstrated the variability of spectral measurements for numerous factors such as the physical state of the observed material, the conditions of observation (geometry, illumination, and weather) and the instrumentation at hand. This is paramount as detection algorithms rely on accurate reference data.

Integrating Compressive Sensing and Machine Learning Techniques for Radar Applications (SET‑ET‑111), 2019 – 2020

This ET examines the potential benefits of integrating Compressive Sensing and Machine Learning techniques for sparse signal recovery in radar applications. Furthermore, the ET proposed utilising CS‑based generative models to fill in the gaps in measured data through predictions.

Integrating Compressive Sensing and Machine Learning Techniques for Radar Applications (SET‑ET‑111) aims to identify application areas where integrated designs with Compressive Sensing and Machine Learning components outperform state‑of‑the‑art techniques. This results from utilising ML and DL techniques to recover sparse signals. Finally, CS‑based generative models could be used to produce vast training data sets required for training algorithms, filling in the gaps in measured data by employing predictions from a compressed database. This CS‑assisted training strategy will significantly widen the scope of problems where AI techniques could be successfully applied. For example, the availability of large, collected radar data training sets in the military domain cannot always be guaranteed.

Assessment of EO/IR Compressive Sensing and Computational Imaging Systems (SET‑ET‑119), 2020 – 2021

This ET aims to conduct a performance assessment of computational imaging and compressive sensing systems for military‑relevant tasks.

Assessment of EO/IR Compressive Sensing and Computational Imaging Systems (SET‑ET‑119) aims to select candidate CI and CS systems to conduct experiments in field settings. The ET concluded that CI technology could enable optimised special‑purpose tactical sensor designs for specific military sensing, allowing defence planners to expand the application space of optical sensing where conventional imaging techniques are limited by physics or technology. In practice, CS has the potential to acquire an image with equivalent information content to a large format array while using smaller, cheaper, and lower bandwidth components.

ISR – COGNITIVE RADAR AND RADIO

“Applying the ideas of cognition to radar has the potential to usher in a new era of sensing, not just improving the performance of existing radar systems but opening up whole new capability areas.”10

10

ISR – COGNITIVE RADAR AND RADIO

OVERVIEW

The research on Cognitive Radar builds upon previous work on Artificial Intelligence to mimic our attributes of learning, memory, attention, and intelligence, all to make the radar “smarter.” Cognitive Radar integrates AI techniques across the whole spectrum of its sub‑systems, ranging from Advanced Algorithms, Multi‑Sensor Data Fusion, Hyperspectral Sensing, and Modelling to Radar Resource Management. Cognition is observing and understanding the change in the environment in which the radar works. This, in turn, allows the adaptation of the radar’s behaviour to the changing conditions based on the knowledge learned by its’ algorithms through observing the environment and the results of previous behavioural changes. In essence, it embraces the “perception‑action cycle” and the explicit generation and exploitation of memories and knowledge. The challenge in developing robust Cognitive radar systems is to prevent the wrong decision from being made during the cognition/ learning process.

On a practical level, the attention of Cognitive Radar should be focused on mission objectives to facilitate problem‑solving and goal‑oriented behaviour. For this purpose, Radar Resource Management (RRM) is used to manage the radar resources of electronically steered phased array radars for mission objectives.

Research Conclusions

• AI techniques enable ubiquitous cognition applicable to all radar systems and have the potential to usher in a new era of sensing.

• AI and ML are used at various levels of intelligent sensing, from detection and tracking to optimising decisions.

• Extension of cognitive techniques to distributed sensing is a natural way forward for modern radar systems.

• Cognitive Radio Networks using AI techniques can provide robust and efficient communications in highly dynamic environments.

• ML and AI techniques identify patterns and learn behaviours, which are essential knowledge for the decision‑making aspect of Cognitive Radar systems.

• Adaptive methods using AI for Radar Resource Management can optimise resource management at the signal level to assist human operators.

• It is safe to assume that potential adversaries have recognised the disruptive operational potential of intelligent radar systems and are developing options to counter them.

Research Challenges

• Cognitive Sensing can be tricked into learning bad habits if the algorithms are exposed to poisoned data sets during the training stage feedback loop.

• ML and AI require significant training data, which is always in short supply for radar applications.

• Enhancing intelligent ISR systems interoperability and integration in joint military operations.

• Limiting the training burden of the operator engaged in radar management by using augmented reality or a human‑machine interface.

• Improving human‑machine interfaces and explainability of AI algorithms to foster trust in Cognitive Radar systems.

Figure 6: ISR‑ Cognitive radar and radio

COMPLETED RESEARCH

The research portfolio focusing on Cognitive Radar and Radio has been evolving simultaneously with other activities in the ISR cluster, to which the research is also complementary. Cognitive Radar and Radio use and integrate advanced computational methods, including Data Fusion and advanced resource management. RTG SET‑182 is an early precursor activity that introduced the concept of cognition in radar systems during its investigation of signal management in 2011. This concept was further explored in the RLS SET‑216 from 2014 to 2017, which developed the idea of ubiquitous cognition. RTG SET‑227 conducted experiments and theoretical investigations on AI techniques as enablers of cognition in distributed sensing from 2015 to 2019. RTG SET‑223 focused on the aspects of resource management or optimisation on signal levels from 2015 to 2019. RTG IST‑140, as the only activity not managed by the SET TP in this cluster, researched efficient solutions for network management in terms of signal modulation. Finally, RSY SET‑241 aimed to diffuse knowledge accumulated by the activities to NATO stakeholders between 2016 and 2017.

Radar Spectrum Engineering and Management (RTG SET‑182), 2011 – 2014

This RTG is a precursor activity that developed experiments and models for exploiting the transmitter, receiver, and waveform design. The RTG was the first to identify the value of Cognitive Radar in improving optimal spectrum use.

RTG SET‑182 describes Cognitive Radar as a potentially very significant set of techniques in spectrum engineering and argues that there is much to gain from the intelligent allocation of resources in spectrum management. Furthermore, the RTG claimed that a universally agreed definition of cognitive radar and understanding its potential military benefits should be focal areas for further research. According to the activity, cognitive Radar’s characteristics are the perception‑action cycle and memory, updated by the information gained from the target scene. The perception‑action cycle means that the radar should dynamically adapt its transmitted waveform in response to its perception of the target scene.

Cognition and Radar Sensing (RLS SET‑216), 2014 – 2017

This RLS raised the awareness of the value of Cognitive Radar sensing by presenting the latest research. Discussions were based on fundamental concepts and potential military applications.

Cognition and Radar Sensing (RLS SET‑216)

overviewed the emerging topic of cognitive radar sensing. RLS SET‑216 concluded that applying the ideas of cognition to radar can usher in a new era of sensing, not just to improve the performance of existing systems but to open new capability areas. AI techniques enable cognition, which is ubiquitous and can be applied to all radar systems. Benefits range from sensitivity enhancements for improved tracking to sensing for autonomous guidance and navigation.

Cognition in many radar sensors is deficient or non‑existent. Human operators provide radar systems with cognition for all intents and purposes; however, human data capacity and reaction speeds are insufficient for modern radars’ decision challenges and timescales. The operator must communicate objectives and requirements effectively, and the radar system must provide the necessary information to justify the system’s decisions. Otherwise, the operator will not trust the radar. In the short term, we need to design better human‑machine interfaces and foster trust in the AI system that underpins cognition by increasing their explainability. However, for radar systems to realise their full potential, they must include a cognitive approach in the long term.

Cognitive Radio Networks – Efficient Solutions for Routing, Topology Control, Data Transport, and Network Management (RTG IST‑140), 2015 – 2017 This RTG sought to increase the robustness and efficiency of military communications via Cognitive Radio Network (CRN) solutions. The RTG demonstrated the potential of CRNs in automatizing network management and solving spectrum scarcity problems.

Cognitive Radio Networks – Efficient Solutions for Routing, Topology Control, Data Transport, and Network Management (RTG IST‑140) connected research efforts from the areas of autonomous frequency adaptation, cognitive radio, and heterogeneous tactical networks. RTG IST‑140 proposed CRNs as a robust solution that can autonomously react and act upon the changes in the spectral environment, thus optimising the use of the resources. To achieve these goals, CRNs should use AI and Machine Learning techniques, due to their learning capability to enhance route selection, topology control, adaptability, and automation. The insights from the RTG will also help define the specification and standardisation of a CRN waveform, usable in joint and combined military operations.

The research on CRN trust management has shown its importance for the authenticity of control information, which should not be based only on observation and measurement, but on a robust architecture framework. The behaviour of highly adaptive CRN systems will have to be constrained by appropriate policies, which need to be developed and evaluated as part of the introduction of CRN technologies.

Cognitive Radar (RTG SET‑227), 2015 – 2018

This RTG explored the benefits and drawbacks of cognitive processing in various experiments and simulations. The RTG focuses on AI and ML as underpinning techniques in radar cognition.

Cognitive Radar (RTG SET‑227) conducted experiments and theoretical investigations to illustrate the benefits and challenges of enabling cognition‑based capabilities of radar systems. Most of the participating nations supported research on Cognitive Radar. However, practical in‑service cognitive radar systems are a long way in the future. This RTG will allow a coherent view of opportunities and routes to exploitation, influencing the design of future radar systems. ML and other AI techniques such as Neural Networks will have a leading role to play in identifying patterns and learning behaviours, which are essential knowledge for the decision‑making, “perception‑action cycle” aspect of cognitive radar. The RTG proposed using ML for spectrum characterisation and awareness, and convolutional neural networks for signal identification. Therefore, AI techniques will be fundamental for cognitive radar effectiveness, specifically by characterising the situation and suggesting actions.

ML and AI require significant training data, which is in short supply for radar applications. As each scene that a radar faces provides a unique signal environment, geometry, and set of targets, it is imperative for AI techniques to incorporate both supervised learning from existing datasets and real‑time unsupervised learning as new scenarios emerge. Conventional ML/AI algorithms process data inputs and yield classification, which is easy to define. In Cognitive Radar applications, ML/AI algorithms are used to output a course of action. Thus, additional modelling is required to assess whether the proposed course of action yields improved performance. In contrast to a simple correct/incorrect assessment, the feedback in a cognitive radar application must include a figure of merit on the proposed course of action. This modelling must be included in the training stage to provide feedback to the algorithm.

Adaptive Radar Resource Management (RTG SET‑223), 2015 – 2019

This RTG mapped Radar Resource Management (RRM) techniques and their benefits in establishing a common understanding of long‑term capability requirements for ISR in the NATO community. The RTG stressed the role of automated RRM assisting operators, particularly on the signal level.

Adaptive Radar Resource Management (RTG SET‑223) examined the benefits and challenges of using Radar Resource Management (RRM) techniques in military operations. RTG SET‑223 also elaborates on conclusions from RLS SET‑216 and RTG SET‑227 by further exploring and examining optimal pathways to integrating more autonomy at the sensor level using AI techniques in RRM. The RTG states that RRM is, in essence, a beam scheduling challenge, where the radar surveys the fog of war to reduce the probability that an undetected target approaches a region of interest. Adaptive Radar Resource Management methods use AI techniques to determine the best course of action for the search beam schedule.

The motivation behind exploring AI and ML algorithms for applications in radar beam scheduling is the intuition that heuristics‑based scheduling algorithms may not be optimal and cannot cope with rapidly changing conditions. The position supported in this report is that an algorithm based on ML methods can incorporate many more features that describe the radar and the environment than a radar designer would typically consider, despite the latter’s expertise. The RTG concludes that management at the mission and situation levels can incorporate real‑time human input. However, management at the signal level will have to be conducted automatically because of the sub‑second time scales involved. Compared to non‑adaptive RRM, results show that adaptive RRM achieves better tracking performance against benchmarks and the same tracking performance against context targets while requiring fewer tracking resources.

9th NATO Military Sensing Symposium (RSY SET‑241), 2016 – 2017

This RSY focused on diffusing information about state‑of‑the‑art sensing technologies. The RSY fostered interoperability within NATO and stimulated joint programmes.

The 9th NATO Military Sensing Symposium (RSY SET‑241), among many sensing‑related topics, this RSY introduced a novel method of RRM based on ML techniques. Unlike conventional methods, which enumerate all viable solutions via branching and bounding, the proposed method utilised ML to eliminate unlikely possibilities, reducing the time spent optimising radar resource management. The RSY proposed an approach to detect weak target signals in the dense clutter environment. The approach builds on compressed sensing sparse signal processing via ML techniques integrated into a Bayesian learning framework for radar detection. ML was used at different levels, from optimising decisions to detection and tracking. Beyond ML, the RSY demonstrated DL techniques that tracked and classified military platforms with a high success rate in various environments.

“The success of NATO forces depends on understanding the environment they are operating in and using that knowledge to make accurate decisions. In many cases, environmental knowledge is incomplete or contradictory, which can lead to poor operational and tactical decisions.”11

11

ISR – MARITIME DOMAIN

OVERVIEW

The Centre for Maritime Research and Experimentation (CMRE) contributes to this critical area by developing cost‑effective integrated solutions that provide NATO militaries with enhanced ISR knowledge. In the field of AI, CMRE contributes mainly but not exclusively through the following three programmes:

• The CMRE Autonomous Naval Mine Counter Measures (ANMCM) programme provides insights about current and upcoming MCM technologies with operationally relevant measures, such as detecting and classifying high‑resolution images and other informative artefacts.

• The CMRE Environmental Knowledge and Operational Effectiveness (EKOE) programme focuses on observing, understanding, describing, and forecasting the marine environment for NATO nations. Beyond oceanography and acoustics, EKOE focuses on analysing the 3D effects on acoustic propagation associated with the variability of sound propagation.

• The CMRE Cooperative Antisubmarine Warfare (ASW) programme transforms NATO’s antisubmarine warfare (ASW) strategy from relying on conventional assets to achieving ASW dominance through affordable, intelligent autonomous ASW networks with platforms fielding both active and passive sensors.

Research Conclusions

• Passive sonar surveillance system with embedded intelligence enables the continuous real‑time capability to detect, localise and classify vessels.

• Approaches based on Machine Learning, Deep Learning, and Neural Networks can analyse high‑fidelity sonar data or images to detect and classify unexploded underwater ordnance.

• Neural Networks can enable the assimilation of underwater acoustic data into ocean predictive models to improve oceanographic forecasts.

• Machine Learning approaches of variational probabilistic inference are a robust method for autonomous data‑driven navigation or tracking of platform motion.

Figure 7: ISR ‑ Maritime domain

COMPLETED RESEARCH

The research portfolio focused on AI applications in the maritime domain is being managed predominantly by the CMRE. The integration of intelligent capabilities in maritime systems has been an active research area since 2014, undergoing a dramatic increase in activities since 2019. The research portfolio splits into the following three main categories. First, research focused on enhancing the automated targeting, recognition, and classification capabilities of underwater objects (e.g., FRATRE for French Automatic target recognition ATR MINES, Automatic Object Classification with Active Sonar, Sonar‑Based Deep Learning for Underwater UXO Remediation). Second, oceanography research seeking to provide detailed information about the environment by exploiting sonar data in novel ways (e.g., Texture‑Based Seafloor Characterisation Using Gaussian Process Classification ISR SEABED OCEANOGRAPHY, Coupled Ocean‑Acoustic Variational Data Assimilation). Last, research enhancing our real‑time sensing and tracking capabilities of vessel and platform motion (Unsupervised learning of platform motion in synthetic aperture sonar, PERSEUS EU Project AUTONOMY MONITORING VESSELS).

PERSEUS EU Project (CMRE), 2014 – 2016

This CMRE activity developed and demonstrated passive underwater acoustic technology concepts for autonomous underwater UVX.

PERSEUS EU Project (CMRE) addressed the research gap in conventional surveillance technologies, which cannot easily detect fast boats, which generally have small radar signatures and do not carry Automatic Identification Systems (AIS).

In the PERSEUS project (Protection of European BoRders and Seas through the IntElligent Use of Surveillance), scientists designed, developed, and demonstrated at sea concepts of continuous, real‑time passive underwater acoustic systems for maritime surveillance integrated on board unmanned mobile platform (Underwater Glider and Wave Glider).

The embedded intelligence in a passive sonar surveillance system proved particularly effective due to its real‑time continuous monitoring capability, ranging from detection and localisation to vessel classification. Furthermore, the platform/system combination has proven persistent and covert, with wide area coverage and minimum environmental impact. Advanced target classification algorithms have been successfully applied in near real‑time during at‑sea demonstrations.

FRATRE For French Automatic Target Recognition (CMRE), 2019 – 2020

This CMRE activity developed intelligent and autonomous mine detection sonar capabilities.

FRATRE for French Automatic target recognition (CMRE) focused on developing autonomous capabilities based on Deep Learning (DL) techniques, which can analyse high‑resolution Synthetic Aperture Sonar (SAS) images provided by Autonomous Underwater Vehicles (AUV) or by Unmanned Surface Vessels (USV) and detect as well as classify mines or other ordnance. CMRE proposed to adapt the latest version of this CMRE classifier on small SAS snippets to large SAS tiles; implemented a new single‑stage detection/classification algorithm based on the CMRE classifier and evaluated the final algorithm performance on a subset of segmented data.

Texture‑Based Seafloor Characterisation Using Gaussian Process Classification (CMRE: ANMCM), 2019 – 2020

This CMRE, ANMCM activity automated characterisation and classification of seafloor sonar imagery.

Texture‑Based Seafloor Characterisation Using Gaussian Process Classification (CMRE: ANMCM) examined the use of machine learning to process sonar imagery and automate the characterisation of seabed types. Automated characterisation of seabed types is beneficial in estimating the performance of the sonar and Automated Target Recognition (ATR). This is useful for evaluating the performance of through‑the‑sensor mine‑hunting missions. Since synthetic aperture sonars can generate high‑fidelity acoustic images of the seabed, machine‑learning techniques can be employed to distinguish different seabed types (rocky, ripples, seaweed, and sand). Furthermore, this improved model could learn from both labelled and unlabelled sonar data, significantly increasing the amount of data that can be used for training the AI algorithms.

Automatic Object Classification with Active Sonar (CMRE: ASW), 2020

This CMRE activity proposed autonomous object classification methods for littoral environments.

Automatic Object Classification with Active Sonar (CMRE: ASW) proposed using Neural Networks to classify underwater objects from active sonar system data collected for underwater surveillance. First, the raw signal is processed, transformed in the time‑frequency domain and classified (object of interest/clutter). Next, the values of the neural network parameters (weights and biases) are learned using data collected during two sea trials

with an Echo‑Repeater as an object of interest. The classifier is then validated using data from a third sea trial in different geographical locations and environmental conditions. In CMRE’s validation dataset, the CNN classifier significantly reduces the number of false alarms and outperforms traditional feature‑based classifiers previously developed.

Sonar‑Based Deep Learning for Underwater UXO Remediation (CMRE), 2020 – 2021

This CMRE, ASW activity used DL approaches to develop novel Unexploded Ordnance (UXO) detection and classification algorithms.

An unfortunate legacy of former military activities is the contamination of aquatic environments with military munitions. Their presence is a serious threat to both humans and the environment, and the return of these waters to public use is contingent upon detailed underwater surveys. This activity aims to exploit sonar data and develop a high probability detection and classification framework for Unexploded Ordnance (UXO) at low false alarm rates.

As a result, applying machine‑learning algorithms to sonar data collected at potentially contaminated underwater sites can guide remediation efforts to effect savings. Specifically, because fewer resources will be spent investigating harmless clutter, the cost of remediation should decrease substantially. In addition, the developed algorithms are functional with measured data from existing systems and are readily deployable in a short time frame for use in actual remediation efforts.

Coupled Ocean‑Acoustic Variational Data Assimilation (CMRE: ASW, EKOE), 2020 – 2021

This CMRE activity assessed the viability of DL approaches for assimilating acoustic underwater propagation measurements.

Coupled Ocean‑Acoustic Variational Data Assimilation (CMRE: ASW, EKOE) investigated the feasibility of assimilating underwater acoustic data into ocean predictive models through extended data assimilation schemes. In particular,

the impact of Transmission Loss (TL) data associated with a simple geometry of source and receivers was assessed. The scenario analysed as part of this activity considered a low‑frequency acoustic propagation signal a sound source of opportunity for improving oceanographic forecasts. Assimilation of TL data was achieved by implementing a neural network‑based observation operator that maps forward and backward temperature increments onto increments of TL. Such an operator is based on the Canonical Correlation Analysis (CCA) of physical and acoustic datasets from an ensemble of oceanic and acoustic simulations.

Unsupervised Learning Of Platform Motion In Synthetic Aperture Sonar (CMRE: ANMCM), 2021

This CMRE, ASW and EKOE activity developed an ML‑based aperture sonar imaging method for platform motion estimation.

Unsupervised learning of platform motion in synthetic aperture sonar (CMRE: ANMCM) introduced a machine learning approach to platform motion estimation from spatiotemporal coherence measurements of diffuse backscatter based on variational probabilistic inference. A Variational Autoencoder (VAE) is used for learning disentangled latent representations directly from the input 3D coherence measurements, without any assumptions. The VAE with unsupervised training learns to infer the relative 3D platform position between two pings with sub‑wavelength accuracy for low‑frequency applications. Increasing the VAE inference accuracy for high‑frequency synthetic aperture sonar requires including a small amount of data labelled with 3D position estimates during training.

The presented analysis was based on a simulated dataset enabling reproducibility. The machine learning approach of variational probabilistic inference constitutes a robust method for autonomous data‑driven navigation, capitalizing on a large amount of spatial coherence measurements of diffuse backscatter obtained during synthetic aperture sonar operations.

PREDICTIVE MAINTENANCE AND LOGISTICS (PML)

“Bitter experience in war has taught the maxim that the art of war is the art of the logistically feasible.”12

12 ADM Hyman Rickover, USN.

PREDICTIVE MAINTENANCE AND LOGISTICS (PML)

OVERVIEW

Military fleet management represents an example of investment under uncertainty. Fleet operation and maintenance costs evolve, constantly reacting to price shocks, emerging technologies, and inevitable obsolescence. As a result, decision‑making concerning fleet management suffers from near irreversibility and illiquidity. Militaries cannot easily decouple themselves from existing procurements or quickly sell costly assets. Military logistics and support costs represent a substantial proportion of a nation’s defence budget. Small investments into better tools for logistics analysis can enable significant savings, reduce the deployed footprint, increase efficiencies and flexibility, ensure the required military effect, and support declared mission goals. Artificial Intelligence (AI) is a tool that can assist decision‑makers by providing better Business Intelligence (BI) and enable significant operational savings by enhancing Enterprise Resource Planning (ERP) tools and methods.

Research Conclusions

• Advanced Analytics (Big Data, AI techniques) can provide confidence and reduced uncertainty and assist in decision‑making for predictive maintenance and logistics at strategic, operational, and tactical levels.

• Even small investments in Advanced Analytics can lead to resource savings, increased efficiencies in business processes, and increased force readiness and effectiveness.

Research Challenges

• Multilateral cooperation and sharing of experience in developing advanced analytical methods will reduce development costs and enable the broader adoption of existing models and approaches.

• Given the varieties of military vehicles and platforms, collaboration is essential in developing reliable datasets for the further development of AI techniques.

• Alliances’ systems must be compatible and interoperable. This compatibility will increase the effectiveness and efficiency of planning and executing sustainment operations.

Figure 8: Predictive Maintenance and Logisitics (PML)

COMPLETED RESEARCH

The SAS Technical Panel (except for one activity conducted by the AVT Technical Panel) manages the portfolio of PML research activities. As the use of AI techniques in PML is an emerging topic, ETs are the most prevalent activity type. This cluster consists of two precursor activities. RTG SAS‑099 examined the challenge of fleet management, and RTG SAS‑132 sought to foster analytical capabilities in this domain. The rest of the activities, including SAS‑ET‑DX, SAS‑ET‑DW, SAS‑ET‑EH and SAS‑ET‑EN, examine various aspects of AI application in PML, ranging from methods to logistics technologies, business intelligence tools and enterprise resource planning systems. AVT‑ET‑184 is the only outlier activity, focusing on assessing the physics of failure for military platforms.

Economics for Evaluating Fleet Replacement (RTG SAS‑099),

2012 – 2015

This RTG is an early precursor activity, focusing on developing a strategy for determining an optimal replacement or significant overhaul time, after which it becomes disadvantageous to maintain the fleet.

RTG SAS‑099 is a precursor activity that suggested a methodology for determining and standardising the economics for evaluating fleet replacement decisions. Timing the fleet replacement decision presents a trade‑off between learning more about the fleet’s performance value and minimising the total fleet ownership cost. Understanding the importance of delay features centrally in optimal replacement decisions, as NATO forces retain fleets for increasingly long service lives. In practice, RTG SAS‑099 uses Bayesian methods to estimate the model parameters with empirical fleet data to show the decision‑maker how uncertainty in terms of costs and operational availability plays a vital role in understanding the timing of a replacement decision. This activity concludes that replacement models based on Bayesian methods with forecasting capabilities can be reliably applied to historical fleet data to gain insight into the total ownership costs of the fleet.

Models and Tools for Logistic Analysis (RTG

SAS‑132), 2017 – 2019

This RTG was a precursor activity that developed a catalogue of models and tools used by NATO nations to analyse military logistics and identify gaps in analytical capabilities.

RTG SAS‑132 is a precursor activity that surveyed models and tools for defence logistics analysis. The collection of substantial amounts of information

in Enterprise Resource Planning (ERP) can be exploited through analytic tools enhanced by AI techniques such as Machine Learning to make predictions useful for planners. Such information can be helpful in the descriptive analysis of a wide range of logistic processes, predictive maintenance, planning, reporting and decision support. It is expected that, in the future, more advanced analytics products based on AI will be developed and integrated into ERP systems for real‑time decision support related to logistics.

The logistics cost is a substantial proportion of a nation’s defence budget. Thus, a modest investment in analysis methods and tools can enable savings, reduce deployed footprint, increase efficiencies, and ensure military effectiveness. Moreover, by learning from other NATO nations’ experience in analysing similar problems, we can save development costs and compare approaches for cross‑validation of methods. Furthermore, nations will be able to focus development efforts on capability gaps and areas of mutual interest to the defence logistics analysis community.

Physics of Failure for Military Platforms (AVT‑ET‑184), 2018

This ET seeks to exploit emerging technologies, including AI and Deep Learning (DL), to assess and predict the Physics of Failure (PoF) for military platforms. The ET claims that advanced analytics provide confidence and reduced uncertainty value.

AVT‑ET‑184 is the only activity not managed by SAS TP. This activity followed previous research from AVT TP focused on predicting the failure mechanisms and managing the health of military platforms. According to this ET, military platform data should be treated as a strategic asset with quality, value, and benefits attributes. Big Data, Data Mining, AI, and Deep Learning should be used as tools to provide confidence and reduce the uncertainty of the relevant data, as well as synthesize this data to aid in decision‑making at strategic, operational, and tactical levels of leadership including S&T.

In practice, advanced analytics can break the challenge into executable chunks, collect, and feed the data into integrated models. Component failure data is one of those eco‑systems domains where the computationally valuable data would be translated into a comprehensive mapping of holistic changes in the component functioning as a system over time. AI‑enhanced failure analysis could prove more accurate than classical failure analysis focuses on predicting failure linearly.

Advanced Analytics for Defence Enterprise Resource Planning (SAS‑ET‑DX), 2018 – 2019

This ET examines Business Intelligence (BI) tools fostered by AI, Cognitive Analytics and Machine Learning (ML) that are being developed and implemented in Allies’ ERP systems. Even small investments in such tools can lead to resource savings.

SAS‑ET‑DX mapped advanced analytics products that NATO nations have been developing to foster their respective ERP systems for support planning and decision‑making. NATO nations rely on ERP systems to manage core business processes. These ERP systems rely on databases containing vast real‑time and historical transaction datasets. In addition, AI, Cognitive Analytics and ML serve as BI tools that can increase the efficiency of business processes.

Given the size of nations’ defence budgets, small investments in analytics can lead to comparatively huge resource savings, increased efficiencies in business processes, and increased forces readiness and effectiveness. In addition, learning from other countries’ experiences in developing advanced analytics for ERP will reduce development costs and enable broader adoption of existing models and approaches. This activity could provide inputs for and benefit from the research summarised in the C4 chapter.

Cost Analysis of Contractor’s Price for Defence Research and Development Projects (SAS‑ET‑DW), 2018 – 2019

This ET explores the necessary design for tools and methods for effective cost and price analysis used in military procurement procedures. Collaboration and sharing of cost analyses between NATO members were identified as being essential to developing novel methods.

SAS‑ET‑DW examined the necessary design for tools and methods for effective cost and price analysis for military procurement procedures. Estimating costs under budgetary and informational constraints is a challenge. Implementing standardised methodologies for benchmarking the contractors’ price proposals can alleviate this issue.

Given the variety of defence platforms and projects, collaboration is essential in developing the datasets for effective research. Sharing each stakeholder’s unique cost analysis methods will create opportunities to compare and develop new cost analysis methods. Developing novel cost analysis methods for defence contracts depends on advances in Data Mining, Big Data and AI techniques. Unfortunately, readily retrievable cost data that could serve in computing cost estimates for new weapon systems are lacking.

Coalition Sustainment Interoperability Study (SAS‑ET‑EH), 2019 – 2020

This ET investigates emerging technologies in logistics and their impact on planning and executing sustainment operations in a tactical environment. Innovative command and control systems must remain interoperable with coalition partners’ systems.

SAS‑ET‑EH investigated emerging technologies in logistics, focusing on how unmanned systems enhanced by AI could be employed to supply units in a tactical environment. This ET outlined the work required to harness emerging technologies within military S&T activities and the commercial sector.

In the distributed, joint battlefield, AI can facilitate the best options for providing sustainment to meet a particular need regardless of organisational hierarchy. However, command and control systems must be compatible and interoperable with coalition partners’ systems to increase the effectiveness and efficiency of joint planning and execution of sustainment operations.

Assessing the Implications of Emerging Technologies for Military Logistics (SAS‑ET‑EN), 2019 – 2020

This ET seeks to examine emerging technologies originating from the civil sector and their potential impacts on military logistics.

SAS‑ET‑EN aims to highlight work across NATO to assess and develop logistics capabilities that use emerging technologies such as AI, autonomy, alternative energy sources and additive manufacturing. This should help shape national acquisition programmes, understand emerging interoperability challenges, and identify analytical and more comprehensive collaboration opportunities.

TRAINING, MODELLING & SIMULATION (TMS)

“[...] current military operational models, the human aspect is still often represented in a mechanistic way, bearing little resemblance to observations, as if all humans always act the same way in a situation much as a machine would. Human behaviour is not deterministic. Without proper representation of behaviour, and the reasons behind the behaviour, the validity of the model may be seriously flawed, making its performance and predictions questionable.”13 13

TRAINING, MODELLING & SIMULATION (TMS)

OVERVIEW

Military operations require capable well‑trained military personnel. Thus, militaries must emphasise training to use state‑of‑the‑art technology under evolving conditions effectively.

Embedded Virtual Simulation applications embedded into military systems can provide a range of training capabilities. For example, Augmented Reality can present virtual targets and maintenance problems. Intelligent agents can drive forces to complete scenarios, and a brilliant tutor can provide training in management functions. By utilising network‑enabled capabilities, collective training and mission preparation are possible. All these systems use AI techniques on their pathways to decrease costs and increase efficiency in military training.

Human Behaviour Modelling (HBM) tackles the challenge of realism in simulated operations via quantitative representation of such variables for individuals and small groups. A particular area of interest is the seamless interaction of humans with realistically simulated human characters governed by AI. Given enough data, the AI algorithms can create Computer Generated Forces (CGFs) without explicitly programming behaviour.

Cost and safety concerns related to product development and innovation constantly drive the Military to embrace synthetic environments to experiment with innovative ideas, mitigate risks, or objectively compare design alternatives. This applies in the technical domains beyond the scope of training and HBM. For example, simulation and modelling of military vehicles and platform performance simulating capabilities of novel ISR systems.

Research Conclusions

• AI techniques can reduce the system development burden and help integrate Embedded Virtual Simulation (EVS) and Intelligent Tutoring Systems (ITS) into military training, even in deployed conditions.

• Applying these simulations to well‑defined, procedural military domains like navigation, marksmanship and combat casualty care is feasible.

• The development of tools and methods to deliver and manage intelligent tutoring of teams engaged in collaborative learning is of paramount importance.

• Incorporating an all‑encompassing “human view” into agent‑based modelling and using Bayesian network analyses benefits the analytical processes employed by NATO militaries.

• ML techniques for the simulation of human behaviour benefit from Reference Architecture models that operate on symbolic representations of information and knowledge.

• Simulating human behaviour using Deep Learning and Neural Networks proved difficult without robust datasets on opponent behaviour. Generating this data for automated model development is a promising area of research.

• Improving intelligent system creation capabilities reduces simulation development workload and associated costs.

• Standards for facilitating interoperability, reusability, and flexibility for operational model architectures also reduce costs.

• Simulations enhanced by artificially intelligent behaviours improve the consistency of scenarios and reduce the need for Subject Matter Experts (SMEs).

• ML techniques improve vehicle and component modelling systems that enhance predictive maintenance, fleet management and design development.

• ML and DL techniques can provide reliable predictions of large‑scale aerodynamic simulations.

Research Challenges

• Technological limitations constrain the efficacy of training connected with the fidelity of the simulation.

• The development of AI tools and methods to train psychomotor tasks involving physical coordination and skill.

• Generating high fidelity human behaviour models that capture all socio‑cultural aspects and simulate complex operational environments (e.g., urban).

• Realistic cognitive modelling of interactions with and between Intelligent Agents within simulation environments.

• The potential of Intelligent Agents to guide embedded team learning in deployed conditions.

• Intelligent Agents’ dependence on interface requirements allows coordination among team members and platforms.

• Reusability of AI methods across several alternatives is still an issue, as many solutions are still hard‑coded into proprietary solutions.

• To reinforce the trust and credibility of AI‑based T&S systems, tailored versions of Verification and Validation (V&V).

COMPLETED RESEARCH

The portfolio of TMS research activities includes a healthy variety of activity types, managed by HFM and MSG TPs. From 2009 to 2016, RTG HFM‑165, HFM‑ET‑120, and RTG HFM‑237 examined the role of AI in virtual tutoring systems. From 2010 to 2017, RSY MSG‑069, RSY HFM‑202, RWS MSG‑107, RWS HFM‑220 and RTG MSG‑127 increased the fidelity of T&S by introducing Human Behaviour Modelling and further reinforcing the cross‑panel synergies in the research. From 2011 to 2018, HFM‑ET‑112, RTG HFM‑216, HFM‑ET‑144 and RTG HFM‑268 exploited advances in AI to examine the potential of synthetic environments. In addition, four related activities occurred between 2018 and 2020: Two activities conducted by the AVT TP focused on simulating the performance of military vehicles, while one conducted by the SET TP focused on ISR systems. The last addition to the TMS cluster was from the CMRE DKOE programme, which focused on forecasting COVID‑19 epidemiological phases. This cluster contains robust research examining many aspects of AI applications in T&S. The goal of all the summarised activities was to decrease costs and increase the efficiency of military T&S. Lessons learned in T&S have crossover potential for better performance decision‑making in the C4 cluster of activities.

Figure 9: Training, Modelling and Simulation (TMS)

TMS – TUTORING

Improving Human Effectiveness Through Embedded Virtual Simulation (RTG HFM‑165), 2007 – 2011

The RTG HFM‑165 is an early precursor activity which provided the foundation for further research on virtual and AI‑augmented simulation technologies for military training. The RTG exploits advances in ML techniques and focuses on integrating training capabilities into operational environments. In conclusion, embedded training enables rapid skills development, retention, and adaptation close to the battlefield. Thus, allowing a greater coherence between the operational environment and training.

The relationship between training and operational environments, that is, the fidelity of the EVS and the consequences for training transfer, remain a concern due to the technological limitations. Team training is an important application area for EVS; therefore, interface requirements must allow coordination among team members and platforms. In addition, intelligent tutors within simulation environments need better interaction design.

Assessment of Intelligent Tutoring System Technologies and Opportunities (HFM‑ET‑120), 2011 – 2012

This ET identifies pathways to instructional efficiencies to be achieved using intelligent tutoring systems in training and educational development.

HFM‑ET‑120 reviewed the state‑of‑the‑art in Intelligent Tutoring Systems (ITS) technology and practice for conducting NATO education and training. Notably, it identifies opportunities for developing instructions that address critical cognitive capabilities needed for military operations. Instructions to produce these capabilities require interaction with ITSs that rapidly adjust to individual learner abilities, prior knowledge, experience, and to some extent, even misconceptions. In addition, such a system must be affordable, effective, and practical at the scale required for military personnel.

Assessment of Intelligent Tutoring System Technologies and Opportunities (RTG HFM‑237), 2013 – 2016

This RTG incorporates lessons learned from RTG HFM‑165 and HFM‑ET‑120 to review further and analyse the nature, extent, availability, and feasibility of opportunities presented by ITS for conducting NATO education and training. The RTG concludes that while ITS shows promise in some areas, developing ITS tools to train psychomotor tasks is challenging.

RTG HFM‑237 builds on the recommendations from HFM‑ET‑120 and conclusions from RTG HFM‑165 RTG HFM‑237 identified opportunities and challenges in four primary areas of ITS: authoring (development), standardisation, data analytics, and adaptive interfaces.

Authoring ITSs is expensive and requires many specialised skills. AI techniques and advances in computer science are reducing the authoring burden through automation and enhanced usability. The RTG used authoring tools to develop prototype tutors for psychomotor and collective task domains relevant to military training needs. Since military operations involve teamwork, the RTG identified collaborative learning through adaptive tutoring of teams by ITSs as an area of importance. Standardisation is a strategy to reduce the cost of ITSs by increasing the reuse of ITSs and their components through promoting interoperability between learner models, standardising instructional strategies, developing common communication protocols, and leveraging existing standards to model learner competencies in various task domains. Understanding and modelling course contents and instructional strategies through advanced analytics allows optimised learning outcomes. Finally, adaptive interfaces should be dynamic to adapt to the needs of individuals and their varying roles in ITS development, deployment, and evaluation.

TMS – BEHAVIOUR MODELLING

Use of M&S in: Support to Operations, Human Behaviour Representation, Irregular Warfare, Defence Against Terrorism and Coalition Tactical Force Integration (RSY MSG‑069), 2009 – 2010

The RSY is a precursor activity, which provides a capabilities and efforts overview in using M&S, including HBM, to decrease costs and increase efficiency in military training.

The RSY MSG‑069 partially focused on developing better Computer‑Generated Forces (CGF), particularly for training applications. CGFs are used to drive the behaviour of simulated forces in a realistic manner. In place of advanced CGFs, human experts usually must provide input. AI‑based CGFs reduce the number of subject matter experts needed to conduct an exercise through AI techniques that provide tools to support this objective. Unfortunately, the reusability of AI methods across several alternatives is still an issue, as many solutions are still hard‑coded into proprietary solutions. Lastly, this precursor activity emphasised collaboration and the alignment of research efforts. It reacted to the state of science and increased the scope of ITS/VTS by introducing the paradigm of human modelling of CFGs in synthetic environments.

Human Modelling for Military Application (RSY HFM‑202), 2010 – 2011

This RSY recommends applying general Human Modelling and AI‑enhanced M&S tools in military applications. The RSY states that comprehensive modelling of realistic agents is challenging.

RSY HFM‑202 examined human modelling and AI integration demands from M&S Military Analysis, Training and Military Systems Development and Acquisition perspectives. RSY HFM‑202 analysed a broad spectrum of literature on agent‑based modelling and Bayesian network analyses of modelling teams, organisations, government actions, societies, cultures, and the resulting military contributions to non‑traditional operations. Based on the literature review, the RSY concluded that incorporating all‑encompassing human behaviour into agent‑based modelling and using Bayesian network analyses benefits the analytical processes employed by NATO militaries. However, increased exposure to the larger body of research on AI techniques is required to conduct more robust social‑cultural modelling. The same goes for research and technology development for the vertical interoperability of tactical‑operational‑strategic models.

Human Behaviour Modelling for Military Training Applications (RWS MSG‑107), 2011 – 2012

This RWS builds upon RSY HFM‑202 and integrates the interests of the various TPs to focus on HBM applications in training and associated challenges. The RWS concludes that integrating human behaviour into military training simulations is challenging.

RWS MSG‑107 gathered ACT, MSG, SAS and HFM TPs to explore the state‑of‑the‑art HBM, software architectures and implementation requirements. From the perspective of the involved TPs, the purpose of the simulations was a key aspect. They collaborated on integration processes and identified barriers to HBM deployment in the military, such as knowledge stovepipes, the lack of reusable tools, the lack of a generic architecture and sometimes the lack of scientific data. Integrating science‑based models that describe only part of human behaviour into complex military training settings and analytic environments was the most significant challenge. This concerns the application domain of training for complex operational environments (e.g., urban). Hence, there is a need for standards for operational model architectures wherein human and military sub‑models can work together. There is also a need for more natural interaction between human trainees and simulated characters.

Human Behaviour Modelling for Military Training Applications (RWS HFM‑220), 2012 – 2013

This RWS provides an overview of the quantitative representation of performance, decision‑making and the behaviour of individuals and groups in military HBM. The RWS emphasised the necessity of robust datasets in developing opponent behaviour simulations.

RWS HFM‑220 is a continuation of RWS MSG‑107

This RWS established quantitative indicators for performance, decision‑making and the behaviour of individuals and groups. Furthermore, the RWS developed a report with recommendations for human modelling in Military T&S and its interaction with live player decision‑making in small team environments.

In a simulated air combat scenario, researchers in HFM‑220 used ML to master the potential complexity of adapting opponent behaviour to new conditions in a simulation. Neural network parameters were used as an input for a cognitive model. As no prior knowledge could serve as the basis for the opponent’s behaviour, the

danger that student behaviour is adapting to the wrong opponent’s behaviour was recognised. In summary, it proved exceedingly difficult to simulate opponent behaviour without access to robust datasets that would serve as the basis for behavioural patterns for the opponent.

Reference Architecture for Human Behaviour Modelling in Military Training Applications (RTG MSG‑127), 2013 – 2017

This RTG developed software architecture for HBM of individual players intended for use in military training applications. The RTG identified the benefits of utilising reference architecture, a model that enables intelligent behaviour based on the symbolic representation of knowledge.

RTG MSG‑127 puts human behaviour models reliably into practice, as was previously suggested by RSY HFM‑202 and RWS MSG‑107. The RTG aimed to reach that goal by supporting the development of Reference Architecture (RA) for human behaviour modelling of individual players intended for use in military training applications. The recommended RA established a common basic structure for HBM. Researchers identified DL learning techniques as promising research areas but collecting and generating ‘Big Data’ for automated model development by training DL‑based Neural Networks was challenging. To create high‑fidelity human behaviour models that capture all socio‑cultural aspects, researchers from the Human and Physical Sciences must work together. Lastly, HBM must act credibly. Unfortunately, Verification, Validation and Accreditation (VV&A) of AI‑based HBM is a significant issue.

This RTG prepared a proposal to develop and implement the RA in a pilot project. As part of this process, the RTG recommended developing a roadmap for improving the RA and implementing assets compliant with the RA across all NATO application domains. Furthermore, standards for facilitating interoperability, reusability, and flexibility must be established to compose operational model architectures and enable CGFs to mimic human behaviour and interact effectively. Lastly, tailored versions of VV&V processes must be developed. This endeavour is linked with AI research in C4, specifically “Artificial Intelligence and Big Data for Decision Support” (RWS IST‑160).

TMS – SYNTHETIC ENVIRONMENTS

TMS – SYNTHETIC ENVIRONMENTS

Advanced Training Technologies for Medical –Healthcare Professionals (HFM‑ET‑112), 2010 – 2011

This ET explores advanced modelling and simulation technologies to develop medical training applications.

HFM‑ET‑112 explored using the latest computer graphics, natural language processing, and AI technologies to develop medical education applications, especially training health care professionals and just‑in‑time care for the warfighter in the field and families at home. For instance, virtual reality‑generated assets are used to treat post‑traumatic stress disorder.

Synthetic Environments for HSI Application, Assessment, and Improvement (RTG HFM‑216), 2011 – 2014

This RTG identifies opportunities for synthetic testbeds and suggests how they can provide cost‑effective assessment techniques for developing, acquiring, and training communities of interest.

RTG HFM‑216 described an M&S approach to experimentation that uses models and operators to measure system performance under targeted critical perturbations. This approach, called Synthetic Environment for Assessment (SEA), is a new way to use simulation to conduct trade‑off analyses and explore complex design spaces. SEA has the potential to progress systems from simply sustaining incremental improvements in favour of disruptive innovation. While this RTG did not address AI applications directly, it established a methodological foundation for HFM‑ET‑144 and RTG HFM‑268

Synthetic Environments for Mission Effectiveness Assessment (HFM‑ET‑144), 2015

This ET follows RTG HFM‑216 and investigates a meaningful way ahead for research on Synthetic Environments for Mission Effectiveness Assessment” conducted by MSG, HFM and SCI TPs.

HFM‑ET‑144 compiled lessons learned from HFM‑ET‑112 and RTG HFM‑216 and investigated a meaningful way ahead for research on Synthetic Environments for Mission Effectiveness Assessment. The main goal was to explore modelling, simulation, gaming, and related technologies to develop medical training and education applications.

Cross Panel Activity on Synthetic Environments for Mission Effectiveness Assessment (RTG HFM‑268), 2015 – 2018

This RTG built SEA accessible to NATO members and demonstrated its benefits for future capability development. In addition, the RTG established a method for collecting realistic mission efficacy data.

RTG HFM‑268 is a cross panel activity between MSG, HFM and SCI. This RTG demonstrated the benefits of SEA at Warrior Preparation Center’s (WPC) Spartan Warrior AWACS exercises. The RTG results showed a workable way to gather realistic mission efficacy data that decision‑makers could use in the acquisition processes, specifically early design decisions. Additionally, automated software with artificially intelligent behaviours dramatically improves the consistency of running scenarios. As a result, it may reduce the need for Subject Matter Experts (SMEs) to play various roles in a scenario.

Figure 10: Training, Simulation and Modelling (TMS): synthetic environments

TMS – OTHERS

Gas Turbine Engine Environmental Particulate Foreign Object Damage [EP‑FOD] (RTG AVT‑250), 2016 – 2018

This RTG uses self‑educating ML techniques to identify the causes and the trajectory of engine degradation.

RTG AVT‑250 examined how the vehicle imposes performance impacts on the propulsion system and more precisely identifies and understands the causes and the trajectory of engine degradation. Furthermore, dedicated aircraft modelling enables the validation and prediction of the level of engine damage and management system, a collection of information in a data lake, combined with phenomenological models of engine and aircraft performance. In summary, the Digital Twins of an aircraft and its engines may be developed by feeding ML models with fused post‑mission engine data, meteorological and atmospheric data, and aircraft position data.

ML techniques improve aircraft modelling systems that focus on predicting the level of engine damage. These systems offer benefits, including fault prognostics, trends in engine behaviour, optimised fleet management, predictive maintenance, and improved next‑generation engine design and development.

Enhanced Computational Performance and Stability & Control Prediction for NATO Military Vehicles (AVT‑ET‑199), 2019

This ET explores large‑scale aerodynamic simulations predicting the performance of military vehicles.

AVT‑ET‑199 explored pathways to generating simulations predicting the performance and dynamic derivatives and Stability & Control characteristics for the design and performance assessment of modern military vehicles. ML and DL techniques are used to provide reliable predictions of large‑scale aerodynamic simulations.

Simulation of Low Photon LIDAR in Complex Environments (SimPL) (SET‑ET‑112), 2019 – 2020

This ET laid the groundwork for next‑generation simulation capability in single‑photon LIDAR.

SET‑ET‑112 aligned community efforts to lay the groundwork for next‑generation simulation capability in single‑photon LIDAR, which provides powerful new capabilities in sensing to combat deliberate obscuration (with low detectability) by an adversary. In addition, this ET uses synthetic training data to accelerate the development of next‑generation AI‑based ATR algorithms.

COVID‑19 Adaptive Learning and Forecasting (CMRE DKOE), 2020 – 2021

This CMRE activity uses Adaptive Bayesian learning and forecasting techniques to reliably forecast the COVID‑19 epidemiological curve.

COVID‑19 Adaptive Learning and Forecasting (CMRE DKOE) focused on the quickest detection of the epidemiological phases of COVID‑19. Specifically, recognising the initiation and termination of an epidemic wave. The activity also examined testing and forecasting methods in operational scenarios. Adaptive Bayesian learning and forecasting approach outperformed available state‑of‑the‑art forecasting solutions.

Figure 11: Training, Simulation and Modelling (TMS): others

systems are rapidly becoming an increasingly important part of the armed forces…”

UNMANNED VEHICLES (UXV)

OVERVIEW

Unmanned systems have become a reality in defence across all domains (land, air, sea, space, and cyber). UxVs will constitute one of the essential characteristics of the future battlespace. Technological readiness levels are already mature, especially if human operators remotely control the UxVs. Increasing levels of autonomy offer the prospect of increased capability for UxVs. This entails using AI techniques to conceive artificially intelligent UxVs, which can sense their environment and act in it to achieve goals assigned by humans. Deploying autonomous UxVs and integrating them in the military context is a live issue, demanding further research and investigation. The gap between robotic technology and its operational use constitutes a significant limitation.

Research Conclusions

• Autonomous intelligent systems can be applied at the strategic, operational, and tactical levels across all military domains.

• Enabling autonomous behaviour for unmanned systems is necessary to create safer and scalable defence capacities.

• Removing the human from the vehicle’s cockpit enables a more efficient and cost‑effective design of UxVs.

• UxVs enhanced by AI techniques can enhance military forces’ ability to project combat power.

• AI techniques can improve real‑time dynamic planning mechanisms for UxV systems.

• There has been significant advances in AI‑enabled cooperative unmanned systems (swarms).

• Using AI techniques for target identification via swarm systems is an up‑and‑coming research area.

Research Challenges

• Finding a commonly agreed definition of meaningful human control, autonomous behaviour, and intelligent systems in the context of military operations is necessary.

• To accelerate the development of interoperable, intelligent UVXs, further collaboration between researchers of different NATO nations is necessary.

• Integrating autonomous agents and soldiers in spatially complex conditions can be addressed by enhancing intelligent autonomy.

• Verification & validation of autonomous intelligent systems represents the biggest challenge.

Figure 12: Unmanned vehicles (UXV)

COMPLETED RESEARCH

The portfolio of UxV research contains a wide range of activities that have been shifting their focus throughout the investigated timeframe. As many of the activities are Exploratory Teams (ET), it is safe to say that using AI in UxVs is an emerging research topic. Consequently, these activities have few clear findings or conclusions, but they have formed the basis for subsequent Research Task Groups (RTG), Symposiums, and ongoing activities. The SAS, IST, and SCI Panels manage the portfolio of activities. For example, RTG SAS‑097 and IST‑ET‑067 researched general intelligent autonomy between 2012 and 2016. SCI‑ET‑015 and b focused on UxVs‑Humans communication and teaming from 2014 to 2016. The most recent activities, SCI‑ET‑022, IST‑ET‑099 and SAS‑ET‑EI, investigated novel methods and applications between 2016 and 2019.

Robotics Underpinning Future NATO Operations (RTG SAS‑097), 2012 – 2016

This RTG analyses the increasing use of unmanned vehicles and platforms in military operations and seeks to bridge the gap between operational requirements and technological possibilities.

RTG SAS‑097 claimed that the main challenge in developing increasingly autonomous systems is the lack of system theories allowing holistic analysis of the overall autonomous systems, involved processes and their interactions. The activity offers an analysis of the operational requirements of robotics. These range from the need to develop concepts of operation to the need for open architectures to allow technology insertion and the need to operate Beyond Line of Sight. Researchers also found that increasing levels of autonomy should be developed to eliminate the need for continuous navigation control of robots.

Drawing on this analysis, researchers also addressed the probable future direction of robotic technology, developing an extensive body of structured notes on three main characteristics: control, sensors, and systems. These notes highlighted strategic challenges and opportunities in the field. Several key themes emerged across all three aspects, including the need for augmented cognition, improved human‑machine teaming architectures, and better human information processing capabilities. Researchers also identified the possibilities offered by alternative energy sources and the role of nanotechnology in enhancing UAV performance. Elsewhere, researchers focused on characterisingsing uncertainty and developing policies and technologies to defend against cyber threats.

The RTG concludes by emphasising the need for consensus on the definition of “Meaningful Human Control” in military operations if research and operationalisation are to move forward. UxVs enhanced by AI have significant potential to improve a force’s ability to project combat power and adapt to more complex and uncertain future environments. Furthermore, removing the human from the vehicle’s cockpit enables more efficient and cost‑effective design for vehicles and platforms. However, centralised management of UxV systems will not be able to deal with the dynamic challenges in information‑poor environments. Still, AI techniques and Bayesian approaches show promise in fostering real‑time dynamic planning mechanisms and autonomous decision‑making.

Machine Learning Techniques for Autonomous Manoeuvring of Weapon Systems (IST‑ET‑067), 2012

The IST‑ET‑067 examines the role of AI techniques as enabling technology for integrating autonomous intelligent solutions into the modern battle space. Significant emphasis is put on the organisation and architectural design of autonomous intelligent battlefield agents for relevant Machine Learning techniques and their applications.

The ET concludes that autonomous intelligent solutions can be applied at the strategic level, e.g., for Course of Action analyses, at the operational level; for simulations of logistic support in deployed operations and at the tactical level; for the control of weapon systems or the simulation of engagements. Unfortunately, the main obstacles to the widespread adoption of autonomous intelligent solutions are requirements definition and the Verification and Validation (V&V) of Machine Learning techniques

Autonomy in Communications Limited Environments (SCI‑ET‑015), 2014 – 2015

This ET analyses how AI‑based embedded computing improves the autonomous behaviour of unmanned systems.

SCI‑ET‑015 seeks to build a framework to facilitate research collaboration, data sharing, and joint experimentation in the subject of embedded machine intelligence for autonomous unmanned systems. The ET highlights that embedded computing is essential for sensor signal processing, vehicle situational awareness, and long‑term mission planning. According to the ET, the scalability of innovative defence capabilities is intricately connected with deploying safe, intelligent, and autonomous UxVs. Furthermore,

AI techniques increase UxV warfighting capacity in settings where communications of unmanned systems are severely limited.

Intelligence & Autonomy (Robotics) (RSM IST‑127), 2014 – 2016

This RSM identifies the gaps between available civilian solutions and military needs in intelligent autonomous systems.

IST‑127 consolidates existing research and identifies gaps in the literature. It notes that the private sector is leading the development of autonomous UxVs, producing affordable sensors, and paving the way toward public and legal acceptance of autonomous vehicles.

Unlike in the civil sector, the military systems must deal with reduced information, spatially complex environments, and malicious threats while interacting with other systems and soldiers throughout operations. Enhancing systems’ communication capabilities, or fostering the systems’ intelligence and decision‑making capabilities, can support the military. Algorithms devised to deal with these challenges must address requirements specific to the military, for example, military rules, tactics, and providing interaction between the intelligent agents.

The RSM makes two main recommendations.

Firstly, finding a commonly agreed definition of autonomous behaviour and intelligent systems is necessary. Close collaboration between researchers of different NATO nations is essential to accelerate the development of intelligent autonomous military systems. Such a collaboration would also lead to interoperable and compatible future solutions.

Novel Employment of Autonomous Military Systems (AMS) (SCI‑ET‑022), 2016 – 2017

This ET conceives system concepts potentially enabled by intelligent autonomy, which would offer a significant military advantage.

SCI‑ET‑022 describes the key characteristics and status of technology adoption for AMS. Furthermore, the ET raises awareness of the various possibilities of employing autonomous military systems in either novel warfighting applications or addressing traditional warfighting challenges in novel ways.

Mission Assurance and Cyber Risk Assessment for UAxS (Multi‑Domain Unmanned/Autonomous Vehicles and Systems) (IST‑ET‑099), 2017

The ET examines the multi‑faced role of autonomous UxVs and C4ISR systems in future NATO operations.

IST‑ET‑099 develops an initial model for synchronised, multi‑domain missions for UxVs; such a model would consider emerging methods and frameworks in mission assurance, cyber security, and risk assessment. This ET concludes that UxVs have the potential to deliver substantial operational value in decision‑making speed for real‑time operations. AI techniques can foster the processing of high‑volume data and course‑of‑action determination in complex multi‑domain missions. Key challenges identified concern response to adversary attacks against UxVs and improving trust in algorithms guiding these systems.

Employing AI to Federate Sensors in Joint Settings (SAS‑ET‑EI), 2019 – 2020

This ET examines methods of managing UxV swarms used for ISR purposes.

SAS‑ET‑EI focuses on enabling operational planners to design an effective federated network of autonomous unmanned ISR vehicles. The ET suggests evaluating the scenario’s physical characteristics and the expected target type and specifying the best mix of sensor types and number of sensors to obtain the highest possible success rate from a target identification system. Using AI for automatic target identification by UxV swarm systems continuously improves the identification success rate.

CONCLUSIONS

CONCLUSIONS

This Chief Scientist report analyses the extensive research led since 2010 by the NATO STO PoW on AI relevant topics in a military context. This NATO OCS document draws upon the published findings of research activities undertaken by the STO’s network of over 5,000 active scientists, analysts, and researchers, the world’s largest collaborative research forum in the field of defence and security.

The report aims to deepen the understanding of military, civilian, and NATO decision‑makers and audiences on AI’s impact on military operations, defence capabilities and decision‑making, especially as the use and application of AI techniques continue to grow. The report has focused on seven central themes:

1. Advanced algorithms

2. Command, Control, Communications, and Computers (C4)

3. Human‑Machine Symbiosis (HMS)

4. Intelligence, Surveillance and Reconnaissance (ISR)

5. Predictive Maintenance & Logistics (PML)

6. Training, Modelling & Simulation (TMS)

7. Unmanned Vehicle (UxV)

The work described in the report provides solid evidence‑based frameworks for ensuring informed decisions are made in AI development and adoption.

SUMMARY OF RESEARCH CHALLENGES

The document outlines the efforts to administer a programmatic overview of research activities in the STO research agenda, exploring the many facets of AI in a military context. The extensive review provides a solid evidence‑based framework for addressing AI’s disruptive impact on military capabilities and forces.

Future research in AI, including using AI techniques to reach other goals, may wish to consider the challenges identified below and build on these in current and future research activities.

Transferability and reusability of commercially available AI

• There is no coordinated endeavour to identify relevant commercial applications of AI in the advanced stages of implementation and the necessary modifications for military applications.

• There is limited research to identifying military requirements not currently undertaken in the commercial field.

• There is limited research on the transferability of AI across several military applications, as many solutions are proprietary.

Adaptability to unforeseen real‑time events

• Currently available AI methods are slow to train, thus slow to adapt to rapidly changing threats. This limitation makes them unsuited to some live encounters.

• ML/DL methods are constrained in complex operational environments, particularly detecting, and classifying unprecedented events.

Simulation of socio‑cultural variables

• ML/DL methods struggle to generate high‑fidelity human behaviour models that capture complex socio‑cultural aspects.

AI Robustness

• Limited research focuses on improving AI robustness, trustfulness, explainability, verification and validation, and deploying AI‑based systems in a military context.

• There is a lack of research exploring customised versions of verification and validation protocols for military contexts.

• Changes to the input data can easily compromise ML/DL approaches.

There is a lack of research on adjusting training, testing, validation, and product phases of ML/DL tools to make them resilient to input manipulation.

Military relevant data and the training of algorithms

• AI training methods rarely use large sets of appropriate annotated data pertinent to military operations.

• There is a lack of research to determine the amount of labelled data required to train an algorithm to solve a given problem.

• There are limited endeavours ensuring wide‑scale availability of reference data repositories and using existing data management infrastructure.

• There is a lack of research on augmenting raw and labelled data with high fidelity simulated/ synthetic data.

Interoperability and standardisation

• There are limited efforts to enhance interoperability and the development of common standards, metrics, and best practices for joint NATO activities.

• There is limited multilateral cooperation and sharing of experience in developing advanced analytical methods to reduce development costs and enable broader adoption of existing models and approaches.

• There is a lack of commonly agreed definitions of AI relevant terminology in the context of military operations.

Explainability of AI

• There are limited efforts to increase the explainability of algorithms using AI techniques to users across the whole chain of command.

• There are limited efforts to improve the understanding of AI, Big Data, Data Science, and advanced analytics to inform political leaders and military decision‑makers about the potential and drawbacks of these techniques and technologies.

• Limited collaboration with educational establishments hinders the attraction of experts to work on military projects.

• There is a lack of projects to explain the use of AI in military operations to the public.

Supervisory control of AI systems

• There is a lack of research on human interactions and autonomous/intelligent systems.

• There are inadequate efforts to standardise optimal mission management practices and control strategies for supervisory tasks.

• There is a lack of research on allocating artificial and human cognition in missions and their phases.

Resilience and AI Countermeasures

• There is a need for research on countering adversarial use of AI.

APPENDIX A –

BIBLIOGRAPHY

ADVANCED ALGORITHMS (AA)

COMPLETED RESEARCH

INTELLIGENCE EXPLOITATION OF SOCIAL MEDIA (

RTG SAS‑IST‑102)

ACTIVITY TYPE:

Research Task Group

DURATION:

Nov 2013 – Nov 2016

OVERVIEW:

This RTG proposes to explore and provide expert opinion on the methods, tools, and techniques best suited for the foraging cycle related to select social media sources. Furthermore, it considers the approaches that could be taken toward improving discovery and the sense‑making cycles.

OBJECTIVES:

To determine Social Media (SM) exploitation techniques as a source for intelligence purposes. Further, the activity examines prospective methods and tools for intelligence analysts exploiting SM sources.

APPROACH:

This task group used a real‑world scenario similar to a country study for a NATO country of concern. The useful information pulled together from selected social media sources will determine the benefit to the situational awareness picture. The activity considered:

• Sources & Methods for data discovery.

• Methodologies, algorithms, and tools for monitoring (searching, filtering, cleaning, indicators), analysis and integration (evaluation of sources, types of analysis) leading to interpretation and possible behaviour estimation (deceptive behaviours, validation of results).

• Automated support for scanning, monitoring, searching, discovery, and mining.

• Language processing.

• Pattern recognition (e.g., trending, relationship analysis, visual analytics).

• Risks and legal caveats associated with collecting, processing, and exploiting social media.

FINDINGS:

Monitoring and analysis of SM data presents an urgency for intelligence agencies. The identification of tools and methods in an emerging intelligence environment poses a trade‑off surrounding the use of openly available or in‑house developed monitoring and analysis tools, methods, and algorithms.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COMPLEX EVENT PROCESSING

FOR CONTENT‑BASED TEXT, IMAGE, AND VIDEO RETRIEVAL (IST‑ET‑086)

ACTIVITY TYPE:

Exploratory Team

DURATION:

April 2016 – May 2020

OVERVIEW:

This ET was formed to advance the text, imagery, and video exploitation and to develop methods to cross‑cue between these domains due to the increasing variety, size, complexity, and uncertainty of data facing decision‑makers.

OBJECTIVES:

To foster the development of theoretical and algorithmic tools supporting joint exploitation of multimedia data sources. The aim is to cover at least three multimedia types: image, video, and text.

APPROACH:

A model‑driven information fusion approach named “Complex Event Processing (CEP)” was identified as promising to analyse patterns that continuously unfold in real‑time. CEP can address foundational scientific and technological aspects of text and video content extraction, including image content analysis, automatic identification of modified images/video, algorithms for content‑based retrieval, indexing methods for video retrieval and improved image analysis.

FINDINGS:

Developing methods for joint exploitation of multiple media requires effort at several levels. These range from identifying real‑world scenarios where two or more multimedia types co‑occur to designing features that best lead to efficient extraction of actionable information.

RECOMMENDATIONS:

• Defining the scope and extent of the future investigations.

• Developing a research approach includes identifying the extent to which a common framework can be developed for representing data from the different classes.

• Developing ML tools to automatically detect and classify events from combinations of one or more data classes.

• Data collection and ground truth labelling.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

CONTENT‑BASED MULTI‑MEDIA ANALYTICS (CBMA) (IST‑144)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2015 – March 2016

OVERVIEW:

This RTG was formed to accelerate situational awareness and decision‑making and deal with the complexity of the defence information space. Furthermore, develop interoperable tools that cross‑cue knowledge from one CBMA method to generate taskings in another.

OBJECTIVES:

To provide better‑automated support to military analysts and decision‑makers in extracting meaningful information from multi‑media sources pertinent to their information environment. The expected impact of the work for NATO is to boost the agility of military operations in the physical and IE through a deep understanding of adversary perspectives, intent, and threats.

APPROACH:

The NLD, NOR UK, and USA teams have shown SOTA advanced capabilities in analytics of heterogeneous media sources through concept. Their utility has been demonstrated by application to a realistic military scenario using a storyboard outlining applied analytics processes. Specific technical capabilities were:

• NLD – Image classification and indexing capability allowed images to be searched for video frames, including target objects.

• NOR – Image classification capability allowed rapid scanning of video frames for location, object, and behavioural activities of interest in a large video repository.

• UK – Text analytics capabilities that extract entities and relationships from various source documents.

• USA – Text analytics capabilities that classifies social media users and followers as likely to belong to a terrorist group.

FINDINGS:

CBMA allows rapidly exploiting data from multiple sources for sense‑making, decision support and knowledge generation. This includes the contextual understanding of events and event prediction.

An overall challenge is how best to integrate with and exploit the current and future commodity distributed systems (e.g., servers, cloud) and distributed services (e.g., computation, storage).

ML systems are critical in decision support and autonomous systems in NATO operations. These systems influence control over vehicles, sensors and weapons, and the decisions made from sensor inputs; therefore, they need to be human‑centric, as human involvement will always remain critical.

There is too little activity in ML research looking at how we can create more robust systems and how such systems might require a fundamental change in training, testing, validation, and product phases to become resilient to input manipulation.

RECOMMENDATIONS:

services that core technical components developed by the RTG need to provide. The central element is identifying some activity by the analyst, e.g., ‘insurgent activity’. Then, identifying such activity in the operational space and extracting meaningful information from the analytics to build the supporting evidential case. For example, CBMD stages and text analytics has indicated potential adversaries and modes of operation. Further steps and video analytics are extracting facts in answer to queries and iterative refinement whereby relevant elements are linked together, relationships inferred, and added to the evidential case. The final product is a final set of correlated and fused information that pieces together and presents a compelling picture to trigger a decision and action.

The key recommendations for further NATO research between panels IST and HFM are to:

• Investigate designs for a human‑centric distributed system of systems, leveraging the commodity cloud and ML developments.

• Determine the state‑of‑the‑art in robustness and accountability for ML systems. Especially DL systems with complex and large models are virtually impossible to manage by humans.

• Explore potential approaches for the coalition to address emerging socio‑technical issues, such as Ethics and Privacy, arising with human‑machine systems.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

Figure 11 captures the essence of the type of

Figure 13: Intelligence cycle for multi‑modal data

CONTENT‑BASED REAL‑TIME ANALYTICS OF MULTI‑MEDIA STREAMS (RESEARCH SPECIALISTS’ MEETING IST‑158)

ACTIVITY TYPE:

Research Specialists’ Meeting

DURATION:

September 2017 – September 2018

OVERVIEW:

This RSM is one of the outputs of RTG IST‑144. This meeting brought together experts from military agencies, industry, and academia to present and discuss analytic capabilities developed for CBMA.

APPROACH:

The RSM was structured to have a keynote and three sessions (Semantic Multi‑Media Analysis, Imagery Exploitation, and Emerging Issues) for technical presentations on intelligent capturing and processing by sensor systems, exploitation of imagery indexing and expanding the DL approach for semantic video analytics. In addition, the RSM had a specific session for inter‑disciplinary working groups to discuss essential technical areas that should be addressed in future research.

FINDINGS:

Four overarching categories were identified to frame a unified construct for multimedia analytics. These groupings are Capture, Analysis, Dissemination, and Training. Cutting across each of those categories are bias, confidence, and ethics issues.

RECOMMENDATIONS:

Two significant outcomes followed from the RSM:

• Continuation of the shared video indexing experiment.

• Follow‑on human subjects experiment to explore the human analysts’ ability to incorporate multi‑modal information rapidly and accurately with various visual displays.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INFORMATION FILTERING AND MULTI‑SOURCE INFORMATION

FUSION (IST‑106)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2011 – December 2014

OVERVIEW:

RTG IST‑106 was established to examine computerised decision support tools and processes for fusing heterogeneous information.

OBJECTIVES:

To examine optimal pathways to synchronise and align heterogeneous information for decision‑making support.

APPROACH:

In a three‑stage process, relevant information must first be discovered, collated, filtered, matched, and assigned to the suitable process and decision‑maker. Secondly, it must be exploited, validated, merged, aligned, inferred, fused, augmented, presented, and stored so that it can be retrieved and shared easily to support the conservation of organisational knowledge.

FINDINGS:

Databases must be organised such that their data can be retrieved and interpreted for real‑time decision‑making support.

A lone source can contain only weak indications, but these single weak indications yield more concrete hints of a possible threat.

It is necessary to have concepts of normality for comparison and authentication of suspicious events. Otherwise, it is difficult to detect and define abnormal or unusual situations.

We need a methodology for creating these anomalous patterns by military experts and AI tools to support the patterns’ creation, development, and maintenance.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

CONTINUOUS PLANNING PROCESS AND DECISION SUPPORT AT TACTICAL LEVELS (IST‑ET‑084)

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2015 – December 2015

OVERVIEW:

IST‑ET‑084 was established to address the challenge of making planning and execution processes more dynamic, to the extent of being able to plan and execute continuously in unremittingly changing situations.

OBJECTIVES:

To examine the status of NATO’s ability to conduct dynamic planning and execution.

APPROACH:

IST‑ET‑084 explores planning and decision support tools (covering operational and tactical planning). Specific technology areas are plan representation (that supports both human collaborative working as well as used by synthetic agents), automated reasoning (to provide intelligent support to human planners) and learning (for plan re‑use).

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COLLECTION AND MANAGEMENT OF DATA FOR ANALYSIS SUPPORT TO OPERATIONS (SAS‑111)

ACTIVITY TYPE:

Research Task Group

DURATION:

2015 – 2018

OVERVIEW:

RTG SAS‑111 was established to identify data management issues affecting NATO operations. It produced a guide for analytical support to operations with recommendations for how NATO deployed HQs could enhance their decision‑making capability.

OBJECTIVES:

To enhance deployed NATO headquarters’ ability to collect and manage the data required for analytical support to operations.

APPROACH:

Modernisation of the battlespace, e.g., the military intranet of things, will introduce numerous new military data sources and the means to collect available data. In addition, the need to train and implement AI systems to speed up decision cycles will create additional requirements for more sophisticated data collection and management in selected areas.

FINDINGS:

Developments in DL/ML will inevitably lead to the development of autonomous weapon systems and drones equipped with AI.

Autonomous systems need reference scenarios to be trained and conditioned for their missions and the corresponding environments. The key to developing such reference scenarios is accurate geospatial and environmental data with fine granularity, a reliable and precise domain‑specific pattern‑of‑life assessment, and an appropriate tactical scenario.

Most analysts spend too much time gathering, structuring, completing, improving, cleansing, verifying, and validating data. AI techniques can alleviate this issue.

Automated data collection and filtering for strategic electronic warfare is a success story. The electromagnetic environment is complex, congested, cluttered and produces high volumes of data; hence, the analyst is overwhelmed with prioritising reports for further analysis. ML techniques can automatically prioritise messages and reports. Consequently, enabling analysts to identify and prioritise high‑interest reports quickly.

Datasets are assets, and the reuse of properly collected and managed data can benefit all parties involved.

Sharing data among various branches of a military HQ, among NATO entities, or non‑NATO entities is essential to deliver comprehensive and insightful analysis results to decision‑makers, reduce duplication of effort, errors, and inconsistencies, and build trust and trust transparency in military operations. Furthermore, AI tools and techniques cannot deliver meaningful results without access to comprehensive training data.

Sharing should not be restricted to only prepared, presented, and archived data.

Raw and collected data should also be made available to analysts and AI applications for them to be able to verify results or discover new insights from the data.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

BIG DATA AND ARTIFICIAL INTELLIGENCE FOR MILITARY DECISION‑MAKING (IST‑160)

ACTIVITY TYPE:

Research Specialists’ Meeting

DURATION:

May 2017 – June 2018

OVERVIEW:

RSM IST‑160 was established to provide a forum for operational experts and scientists to explore potential research areas enabling NATO information superiority.

OBJECTIVES:

Identify areas of interest and concern associated with Big Data and AI within NATO decision‑making and establish a standard road map.

APPROACH:

Data from a broad spectrum of sources are becoming available and requires analysis to present the information to the operators. Users need information quickly but defining what is relevant can be difficult. AI is sufficiently mature to achieve some of the desires. Critical to the models is the training data, which needs to be extensive. In the military field, some data can be sparse or extremely sensitive. In this instance, the training data could be supplemented with synthetic data derived from models or data farming.

FINDINGS:

The conclusions from the RSM can be summarised under the following broad categories:

State‑of‑the‑art:

The current state‑of‑the‑art AI can provide high‑quality support only if the entities of interest lie within a narrow scope, i.e., are characterised by a well‑defined signature (shapes or family of patterns).

Numerous AI tools are already in commercial use, and with appropriate modifications, they could form the basis for developing robust tools for the Military.

AI development still lacks a cohesive, overreaching theoretical basis and is being explored ad hoc manner

Data:

Development of structured learning may assist where training data is not readily available. In addition, synthetic data to replace sparse data in specific areas can be employed, but care is advised to ensure the appropriate statistics are presented.

Appropriate data and access to data are critical for training and establishing meaningful and comprehensive output in a timely fashion.

Adversaries with fewer restrictions on access to data will harvest the information and could readily achieve information superiority.

Trust, Explainability and Verification:

It is unclear how an AI system designed to be statistically adaptive could be verified and validated, given the complexity of situations to which it could be exposed. Furthermore, the quantity of code and complexity has exceeded any formal analysis capability, which can be undertaken in a reasonable time.

Interactions with humans are strongly recommended, especially where intuition plays a part, which is challenging to capture in an AI tool. The interjection of the operator or decision maker is desirable, but the penalty is that the involvement may influence the reaction time.

The use of AI in military operations will also need to be better explained to the public.

Training and Education:

AI, in its current state of progress, with its operational requirements, anticipated benefits and risks, requires that its users exercise an important level of critical judgement and awareness if it is to be used as a component in critical decision‑making.

Confidence in the presented information derived from the Big Data sets is essential with operators gaining confidence with use. AI as a discipline can provide the appropriate aid; however, users must understand the limitations.

Counter AI:

Uncertainty in the results needs to be recognised, and third parties injecting false and “poisoned” data will influence a measure of confidence expressed as the output of the tools. In addition, the detection of “poisoned” data is non‑trivial.

RECOMMENDATION:

Identify military requirements not currently under research in commercial fields. Identify relevant tools in AI, extant or an advanced stage of implementation, and identify the necessary modifications for military applications.

Consider means to train the tools with the appropriate data. Ensure that the data repositories are available to as broad an audience as feasible and benefit from the existing gathering establishments.

Develop the training methods – for future users – that include a considered, basic understanding of the underpinning of the various forms of AI, advantages, and risks, as well as benefits and costs of exploiting it in the process of decision‑making.

Contribute to the ongoing efforts in improving AI robustness, trustfulness, explainability, verification and validation in its underpinning and deployment in a military context.

Develop adaptive interfaces to match the user and the environment. Potential use of sensors to determine the users’ psychological state.

In some instances, some objects of interest to the military will be uncommon, and the data for training will be rare. A paper proposed employing data derived from modelling to fill the void. Data Farming methods were suggested as an alternative approach.

We should encourage educational establishments to introduce more courses and attract experts to work on military projects with a higher interest level.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MISSION‑ORIENTED RESEARCH FOR AI AND BIG DATA FOR MILITARY DECISION MAKING (IST‑173)

ACTIVITY TYPE:

Research Specialists’ Meeting

DURATION:

December 2018 – June 2020

OVERVIEW:

RSM IST‑173 was established to forge synergies between relevant TPs and develop the S&T Roadmap for AI in NATO.

OBJECTIVES:

• To develop a mission‑oriented research roadmap for Artificial Intelligence and Big Data for Military Decision Making.

• To strengthen the CPoW in the respective research area by increasing cross‑panel activities.

APPROACH:

This RSM describes the methodology of the mission‑oriented research design process, the lessons identified and learned during its application, as well as recommendations and guidelines on how to use and maintain the “living roadmap.”

FINDINGS:

The primary product of this RSM is a NATO S&T “living roadmap.” This roadmap is in the shape of a wiki that can be easily corrected, augmented, and updated. This roadmap is found on NATO Science Connect15 .

There is an urgent need to increase the awareness of what AI and Big Data Analysis is among military decision‑makers.

Improving the availability and sharing of resources like copious amounts of annotated, labelled, and validated data, computing resources, and evaluations across NATO members would greatly benefit the AI training. We need a better understanding of human‑machine interactions and interfaces that enable them.

15 Wiki Living S&T Roadmap for AI and Big Data for Military Decision Making.

RECOMMENDATIONS:

This RSM recommended the following four areas for further research.

• Developing an overall architecture for a NATO Machine Learning Ecosystem.

• Ensuring robustness and accountability in Machine Learning Systems.

• Coordinate and systematize the development and deployment of AI systems.

• Exploration of the potential of Unsupervised Machine Learning in the Military Domain.

Beyond recommendations, the RSM identified questions essential to be addressed for further developing the AI‑enhanced C4 paradigm. For example, how to determine the amount of labelled data required to solve a given ML problem? How to reduce the amount of labelled data necessary to build a machine‑learning model? How to counter AI bias? How to build trust in AI products? How to increase the transferability of AI models between applications.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

BIG DATA CHALLENGES: SITUATION AWARENESS AND DECISION SUPPORT (IST‑178)

ACTIVITY TYPE:

Research Workshop

DURATION:

January 2019 – April 2020

OVERVIEW:

RWS IST‑178 was established to raise awareness of challenges in Big Data research and enable further collaboration between different TPs.

APPROACH:

This RWS brought together members from multiple RTGs and ETs to present, discuss and share their work, problems and challenges relating to exploring, exploiting, and using dynamic Big Data in their respective tasks and mission domains.

AI tools have been presented and demonstrated with the recommendation that more research is undertaken to allow tools to evolve.

FINDINGS:

The volume of raw data is expanding rapidly, increasing the need for AI assistance to decision‑makers in managing the information glut.

Trust in AI is an existential issue; thus, human involvement in the decision‑making remains essential Smart filters and visual analytics play a significant role in assisting the users.

Due to the vast amount of data already exploited for commercial and state purposes, users must be educated to be aware of the impact that third parties can exert.

Social media permits rapid access and dissemination of false information, enabling adversaries to create division and influence the opinion of broad audiences by using bots relaying the data as if it were a true actor. Therefore, NATO needs to consider a proactive stance.

A limited set of training data can be mitigated by introducing simulated data. However, care must be exercised to avoid bias and skewing the outcome.

AUTONOMY TO ACCELERATE THE INTELLIGENCE CYCLE (SAS‑ET‑EG)

ACTIVITY TYPE:

Exploratory Team

DURATION:

July 2019 – July 2020

OVERVIEW:

SAS‑ET‑EG was established to explore the potential of AI‑based autonomy in supporting and speeding up the intelligence cycle.

OBJECTIVES:

To conduct a mapping and analysis of the role of different autonomy levels in the intelligence process.

APPROACH:

The ET suggests analysing the potential of the different autonomy levels in the four phases of the intelligence cycle (Direction, Collection, Processing and Dissemination). Focusing on which level of autonomy is required and desirable in each context, for instance:

• Collection, processing, and fusion of data from multiple sources that consider crosschecking all data sources to reduce deception.

• Processing data that include anomaly detection or pattern recognition and classification of intelligence products.

• Secure dissemination and interoperability of data sharing within NATO.

FINDINGS:

This activity would help lay the foundation for integrating different Autonomy levels into the Intelligence cycle. Given the inherent multi‑disciplinary character of the research questions, it is essential to set up this activity as a crossover activity and involve the Intelligence community to address this topic effectively.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

A HACKATHON TO DETERMINE HOW LARGE EXERCISE DATASETS CAN BE USED TO RECONSTRUCT OPERATIONAL DECISION‑MAKING TO IMPROVE TRAINING AND ANALYSIS VALUE (SAS‑IST‑162)

ACTIVITY TYPE:

Specialist Team

DURATION:

February 2020 – February 2022

OVERVIEW:

This ST was established to explore the potential of a data driven approach for automatic reconstruction of operational decision‑making and lessons learned extraction.

OBJECTIVES:

To explore the potential of a data‑driven approach for automatic reconstruction of operational decision‑making and lessons learned extraction.

APPROACH:

The ST will examine pathways to identifying patterns, operational workflows and gaps in the data related to operational decisions and activities and focus on translating operational subject matter expertise into machine learning formats. The work will produce a technical report that summarises the findings of the Hackathon and describes a structured approach to collecting and analysing substantial amounts of exercise data to reconstruct operational decision‑making and, if possible, automatically extract lessons learned. The ultimate exploitation goal is that the results of this activity will be used in future NATO exercises by NATO bodies, e.g., NCIA, CMRE, JALLC, JFC Brunssum, JFC Naples, LANDCOM, MARCOM, AIRCOM and Nations.

FINDINGS:

This study will conclude in February 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AI, ML AND BD FOR HYBRID MILITARY OPERATIONS

(AI4HMO) (RSY IST‑190 (IWA))

ACTIVITY TYPE:

Research Symposium

DURATION:

April 2021 – April 2022

OVERVIEW:

This RSY was established to provide a forum for experts to present and discuss the state‑of‑the‑art developments and challenges in the vast field of AI, ML and BD in the context of Hybrid Military Operations. The goal of the RSY is to pursue a combined approach to strengthening NATO and NATO nations and establish a road map on “AI, ML and BD for Hybrid Military Operations (AI4HMO).”

APPROACH:

NATO nations face the challenge of hybrid military scenarios increasingly dominated by AI‑based technology. Such scenarios often combine AI technologies in electronic, cyber, and information operations to influence various levels (technical, decision‑making, leadership, population).

This RSY will cover the following scientific topics: AI against hybrid Threats, AI Security, AI Transparency and Traceability, Understandable AI, Adversarial ML and AI, AI and BD based dominance in the Electromagnetic Spectrum, Information Assurance and Cyber Defence, Information Fusion for Hybrid Scenarios, Use of the Electromagnetic Spectrum, Communication Technologies (e.g., 5G, SatCom, WBHF, Terahertz), Human Machine Integration, Hyper automation, Cyber Situational Awareness, Moving Target Defence (MTD), Open Source Intelligence (OSINT), Hybrid Situational Awareness and Hybrid Threats, BD Technologies for Hybrid Scenarios, Detection of Misinformation

FINDINGS:

This study will conclude in April 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

DEEP MACHINE LEARNING FOR CYBER DEFENCE (RTG IST‑163 (IWA))

ACTIVITY TYPE:

Research Task Group

DURATION:

June 2018 – June 2022

OVERVIEW:

This RTG was established to exploit Deep Machine Learning to enhance the strategic cyber position of the military and create a defence that addresses the threats of today and the future.

OBJECTIVES:

Contribute to the improved understanding of how to apply Deep Learning (feedback based machine learning mechanism) to cyber defence.

APPROACH:

The RTG will consolidate the NATO wide knowledge in the field of deep ML and cyber defence, identify the gaps between civilian solutions and military needs, foster joint data processing and data sharing and pursue the transfer of civilian technologies and applications to the military domain.

On a practical level, the RTG will focus on the following. Improving, tailoring, or optimising existing Deep ML techniques for cyber defence applications. Architectural design to enable ML in simulations. Critical factors and potential barriers to the application of ML. Deep ML joint experimentation with varying levels of Deep ML applied to missions with military relevance. The activity will also consider data collection for AI and ML platforms that participating nations can share in developing improved AI and ML algorithms.

FINDINGS:

This study will conclude in June 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

USING SIMULATION TO BETTER INFORM DECISION MAKING FOR WARFARE DEVELOPMENT, PLANNING, OPERATIONS AND ASSESSMENT (RTG MSG‑SAS‑178)

ACTIVITY TYPE:

Research Task Group

DURATION:

July 2021 – July 2024

OVERVIEW:

This RTG was established to enable utilising Modelling and Simulation (M&S) in conducting Operations Research and Analysis to support and enhance decision‑making.

OBJECTIVES:

To develop an action plan for NATO to identify solutions across the DOTMLPFI spectrum to allow NATO to better support decision‑making.

APPROACH:

NATO’s reliance on M&S capabilities must keep pace with the new demands of rapidly evolving changes in the security environment. Accordingly, this RTG will produce recommendations for how NATO should best use M&S to support decision‑making across its enterprise. Examples of use cases should illuminate these. Furthermore, the study will identify appropriate mechanisms for NATO to access and employ national M&S capabilities.

This RTG will cover the following requirements for application domains: Defence Planning, Operations Planning, Decision Support, Warfare Development, Concept Development and Experimentation.

FINDINGS:

This study will conclude in July 2024, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AGILE, MULTI‑DOMAIN C2 OF SOCIO‑TECHNICAL ORGANISATIONS IN COMPLEX ENDEAVOURS (RTG SAS‑143)

ACTIVITY TYPE:

Research Task Group

DURATION:

April 2018 – April 2022

OVERVIEW:

This RTG was established to provide a way to systematically describe and assess the appropriateness of different Multi‑Domain C2‑Harmonisation Approaches appropriate for Kinetic, Cyber, and Non‑Kinetic operations.

OBJECTIVES:

To explore the nature of agile multi‑domain C2 of a socio‑technical enterprise that includes humans, intelligent networks, and autonomous entities in a cyber‑contested and hostile environment.

APPROACH:

RTG SAS‑143 is developing the C2 concepts and tools necessary to achieve harmonisation across operations in multiple domains with various human and non‑human partners. These will be assessed in simulations, war games and exercises. Scientific objectives are to investigate the following:

• C2 Agility implications of socio‑technical enterprises – collaborating teams of human, autonomous entities, and intelligent networks.

• Challenges associated with agile Multi‑domain C2 of Hybrid Operations.

• Harmonisation of C2 Approaches appropriate for Kinetic, Cyber, and Non‑Kinetic operations.

The research results will be produced in a final report with findings and conclusions, updates to C2 Agility Theory and recommendations for C2 Practice.

FINDINGS:

This study will conclude in April 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AUTOMATION IN THE INTELLIGENCE CYCLE (SAS‑157)

ACTIVITY TYPE:

Research Task Group

DURATION: January 2020 – January 2023

OVERVIEW:

This RTG was established to investigate how techniques from the fields of Artificial Intelligence (AI) and autonomous systems might be applied in support of the intelligence process and in which part of the intelligence cycle should they be used to achieve improved outcomes.

OBJECTIVES:

To identify opportunities to accelerate the Intelligence cycle by applying AI enabled systems and automation.

APPROACH:

The RTG will examine at which particular phases/ (sub)functions/activities of the intelligence cycle could AI and automation play a significant role and which combination of technologies is deemed the most promising for each phase and forms an opportunity for improvement. Secondly, the RTG will select a subset of identified opportunities and qualitatively as well as quantitatively assess the benefit of automation in specific cases within national and NATO experiments/exercises.

To answer tendered research questions, the RTG will conduct conceptual mapping of AI technologies against functionalities of the intelligence cycle, based on the benefit they could bring. Assess the hypothesized benefits of AI and autonomy for the intelligence cycle through field experiments with intelligence professionals. Finally, report overall conclusions and make future recommendations to improve the intelligence cycle, which is valuable for NATO nations and the NATO intelligence community.

FINDINGS:

This study will conclude in January 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

EMPLOYING AI

TO FEDERATE

SENSORS IN JOINT SETTINGS (SAS‑158)

ACTIVITY TYPE:

Research Task Group

DURATION:

June 2020 – June 2023

OVERVIEW:

This RTG was established to improve mission planning through a multi‑sensor federation, a combined planning and mission optimisation process actively selecting and allocating multiple complementary sensors.

OBJECTIVES:

To design a system that enables optimal multi‑sensor mission planning.

APPROACH:

This RTG will combine systems and component level considerations as part of the research activity. The design will be based on simulated scenarios, multiple sensors, connectivity options and platforms, and metrics representing specific mission objectives. This objective includes utilising Machine Learning to fuse the resultant data and steps to validate that the mission objectives have been fulfilled with the predicted asset allocations. Furthermore, the dynamics and complexity of the mission planning optimisation techniques and Artificial Intelligence techniques like reinforcement learning will be explored to derive this complex planning. In practice, the RTG will specifically focus on the following:

• Practical optimisation methods for sensor management, including methods for defining operationally relevant metrics.

• Modelling and simulation to characterise operational scenarios based on their physical characteristics.

• Model validation using high fidelity simulation.

• The use of Machine Learning to process and fuse heterogeneous data from multiple sensors.

FINDINGS:

This study will conclude in June 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ETHICAL, LEGAL, AND MORAL (ELM) IMPACTS OF NOVEL TECHNOLOGIES ON NATO’S OPERATIONAL ADVANTAGE –THE “ELM TREE” (SAS‑160)

ACTIVITY TYPE:

Research Task Group

DURATION:

July 2020 – July 2023

OVERVIEW:

This RTG was established to provide insights into the ELM constraints assumed to be imposed by novel and emerging technologies on NATO’s operational advantage.

OBJECTIVES:

To develop a conceptual framework providing situational awareness of ELM risks and issues.

APPROACH:

The activity will provide an understanding of operational challenges shaped by the ELM issues and deliver the ‘ELM Tree’ as a framework for considering these concerning game changing technologies in the future operating environment. The emphasis will be on strategic analysis relating to ELM issues in the military context, principally applied to disruptive technologies. The RTH will conduct case studies, historical assessments, scenario and vignette analyses and decision support.

The ‘ELM Tree’ framework will identify the key ELM implications for critical decision‑makers and capability risk holders within NATO, with the output oriented toward recommended actions for resolving the threats and vulnerabilities identified from the discrete results of the research.

FINDINGS:

This study will conclude in July 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HOW COULD TECHNOLOGY DEVELOPMENT TRANSFORM THE FUTURE OPERATIONAL ENVIRONMENT (SAS‑159)

ACTIVITY TYPE:

Research Task Group

DURATION:

March 2020 – December 2023

OVERVIEW:

This RTG was established to provide insights into the potential of the ongoing technological developments and identify the technological developments that will significantly alter the conduct of conflict, adaptation of planning processes and change long term goals, strategies, and concepts.

OBJECTIVES:

To explore the possible consequences of disruptive technology development on the future operational environment in the 2040 to 2050 perspective.

APPROACH:

The work will build upon existing analyses of technology and strategic trends, including identified science and technologies of interest from each member nation. The results can be used to update the STO tech watch cards and NATO Tech Trends Report. The results from the task group will also inform national and collective capability development to close gaps identified across DOTMLPFI and potentially increase interoperability across nations and the Alliance.

FINDINGS:

This study will conclude in December 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ROBUSTNESS AND ACCOUNTABILITY IN MACHINE LEARNING SYSTEMS (RTG IST‑169 (AI2S))

ACTIVITY TYPE:

Research Task Group

DURATION:

June 2018 – June 2022

OVERVIEW:

This RTG was established to provide an overview of the potential for building and documenting accountability in Machine Learning systems and the possible impact that this will have.

OBJECTIVES:

To determine the state‑of‑the‑art in robustness and accountability for Machine Learning and Deep Learning systems.

APPROACH:

The RTG will examine the robustness of ML systems during training and operations, accountability of such ML systems and methods of interaction between different nations’ ML systems in a federated military environment concerning robustness and accountability. The RTG will also investigate vulnerabilities and countermeasures to manipulating ML systems.

The activity will examine whether a methodology can be designed to verify commercial ML systems compliance with military criteria to enable the transfer of commercial ML systems to the military, including defining what kind of criteria should be mandatory in a military setting.

FINDINGS:

This study will conclude in June 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MULTI‑DIMENSIONAL DATA FARMING (RTG MSG‑186)

ACTIVITY TYPE:

Research Task Group

DURATION:

July 2021 – July 2024

OVERVIEW:

This RTG was established to explore how data farming allows NATO military decision‑makers in defence planning, operations, training, and capability development to reduce uncertainty.

OBJECTIVES:

To develop analytical methods and simulation tools enabling Data Farming for M&S of multiple areas affecting military operations.

APPROACH:

This RTG will provide results extending data farming capability through advanced analytics using Bayesian approaches and AI. These advancements will affect capability gaps in decision support for understanding future NATO efforts in complex environments such as Hybrid Warfare. The central theme will be leveraging multiple models capturing multiple elements of warfare relevant to decision‑making. This task group will exploit the development of cyber data farming use cases.

After the work of this RTG, it is expected that data farming and associated services will enable more precise M&S of multiple areas influencing operational questions to support decision‑makers who need to consider the interactions inherent in various domains.

FINDINGS:

This study will conclude in July 2024, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN MACHINE SYMBIOSIS (HMS)

COMPLETED RESEARCH

UNMANNED MILITARY VEHICLES: HUMAN FACTOR ISSUES IN AUGMENTING THE FORCE (HFM‑078)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2002 – March 2006

OVERVIEW:

This RTG was established to assess how Unmanned Military Vehicles (UMVs) can effectively augment the manned forces.

OBJECTIVES:

To improve operational effectiveness and efficiency through adequate force augmentation with UMVs. At the same time, providing a single point of focus for identifying, prioritising, and addressing human factors challenges associated with UMVs. Finally, to augment the force using UMVs by leveraging the potential advantages of UMVs to function as force multipliers.

APPROACH:

This RTG revised traditional manned systems theoretical frameworks and highlighted aspects directly applicable to optimising operator/vehicle ratios and interoperability of uninhabited systems.

FINDINGS:

Collaborative Human Machine Teamwork

– Optimal Task Distribution, Virtual team performance, Manned/Unmanned collaboration, Interoperability, Flexible level of automation, Optimisation of operator/vehicle ratio. .

Human automation and Control Stations

– Intelligent Operator Support, Operator functional state assessment, Intelligent adaptive interfaces, Cognitive co operation.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SUPERVISORY CONTROL OF MULTIPLE UNINHABITED SYSTEMS: METHODOLOGY AND ENABLING HUMAN‑ROBOT INTERFACE (HFM‑170)

ACTIVITY TYPE:

Research Task Group

DURATION:

April 2008 – May 2011

OVERVIEW:

RTG HFM‑170 was established to address research gaps in the coupling of intelligent autonomy with the decision‑making responsibilities of the operator to maximise mission effectiveness.

OBJECTIVES:

To develop and demonstrate the pertinent supervisory control system, methodologies, interface design practices and concepts for UV operations. RTG HFM‑170 aims to enable single operator control of multiple UxVs with varying degrees of autonomy.

APPROACH:

This RTG considers a paradigm whereby a single operator simultaneously supervises multiple autonomous UxVs.

Various critical issues pertaining to the AI applications in supervisory control and enabling technology were addressed in the RTG HFM‑170 technical demonstrations. Such as:

• Applied Artificial Impedance Control (AIC) for the trajectory generation for Unmanned Ground Vehicles (UGVs).

• Principles of the Self Organising Multi Agent System (MAS).

• Advancing pilot assistant systems for reconnaissance.

• Using a hybrid intelligent agent system to coordinate a team of ground robots.

FINDINGS:

In four specific Technology Demonstrations (TDs), RTG showed that the operator’s role is becoming supervisory since future UxVs will be increasingly automated due to AI associated systems’ advances.

Artificial Impedance Control (AIC):16 AIC is applied for the generation of trajectories of UGVs instead of pre‑planning the path. AIC enables UGVs to perform obstacle avoidance tasks without knowing the obstacles’ entire geometry and the environment’s geometry. Once the AIC for a single robot was developed, functionality was extended to multiple robots.

• Communication links are essential to control multiple unmanned vehicles; thus, distributed computing power is necessary for system stability.

• The AIC enables UGVs to avoid obstacles without requiring a complex vision system. The closest point of the obstruction surface from the vehicle at each time provided by simple range sensors is a sufficient reference point for AIC.

• Autonomous mission management and robust ground control station interface are focal areas for future research.

16

Multi Agent Systems (MAS):17

MAS are a sub field of AI devoted to enabling a group of artificial agents to accomplish tasks in cooperation. Supervising actual MAS requires entities to make decisions autonomously.

The study demonstrated the adaptability of swarm intelligence in simple missions, such as surveillance and intrusion tracking. Secondly, it analyses the gap between the swarm algorithm’s performance and the perception operators may have from these elements.

• Swarm intelligence is a promising approach for multiple UxVs control in terms of algorithmic performance and robustness.

• The most important lesson relates to the behaviour understanding gap and the operators’ trust in the swarm’s algorithm.

• Man machine interfaces need better designs and modalities to adapt to commands and support meaningful interaction.

Assistant UAV Systems:18

The study examines the deployment of multiple UAVs as remote sensor platforms in military helicopter missions. The goal is to reduce the workload of the helicopter commander. UAVs provide real‑time reconnaissance without exposing humans to threats. The commander aboard the mission leading helicopter will conduct the guidance of these tasks.

An artificial cognition based agent aboard each UAV would understand the operator given tasks and generates tactical sense making behaviours. The commander provides a single or a series of tasks to each UAV via a graphical user interface based on a moving map display.

• The introduction of the assistant system improves human factors related variables like situation awareness and workload, improves performance and safety and is well accepted.

• Since multi‑UAV guidance from the cockpit is a novel task, it requires further development of generic approaches, mission procedures and proper training for the pilots and commanders.

Intelligent agents:19

The research examines two main approaches to supervisory control, distributed intelligence using swarm technologies and centralised intelligence using an intelligent agent as an intermediate supervisor. The study demonstrates the dynamics of an intelligent agent interacting with the human operators to coordinate robots conducting a reconnaissance mission.

The experiment proved the feasibility of using the hybrid supervisory system to control up to eight robots. Examined systems reliability level and type of errors. Showed the system’s efficacy in aiding the operator conduct more complex missions, which required four robots to entrap a moving target.

• Research showed the advantages of a hybrid supervisory system with a centralised agent controlling up to 8 UxVs in an urban combat environment.

• AI agents improve the operator‑to‑vehicle ratio in a military environment if there are weak dependencies between the task that the operator performs and the task that the autonomous function performs.

• The research demonstrator proved that a better, compact interface in the current experiment allowed verification to be accomplished more efficiently.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

17 Chapter 6 – Fra 1: UAV Swarm Control – Smaart Project “Interacting With Multi Agent Systems/UAV Swarms.”

18 Chapter 8 – Ger 1: Cognitive and Cooperative Assistant System for Aerial Manned Unmanned Teaming Missions.

19 Chapter 16 – US 3: Intelligent Agents as Supervisory Assets for Multiple Uninhabited Systems: Roboleader.

ROBOTS UNDERPINNING FUTURE NATO OPERATIONS (ANALYSIS OF HUMAN MACHINE INTERFACES) (SAS‑ET‑BX)

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2011 – January 2012

OVERVIEW:

SAS‑ET‑BX was formed to define research areas affecting the use of robotic systems in military operations and reassess their advances in NATO operations.

OBJECTIVES:

Identify the research gaps in the role of robotics in military operations.

Develop new AI control tools and perception algorithms applicable to military scenarios and embed them into the cognitive framework.

APPROACH:

This ET explored the extensive usage of robotics in NATO operations and analysed the gap between requirements and technical possibilities compared to the Long Term Capability Requirements (LTCRs) set.

FINDINGS:

The ET identified the following research areas of importance: robots’ cognitive abilities, self‑localisation and navigation, robotic coordination and teamwork, scalable robotic simulations, and robotics applications.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SUPERVISORY CONTROL OF MULTIPLE UNINHABITED SYSTEMS – METHODOLOGIES AND HUMAN‑ROBOT INTERFACE TECHNOLOGIES (HFM‑217)

ACTIVITY TYPE:

Research Workshop

DURATION:

September 2011 – June 2013

OVERVIEW:

RWS HFM‑217 was established to disseminate the results and lessons learned associated with the technical demonstrations conducted by RTG HFM‑170

APPROACH:

The technical demonstrations that had taken place at member nations under RTG HFM‑170 served as the basis for the RWS. The technical part of the program consisted of two keynote presentations and thirteen interactive Technology Demonstrations followed by an overall summary.

FINDINGS:

We need not focus only on supervisory control of uninhabited systems but also on the multi‑faceted and reciprocal interactions between humans and automated processes. Such as understanding and deciding which type of task distribution in controlling multiple UxVs best matches the real‑world scenarios.

A model like that could also provide consistency in user interface design decisions around NATO. The difference between the roles of humans and machines and between the notions of uninhabited systems and other automated processes will inevitably blur.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

Figure 14: Some critical success factors for human machine symbiosis

HUMAN‑AUTONOMY TEAMING: SUPPORTING DYNAMICALLY ADJUSTABLE COLLABORATION (HFM‑247)

ACTIVITY TYPE:

Research Task Group

DURATION:

April 2014 – December 2018

OVERVIEW:

RTG HFM‑247 was established to explore the rapidly developing area of Human Autonomy Teaming (HAT, driven by the progress in AI (particularly ML)), Robotics, Cloud Computing, Conversational Agents, Mixed Reality, Internet of Things and Sensing.

OBJECTIVES:

Identify and demonstrate successful teaming methodologies and interface design practices that allow for shared situational awareness of the task and environment, bi directional understanding of intent, dynamic work distribution, and effective mission collaboration.

APPROACH:

RTG HFM‑247 directly leverages RTG HFM‑170, which developed a framework for supervisory control of human‑robotic systems. First, the RTG overviews standardisation of terminology describing highly automated Human Machine Systems along with top level reoccurring problems faced in HAT applications and then proposes a few candidate HAT solutions that might serve as design pattern instantiations.

The RTG also defines abstract patterns, which could be considered solutions to the main challenges related to introducing an artificial agent within a collective of human teams.

FINDINGS:

Strengths and weaknesses in hybrid human‑AI teams are being supplemented and compensated. For example, humans and AI are mutually adaptive (learning) agents in such teams.

As modern systems take their autonomy from learning abilities (DL), the respective modes of reasoning developed by artificial autonomous systems and expert operators have a considerable risk of not being aligned and mutually understandable.

Human machine interactions are based on interdependency management, coordination, conflict resolution, team mutual and self‑monitoring and evaluation. These operations rely on effective communication. Similarly, the autonomous system must be equipped with emotion detection and coping so that it will understand stress levels and adapt accordingly.

Technological limitations, legal and moral constraints, and most significant human involvement suggest that some form of human‑in‑the‑loop control always will be required. These challenges mean finding the right balance of human supervision and automation must be identified in context.

Figure 15: Three human agent partnership interactions: objective sharing, harmonising activities via work agreements and explaining task outcomes

The following areas of research require further development:

• Meaningful Human Control: How to establish and maintain across all AI systems?

• Explainable AI in human agent teamwork. AI need to be further explored to facilitate trust calibration and appropriate reliance.

• Evolving hybrid intelligence by co learning. Human and machine actors should be able to feed and consult the models in such a way that it contributes to the team’s performance.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AUTONOMY FROM A SYSTEM PERSPECTIVE (RSM SCI‑296), 2016

ACTIVITY TYPE:

Research Specialists Meeting

DURATION:

August 2016 – December 2017

OVERVIEW:

RSM SCI‑296 was established to provide a system‑level perspective for the autonomous platform and pertinent sub systems.

OBJECTIVES:

• To identify areas where NATO should increase scientific focus and use autonomy and autonomous systems.

• To raise awareness of all the ongoing & planned NATO activities in the autonomy area and piece together synergies between various S&T areas.

APPROACH:

Three underlying approaches to accomplish the objectives were defined: 1) to understand what levels of human‑machine interdependence are currently enabled by the state‑of‑the‑art. 2) Within a framework of mission classes, to assess where increases in human machine interoperability would cause an improvement in existing applications. 3) To assess the underlying S&T advances, limitations, shortfalls, and open issues identified in the first two goals and plot a way forward.

FINDINGS:

There is an urgent requirement for capability development concerning autonomous systems, specifically to flatten the cost curve. Currently, the commercial sector is driving the innovation, but the military sector must remain knowledgeable and seize opportunities to influence the utility of these innovations in a military context.

AI’s inability to infer context is derived from the lack of annotated ground truth data for military applications of autonomy. It is impossible to train and validate ML algorithms without this data.

AI will be inherent in human machine systems as the prerequisite for optimal situational awareness. Therefore, the design must accommodate adaptation to evolving circumstances and the associated training, verification, and validation of the human‑machine team.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN AUTONOMY TEAMING (HFM‑300)

ACTIVITY TYPE:

Symposium

DURATION:

January 2018 – January 2019

OVERVIEW:

RSY HFM‑300 was established to disseminate the results and lessons learned from RTG HFM‑247

APPROACH:

RSY HFM‑300 includes presentations and demonstrations of selected TAs conducted throughout RTG HFM‑247. The RSY comprised five themes: operational requirements; Human‑autonomy teaming structure; autonomous capabilities that support teaming; HAT interaction and design; and HAT institutional integration.

FINDINGS:

Key enabling HAT technologies include perceptual processing, planning, learning, interaction, natural language understanding and multi‑agent coordination.

The challenge of augmentation of Situation Awareness promises soon to be ameliorated by advances in AI sensor fusion capability and mission data fusion. Computer algorithms can acquire, distil, organise, and present otherwise disparate pieces of intelligence into a single picture.

Autonomy can be employed in diverse ways for various mission phases and affects how different agents synchronise activities across mission phases.

The balance in allocating cognitive functions varies between the missions and phases and the phases of cognition in decision‑making. More data‑intensive tasks tend to be associated with AI applications, improving efficiency, and reducing workforce demands.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MEANINGFUL HUMAN CONTROL (MHC) OVER AI‑BASED SYSTEMS (HFM‑ET‑178)

ACTIVITY TYPE:

Exploratory Team

DURATION:

October 2018 – October 2019

OVERVIEW:

HFM‑ET‑178 was formed to address MHC suggestions mentioned during RSM SCI‑296 This ET AI builds upon work on human autonomy teaming conducted in HFM‑247, specifically upon suggested design patterns.

OBJECTIVES:

• To develop a definition and scope for MHC research for broader use within NATO.

• To delineate factors impeding/contributing to MHC in future military applications of AI‑based systems.

APPROACH:

MHC was examined from multiple perspectives to better identify the essential features and drivers for effective MHC over automated and AI‑based systems. In addition, emphasis was put on researching how human machine cooperation and interaction takes place and how human machine teams will be trained.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AIR VEHICLES CREW’S NEURO‑PSYCHOPHYSIOLOGICAL BASED REAL‑TIME STRESS MONITORING FOR HUMAN MACHINE INTERFACES WORKLOAD EVALUATION ENFORCING MISSION EXECUTION (HFM‑AVT‑ET‑185)

ACTIVITY TYPE:

Exploratory Team

DURATION:

October 2019 – October 2020

OVERVIEW:

HFM‑AVT‑ET‑185 was established to address the interaction between pilot and aircraft, which is considered a milestone in achieving better operational performance while increasing safety and mission effectiveness.

OBJECTIVES:

• To define a common set of requirements and standards for the Airborne Infrared Stress Monitoring System (A‑ISMS).

• To obtain a recognised Bedford Workload Rating Scale objective metrics with specific Open System Architecture compliant OPerational SoftWare (OPSW) data from different NATO partners.

APPROACH:

Re thinking human machine interaction based on advancements in AI, DL and neuroscience opens the possibility of generating more reliable methodologies and methods, with specific metrics used to evaluate the Human Machine Interface’s effectiveness, to address and validate the solutions for creating a new generation of cockpits centred on the human factor.

The use of clearly defined and validated standards, as well as auxiliary systems such as AI and DL that are easily installable and usable on military aircraft, will ease the adoption of A ISMS and improve the operational use and training effectiveness of existing and newly designed air vehicles and flight/ mission simulators.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN MACHINE SYMBIOSIS (HMS)

ONGOING RESEARCH

MEANINGFUL HUMAN CONTROL OF AI‑BASED SYSTEMS: KEY CHARACTERISTICS, INFLUENCING FACTORS AND DESIGN CONSIDERATIONS (RWS HFM‑322)

ACTIVITY TYPE:

Research Workshop

DURATION:

October 2019 – February 2022

OVERVIEW:

This RWS was established to exploit previous STO’s research on meaningful human control of AI systems, build synergies and explore pathways to address the unbridled proliferation of AI technology and the urgent need to develop AI‑based military capabilities.

OBJECTIVES:

To identify areas where NATO should increase scientific and technical focus on autonomy and meaningful human control of autonomous systems.

APPROACH:

This RWS builds upon RSM SCI‑296 and HFM‑ET‑178, which came to a consensus on the description of MHC, which is “Humans have the ability to make informed choices in sufficient time to influence AI‑based systems in order to enable a desired effect or to prevent an undesired immediate or future effect on the environment.”

RWS HFM‑322 aims to apply the architecture proposed by HFM‑ET‑178 to organise the input of the different domain experts. Furthermore, it allows us to identify and map different vocabularies used in various domains, develop methods to foster cross domain fertilisation between other domains, validate the integration architecture, and determine shortcomings. Finally, the RWS will generate recommendations for follow‑on activities to inform NATO on identifying, achieving, maintaining, and regaining meaningful human control across various AI applications.

FINDINGS:

This study will conclude in February 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ASSESSMENT OF AUGMENTATION TECHNOLOGIES FOR IMPROVING HUMAN PERFORMANCE (RTG HFM‑297)

ACTIVITY TYPE:

Research Task Group

DURATION:

September 2017 – June 2022

OVERVIEW:

The RTG HFM‑297 was established to assess the impact of emerging augmentation technologies that enhance the assistive and immersive properties of virtual, augmented, and mixed reality environments, thereby improving learning, performance, retention, and transfer of skills from training to operational environments.

OBJECTIVES:

To assess the effect of emerging technology interaction capabilities for individuals and units on learning, retention, performance, and transfer.

APPROACH:

To move emerging augmentation technologies from state‑of‑art to state‑of‑practice for training and operations within the NATO Alliance nations, this RTG will conduct the following steps.

Identify technologies to interact with, stimulate the user’s perceptual systems, and measure the resulting improvements in higher learning, performance, retention, and transfer. Identify any adverse effects of augmentation technologies and tools/methods to ameliorate any adverse effects. Examine Return on Investment (ROI) to deploy any promising augmentation technologies identified. Moreover, as these technologies evolve, consideration of the requirements for standardisation and interoperability will be addressed.

FINDINGS:

This study will conclude in June 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN SYSTEMS INTEGRATION FOR MEANINGFUL HUMAN CONTROL OVER AI‑BASED SYSTEMS (RTG HFM‑330)

ACTIVITY TYPE:

Research Task Group

DURATION:

February 2020 – February 2023

OVERVIEW:

This RTG was established to examine the effect national cultures and procedural differences have on the shared understanding of Meaningful Human Control during coalition operations and distil a set of practical human centred guidelines to inform future NATO.

OBJECTIVES:

To develop Human System Integration principles and guidelines for Meaningful Human Control in future NATO operations and acquisitions.

APPROACH:

The RTG aims to establish beneficial and usable guidelines that assist NATO nations in conceptualising, developing, organising, assessing, and validating the MHC of future AI‑based systems.

To provide such guidelines, the RTG will undertake the following: First, investigate current approaches from defence and other domains. Second, determine how MHC can be embedded in an accreditation process to ensure that human accountability is maintained through information management, fusion, decision‑making and action processes. Third, investigate what individual measures of various aspects of human control are and how can these be integrated. Finally, based on mentioned actions, the RTG will develop a model containing the main concepts such as trust, training, accountability, and situation understanding.

FINDINGS:

This study will conclude in February 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COGNITIVE NEUROENHANCEMENT: TECHNIQUES AND TECHNOLOGY (RTG HFM‑311)

ACTIVITY TYPE:

Research Task Group

DURATION:

December 2019 – December 2023

OVERVIEW:

This RTG was established to examine cognitive enhancement techniques and determine paths to their military deployment to maintain a strategic performance advantage over adversaries.

OBJECTIVES:

To examine the state‑of‑the‑art research, techniques, and technologies in cognitive neuroenhancement and report on development efforts and lessons learned.

APPROACH:

The RTG will collate and examine neuromodulation, cognitive training, and biofeedback. Report on research and development efforts, lessons learned, strengths and weaknesses of each approach and combinations of methods, best practices among the NATO participants, technological challenges, and other important considerations for deployment. The topics will encompass techniques, technologies, and interventions that target cognitive performance enhancement, readiness, resilience, and accelerated recovery.

Each of the NATO members currently committed to this RTG has different areas of expertise (e.g., non‑invasive stimulation, artificial intelligence, biofeedback, cognitive training, etc.). By bringing these diverse cognitive enhancement methods together, we may be able to develop synergies between them and combined techniques that far outpace the benefits of any single cognitive enhancement method time.

FINDINGS:

This study will conclude in December 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE, SURVEILLANCE AND RECONNAISSANCE (ISR) – DATA FUSION

COMPLETED RESEARCH

MULTISENSOR FUSION: ADVANCED METHODOLOGY AND APPLICATIONS (RLS SET‑157)

ACTIVITY TYPE:

Research Lecture Series

DURATION:

March 2010 – December 2012

OVERVIEW:

RLS IST‑157 presented the state‑of‑the‑art in data fusion technology and its applications. Thereby increasing awareness of its value to the NATO scientific and engineering communities.

OBJECTIVES:

To introduce modern distributed sensor networks and sophisticated current tracking and data fusion technologies.

APPROACH:

Lectures by leading experts in the field discussing advanced applications relevant to covert surveillance by distributed active or passive radar/sonar networks; security assistance systems for NATO DAT, or high‑precision and reliable multisensory fusion products for better production situation pictures in NATO’s ISTAR systems, for instance.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INFORMATION FUSION (HARD AND SOFT) FOR ISR (RSY

IST‑SET‑126), 2014 – 2015

ACTIVITY TYPE:

Symposium

DURATION: October 2014 – December 2015

OVERVIEW:

The IST‑SET‑126 offered a forum discussion on Information Fusion (Hard and Soft) and assessing the combination of both disciplines.

APPROACH:

The Symposium included lectures by invited speakers from research and practice and a selection of submitted papers.

OBJECTIVES:

To provide an interdisciplinary forum for research scientists, military experts, and system engineers to present the state‑of‑the‑art research and technology in military data and information fusion for ISR, both Hard and Soft.

FINDINGS:

Common methodologies, algorithms, and tools in both hard and soft in the ISR domain

Standardising definitions of ambiguous terminology (reconciliation of terminology discrepancies)

Data/information representation strategies for data and information derived from hard fusion, soft fusion, and the fusion of both

Combining results from multi‑modal sensing data fusion and distributed sensor networks and data with high‑level intelligence information

Integration of unmanned ISR sensors and systems into comprehensive hard/soft fusion

Challenges for a seamless fusion of hard and soft data

RECOMMENDATIONS:

The RSY identified the following short‑term topics:

• Retaining a measure of confidence with each entity.

• Handling of subjective information.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTEROPERABILITY & NETWORKING OF DISPARATE SENSORS AND PLATFORMS FOR ISR APPLICATIONS (RTG SET‑218)

ACTIVITY TYPE:

Research Task Group

DURATION:

September 2014 – December 2017

OVERVIEW:

The RTG SET‑218 was established to address the lack of easy integration, interoperability, and networking of disparate ISR sensor assets and sensing platforms for coalition operation.

APPROACH:

The Research team explored and assessed the following topics:

• Implementation of middleware solutions

• Map solutions to appropriate STANAG requirements and make common data schema available for data ingest

• Conduct interoperability events using solutions in a real‑world environment

OBJECTIVES:

To achieve coalition interoperability of widely disparate ISR sensors from different nations by developing interoperability standards and demonstrating middleware for ease of integration and interoperability of the ISR assets to increase military effectiveness.

FINDINGS:

The integration of disparate coalition sensors with proven middleware solutions enables true interoperability. Exploitation solutions can ingest and view sensor data across the national spectrum, making such systems sensor agnostic. If implemented correctly, the notion of stove‑piped national tactical level systems is moot. This would allow NATO teams to use the “best of breed” of available sensors rather than relying solely on a national toolkit.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ADVANCED ALGORITHMS FOR EFFECTIVELY FUSING HARD AND SOFT INFORMATION (RLS IST‑134) AND ADVANCED ALGORITHMS FOR EFFECTIVELY FUSING

HARD AND SOFT INFORMATION (RLS‑IST‑155)

ACTIVITY TYPE:

Research Lecture Series

DURATION:

January 2015 – December 2015 and 2016 – 2017

OVERVIEW:

RLS IST‑134 was established to disseminate the existing knowledge on sophisticated algorithms for hard and soft fusion among researchers and systems engineers in NATO’s member states. RLS IST‑155 is a continuation of RLS IST‑134

APPROACH:

The RLS Team presented core methodologies and algorithms that solve the various aspects of “hard” and “soft” information fusion. Further, the RLS team discussed the integration of context information that plays a fundamental role in emerging information fusion systems. This RLS focuses on both knowledge representation by advanced formalisms and computer linguistic approaches.

FINDINGS:

The Bayesian approach was an effective solution to extended object and cluster tracking.

Natural language processing algorithms enhanced by AI and advanced analytics allow much helpful information gleaned from the text. It is challenging to find access to the underlying general methodology and to apply the inventory of various fusion techniques to solving individual application problems.

Defining “context” for soft data is complex and variable.

Research articles addressed various challenges associated with advanced algorithms and data fusion processes. The Bayesian approach was an effective solution to extended object and cluster tracking. The resulting tracking algorithms model ellipsoidal object extensions from simulated sensor data of a partly unresolvable formation. This method is relevant for tracking large, collectively moving target swarms.

Effective use of soft data in the fusion process depends upon understanding information uncertainty. Textual markers of uncertainty already exist in the natural language sources and should be identified and exploited by algorithms to ensure a realistic assessment of information credibility. Natural language processing algorithms enhanced by AI and advanced analytics allow beneficial information gleaned from the text. However, the differing needs of the short timeframe situation awareness and long intelligence time horizon, as well as the changeability of human activity being observed, requires not only the storage of information which may seem irrelevant, but also flexibility in the underlying storage structures.

The RLSs addressed numerous opportunities and challenges associated with advanced algorithms and data fusion processes in ISR. The RLSs agree that it is particularly challenging to find access to the underlying general methodology and apply the inventory of various data fusion techniques to solving individual military application challenges.

Contextual knowledge has been pointed out as a key element to reason about the properties and relations of the entities of interest in several ISR domains, particularly computer vision and anomaly detection. In addition, natural language processing algorithms and text analytics allow much useful context to be gleaned from the text. The RLS suggested a layered architecture to formalise contextual knowledge in fusion processes.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ARTIFICIAL INTELLIGENCE FOR MILITARY MULTISENSORY FUSION ENGINES (RSM SET‑262)

ACTIVITY TYPE:

Research Specialists’ Meeting

DURATION:

March 2018 – November 2018

OVERVIEW:

RSM SET‑262 was established to foster further research and build synergies in the research on advanced methodologies and algorithms of AI‑inspired sensor informatics.

OBJECTIVES:

To share the latest research on sensor AI topics addressing the needs of NATO’s future missions.

APPROACH:

With an emphasis on military applications of AI methods, the RSM presented the latest research such as probabilistic reasoning over time, statistical decision‑making, sparse data in multiple sensor data fusion for tracking, classification, anomaly detection, Bayesian and machine learning methods, knowledge‑representation, multiple hypothesis and logical analysis, sensor and resources management and the multi‑functionality, i.e., the shared use of sensing hardware to achieve specialised goals.

FINDINGS:

Multi‑sensor Fusion Engines are and will be the backbones of NATO’s situational awareness capabilities with disruptive effects on friendly and adversary forces.

Enormous quantities of labelled data are needed to train AI, necessitating large amounts of actual data, supplemented by high fidelity simulated data.

Outputs from algorithms using AI techniques, whether based on ML or DL approaches, must be explainable to end‑users.

Minor changes to the input data can compromise deep Learning Neural Networks; therefore, it is critical for any AI method to be robust against adversarial attacks.

Currently available AI methods are slow to train, thus slow to adapt to rapidly changing threats, making them unsuited to some live encounters.

The availability of massive quantities of real‑time sensor data brings opportunities for new data fusion algorithms optimised for big data processing. AI‑based data analytics are proficient in detecting anomalous behaviour and tracking extended and group objects such as pedestrians (vehicle convoys, bio‑chemical plumes, etc. using multiple heterogeneous sensors. Furthermore, more advanced ML and DL methods can alleviate issues with target classification based on strongly conflicting sensor information. Moreover, AI techniques demonstrated value in other optical image classification of unexploded ordnance, especially because manual methods are not easily scalable. Lastly, the object detection performance in hyperspectral imaging can be significantly improved by ML and DL techniques.

RSM show that copious quantities of actual data in proper context are needed to train AI. Military applications often have less data than the “big data” of commercial applications. Therefore, recording as much data as possible and generating high‑fidelity simulated information is essential. As minor changes to the input data can compromise Deep Learning Neural Networks’ performance, it is critical for AI methods to be robust against adversarial attacks. Investing in AI methods that are not potentially robust will result in little added value to fielded applications. Careful field‑testing in realistic settings will be required to develop confidence.

Neural Network outputs must be explainable to end‑users, and different explanations are needed for other users. Significant challenges to deep learning are finding the proper NN structure, reducing resource consumption, assessing confidence in the results, effective use of the available bandwidth, and minimising the need for labelled data.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

DATA FUSION AND ASSIMILATION FOR SCIENTIFIC SENSING AND COMPUTING (AVT‑ET‑204)

ACTIVITY TYPE:

Exploratory Team

DURATION:

June 2019 – May 2020

OVERVIEW:

AVT‑ET‑204 was founded to provide the NATO community with information on the state‑of‑the‑art data fusion and assimilation methodologies for applications to scientific experiments.

OBJECTIVES:

To provide state‑of‑the‑art information on data fusion methodologies for applications in computation.

To devise experiments and computations focusing on the optimised deployment of sensors and assimilation of data into models.

APPROACH:

The advancement of low‑cost sensors and multi‑sensor data fusion algorithms developed by military and private industry brings a paradigm shift in scientific instrumentation and computation.

A network of multiple inexpensive sensors integrated with advanced algorithms could bridge several gaps in traditional scientific instrumentation. The recent development and application of Machine Learning surrogate modelling and optimisation methods for various problems in scientific modelling and computing offer a unique opportunity for further advancement of the effective use of low‑cost instrumentation and multi‑sensor data. Potential applications have complementary nature. On the one hand, low‑cost sensors and multi‑sensor data are used to collect data to validate existing computational models. On the other hand, low‑cost sensors and multi‑sensor data are assimilated into computer simulations to develop new/refined computational models.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AUTOMATED SCENE UNDERSTANDING FOR BATTLEFIELD AWARENESS (SET‑ET‑107)

ACTIVITY TYPE:

Exploratory Team

DURATION:

June 2018 – June 2019

OVERVIEW:

SET‑ET‑107 was established to develop common standards and metrics for joint activities in data fusion, image processing and sensing.

OBJECTIVES:

To develop a strategic plan for standardising annotated data sets and evaluating metrics to assess algorithm performance of scene understanding.

To identify established algorithms from industry for potential use in military scenarios.

APPROACH:

The key to automated scene understanding for battlefield awareness dwells in the simultaneous growth of deep learning techniques and the large sets of annotated data relevant to the military.

This ET uses modelling and simulations to augment collected data sets and improve algorithm training. Collected and simulated data should be shared in subsequent tasks with participating nations for algorithm development and evaluation purposes. The activity also examines the conversion of open source and commercially available algorithms for training on data relevant to the military.

Industry successes have not yet been fully leveraged to address military problems. This is partly because relevant military data does not exist in sufficient quantity and is typically unavailable or of interest to industry leaders. Robust data sets will need to be developed with multi‑modal sensors, scenarios, and targets relevant to the military. This development is necessary if there is to be a transition of these algorithms to military systems.

Successful transition of algorithms using AI techniques would allow NATO members to develop sensor systems that can automatically condense sensor data into timely, actionable information for user consumption, resulting in reduced user fatigue while increasing effectiveness.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MACHINE LEARNING FOR WIDE AREA SURVEILLANCE (SET‑ET‑110)

ACTIVITY TYPE:

Exploratory Team

DURATION:

October 2018 – October 2019

OBJECTIVES:

To define the scope and goals of a follow‑on RTG to develop ML structures to support wide‑area surveillance sensor processing. Specifically, identify necessary data sets and shortlist promising ML strategies.

APPROACH:

Machine Learning (ML) is highly applicable in the surveillance of urban areas and detecting and tracking small manoeuvrable targets in support of activity‑based intelligence.

Despite developments in sensor technology, the continued evolution of the problem space raises challenges to achieving robust surveillance. For example, the migration of sensors to high‑altitude platforms leads to surface clutter interference resulting in degradation of radar performance. In addition, surveillance of urban areas and the desire to detect and track small manoeuvrable targets is exceptionally challenging.

The ET provides the most effective ML strategies, architectures, and insights on anticipated performance and potential adversarial threats. Developed architectures will provide an opportunity for transferred learning to operational systems. A key desired outcome is establishing a reference database to support future ML for wide‑area surveillance S&T research.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE, SURVEILLANCE AND RECONNAISSANCE (ISR) –COMPUTATIONAL IMAGING AND COMPRESSIVE SENSING

COMPLETED RESEARCH

COMPUTATIONAL IMAGING AND COMPRESSIVE SENSING FOR EO/IR SYSTEMS (RTG SET‑232), 2016 – 2019

ACTIVITY TYPE:

Research Task Group

DURATION:

March 2016 – March 2019

OVERVIEW:

RTG SET‑232 was established to provide standard tools for characterising Computational Imaging (CI) and Compressive Sensing (CS) techniques for EO/IR imaging sensors and their potential applications.

OBJECTIVES:

To provide a toolbox for assessing CI and CS techniques.

To design concepts on the applicability of CI and CS techniques to imaging systems.

To conduct joint research activities on CI and CS techniques.

APPROACH:

Research focuses on algorithm designs, laboratory assessment, field performance assessment, performance modelling, and conceptual design. Anticipated tools include image sequences allowing evaluation of CI and CS techniques through simulation and test set‑up configurations for end‑to‑end sensor performance characterisation in the laboratory and field. In addition, efforts were made to exchange data, algorithms, and techniques to establish a common basis for research.

FINDINGS:

Computational Imaging and Compressive Sensing techniques have significant military potential when used for non‑line‑of‑sight imaging to see around corners and multiplexed imaging to gain wide field‑of‑view situational awareness.

Employing AI techniques in optics and image processing led to significant advances in the capability of EO/IR systems.

Deep neural networks had been proposed as a practical innovative approach to learning a mapping from low‑resolution to high‑resolution images.

Further assessments of CI and CS’s impacts on applicable military systems (e.g., automatic image understanding) are needed.

The main challenge for implementing CI and CS in military applications is the lack of representative learning samples to train the AI algorithms that underpin them.

Enhancing interoperability via joint data collection and exploitation is required to develop the infrastructure and best practices needed to advance CI and CS among NATO members.

In conventional optical design methods, high‑quality imagers often require high system complexity. Recent advances in computational imaging and compressive sensing addressed this issue, extending the capability and reducing the complexity of EO/IR systems. Mainly extending the depth of field of an imaging system far beyond normal limitations. Compressive sensing offers a method for directly acquiring a compact signal representation without a conventional sampling of the signals. Obtaining an image of equal quality to a large format array while providing smaller, cheaper, and lower bandwidth imagers.

CI and CS require non‑linear reconstruction algorithms with high computational costs. AI‑based and deep neural methods for image reconstruction can help solve an optimisation problem and better model the scene of interest. AI can also be employed to reconstruct an image from data directly, i.e., without using an optimisation framework. For this scenario, a learning neural network can be trained to recover a sparse signal from measurements using deep learning.

For a CI or CS system to reach the battlefield, it must fill a gap where current technologies are inadequate or accomplish a particular task more efficiently than traditional EO/IR systems concerning system size, weight, power consumption, performance, latency, or cost. CS approaches can also be used to develop adaptive multi‑spectral imaging sensors for high‑confidence target identification in ISR applications. The combined benefits of these new capabilities have the potential to revolutionise imaging sensor technologies across a wide variety of NATO applications.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

https://doi.org/10.1117/ 12.919036.

Figure 16: Example where a sparse representation of an overcomplete dictionary is used to significantly increase the resolution of a single low‑resolution image20

20 Kruithof, M.C., van Eekeren, A.W., Dijk, J., and Schutte, K. (2012). Single Image Super Resolution via Sparse Reconstruction. Proc. SPIE8365, Compressive Sensing, 83650F.

EXPLOITATION OF LONGWAVE INFRARED AIRBORNE HYPERSPECTRAL DATA (RTG SET‑240)

ACTIVITY TYPE:

Research Task Group

DURATION:

March 2016 – March 2020

OVERVIEW:

RTG SET‑240 was established to advance understanding of the phenomenology associated with hyperspectral target detection and developed novel exploitation methodologies adapted to specific contexts.

OBJECTIVES:

To establish recommendations for applying the hyperspectral remote sensing technology for CBRNE/ IED threat detection.

APPROACH:

This RTG is the follow‑on of the SET‑190 RTG on ‘Phenomenology and Exploitation of Thermal Hyperspectral Sensing. The RTG focused on analysing datasets through a comparison of the exploitation methodologies and algorithms, evaluation of their detection/identification capability and sharing results and writing joint recommendations on how hyperspectral technology will serve to address the detection of CBRNE/ IED threats.

FINDINGS:

Machine Learning and Deep Learning have provided remarkable results in big data analysis, especially in applications related to image analysis and computer vision.

Deep learning has also emerged as a relevant approach for hyperspectral imaging and remote sensing applications.

Deep learning methods need voluminous annotated data to succeed in solving classification or segmentation problems. Labelling large amounts of data is a considerable effort and can be infeasible for many tasks.

Unsupervised Machine Learning strategies have been applied successfully for atmospheric compensation and anomaly detection in terrain textures. Indirect illumination and adjacency effects have demonstrated that neighbouring structures significantly influence target signatures and affect the performance of hyperspectral target detection algorithms. In short, ML algorithms have achieved state‑of‑the‑art classification performance on benchmark hyperspectral data sets. On the other hand, they do not consider varying atmospheric conditions experienced in a real‑world detection scenario. However, at least partially, Deep Learning approaches have shown promise in addressing such situations.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COMPRESSIVE SENSING APPLICATIONS FOR RADAR, ESM AND EO/IR

IMAGING

ACTIVITY TYPE:

(RSM SET‑265), 2019

Research Specialists Meeting

DURATION:

March 2019

OVERVIEW:

RSM SET‑265 was established to identify the most promising future directions for further development of the investigated technology and help cross‑fertilise ideas for future research. DL and ML techniques were identified as the capable tools that enhance CS applications.

APPROACH:

This RSM provided an opportunity for specialists in this research field to come together, present the current state‑of‑the‑art of CS and identify where investment is needed to propel CS radar and EO/ IR systems to the next level.

FINDINGS:

The applications discussed during the RSM emphasised the advantages and challenges of using Compressive Sensing based systems, and as a result, two Exploratory Teams (ETs) were started to investigate further the potential of CS applied to radar.

The prospective benefits of approaches based on Deep Learning via convolutional neural networks lead to fast event detection in machine vision. This advantage is due to the need for fewer measurements and tuneable resolution. Deep learning approaches extracted visual information from an image bounced off the reflecting wall by a virtualised light source and a virtualised focal plane. The technique relies on temporal modulation and subsequent coherent integration on receive mode, conceptually like radar pulse compression.

Deep Learning for compressive infrared and hyperspectral machine vision can lead to faster event detection.

Deep Learning approaches in spatially controlled coherent illumination can acquire spatially resolved imaging of non‑line‑of‑sight objects.

A CS method enhanced by Machine Learning has the potential to achieve automatic target recognition that could be trusted to some extent.

The future use of CS suffers from an unclear path towards verification and validation that hinders its uptake for operational applications.

Many publications with few practical systems imply that CS may still be around the “peak of inflated expectations,” with the practical applications arising within the “plateau of productivity” still to come.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

WORKSHOP ON PHENOMENOLOGY AND EXPLOITATION OF HYPERSPECTRAL SENSING WITHIN NATO (RWS SET‑277)

ACTIVITY TYPE:

Research Workshop

DURATION:

October 2019

OVERVIEW:

RWS SET‑277 was established to explore the state‑of‑the‑art hyperspectral technology, including recent developments in Machine Learning for physics‑based signature modelling and target detection.

OBJECTIVES:

To provide a platform for current and past NATO initiatives interested in hyperspectral technology.

APPROACH:

The RWS provided a platform for all the NATO initiatives involving hyperspectral technology to share their findings, get feedback, discuss the technology’s evolution, and define a way ahead. This scope covers many topics, including deep learning, fusion frameworks, and data compression.

FINDINGS:

Machine learning approaches show enormous potential in physics‑based signature modelling for target detection in an operational context.

Deep Learning requires a large amount of labelled training data to learn specific objects or material categories.

Deep neural networks trained to reconstruct the input data in the output (autoencoders) are a promising approach to anomaly detection that does not require a large volume of labelled data.

The workshop reported the developments in Machine Learning for physics‑based signature modelling and target detection. The workshop concluded that Machine Learning has the potential to capture the complexity of physics‑based spectral characterisation of materials that is necessary to address the spectral variability encountered in an operational context to achieve high target detection scores.

An autoencoder is a deep neural network trained to reconstruct the input data in the output. It consists of an encoder, which tries to encode the input data into a low dimensional compact representation and a decoder striving to reconstruct input data from the condensed representation.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTEGRATING

COMPRESSIVE

SENSING AND MACHINE LEARNING TECHNIQUES FOR RADAR APPLICATIONS

(SET‑ET‑111)

ACTIVITY TYPE:

Exploratory Team

DURATION:

April 2019 – April 2020

OVERVIEW:

SET‑ET‑119 was established to identify the potential benefits of integrating Compressive Sensing and Machine Learning techniques for radar applications.

OBJECTIVES:

Identify application areas where integrated designs with Compressive Sensing and Machine Learning components outperform state‑of‑the‑art techniques.

APPROACH:

Integrating CS and ML for radar applications offers the potential to combine algorithms for sparse signal recovery with computationally efficient ML Models such as Deep Neural Networks to replace the expensive iterations with fixed networks learned from the data. This learning process permits the extraction of better representation from sparse training data. On the other hand, CS‑based generative models of target and clutter could be used to produce vast training sets required for ML algorithms to fill in the gaps in measured data by employing online predictions from a “compressed” database, improving the generalisation. This CS‑assisted training strategy will significantly widen the scope of problems where ML could be successfully applied, as in the military domain, the availability of substantial measured radar data training sets cannot always be guaranteed.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ASSESSMENT OF EO/IR COMPRESSIVE SENSING AND COMPUTATIONAL IMAGING SYSTEMS (SET‑ET‑119)

ACTIVITY TYPE:

Exploratory Team

DURATION:

November 2020 – November 2021

OVERVIEW:

SET‑ET‑119 was established to conduct an end‑to‑end performance assessment of a military‑relevant task using candidate computational imaging and compressive sensing systems.

OBJECTIVES:

Select candidate computational imaging and compressive sensing systems, define relevant military tasks, conduct joint field trials, and apply assessment metrics.

APPROACH:

Computational imaging technology can enable special‑purpose tactical sensor designs for specific military sensing tasks, allowing defence planners to expand the application space of optical sensing where conventional imaging techniques are limited by physics or technology.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE, SURVEILLANCE AND RECONNAISSANCE (ISR) – COGNITIVE RADAR AND RADIO

COMPLETED RESEARCH

RADAR SPECTRUM ENGINEERING AND MANAGEMENT (RTG SET‑182)

ACTIVITY TYPE:

Task Group

DURATION:

January 2011 – December 2014

OVERVIEW:

SET‑182 encourages a perspective of NATO forces being able to operate radars in the presence of other users of the electromagnetic spectrum.

OBJECTIVES:

To mature research and models exploiting transmitter, receiver, and waveform designs toward more optimal spectrum use.

APPROACH:

The RTG draws upon the effort of RTG SET‑066 on “Frequency Sharing Between Communication and Radar Systems.” While working in parallel with RTG SET‑179 on “Dynamic Waveform Diversity and Design.”

RECOMMENDATIONS:

• Communication and military governing bodies need to ensure more collaboration.

• NATO nations should identify and allocate significant funding for spectrum R&D.

• NATO governing organisations for radar and military should take a greater role in helping the wireless community develop systems and standards that are robust to radar emissions.

• Acquisition program managers for military systems must have greater involvement in radar spectrum management.

• The wireless community and the policymakers need a better understanding of radar requirements and how wireless operations can adversely affect radar.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COGNITION AND RADAR SENSING (RLS SET‑216)

ACTIVITY TYPE:

Research Lecture Series

DURATION:

September 2014 – September 2017

OVERVIEW:

RLS SET‑216 was established to overview developments in cognitive radar sensing, improve the performance of existing radar systems and open new capability areas.

APPROACH:

The Lecture Series reviewed emerging developments in cognitive radar sensing. Discourses were based on fundamental concepts and advanced applications using various examples relevant to NATO’s missions.

FINDINGS:

AI techniques enable ubiquitous cognition, which can be applied to all radar systems and has the potential to usher in a new era of sensing.

Cognitive radar systems increase situational awareness by enhancing sensing, tracking, autonomous guidance, and navigation.

Challenges in developing Cognitive Radar systems dwell in designing better human machine interfaces and fostering trust in the AI system by increasing their explainability.

Attempts to produce cognitive architectures within the artificial intelligence community capture the essence of the cognitive process based on a biomimetic approach. The research summarised by the activity memory‑driven perception‑action cycle is presented and applied to radar. The cognitive radar must communicate with the operator through the human‑machine interface. The operator must communicate objectives and requirements effectively, and the radar system must provide the necessary information to justify the radar system decisions. Otherwise, the operator will not trust the radar.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COGNITIVE RADIO NETWORKS – EFFICIENT SOLUTIONS FOR ROUTING, TOPOLOGY CONTROL, DATA TRANSPORT, AND NETWORK MANAGEMENT (RTG IST‑140)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2015 – December 2017

OVERVIEW:

RTG IST‑140 was established to investigate Cognitive Radio Network (CRN) technology as a promising network management solution to achieve adaptability in infrastructure‑based and ad‑hoc military requirements.

OBJECTIVES:

To improve the robustness and efficiency of military communications via Cognitive Radio Networks (CRN).

APPROACH:

CRNs are a promising solution for the spectrum scarcity problem, as they autonomously identify the optimal parameters (frequency, modulation, etc.) for a transmission. CRNs go one step further by adapting the low‑level transmission parameters between two devices and networking aspects like routing, topology control, and data transport to achieve end‑to‑end goals. Based on this technology, the robustness and efficiency of military communications can be improved.

FINDINGS:

CRNs are applicable to highly dynamic environments while pursuing end‑to‑end goals, thus leading to robust and efficient communication.

CRNs should use AI or machine‑learning techniques to enhance route selection, topology control, adaptability, and automation. The behaviour of highly adaptive CRN systems will have to be constrained by policies.

CRNs will become the next step in the evolution of radio networks, as their autonomous adaptation capability promises robust, reliable, and efficient communications while requiring fewer management efforts than conventional radio networks. These properties are particularly advantageous for tactical networks in national and multi‑national operations.

The application of AI techniques in the CRNs seems promising because of its learning capability. For example, routing mechanisms are based on the AI learning method known as reinforcement learning. The reinforcement learning mechanism supports efficient routing strategies in fixed or slow‑changed network structures. Frequency management enhanced by AI techniques helps automate the management before, during and after a mission. In addition, autonomous frequency management becomes more critical as the amount of information from different layers proliferates. Therefore, conventional management approaches and policies must be revised to account for the interaction between autonomy and humans.

The behaviour of highly adaptive systems will have to be constrained by appropriate policies. Appropriate policies must be developed and tested to introduce CR technologies into the marketplace by industry. NATO shall consider applicable coalition policies for dynamic spectrum access. Considering CRN’s enhanced cognitive capabilities, policies may concern spectrum access and touch all system aspects.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ADAPTIVE RADAR RESOURCE MANAGEMENT (RTG SET‑223)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2015 – January 2019

OVERVIEW:

RTG SET‑223 was established to address a wide range of capability requirements for ISR systems in a military context, for example, Active Ballistic Missile Defence or Defence Against Terrorism Requirement for Protecting Harbours and Vessels from Surface Threats.

OBJECTIVES:

To establish a collective understanding of Radar Resource Management (RRM) techniques and their respective benefits in the NATO community.

APPROACH:

The RTG researched Adaptive RRM for phased array radars by defining key elements and metrics of “RRM services,” considering how adaptive RRM algorithms could be evaluated in simulation, using benchmark test scenarios of increasing complexity and assessing their performance against the benchmark scenarios.

FINDINGS:

Compared to non‑adaptive RRM, results show that adaptive RRM achieves better tracking performance against benchmarks and the same tracking performance against context targets while requiring fewer tracking resources. Management at the mission and situation levels can incorporate real‑time human input, while managing at the signal level will have to be conducted automatically because of the sub‑second time scales involved.

As it is likely, potential adversaries recognise the disruptive operational potential of intelligent radar and are developing options for countering it.

The adaptive techniques for surveillance are rule‑based rational decision‑making artificial intelligence approach which develops, maintains, and manages information about the landscape on multiple timescales in a tactical decision aide hierarchy. The position supported in this report is that an algorithm based on Machine Learning methods can incorporate many more features that describe the radar and the environment than a radar designer can do, despite the latter’s expertise.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

Figure 17: Artificial intelligence hierarchy21

21 Huizing, A. “Deep Learning in Military Sensing,” 39th NATO Sensor and Electronic Technology Panel Business Meeting, Helsinki, Finland, 10 May 2017.

COGNITIVE RADAR (RTG SET‑227)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2015 – January 2019

OVERVIEW:

RTG SET‑227 was established to develop and conduct experiments and theoretical investigations to illustrate the benefits and challenges of enabling cognition‑based capabilities in radar systems.

OBJECTIVES:

Advantage bio‑mimetic approaches borrowed from nature and memory‑based learning and control paradigms.

To conduct experiments and theoretical investigations into cognition‑based capabilities in radar systems.

APPROACH:

The overarching theme of the RTG is incorporating broader significant autonomous decision‑making and feedback‑controlled adaptivity into the sensor. The bulk of the work of the report has been to explore the benefits and drawbacks of cognitive processing in various experiments and simulations. The experimental work represents some of the first of its kind and demonstrates actual cognitive behaviour.

The work has also highlighted the difficulty of experimental work on cognitive sensing. Despite extensive research on ML for processing sensor data, the deployment of ML techniques in cognitive radars and the respective military operational consequences are poorly understood. Furthermore, at the outset of this study, there was little or no experimental work to demonstrate practical cognitive behaviour. This study has provided a significant step in addressing both of those issues.

As a continuation of a similar effort, this activity also leverages technical accomplishments of the SET‑179 and SET‑182 RTGs on Waveform Diversity and Radar Spectrum, respectively.

FINDINGS:

Cognition has become an established subject in modern radar systems and signal processing because the extension of cognitive techniques to distributed sensing is a natural way forward. ML and other AI techniques can be used to identify patterns and learn behaviours, which are essential knowledge for the decision‑making aspect of cognitive radar.

Cognitive radar applications require ML/AI algorithms that employ extra modelling to propose a course of action, self‑correct, and provide a figure of merit on the proposed course of action.

ML and AI require significant training data, which is always in short supply for radar applications.

Cognitive Radars employing AI can be tricked into learning bad habits if trained on poisoned data sets. Thus, it is essential to safeguard them against adversarial actions. There has not been a clear consensus on the exact definition and utility that cognitive processing may provide to military sensing systems.

As each scenario that a radar faces provide a unique set of information and variables, AI techniques must be able to incorporate both supervised learning from existing datasets and online real‑time learning as new scenarios are presented. How to train, implement, and protect cognitive radars employing AI is still an open question. Some have expressed concern that cognitive radars, especially those using AI, could be tricked into learning bad habits. Therefore, AI techniques for cognitive radar should have some bounds on the radar’s acceptable behaviour. Such bounds, however, could limit the potential impact of cognitive systems by restricting their ability to learn and adapt. Furthermore, the feedback in a traditional AI application might be a straightforward correct/incorrect assessment; in a cognitive radar application, the feedback must include a figure of merit on the proposed course of action. Therefore, additional research is needed to use AI as a decision agent in cognitive radar’s closed‑loop or perception‑action cycle.

Future radars are likely to be distributed, intelligent and spectrally efficient. The extension of cognitive techniques to distributed sensing is a natural way forward. Resource management and information exchange between nodes in such a distributed network must be fully understood and developed, especially in a GPS‑denied environment. The experimental work undertaken in this Task Group can be extended to distributed sensing networks.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

9TH NATO MILITARY SENSING SYMPOSIUM (RSY SET‑241)

ACTIVITY TYPE:

Research Symposium

DURATION:

January 2016 – June 2017

OVERVIEW:

RSY SET‑241 was established to stimulate joint programs and agreements and improve the quality of national programs related to military sensing.

OBJECTIVES:

To enable comparison of sensing systems from different NATO countries, generate expert critique of programs and facilitate future collaboration between NATO countries.

APPROACH:

This RSY gathered leading and allowed transdisciplinary discussions between technical and NATO military experts. Ten nations attended the RSY, and its program included seventy‑nine papers, five keynote speakers, and the exhibition of eighteen posters. The subjects of the presentations were: passive/active RF sensing for detection, tracking and imaging, focal plane arrays, hyperspectral/multispectral technologies, laser technology, soldier systems, targeting systems, counter CBRN technology, low‑light systems, image processing, and interoperable systems.

FINDINGS:

AI techniques that enabled data fusion approaches, integration of multiple sensors and development of advanced algorithms for signal processing add to increased sensor performance and improved situational awareness. Machine learning is used at various levels, from optimising decisions to detection and tracking. Future research should focus on using these algorithms in real‑time.

ISR system interoperability and integration are paramount for the success of future military operations implicating multiple nations and requiring various sensors.

Limiting the training burden of the operator engaged in radar management by using augmented reality or a human‑machine interface.

A new method of radio resource management based on machine learning resource management led to improvements in cognitive & adaptive radar technology. Conventional branching and bounding methods that enumerate all viable solutions on a search tree take an enormous time to run. Thus, by using machine learning, some nodes can be eliminated, reducing the time spent to optimise radar resource management. Furthermore, recent advances in compressed sensing, sparse signal processing and machine learning enable approaches to detect weak target signals in a clutter‑dense environment. The models are integrated into a Bayesian learning framework for radar detection.

The activity also focuses on a deep learning technique that tracked and classified military platforms in various environments, where the algorithm could detect a military target with a probability of higher than 95 percent. When combined with natural language processing and visual semantics, this is an interesting and challenging battlefield use case. A potential application for this architecture is lip‑reading and enhanced object tracking in environments with the occlusion of one of the modalities.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE, SURVEILLANCE AND RECONNAISSANCE (ISR) –COMPUTATIONAL IMAGING AND COMPRESSIVE SENSING ONGOING RESEARCH

ELECTRONIC SUPPORT (ES)

TECHNIQUES ENABLING COGNITIVE ELECTRONIC WARFARE (EW) (SCI‑326)

ACTIVITY TYPE:

Research Task Group

DURATION:

November 2019 – November 2023

OVERVIEW:

This RTG was established to assess combined communication and radar signal detection, classification and network topology identification using Machine Intelligence (MI) approaches in NATO‑relevant scenarios.

OBJECTIVES:

To formally analyse and document the utility of MI, including strengths and weaknesses of various algorithmic approaches, in electronic support operations.

APPROACH:

The investigation conducted by this activity will guide NATO on those approaches that provide the most promising potential for multi‑national NATO operations with recommendations for future activities. The analysis will include identifying insights into the practical feasibility of implementing MI, considering hardware and software requirements that may affect integration and deployment. Specifically, the RTG will also identify techniques and methods for generating and collecting training data and the impact of this data on MI performance. This RTG intends to conduct Cooperative Demonstrations of Technology (CDT) to support analytic results and technology recommendations.

FINDINGS:

This study will conclude in November 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MACHINE LEARNING FOR WIDE AREA SURVEILLANCE (SET‑278)

ACTIVITY TYPE:

Research Task Group

DURATION:

November 2019 – November 2022

OVERVIEW:

This RTG was established to examine the application of machine learning to improve the performance and versatility of wide‑area sensors to provide improved ISR capabilities.

OBJECTIVES:

To develop efficient ML techniques and hybrid strategies tailored to Wide Area Surveillance.

APPROACH:

This activity will collect and identify standard data sets to support the development and comparison of data sets across the multinational effort. Develop truthing strategies and tools to help labelling and annotation of data sets. Develop ML performance metrics for detection, tracking and classification. The baseline performance of machine learning against traditional approaches develops, modifies, and improves ML algorithms to address identified challenges.

The outcomes of the RTG will directly impact member nation WAS processing strategies. State‑of art knowledge will be provided on the most effective ML strategies and architectures, as well as insights on anticipated performance and potential adversarial threats. In addition, developed architectures will provide an opportunity for transferred learning to operational systems. A key desired outcome is establishing a reference database to support future ML for WAS S&T research.

FINDINGS:

This study will conclude in November 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ADVANCED MACHINE LEARNING ATR USING SAR/ISAR DATA (SET‑283

)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2020 – January 2023

OVERVIEW:

This RTG was established to focus on various aspects of modern Machine Learning (ML) and advanced signal processing for Synthetic Aperture Radar (SaR)/ISAR Automatic Target Recognition (ATR).

OBJECTIVES:

To examine and compare the use of various ML and Deep Convolutional Networks and the influence of their usage on ATR performances.

APPROACH:

The RTG will cover the following scientific topics: Comparison of real and modelled signatures of various reference targets, using multiple simulators or simulation schemes, and different CAD models or target variants. ATR performance metrics definition. Interest in modern Machine Learning algorithms for radar imaging detection, classification, and identification. The activity aims to propose a workflow, demo and interface that will enlighten the ML solutions found by the group and their “natural” interaction with end‑users.

FINDINGS:

This study will conclude in January 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTEGRATING COMPRESSIVE SENSING AND MACHINE LEARNING TECHNIQUES FOR RADAR APPLICATIONS (SET‑288)

ACTIVITY TYPE:

Research Task Group

DURATION: February 2020 – February 2024

OVERVIEW:

This RTG was established to examine novel techniques that can improve radar performance in congested environments, requiring fewer radar measurements.

OBJECTIVES:

To identify the potential benefits of integrating Compressive Sensing (CS) and Machine Learning (ML) techniques for radar applications.

APPROACH:

The RTG will assess the implementation, performance and robustness of integrated CS and ML architectures and algorithms for radar applications and quantify the capabilities of such systems across a wide range of operational conditions. This activity will also create data repositories and algorithm libraries to enable learning techniques.

CS/ML integration could improve the speed of CS detectors. Alternatively, CS‑based sparse modelling strategies could be used to improve the generalisation capability of deep learning classifiers when training data is limited.

FINDINGS:

This study will conclude in February 2024, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AI FOR MILITARY ISR DECISION‑MAKERS (SET‑290)

ACTIVITY TYPE:

Research Lecture Series

DURATION:

May 2020 – September 2022

OVERVIEW:

This RLS was established to establish core methodologies and proven artificial intelligence algorithms that solve the various aspects of military situational awareness capabilities.

APPROACH:

From a systems‑of‑systems point of view, the participants from the military domain will receive guidelines to assess the chances and limitations of AI‑empowered fusion engines to be embedded into overarching ISR systems. In addition, disruptive effects on military situational awareness, decision‑making and Concepts of Operations are expected to be evaluated. The most important achievement of the RLS is a contribution toward critical judgement in dealing with artificially intelligent and technically autonomous systems under development.

FINDINGS:

This study will conclude in September 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COGNITIVE RADAR (SET‑302)

ACTIVITY TYPE:

Research Task Group

DURATION:

April 2021 – April 2024

OVERVIEW:

This RTG was established to perform theoretical investigations and experiments and develop new methods to demonstrate the benefits of cognition in radar systems.

OBJECTIVES:

• To investigate the role of learning in cognitive radar and distributed cognitive radar systems.

• To facilitate international collaborative experiments.

APPROACH:

The RTG will focus on learning from signal to mission levels within a cognitive radar perception‑action cycle. This activity will be supplemented by modelling the operational environment, action space, learning rewards, objectives, and optimisation constraints. Resource allocation and scheduling for distributed cognitive radar and related learning will also be considered. This RTG will extend the work conducted by SET‑227

FINDINGS:

This study will conclude in April 2024, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE, SURVEILLANCE AND RECONNAISSANCE (ISR) – MARITIME

DOMAIN COMPLETED ACTIVITIES

PERSEUS EU PROJECT

ACTIVITY TYPE:

CMRE

DURATION:

2014 – 2016

OVERVIEW:

This activity was established to integrate passive underwater acoustic technologies into autonomous mobile underwater platforms.

OBJECTIVES:

Design, develop and integrate the passive acoustic system into autonomous vehicles, enabling them to detect and classify acoustic noise sources.

APPROACH:

Two concepts were developed, integrated, evaluated in a laboratory setting, and successfully demonstrated at sea. First, the CMRE project objectives were achieved through the real‑time implementation of ad‑hoc array processing methods based on the exploitation of time‑coherence between the signals received on each hydrophone of the antenna.

FINDINGS:

The maximum detection and localisation range experimentally measured covered an area of approximately three km2. A supervised statistical pattern recognition algorithm was successfully fed with a set of numerical features extracted from vessel signatures for the classification. Three main vessel classes were considered: small motorboats, mid‑sized motorboats, and larger vessels.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

FRATRE FOR FRENCH AUTOMATIC TARGET RECOGNITION

ACTIVITY TYPE:

CMRE

DURATION:

2018 – 2020

OVERVIEW:

The activity was established to develop autonomous capabilities for mine detection and classification.

OBJECTIVES:

To develop a deep learning classification algorithm for sonar imagery.

APPROACH:

The activity focuses on using Deep Learning (DL) to develop an algorithm to detect and classify mines in high‑resolution Synthetic Aperture Sonar (SAS) images provided by Autonomous Underwater Vehicles (AUV) or by Unmanned Surface Vessel (USV).

FINDINGS:

CMRE proposed to adapt the latest version of the CMRE classifier on small SAS snippets to large SAS tiles; implemented a new single‑stage detection/classification algorithm based on the CMRE classifier and evaluated the final algorithm performance on the MANEX’14 subset of segmented data.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

TEXTURE‑BASED SEAFLOOR CHARACTERISATION

USING GAUSSIAN PROCESS CLASSIFICATION

ACTIVITY TYPE:

CMRE – ANMCM programme

DURATION:

2019 – 2020

OVERVIEW:

This activity was established to automate seabed characterisation using sonar imagery.

OBJECTIVES:

To examine the use of Machine Learning techniques for sonar imagery analyses.

APPROACH:

This work was presented at a scientific conference and subsequently written in the form of a journal paper and submitted, and it is currently under review. Follow‑on work consists of the development of a semi‑supervised extension of the algorithm.

FINDINGS:

Automated characterisation of seabed type labels is useful in estimating the performance of the sonar and ATR combination. This is useful for through‑the‑sensor performance evaluation of a mine‑hunting mission.

Since synthetic aperture sonars can generate high‑fidelity acoustic images of the seabed, machine‑learning techniques (based on a Gaussian process) can be employed to distinguish different seabed types (rocky, ripples, seaweed, and sand). Furthermore, this improved model could learn from both labelled and unlabelled sonar data, significantly increasing the amount of data that can be used for training the algorithm.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AUTOMATIC OBJECT CLASSIFICATION WITH ACTIVE SONAR

ACTIVITY TYPE:

CMRE ASW programme

DURATION: 2020

OVERVIEW:

The activity was established to design an automatic object classification method and shows its performance in shallow water.

OBJECTIVES:

To develop an unsupervised learning method based on anomaly detection to classify objects.

APPROACH:

In the littorals, there are significant amounts of clutter contacts from the seafloor and coastal reverberation. This considerable number of undesired contacts can be exploited to learn the “fingerprint” of the clutter and identify the object‑related contacts as anomalies. The paper describes the proposed classification method and shows its performance with real data collected at sea using an echo‑repeater as an artificial object.

FINDINGS:

The proposed method relies on machine learning, exploits the clutter contacts to learn the clutter signature, and then classifies the object contacts as anomalies if their signature is not like the learned one. The signature is a vector that collects several features obtained by processing the acoustic response of the sonar contact.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SONAR‑BASED DEEP LEARNING FOR UNDERWATER UXO REMEDIATION

ACTIVITY TYPE:

CMRE

DURATION:

2020 – 2021

OVERVIEW:

This activity was established to address water contamination of military ammunition.

OBJECTIVES:

To develop novel Unexploded Ordnance (UXO) detection and classification algorithms based on AI techniques.

APPROACH:

The general‑purpose detection algorithm created as the activity’s outcome exploited the concept of integral images to flag suspicious regions in a given data volume quickly and computationally efficient. The follow‑on classification algorithm was based on deep‑learning techniques, specifically deep Convolutional Neural Networks (CNNs) that were extended to function with three‑dimensional (i.e., volumetric) input data cubes.

FINDINGS:

The key outcomes are the novel Unexploded Ordnance (UXO) detection and classification algorithms. These have been designed explicitly for volumetric sonar data from two experimental systems, the Sediment Volume Search Sonar (SVSS) and the Multi‑sensor Towbody (MuST).

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COUPLED OCEAN‑ACOUSTIC VARIATIONAL DATA ASSIMILATION

ACTIVITY TYPE:

CMRE ASW and EKOE programmes

DURATION:

2020 – 2021

OVERVIEW:

This activity was established to assess a new data‑driven approach to assimilating acoustic underwater propagation measurements (transmission loss) into a regional ocean forecasting system.

OBJECTIVES:

To assess deep learning approaches to solve automatic classification and regression problems.

APPROACH:

The activity presented and assessed a strategy to introduce an observation operator based on neural networks in data assimilation. Furthermore, the linearisation of such an operator, required by variational schemes, is also discussed, and implemented. Finally, the methodology is applied to the coupled oceanic–acoustic data assimilation problem and provides promising results.

FINDINGS:

A new data‑driven approach to assimilating acoustic underwater propagation measurements into a regional ocean forecasting system may be a valuable help for Anti‑Submarine Warfare (ASW) missions. The CNN solution has outperformed traditional techniques, such as canonical correlation analysis.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

UNSUPERVISED LEARNING OF PLATFORM MOTION IN SYNTHETIC APERTURE SONAR

ACTIVITY TYPE:

CMRE – ANMCM programme

DURATION:

2021

OVERVIEW:

This activity was established to develop a data‑driven method based on a variational auto‑encoder (VAE) architecture that could disentangle platform motion in 3D purely based on the data from a 2D sonar array.

OBJECTIVES:

To develop a data‑driven micro‑navigation method for synthetic aperture processing and provide a complete, 3D description of the relative platform translation between successive pings considering the spatial‑temporal coherence of diffuse backscatter.

APPROACH:

Synthetic aperture sonar requires precise knowledge of the platform motion during image acquisition. Micro‑navigation techniques, therefore, exist to estimate this motion based on temporal correlations among the signals recorded from different elements in the sonar array. Most of these methods are linear. The findings are documented in a journal paper submitted to the Journal of Acoustical Society of America and in the CMRE memorandum report CMRE‑MR‑2021‑015.

FINDINGS:

The proposed machine learning method provides platform motion estimation with sub‑wavelength accuracy even with inaccurate or missing inertial navigation measurements offering a robust micro‑navigation solution. Furthermore, measuring the spatial‑temporal coherence of diffuse backscatter on overlapping elements of a two‑dimensional receiver array provides a complete 3D platform motion estimation necessary for synthetic aperture sonar imaging.

The VAE model can learn platform motion. Injecting noisy platform motion information (i.e., using the model in a semi‑supervised fashion) leads to sub‑wavelength accuracy even for high‑frequency sonar applications.

The suggested follow‑on activity investigated the platform’s motion estimation accuracy with a fully unsupervised machine learning method based on normalizing flows and adapting the learning task by considering motion dynamics. Follow‑on activity includes improving the model’s accuracy by extending the encoder model with an autoregressive flow and using prior information on the dynamics of the platform motion to guide the model in disentangling motion in different dimensions during model training. This work was written up as a journal paper and submitted. It is currently under review.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE, SURVEILLANCE AND RECONNAISSANCE (ISR) – MARITIME DOMAIN

ONGOING RESEARCH

IMPACT OF THE COVID‑19 ON THE GLOBAL MARITIME TRAFFIC

ACTIVITY TYPE:

CMRE DKOE programme

DURATION:

2020 ‑ Ongoing

OVERVIEW:

This activity was established to estimate the impact of the pandemic on global maritime traffic.

OBJECTIVES:

To conduct big data analyses of the Automatic Identification System (AIS) messages.

APPROACH:

Following the outbreak, an unprecedented drop in maritime mobility across all categories of commercial shipping occurred. The research relies on multiple data‑driven maritime mobility indexes to assess ship mobility quantitatively in a given time. The study also computes and compares global and local vessel density maps to highlight significant changes in shipping routes and operational patterns.

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COLLABORATIVE AUTONOMOUS MINE COUNTERMEASURES

ACTIVITY TYPE:

CMRE – ANMCM programme

DURATION:

2014 – Ongoing

OVERVIEW:

This activity was established to develop advanced sonar ATR capabilities based on AI techniques.

OBJECTIVES:

To develop Convolutional Neural Networks (CNNs) to recognise automatically targets on sonar images.

APPROACH:

The activity will cover the following scientific topics:

• Underwater target classification in Synthetic Aperture Sonar (SAS) using deep CNNs.

• Exploiting phase information in SAS

• Target classification using multi‑view SAS imagery.

• Transfer learning with SAS‑image CNNs for improved underwater target classification.

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SCALABLE DATA FUSION AND TRACKING BASED ON FACTOR GRAPH REPRESENTATION

ACTIVITY TYPE:

CMRE DKOE programme

DURATION:

2016 – Present

OVERVIEW:

This activity was established to design a flexible and scalable methodology to fuse heterogeneous data sources and track multiple dynamic targets.

OBJECTIVES:

To develop the capability able to exploit deep‑learning classification outputs.

APPROACH:

The activity will focus on the following research topics:

• Improvement of surveillance performance compared to state‑of‑the‑art alternative solutions

• Self‑tuning algorithm capability

• Adaptive target detection probability

• Capability to efficiently use many sensors/data

• Capability to process both Radar and Sonar target detections

• Results published in the most prestigious journals of the IEEE

There are several ways to expand upon this research, such as:

• Extended Target Tracking (ETT)

• Integration with the prediction of highly manoeuvring targets

• Deep‑learning‑based fusion of multiple sensor data

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MARITIME ANOMALY DETECTION

ACTIVITY TYPE:

CMRE DKOE programme

DURATION:

2018 – Ongoing

OVERVIEW:

This activity was established to detect anomalous vessel behaviours.

OBJECTIVES:

To analyse and simulate scenarios of “stealth” (Automatic Identification System (AIS) switched‑off) deviations from the nominal route.

APPROACH:

The activity will focus on the following research topics:

• Detection of AIS data spoofing/falsification

• Real‑world and simulated scenarios

• Ever Given grounding

• Capability to control the anomaly detection characteristic curve – False Alarm Probability vs Detection Probability

There are several ways to expand upon this research, such as:

• Integration with the deep‑learning prediction

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COGNITIVE UNDERWATER COMMUNICATIONS

ACTIVITY TYPE:

CMRE MUSE programme

DURATION:

2017 – Ongoing

OVERVIEW:

This activity examined how AI could enhance real‑time underwater acoustic communications under specific requirements and mission constraints.

OBJECTIVES:

To develop an intelligent underwater acoustic communications system based on machine learning techniques.

APPROACH:

CMRE conducted several data collection campaigns throughout the years and built a vast catalogue of acoustic waveforms distorted by different acoustic communications channels. This allowed the creation of a machine‑learning system that was trained on the achievable performance of each waveform for a wide range of acoustic propagation conditions. The system was integrated into a deployable hardware format used at sea for the first time during the exercise REP(MUS)21. The system could autonomously select which waveform to employ to best match the acoustic communications conditions.

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SPACE‑BASED SURVEILLANCE

ACTIVITY TYPE:

CMRE DKOE programme

DURATION:

2018 – Ongoing

OVERVIEW:

This activity was established to examine utilising space‑based sensors to improve Maritime Situational Awareness (MSA).

OBJECTIVES:

To propose AI‑based algorithmic solutions to exploit space‑based sensors.

APPROACH:

The activity will focus on the fusion of heterogeneous space‑based sensors, including classification features, provided by deep‑learning imagery target classification. The activity also suggests testing and evaluating these algorithms in MSA operational scenarios.

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

DEEP LEARNING‑BASED LONG‑TERM VESSEL PREDICTION

ACTIVITY TYPE:

CMRE DKOE programme

DURATION:

2019 – Ongoing

OVERVIEW:

This activity utilised deep learning approaches to improve Maritime Situational Awareness MSA.

OBJECTIVES:

To enable driven long‑term prediction (3h, 6h, and 24h) of vessel positions based on Automatic Identification System (AIS) historical data.

APPROACH:

The activity will focus on the following research topics:

• Improvement of the Maritime Situational Awareness (MSA)

• Capability to forecast the commercial maritime traffic

• Improvement of Search‑and‑Rescue (SaR) operations

• Improvement of the multi‑sensor data association capability

• Results published in the most prestigious journals of the IEEE

There are several ways to expand upon this research, such as:

• Extension to the case of highly manoeuvring targets

• Input to the anomaly detection mechanism

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

EXPLOITING AUXILIARY INFORMATION FOR IMPROVED UNDERWATER TARGET CLASSIFICATION WITH CONVOLUTIONAL NEURAL NETWORKS (CNNS)

ACTIVITY TYPE:

CMRE – ANMCM programme

DURATION:

2020 – Ongoing

OVERVIEW:

The activity was established to represent the situation better and improve the subsequent target identification stage of Mine Countermeasure (MCM) operations.

OBJECTIVES:

To modify the CNNs to predict as much information as possible besides the binary class label (i.e., target or clutter).

APPROACH:

The architecture of eight CNNs that constitutes the CMRE classifier was modified, and the new models, initiated with the pre‑trained CNNs parameters, were trained to estimate the image quality and the specific object shape in addition to the binary label.

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SIMULATION OF SONAR IMAGES USING GENERATIVE ADVERSARIAL NETWORKS

ACTIVITY TYPE:

CMRE – ANMCM programme

DURATION:

2021 – Ongoing

OVERVIEW:

This activity was established to enable data farming of realistic sonar images.

OBJECTIVES:

To develop a Generative Adversarial Network (GAN) deep learning architecture.

APPROACH:

Because sonar data is expensive to collect at sea, there is also an express need to be able to create realistic sonar images artificially. Developing Generative Adversarial Networks (GANs) supports the rapid creation of sets of sonar images, which can augment the actual (measured) sonar data and be used interchangeably during classifier training.

This activity will develop a new architecture of the deep‑learning‑based GAN, allowing the simulation of small seabed images.

FINDINGS:

This study is ongoing, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

PREDICTIVE MAINTENANCE AND LOGISTICS (PML)

COMPLETED RESEARCH

ECONOMICS FOR EVALUATING FLEET REPLACEMENT (RTG SAS‑099)

ACTIVITY TYPE:

Task Group

DURATION:

January 2012 – December 2015

OVERVIEW:

The Task Group was established to develop a standardised process to gather and process data for high‑level replacement or upgrade decision support.

OBJECTIVES:

Analysing and disseminating a methodology to determine and standardise economic aspects to evaluate fleet or equipment replacement decisions.

APPROACH:

SAS‑099 builds upon the Greenfield and Persselin model, turning it into a practical decision‑making tool through a first passage time approach and extending it with military factors. The model and fleet analysis are applied to three military case studies. Bayesian and Frequentist time series methods are used to estimate model parameters of empirical fleet data.

FINDINGS:

The RTG revealed to decision makers how trade‑offs under uncertainty influence the fleet or equipment replacement timing decision. Specifically, it enables and supports decision‑makers in real replacement decisions, prioritising replacement decisions through a deeper understanding of the value of delay.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MODELS AND TOOLS FOR LOGISTIC ANALYSIS (RTG SAS‑132)

ACTIVITY TYPE:

Task Group

DURATION:

April 2017 – January 2019

OVERVIEW:

The RTG SAS‑132 elaborates on methods and models for defence logistic analysis as part of Operational Research (OR) or Operational Analysis (OA).

OBJECTIVES:

To develop a matrix of analysis coverage to assess the gaps, overlaps and areas of collaborative opportunities.

APPROACH:

SAS‑132 surveyed models and tools for defence logistics analysis in a joint exercise within the nations participating in this activity. Further exchanging knowledge and expertise on those models and tools. Finally, identifying opportunities for involvement and future collaborations.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

PHYSICS OF FAILURE FOR MILITARY PLATFORMS (AVT‑ET‑184)

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2018 – December 2018

OVERVIEW:

AVT‑ET‑184 was established to exploit emerging technologies (Big Data, Data Mining, AI, and Deep Learning) to assess and predict the Physics of Failure (PoF) for military platforms. This activity is an extension of foundational endeavour aimed at predicting the failure mechanisms and managing the health of military platforms. These activities include AVT‑211, AVT‑172, AVT‑212, AVT‑220, AVT‑222, AVT‑223 and AVT‑242

OBJECTIVES:

To influence and shape the exploitation of emerging technologies for the assessment of PoF.

APPROACH:

The ET assesses current capabilities, gaps and development priorities within an asset and information management strategy. This goal is achieved by appropriately integrating the needs, experiences, and issues from AVT, MSG, SET, IST, and other technical panels.

The approach of AVT‑ET‑184 can be summarised in the following key points. First, identify PoF data. Second, collect PoF data and feed it into the integrated models. Third, translate high asset data into scientific narratives or computationally valuable data. From there, the failed processes could be discovered, respective methods described, and follow‑on actions recommended. Showcase novel methods for accurate prediction. Focus on high‑value assets, high failure rates, UAVs, and information sources collected in a disciplined data‑centric approach.

Data Analytics, as a collective term for Big Data, Data Mining, AI, and Deep Learning, can provide confidence and reduced uncertainty value and report data that assists in decision‑making at strategic, operational, and tactical levels of leadership, including S&T.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ADVANCED ANALYTICS FOR DEFENCE ENTERPRISE RESOURCE PLANNING (SAS‑ET‑DX)

ACTIVITY TYPE:

Exploratory Team

DURATION:

April 2018 – April 2019

OVERVIEW:

SAS ET DX was established to develop a shared understanding of advanced analytics products to integrate with Enterprise Resource Planning (ERP) systems. To support planning and decision‑making related to various business processes (e.g., materiel, infrastructure, maintenance, procurement, sustainment, etc.).

OBJECTIVES:

To closely examine Business Intelligence (BI) tools (AI, Cognitive Analytics, Machine Learning) being developed and implemented into Allies’ ERP systems.

APPROACH:

Participating nations delivered presentations covering the described topics. Collaboration opportunities were identified, and a future collaborative work format was agreed upon. Emphasis is put on advanced analytics models and tools developed in‑house that could be shared or collaboratively developed. Several current activities in the C4 domain (IST‑160 Specialists’ Meeting on Big Data and Artificial Intelligence for Military Decision Making) could provide inputs and benefit from this work.

Given nations’ defence budgets, small investments in advanced analytics can lead to resource savings, increased efficiencies in business processes, and increased forces readiness and effectiveness.

Learning from other countries’ experience in developing advanced analytics for ERP will reduce development costs and enable broader adoption of existing models and approaches.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COST ANALYSIS OF CONTRACTOR’S PRICE FOR DEFENCE RESEARCH AND DEVELOPMENT PROJECTS (SAS‑ET‑DW)

ACTIVITY TYPE:

Exploratory Team

DURATION:

April 2018 – April 2019

OVERVIEW:

SAS‑ET‑DW was established to examine the necessary design for tools and methods for effective cost and price analysis for military procurement procedures.

OBJECTIVES:

To identify processes, methodologies, models, and tools to analyse price proposals of contractors in different types of defence procurement projects.

APPROACH:

When agencies receive requirements for system development projects, they must estimate costs under budgetary and informational constraints. With a large enough budget, they can request more information and collect price proposals to benchmark. Therefore, there is a need for benchmarking contractors’ price proposals. The price is not the only criterion for contractor selection. The other factors will also be included in the contractor selection process. It is also essential to monitor and evaluate the costing issues in a selected contractor for project development.

Given the variety of defence platforms and projects, collaboration is essential in developing the datasets for effective research. Sharing each stakeholder’s unique cost analysis method will create an opportunity to compare and develop new cost analysis methods. The development of novel cost analysis methods for defence contracts is driven by advances in Data Mining, Big Data and AI techniques. However, readily retrievable cost data that could serve in computing cost estimates for new weapon systems are lacking.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COALITION SUSTAINMENT INTEROPERABILITY STUDY (SAS‑ET‑EH)

ACTIVITY TYPE:

Exploratory Team

DURATION:

July 2019 – July 2020

OVERVIEW:

SAS‑ET‑EH was established to investigate emerging logistics technologies and how to employ those to supply units in a dismounted tactical environment.

OBJECTIVES:

To investigate the employment of AI for sustainment information management, tracking and tasking systems across NATO Allies.

APPROACH:

Unmanned systems enhanced by AI are poised to revolutionise how people and supplies are moved in military operations. This activity focused on outlining the work required to harness and use unmanned delivery technologies emerging from both military S&T activities and the commercial sector by reasoning over the complex combinations of available options and recommending courses of action. In distributed, joint battlefield, AI can facilitate the best choice for providing sustainment to meet a particular need regardless of organisational hierarchy.

Command and control systems must be compatible and interoperable with coalition partners’ systems to increase the effectiveness and efficiency of planning and executing sustainment operations.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ASSESSING THE IMPLICATIONS OF EMERGING TECHNOLOGIES FOR MILITARY LOGISTICS (SAS‑ET‑EN)

ACTIVITY TYPE:

Exploratory Team

DURATION: October 2019 – October 2020

OVERVIEW:

SAS‑ET‑EN was established

OBJECTIVES:

• To share insights on the impact of innovative technologies, cognisant of changes to how future operations will be conducted.

• To understand the extent to which changes in the provision of logistics support in the Civil sector can be applied in a Defence and Security context.

APPROACH:

The research symposium planned as an outcome of this activity is expected to highlight work across NATO to assess and develop logistics capabilities that use emerging technologies (AI, autonomy, alternative energy sources, additive manufacturing). This should help shape national acquisition programmes, understand emerging interoperability challenges, and identify analytical and wider collaboration opportunities.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

PREDICTIVE MAINTENANCE AND LOGISTICS (PML)

ONGOING RESEARCH

ASSESSING THE IMPLICATIONS OF EMERGING TECHNOLOGIES FOR MILITARY LOGISTICS (SAS‑165)

ACTIVITY TYPE:

Research Symposium

DURATION:

July 2020 – September 2022

OVERVIEW:

This RSY was established to provide insights on emerging technologies and their potential impact on Military logistic operations, cognisant of changes to how future operations will be conducted.

OBJECTIVES:

To highlight work across NATO to assess and develop logistics capabilities using emerging technologies.

APPROACH:

The symposium will produce meeting proceedings to advise NATO and national logistics and logistics assessment communities on completed research in this area. Furthermore, the RSY will highlight promising assessment approaches and develop recommendations for follow‑on logistics OR&A collaborations. Ideally, the outcome might achieve a SWOT (Strengths, Weaknesses, opportunities, threats) analysis of the different technologies and their impact on Military Logistics.

This activity will also contribute to the NATO Emerging and Disruptive Technologies Roadmap in identifying potential applications of AI, Data, Novel Materials and Autonomy to logistics functions.

FINDINGS:

This study will conclude in September 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COALITION SUSTAINMENT INTEROPERABILITY STUDY (SAS‑168)

ACTIVITY TYPE:

Research Task Group

DURATION:

July 2020 – December 2023

OVERVIEW:

This RTG was established to analyse coalition sustainment state‑of‑the‑art systems among NATO Allies and pave the way for collaborative AI technologies to facilitate optimised resupply operations.

OBJECTIVES:

To develop a baseline for utilising AI for sustainment information management, tracking and tasking systems across NATO Allies.

APPROACH:

All systems that facilitate order generating and order‑fulfilment at the tactical level and distribution management at higher levels will be identified. This identification will then be utilised to develop a guiding set of principles (architecture, interoperability requirements, interfaces, etc.) that will allow coalition forces to effectively integrate and employ Artificial Intelligence to facilitate the planning and execution of tactical sustainment operations directly to the point of need/ consumption.

FINDINGS:

This study will conclude in December 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COMPLETED RESEARCH

IMPROVING HUMAN EFFECTIVENESS THROUGH EMBEDDED VIRTUAL SIMULATION (RTG HFM‑165)

ACTIVITY TYPE:

Task Group

DURATION:

October 2007 – October 2011

OVERVIEW:

This precursor RTG examines the efficient use of virtual and augmented simulation technologies for integrating training functionality into operational equipment. Embedded training integrates training into operational equipment, allowing military personnel to train while deployed.

OBJECTIVES:

To explore the potential of innovative virtual simulation technologies for embedded training to investigate the relevant concepts and system capabilities.

When using embedded virtual simulation in the military domain, address human effectiveness issues across conceptual, functional, and technological levels.

RECOMMENDATIONS:

The RTG endorse embedded virtual simulation as a tool to augment deployed training and improve military effectiveness.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ASSESSMENT OF INTELLIGENT TUTORING SYSTEM TECHNOLOGIES AND OPPORTUNITIES (HFM‑ET‑120)

ACTIVITY TYPE:

Exploratory Team

DURATION:

September 2011 – September 2012

OVERVIEW:

HFM‑ET‑120 was established to identify pathways to instructional efficiencies regarding reduced costs and enhanced effectiveness using intelligent computing in training and educational development.

OBJECTIVES:

To review and provide an analysis of the nature, extent, availability, and feasibility of opportunities presented by the Intelligent Tutoring System (ITS) for conducting NATO education and training.

APPROACH:

Military operations, especially those characteristics of current irregular warfare environments, require, among other things, improvisation, rapid judgement, and the ability to deal with the unexpected. Instruction to produce these capabilities requires interaction with “intelligent” systems that rapidly adjust to individual learner abilities, prior knowledge, experience, and, to some extent, misconceptions. As with basic instruction, technology is required to make this education and training affordable and effective at the scale needed for military personnel.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

ASSESSMENT OF INTELLIGENT TUTORING SYSTEM

TECHNOLOGIES AND OPPORTUNITIES (HFM‑237)

ACTIVITY TYPE:

Research Task Group

DURATION:

March 2013 – March 2016

OVERVIEW:

RTG HFM‑237 was established to map research and development efforts worldwide for adaptive training and education. This will improve the understanding of the antecedents of effective and affordable ITSs.

OBJECTIVES:

To review and analyse the nature, extent, availability, and feasibility of opportunities presented by ITSs for conducting NATO education and training.

To suggest solutions to challenges in the widespread adoption of ITS in military training and education.

APPROACH:

Both ability and prior knowledge affect learning rates. The learner’s ability accounts for the acquisition of knowledge, and prior knowledge is a determining factor for learning rates. The disparities in learning rates and prior knowledge increase the need for individualised instruction. This would require the instruction developers to foresee and program for every state of the learner and the learning system during instruction.

A solution is for the computer to assume ‘authoring’ and perform it in real‑time as needed by the state of the learner and the instructional system. Doing so requires the application of artificial intelligence, which is the basis for developing intelligent tutoring systems.

FINDINGS:

It is easier to apply ITS technologies to well‑defined domains like mathematics, physics, and software programming, where there are a limited number of paths to success. On the other hand, procedural military tasks (e.g., land navigation, marksmanship, and combat casualty care) have proved more amenable to tutoring approaches.

The challenges to the practical use of ITSs in the military include the development of tools and methods to train psychomotor tasks involving physical coordination and skill and conducted beyond classroom environments.

Since most military operations involve teams and teamwork, NATO should endeavour to discover methods to the author, deliver, and automatically manage adaptive tutoring of teams by ITSs and computer‑based guidance of units engaged in collaborative learning. Furthermore, as embedded training concepts, develop research shall explore the potential of artificially intelligent agents to guide embedded team learning.

The RTG makes a functional recommendation to enhance authoring automation, identify/develop, and use AI‑based authoring capabilities to reduce authoring workload. Authoring tools are the single most critical area of challenge and impact. Therefore, RTG HFM‑237 recommends that NATO countries invest in ITS authoring tools to affect ITS affordability and usability. Without a focus on ITS authoring tools, ITSs will continue to be difficult, costly to develop, and impractical for widespread NATO use.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

USE OF M&S IN: SUPPORT TO OPERATIONS, HUMAN BEHAVIOUR REPRESENTATION, IRREGULAR WARFARE, DEFENCE AGAINST TERRORISM AND COALITION TACTICAL FORCE INTEGRATION (RSY MSG‑069), 2009 – 2010

ACTIVITY TYPE:

Research Symposium

DURATION:

September 2009 – September 2010

OVERVIEW:

RSY MSG‑069 was dedicated to using advanced M&S, including HBM and CGF, to decrease costs and increase military training efficiency. The Symposium audience included experts from NATO countries, Partners‑for‑Peace (PfP) nations, and invited nations.

APPROACH:

The symposium collected documented research contributions on central M&S key topics across nations. Thus, offering a common forum of discussion on the state‑of‑art of M&S, impact and lessons learned. It considered twenty‑one papers in separated sessions grouping the contributions in clusters.

OBJECTIVES:

To provide an opportunity to extract and expand knowledge and resolution of issues on NATO’s M&S activities concerning both development and effective employment of M&S.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN MODELLING FOR MILITARY APPLICATION (HFM‑202)

ACTIVITY TYPE:

Research Symposium

DURATION:

January 2010 – May 2011

OVERVIEW:

RSY HFM‑202 was organised to consider the state‑of‑human science and its application to M&S technology and recommend to NATO to make science‑based M&S tools available for military application. This RSY brought together MSG, SAS and HFM TPs.

APPROACH:

The symposium brought together domain experts from different human modelling and AI disciplines to develop a cross‑domain understanding of integration demands. Forty‑three technical papers were scheduled, representing research and development efforts in eight nations. The presentations and papers contained applications to one or more of three general capability areas in M&S Military Analysis, Training and Military Systems Development and Acquisition.

FINDINGS:

Incorporating a fully parameterised “human view” with representative attributes into agent‑based modelling and using Bayesian network analyses benefits essential analytical processes employed by NATO militaries. Many papers addressed agent‑based modelling and Bayesian network analyses of modelling teams, organisations, government actions, societies, cultures, and the resulting military contributions to non‑traditional operations. In addition, new algorithms were presented that can insert into human behaviour models as moderators for heat, strain and fatigue, hydration levels, and terrestrial altitude effects. Much discussion and technical interchange are needed to address the issue of architectures and validation strategies for vertical and horizontal integration of multiple advanced models. Not enough exposition to the larger body of research on models of human behaviour, which might be essential for robust social‑culture modelling (hybrids using agent, systems dynamic and game‑theoretic models). Need for research and technology development for the vertical interoperability of tactical‑operational‑strategic models.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN BEHAVIOUR MODELLING FOR MILITARY TRAINING APPLICATIONS (MSG‑107)

ACTIVITY TYPE:

Research Workshop

DURATION:

June 2011 – June 2012

OVERVIEW:

RWS MSG‑107 was organised to consider the state‑of‑human science and its application to M&S technology and recommend to NATO to make science‑based M&S tools available for military application.

OBJECTIVES:

To follow up on RSY HFM‑202, integrate the various panels’ interests, focus on military training, and investigate the challenges and solutions of this specific application of HBR.

APPROACH:

To continue with the integration process and propose recommendations, this RWS brought together domain experts from disciplines in human modelling, computer science, modelling standards and artificial intelligence and innovative man‑machine interaction technologies (e.g., speech, gestures, mixed reality, natural language understanding).

FINDINGS:

There is a clear demand for compact, re‑usable cognition models that could provide virtual players in training with humanlike intelligence. Integration of science‑based models describing part of human behaviour into complex military training settings and analytic environments has been proven difficult.

There is also a need for more natural interaction between human trainees and simulated characters (e.g., advanced interface).

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

HUMAN BEHAVIOUR MODELLING FOR MILITARY TRAINING APPLICATIONS (HFM‑220)

ACTIVITY TYPE:

Research Workshop

DURATION:

January 2012 – December 2013

OVERVIEW:

RWS HFM‑220 was established to overview the quantitative representation of performance, decision‑making and behaviour of individuals and groups.

OBJECTIVES:

To develop a report with recommendations for human modelling in Military T&S and its interaction with live player decision‑making in small team environments.

APPROACH:

To devise recommendations, this RWS brought together domain experts from disciplines in human modelling, computer science, modelling standards and artificial intelligence and innovative man‑machine interaction technologies (e.g., speech, gestures, mixed reality, natural language understanding).

FINDINGS:

It proved difficult to simulate the opponent behaviour using ML and Neural Networks without robust datasets that would serve as a basis for opponent behaviour.

In a simulated air combat scenario, ML was used to master the potential complexity of adapting opponent behaviour to new conditions. Neural network parameters were used as input to a mixed cognitive model. There is a danger that student behaviour is adapting to incorrect opponent behaviour was recognised, as there was no prior knowledge on which the opponent behaviour could be based.22

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored. 22

Jan Joris Roessingh and Roel Rijken.

REFERENCE ARCHITECTURE FOR HUMAN BEHAVIOUR MODELLING IN MILITARY TRAINING APPLICATIONS (MSG‑127)

ACTIVITY TYPE:

Research task Group

DURATION:

August 2013 – December 2017

OVERVIEW:

RTG MSG‑127 was established to investigate the seamless interaction of players with realistically simulated human characters. Applications range from training for urban operations (to red force representation in tactical air‑to‑air training.

OBJECTIVES:

The development of a Reference Architecture (RA) for Human Behaviour Modelling (HBM) of individual players intended for use in military training applications.

APPROACH:

A particular area of interest is the seamless interaction of Live players with realistically simulated human characters. This capability has broad applications, ranging from training for urban operations to red force representation in tactical air‑to‑air training. The required level of fidelity in such models may vary across use cases. However, they all represent important characteristics of human cognition and performance.

FINDINGS:

Despite advances in the domains of AA and AI techniques, in terms of M&S, the research on HBM did not advance sufficiently forward. It is expected that DDBM shall dramatically increase its TLR within ten years.

ML techniques benefit from models that operate on symbolic representations of information and knowledge to behave intelligently, therefore greatly benefit from RA.

Intelligent behaviour will require representations of knowledge that facilitate grounding that knowledge into semantically meaningful abstractions of the real world. However, the knowledge engineering challenges are a serious barrier to implementing such systems for complex domains. As a result, it will first be necessary to develop knowledge representations linked to ML mechanisms to allow intelligent systems to acquire the knowledge they need to perform neuro‑symbolic modelling.

DL‑based Neural Networks do not remove the need for and value of a basic common structure of an HBM that a RA provides.

Unless Neural network models based on DL rely on a highly abstracted sense of neural, they are computationally intensive to run, limiting their capacity to simulate human cognition and performance in complex tasks. As a result, their use in HBMs is still rare. However, further developments in both software and hardware may change that situation.

Collecting and generating ‘Big Data’ for automated model development by training DL‑based Neural Networks is a promising area of ongoing research. However, available data is often not machine‑readable or lacks metadata. Moreover, sharing of collected information is also not easily done.

There is a need to bring together not just Human and Physical Sciences but also elements of understanding diverse cultures and religions and the impact on how the models work and interact.

The human behaviour models should act credibly to provide training value. Therefore, verification, Validation and Accreditation (VV&A) of HBM is a major issue.

The following is recommended to ameliorate research gaps:

• Prepare a follow‑up proposal to develop and implement the RA in a pilot project, run the models and analyse the outputs to validate the concept.

• NATO should develop a roadmap for improving the RA for HBM and implementing HBM assets compliant with the RA across NATO and Nations for all application domains.

• This RTG should be linked with the thematic approach to “Artificial Intelligence and Big Data for Decision Support” (RWS IST‑160). The link between Training and Simulation and C4 should be further exploited.

• There is a need for standards for facilitating interoperability, reusability, and flexibility in composing operational model architectures wherein humans and CGFs can work together.

• CGFs should mimic behaviour based on general human characteristics (e.g., cognition, emotion, and physiology), cultural background, and societal role.

• Develop tailored versions of V&V processes and methods for HBM. An important aspect to consider is collecting, vetting, and discovering the data that will be the baseline for the models.

ACCESS THIS ACTIVITY:

and linear.

Please click here to read more about this activity’s findings and other issues explored.

Figure 16 depicts the architecture of human behaviour, consisting of mechanisms (behaviour engine) that operate upon stored mental and physical contents (state representation). Processes should be considered cyclic and iterative rather than sequential

Figure 18: A general architecture of human behaviour

ADVANCED TRAINING TECHNOLOGIES FOR MEDICAL –HEALTHCARE PROFESSIONALS (HFM‑ET‑112

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2010 – January 2011

OVERVIEW:

HFM‑ET‑112 was established to explore rapidly advancing M&S technologies to develop medical outreach tools using the latest computer graphics, natural language processing, and AI technologies.

OBJECTIVES:

To explore the use of modelling, simulation, gaming, and related technologies to develop medical education applications, especially associated with the training of health care professionals.

APPROACH:

)

Leverage video games and related computer‑based technologies and concepts to create compelling applications to teach health care professionals, warriors and their families about medical treatment options and educate them about the signs and symptoms of psychological health, traumatic brain injury and other medical issues from deployment. The same technologies can also be leveraged to encourage healthy behaviours and resilience strategies before deployment.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SYNTHETIC ENVIRONMENTS FOR HSI APPLICATION, ASSESSMENT, AND IMPROVEMENT (RTG HFM‑216)

ACTIVITY TYPE:

Research Task Group

DURATION: September 2011 – September 2014

OVERVIEW:

RTG HFM‑216 was established to describe a modelling and simulation approach to conducting trade‑off analyses and exploring complex design spaces.

OBJECTIVES:

To progress modelling and simulation systems from sustaining incremental improvements in favour of disruptive innovation.

APPROACH:

This report presents a conceptual model of Synthetic Environment for Assessment (SEA), a synthetic testbed that can be used to identify the key issues of concern. Use‑cases were provided to show how SEA could be used with significant benefit to its users. This activity explored the descriptions of ongoing SEA activities to derive a detailed architecture for SEA that identifies resources, models and systems, input, outputs, and user communities, resulting in the core technology areas within SEA and the unique technical barriers that need to be crossed to realise SEA in its fullest form.

FINDINGS:

SEA provides a workable way to solve the practical problems involving simulation in capability development. Calibrated scenarios from one lab can be used in another, resulting in data that can be fairly compared. As a result, researchers can have realistic test environments available to test and compare theories of human performance in various disciplines without having to expend precious resources learning the domain or building “throw away” simulations.

Most importantly, the capability development and procurement community can use SEA as a communication mechanism. For example, if capability development proposes that a recent technology for unmanned vehicles has a specific benefit, they could communicate this to acquisition through realistic combat scenarios within SEA.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SYNTHETIC ENVIRONMENTS

FOR MISSION EFFECTIVENESS

ASSESSMENT (HFM‑ET‑144)

ACTIVITY TYPE:

Exploratory Team

DURATION:

September 2015 – November 2015

OVERVIEW:

HFM‑ET‑144 was established to provide cross‑panel activity in Synthetic Environments. To do so, this ET leverages advanced M&S technologies.

OBJECTIVES:

To explore M&S, gaming, and related AI‑based technologies to develop training and system engineering applications critical to effective human‑system design and analysis. There is a strong emphasis on the user interface and the warfighter in the field.

APPROACH:

Rapid advances in M&S have provided NATO nations with the ability to develop many M&S outreach tools using the latest computer graphics, natural language processing, web content and AI technologies. In addition, using M&S to create a realistic synthetic environment provides the means to design and evaluate new alternative military systems in terms of real scenarios and to develop an effective training capability response to our military personnel will use and maintain these new systems.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

CROSS PANEL ACTIVITY ON SYNTHETIC ENVIRONMENTS FOR MISSION EFFECTIVENESS ASSESSMENT (HFM‑268)

ACTIVITY TYPE:

Research Task Group

DURATION:

December 2015 – December 2018

OVERVIEW:

RTG HFM‑268 demonstrated the potential benefits of Synthetic Environments (SEA) as a tool for capability development.

OBJECTIVES:

To build a realistic synthetic environment, accessible/reproducible by NATO members, which provides the means to design and evaluate new alternative military systems and assess their effectiveness in mission‑relevant scenarios.

To develop the ability to measure and analyse the human‑system performance and the effective interaction of the human as they utilise the system’s hardware and software.

APPROACH:

This RTG focused on Warrior Preparation Center’s (WPC) Spartan Warrior AWACS exercises specifically designed for NATO tactical training. As a result, all the key components of a SEA were installed at the WPC.

FINDINGS:

The RTG results demonstrated a workable way to gather realistic mission efficacy data that decision‑makers could use in the acquisition processes, specifically early design decisions. The SEA should ideally have the capability to be fully automated with artificially intelligent behaviours.

Software automated with artificially intelligent behaviours improves the consistency when running scenarios and may reduce the need for Subject Matter Experts (SMEs) to play various roles in a scenario.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

GAS TURBINE ENGINE ENVIRONMENTAL PARTICULATE FOREIGN OBJECT DAMAGE [EP‑FOD] (AVT‑250)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2016 – December 2018

OVERVIEW:

RTG AVT‑250 was established to examine the operational deployment of military aircraft in challenging environments containing airborne particulates.

OBJECTIVES:

To understand how the vehicle imposes performance impacts on the propulsion system and more precisely identify and understand the causes and the trajectory of engine degradation.

APPROACH:

In a dedicated aircraft modelling and management system, a collection of information in a data lake, combined with phenomenological models of engine and aircraft performance, enables validation and prediction of the level of engine damage.

FINDINGS:

Self‑educating, ML techniques improve aircraft modelling systems that focus on validating and predicting the level of engine damage. These systems offer benefits, including fault prognostics, trends in engine behaviour, optimised fleet management, predictive maintenance, and improved next‑generation engine design and development.

Fusing post‑mission engine data with meteorological and atmospheric contamination data and aircraft position data along its flight route can provide insights into the level of exposure to environmental particulates. These data groups then feed mathematical plug‑in models of engine damage and resulting aircraft performance. Engine, airframe, and atmospheric contamination models can be iterated and improved via a self‑educating, machine‑learning loop. In effect, a Digital Twin of an aircraft and its engines can be created such that key performance parameters can be modelled, effectively providing virtual sensing within an engine. Engines and airframes act as in situ measurement instruments. The system and data lake can be extended across aircraft fleets to pick up enterprise‑wide trends.

Figure 19: An aircraft modelling and management system23

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

Figure 20: Schematic diagram of a big data/artificial intelligence (AI) system for determining current and forecast engine environmental particulate damage24

24 Clarkson, R. and Venti, M. (2018). AVT‑250 Work Product, Schematic Diagram of a Big Data/Artificial Intelligence (AI) System for Determining Current and Forecast Engine Environmental Particulate Damage.

ENHANCED COMPUTATIONAL PERFORMANCE AND STABILITY & CONTROL PREDICTION FOR NATO MILITARY VEHICLES (AVT‑ET‑199)

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2019 – December 2019

OVERVIEW:

AVT ET‑199 was established to explore pathways to generating simulations predicting the military vehicle design’s performance, stability, and control characteristics.

OBJECTIVES:

To identify the activities among NATO nations in the stability and control prediction capabilities pertaining to advanced analytical techniques.

APPROACH:

Reduced‑Order Modelling (ROM) or Surrogate Modelling techniques use the knowledge generated by highly accurate numerical CFD simulations but yield a low‑dimensional approximation to reproduce the characteristics of the physical model of higher complexity or using ML and DL techniques to provide reliable predictions of large‑scale aerodynamic simulations.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

SIMULATION OF LOW PHOTON LIDAR IN COMPLEX ENVIRONMENTS (SIMPL) (SET‑ET‑112)

ACTIVITY TYPE:

Exploratory Team

DURATION:

April 2019 – April 2020

OVERVIEW:

SET‑ET‑112 was established to align community efforts to lay the groundwork for next‑generation simulation capability in single‑photon LiDAR, which provides powerful new capabilities in sensing behind obscurants with low detectability by an adversary.

OBJECTIVES:

• To explore community consensus on the best approach in simulating single‑photon lidar in sensing and Automated Target Recognition (ATR) under complex environmental conditions.

• To accelerate the development of next‑generation AI‑based ATR algorithms by providing synthetic training data.

APPROACH:

Active imaging models simulate the outcome under complex conditions, including scattering events such as dust, rain, and turbulence. Existing models are designed for specific needs. The choices of sensors, sources, and configurations reflect these needs and limitations. This ET will allow comparisons and examinations of various benefits versus trade‑offs to strengthen individual models where expanded capabilities will help mature the simulation skillset to shorten the development cycle and reduce the cost.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

COVID‑19 ADAPTIVE LEARNING AND FORECASTING (CMRE DKOE)

ACTIVITY TYPE:

CMRE DKOE programme

DURATION:

2020 – 2021

OVERVIEW:

This activity was established to forecast the COVID‑19 epidemiological curve reliably.

OBJECTIVES:

To examine applying Adaptive Bayesian learning and forecasting techniques to detect critical epidemiological phases.

APPROACH:

The activity focused on the quickest detection of the epidemiological phases: initiation and termination of an epidemic wave and testing and evaluating forecasting methods in MSA operational scenarios.

FINDINGS:

Adaptive Bayesian learning and forecasting outperform state‑of‑the‑art solutions available in the literature.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

TRAINING, MODELLING & SIMULATION (TMS)

ONGOING RESEARCH

PERSONALISED MEDICINE IN MENTAL HEALTH AND PERFORMANCE (RTG HFM‑281)

ACTIVITY TYPE:

Research Task Group

DURATION:

September 2018 – April 2023

OVERVIEW:

This RTG was established to exploit EDTs to leverage people’s variations in biological makeup for disease prevention, diagnosis and treatment, and optimisation of military performance.

OBJECTIVES:

To harness and encourage new advances in personalised approaches to optimise mental health, medical readiness and prevention/ diagnosis/treatment of disorders related to military‑relevant mission performance.

APPROACH:

This RTG will address “Advanced Human Performance and Health” in the following areas: Human Resiliency, Medical Solutions for Health Optimisation, and Enhanced Cognitive Performance. In addition, this RTG will address other NATO S&T areas, including Big Data & Long Data Processing, Analysis, and Sensor Integration & Networks.

The outcome of this RTG will be identifying innovative precision medicine techniques that will improve how NATO member nations provide mental health problem prevention, diagnoses, and treatment, as well as improvements in focused concentration and mental endurance related to mission performance.

FINDINGS:

This study will conclude in April 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

PERSONALISED MEDICINE IN MENTAL HEALTH AND PERFORMANCE (RTG HFM‑294)

ACTIVITY TYPE:

Research Task Group

DURATION: April 2018 – April 2023

OVERVIEW:

This RTG was established to prove the value of computation and genomics in assessing the effect of operational stressors on a soldier’s health and performance, resulting in resource and time management inefficiencies. These capabilities would help to prepare the soldier or unit before an operation.

OBJECTIVES:

To prove the feasibility and value of using high‑computational power and genomics as a tool in the military.

APPROACH:

Output from this RTG will outline how to use high‑computational power with genomics as a tool, address the ethical challenges, inform policymakers, maintain, and ensure privacy and personnel security, the safeguarding data exchange and storage within an IT department.

FINDINGS:

This study will conclude in April 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

EXPERT PANEL FOR STATE‑OF‑THE‑ART CARDIOVASCULAR RISK ASSESSMENT IN AIRCREW AND OTHER HIGH‑RISK OCCUPATIONS. (RTG HFM‑316)

ACTIVITY TYPE:

Research Task Group

DURATION:

September 2019 – September 2023

OVERVIEW:

This RTG was established to produce consensus recommendations for assessing aircrew with suspected or proven cardiovascular disease.

OBJECTIVES:

To undertake retrospective research on pooled multi‑national datasets, combining data on occupational aviation cardiology.

APPROACH:

This working group will investigate all areas of cardiovascular medicine (screening, coronary artery disease, valvular heart disease, arrhythmias, and heart muscle disease). It will also explore the potential value of novel advanced imaging tools such as computational flow dynamics, machine learning, and artificial intelligence tools in assessing cardiovascular disease in aircrew.

The resulting data will benefit all NATO countries by aiding national experts to standardise aircrew selection, keep aircrew flying if safely possible, optimise the financial and training investments into this high‑value group, and enhance flight safety.

FINDINGS:

This study will conclude in September 2023, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

GOAL‑DRIVEN, MULTI‑FIDELITY APPROACHES FOR MILITARY VEHICLE SYSTEM‑LEVEL DESIGN (RTG AVT‑331)

ACTIVITY TYPE:

Research Task Group

DURATION:

January 2020 – December 2022

OVERVIEW:

The RTG was established to extend the STO’s knowledge of multidisciplinary methods that can fuse data to improve the design of military vehicles. NATO decision‑makers need this information to assess the capability of innovative vehicles of the future.

OBJECTIVES:

To identify and extend the knowledge of design frameworks, improving the adaptive selection information sources of various levels of fidelity for system‑level vehicle design based on data and physics.

APPROACH:

This RTG will examine mathematically rigorous frameworks for fusing information sources of different fidelity (e.g., advanced algorithms). It will also address system‑level considerations, including mixing data to overcome resource constraints and evaluating test scenarios using other relevant methods and frameworks to compare and assess.

New multidisciplinary design techniques are anticipated and applicable to air, ground, sea, and space vehicles. These techniques will build on emerging developments in artificial intelligence, machine learning, surrogate modelling, statistical analysis, sensitivity analysis, and other disciplines.

FINDINGS:

This study will conclude in December 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MULTI‑FIDELITY METHODS FOR MILITARY VEHICLE DESIGN (RWS AVT‑354)

ACTIVITY TYPE:

Research Workshop

DURATION:

January 2020 – December 2022

OVERVIEW:

The RWS was established to broaden the dialogue regarding the technical and military relevance of AVT‑331 findings and help identify potential follow‑on efforts to heighten the benefits of the target methodologies.

OBJECTIVES:

To broaden the AVT‑331 perspective by facilitating the extension of the current state‑of‑the‑art frameworks, architectures, and methodologies. This extension will support the adaptive selection of various sources of information for the design of military vehicles.

APPROACH:

The time and effort to develop next‑generation systems needs to be reduced while simultaneously reducing the rate at which problems arise in system development of evolving military requirements. This RWS seeks to identify capabilities for a quick, accurate and thorough assessment of the design space. For example, this will improve the discovery of system defects due to misunderstood physics through improved physics modelling in the design. In addition, increasing the available design space with more disciplines and potentially leveraging physical interactions. Finally, goal‑oriented approaches will decrease the time and resources required to execute the optimisation process.

The information exchange within the RWS will inform AVT‑331 by discussing the capabilities, opportunities, and future needs. On the other hand, the RWS will benefit from technical activities within AVT‑331

FINDINGS:

This study will conclude in December 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MODELLING AND SIMULATION STANDARDS IN FEDERATED MISSION NETWORKING (FMN) (ST MSG‑193)

ACTIVITY TYPE:

Specialist Team

DURATION: January 2020 – January 2022

OVERVIEW:

The ST was established to integrate the developed capabilities of NATO Modelling and Simulation Group (NMSG) Federated Mission Networking (FMN).

OBJECTIVES:

• To identify and define standards required for the FMN and the use of NMSG products in the FMN.

• To propose an initial operational architecture to employ NMSG products.

APPROACH:

Modelling and Simulation will provide a significant part of achieving interoperability in collective training, course of action analysis, mission rehearsal, and evaluation of proposed/planned systems. Furthermore, the envisioned capability should include state‑of‑the‑art information technologies that use AI techniques. Therefore, participation in the FMN specification process by the NMSG is important.

NMSG is particularly well equipped to help in the areas of Command and Control to Simulation Interoperation (C2SIM), Modelling and Simulation as a Service (MSaaS), Correlated Dynamic Synthetic Environments for Distributed Simulation, and Cyber‑Electronic Warfare (CyEW).

FINDINGS:

This study will conclude in January 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

UNMANNED VEHICLES (UVX)

COMPLETED RESEARCH

ROBOTICS UNDERPINNING FUTURE NATO OPERATIONS (SAS‑097)

ACTIVITY TYPE:

Research Task Group

DURATION:

March 2012 – December 2016

OVERVIEW:

RTG SAS‑097 was established to analyse the growing use of robotic systems in military operations and reassess their advances in NATO operations.

OBJECTIVES:

To analyse and bridge the gap between military operational requirements and cutting‑edge technological possibilities.

To provide experimentation support and open opportunities for further research.

APPROACH:

This RTG conducted a trends analysis in autonomous control, sensor, and platform systems, completing a study of operational requirements. Furthermore, the RTG participated in NATO exhibitions and exercises, cooperated with other NATO research organisations, and participated in numerous other NATO and EU symposiums, demonstrations, and workshops. In addition, the group conducted joint experiments between the Czech University of Defence and the United States Army Tank Automotive Research, Development and Engineering Center (TARDEC), worked on the multipurpose platform development, and supported real mission deployments and joint exercises with National entities.

FINDINGS:

Robotic systems will redefine how modern warfare is conducted and may render existing capabilities obsolete. Future operating environments will be more complex and uncertain, and future NATO forces must adapt. Current models of systems suffer from significant scalability problems and the problem of derivation, use and integration of representations for modelling the environment, control, and motivation. In short, better theories are needed to conceptualise robot systems in operations.

A dialogue between humans and machines and interactions between physical and emotional features with moral, ethical, and legal aspects has not been solved completely. There is a need for generating sophisticated criteria for automated decision‑making, whereas human abilities are limited. The challenge is the lack of system theories allowing holistic analysis of the overall systems, involved processes and their interactions.

A growing number of deployed UxVs increases demands on human operators and communications. Even if successfully transferred, human operators cannot process the amounts of data generated by UxVs. Therefore processing, simulation and modelling play a crucial role. Deep learning methods are an example of an approach designed for Big Data processing. Further effort should be made to effective exploitation and cyber protection of data links.

UxVs enhanced by AI can potentially improve a force’s ability to project combat power.

Removing the human from the vehicle’s cockpit enables a more efficient and cost‑effective design.

Centralised network management systems will not be able to cope with new military challenges.

AI techniques and the Bayesian approach can enhance autonomous capabilities and improve real‑time dynamic planning mechanisms. Consensus must be reached on the definition of “Meaningful Human Control” in military operations, command, and control, and technical or other angles.

An autonomous lethal decision should be subject to severe constraints and strict supervision.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored

MACHINE LEARNING TECHNIQUES FOR AUTONOMOUS MANOEUVRING OF WEAPON SYSTEMS (IST‑ET‑067)

ACTIVITY TYPE:

Exploratory Team

DURATION:

March 2012 – December 2012

OVERVIEW:

IST‑ET‑067 was established to examine how AI techniques provide the means for intelligent autonomous agents to reduce workloads for the warfighter significantly.

OBJECTIVES:

To initiate guidance among researchers and developers in line with military end‑user requirements.

To initiate standardisation and harmonisation in developing autonomous intelligent battlefield agents with Machine Learning capabilities.

APPROACH:

This ET addresses integrating autonomous intelligent systems into the battle space. Furthermore, contributes to an understanding of the role of AI in improving Modelling and Simulation, Systems Analysis, Knowledge Development, Distributed Training and Exercise. The special focus is on the organisation and the architectural design of autonomous intelligent battlefield agents, specifically concerning existing and relevant ML techniques and their current applications. Potential applications of ML techniques to defence domains, such as C2, decision‑making, simulation and cyber defence, potential barriers for application of ML, critical factors, and appropriate ML techniques in other domains.

Autonomous intelligent solutions can be applied at the strategic level (course of action analysis), at the operational level (simulation of logistics), and at the tactical level (simulation of engagements).

Systems engineering aspects of Machine Learning techniques, such as requirements definition and verification & validation of autonomous intelligent systems, have been identified as challenging.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

AUTONOMY IN COMMUNICATIONS LIMITED ENVIRONMENTS

(SCI‑ET‑015)

ACTIVITY TYPE:

Exploratory Team

DURATION:

June 2014 – December 2015

OVERVIEW:

SCI‑ET‑015 was established to examine the role of embedded computing in enabling the autonomous behaviour of unmanned systems.

OBJECTIVES:

To create a framework to facilitate research collaboration, data sharing, and joint at‑sea experimentation on autonomy, machine intelligence, and unmanned systems.

APPROACH:

The ET examines how AI can alleviate problems or limitations of a military UxV’s warfighting capacity in settings where communications to unmanned systems are severely limited (undersea domain). Embedded computing is particularly important for sensor signal processing, vehicle situational awareness, and long‑term mission planning.

Embedded computing enables the autonomous behaviour of unmanned systems, which is necessary to create safer and scalable defence capacities.

The expectation of continuing technological advances makes autonomy a pressing topic to stay ahead of the potential opponents.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

INTELLIGENCE & AUTONOMY (ROBOTICS) (RSM IST‑127)

ACTIVITY TYPE:

Research Specialists’ Meeting

DURATION:

October 2014 – December 2016

OVERVIEW:

RSM IST was established to map the research landscape and foster collaboration in intelligent autonomous systems and their military applications.

OBJECTIVES:

To consolidate the knowledge in the field of intelligent and autonomous robots, identify the gap between civilian solutions and military needs and pursue the transfer of technologies and applications into the military domain.

APPROACH:

Experts from eleven nations presented aspects of their recent research on intelligence & autonomy software for unmanned ground, surface, underwater, and air vehicles. The contributions gave novel algorithms, software frameworks, and enhanced military capabilities, like reconnaissance robots, autonomous military transport, and soldier robot teaming.

FINDINGS:

There is substantial interest in unmanned systems with autonomous capabilities, but no current cooperation across nations in the development process. Finding a commonly agreed definition of autonomous behaviour and intelligent system is challenging.

The civil and automotive industry drives the development of autonomous UxVs. New sensors at affordable prices offer solutions to standing problems. Furthermore, the civil sector is already paving the way for the public and legal acceptance of autonomous systems. These developments push military research a huge step forward. However, the requirements for autonomy in military applications are much higher. Unlike in the civil sector, in the military domain, the systems must deal with information‑poor yet spatially complex environments and additional systems requirements to resist attacks from intelligent adversaries.

Further challenges are linked to the interaction of robots and soldiers and the integration with modern command systems. These challenges can be addressed by either enhancing the communication capabilities of the methods or by overcoming the strong dependence on communication links by fostering the intelligence and autonomy of the vehicles. Algorithms devised for the intelligence and independence of the systems must address requirements specific to the military. Specifically considering military rules and tactics and being able to interact with each other and the soldiers in the field.

Image recognition based on segmentation and classification is advancing due to machine learning techniques like deep neural networks. These approaches allow distinguishing classes of objects and linking image content to background knowledge, thus enabling interpreting the sensor data of the environment. However, the nations develop similar capabilities with different assumptions and operational requirements, such as regional weather or vegetation. Therefore, to deploy an autonomous capability, it is necessary to understand optimal operating conditions and employ agile systems suitable for many situations.

To strengthen and speed up the development of intelligent autonomous military systems, close collaboration between researchers of different NATO nations is necessary, especially to keep upcoming solutions flexible, compatible, and interoperable.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

NOVEL EMPLOYMENT OF AUTONOMOUS MILITARY SYSTEMS (AMS) (SCI‑ET‑022)

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2016 – December 2017

OVERVIEW:

SCI‑ET‑022 was established to raise awareness of the various possibilities of employing autonomous military systems in warfighting.

OBJECTIVES:

To examine technical and operational opportunities and challenges presented by increasingly intelligent military autonomous systems.

APPROACH:

The activity focuses on conceiving new system concepts and the ability to assess their potential military utility realistically. The study seeks an understanding of how innovative ideas could be researched, developed, and transitioned to being in the inventory of the military user. ET leverages expertise from the IST lecture series on Horizon Scanning (IST‑135) and the SAS TP to reach that goal.

This ET describes AMS as an embedded hardware or software sub‑system within a larger system, employing bottom‑up and top‑down research. For the bottom‑up investigation, the ET surveys the existing works that detail the novel and powerful aspects of AMSs and what these aspects bring to warfare in terms of performance or capability. The bottom‑up approach investigates current and future threats and attempts to conceive new system concepts that could be enabled by autonomy and offer a significant military advantage. The survey includes understanding the key characteristics and status of technology adoption.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

MISSION ASSURANCE AND CYBER RISK ASSESSMENT FOR UAXS (MULTI‑DOMAIN UNMANNED/AUTONOMOUS VEHICLES AND SYSTEMS) (IST‑ET‑099)

ACTIVITY TYPE:

Exploratory Team

DURATION:

January 2017 – December 2017

OVERVIEW:

IST‑ET‑099 was established to examine the role of Multi‑Domain Unmanned and Autonomous Vehicles and C4ISR Systems (UAxS) in future NATO operations.

OBJECTIVES:

Consider emerging methods and frameworks in UAxS mission assurance, cyber security, and risk assessment.

To develop an initial model of a synchronised, multi‑domain mission conducted by UAxSs and Cyber‑Physical‑Systems (CPS).

APPROACH:

UAxS have the potential to deliver substantial operational value across a diverse array of missions, including (autonomous) decision‑making speed for real‑time cyber and AUxS operations, processing of heterogeneous and high‑volume data, and course‑of‑action determination in complex multi‑domain missions, and endurance in unmanned vehicles. NATO recognises this by supporting several projects and programs.

Responding to attacks against UAxSs and trust in autonomous systems were key challenges. The study suggests utilising formal methods and techniques toward more holistic security and cyber risk assessment in robust multi‑domain UAxS mission models.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

EMPLOYING AI TO FEDERATE SENSORS IN JOINT SETTINGS (SAS‑ET‑EI)

ACTIVITY TYPE:

Exploratory Team

DURATION:

July 2019 – July 2020

OVERVIEW:

SAS‑ET‑EI was established to enable operational planners to design an effective federated network of autonomous unmanned ISR vehicles.

OBJECTIVES:

Identify unique characteristics of different targets in each operational scenario.

Examine methods to set up and manage an ISR swarm in each scenario.

APPROACH:

Validation of target identification based on diverse types of sensors occurs late in the intelligence cycle. ET suggests first evaluating the physical characteristics of the scenario and the expected type of target, and then, based on this analysis, specifying the best mix of sensor types and number of sensors to obtain the highest possible success rate from a target identification system using unprocessed data from the selected sensors. Traditional sensor systems may be augmented by swarms of unmanned ISR vehicles, as being investigated in SET‑263. This activity will focus on selecting types and numbers of sensors to federate.

Using machine learning in UxV automatic target identification systems continuously improves their success rate.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

UNMANNED VEHICLES (UVX)

ONGOING RESEARCH

NEXGEN ROTORCRAFT IMPACT ON MILITARY OPERATIONS (LTSS AVT‑329)

ACTIVITY TYPE:

Long‑Term Scientific Study

DURATION:

January 2020 – December 2022

OVERVIEW:

The LTSS was established to assess the medium and long‑term impact on military operations that Science and Technology (S&T) developments for future rotorcraft are expected to deliver.

OBJECTIVES:

To identify the trade space applicable to next‑generation rotorcraft operational capability concerning technology, financial and schedule information.

APPROACH:

A sizeable portion of the NATO nations’ fleet of rotary‑wing assets will require replacement or retirement in the 2035+ timeframe. This represents a risk to existing capabilities and an opportunity to introduce next‑generation capabilities.

The activity will assess projected S&T development on the military operational impact of next‑generation NATO rotorcraft. The assessment will include qualitative and quantitative evaluations to link EDTs with operational impact. This will include how EDTs, systems and methods may affect tactical concepts and doctrines. Reciprocally, recommendations on the influence of emerging military doctrine on future S&T priorities will be made.

FINDINGS:

This study will conclude in December 2022, and a final technical report will be published.

Access this Activity:

Please click here to read more about this activity’s findings and other issues explored.

ARTIFICIAL INTELLIGENCE IN COCKPITS FOR UAVS (RWS AVT‑353)

ACTIVITY TYPE:

Research Workshop

DURATION:

January 2020 – December 2022

OVERVIEW:

The RWS was established to explore the underlying theory, computational models, systems integration, and field applications of AI‑enhanced guidance, navigation, and control of UAVs to promote trust in autonomous decision‑making.

OBJECTIVES:

To create a shared understanding between experts of different technologies to improve levels of autonomy of UAVs.

APPROACH:

The wider use of UxVs in military applications is resulting in a paradigm shift in operations, execution methods and strategies. UASs depend on the availability of real‑time data and AI techniques to provide filtering, fusing, learning and decision‑making capabilities to derive the necessary information while ensuring safe operations.

On a strategic level, AI solutions for planning and learning can be integrated into self‑adaptive frameworks that monitor, analyse, plan, and execute airborne missions continuously. Due to the human‑machine interactions between UAVs and ground crews (e.g., air traffic control and mission control), communications infrastructure and situation‑aware levels of autonomy will also have to be addressed by future airborne decision support systems.

FINDINGS:

This study will conclude in December 2022, and a final technical report will be published.

ACCESS THIS ACTIVITY:

Please click here to read more about this activity’s findings and other issues explored.

APPENDIX B – NATO’S AI INTERESTED PARTIES IN THE DEVELOPMENT AND ADOPTION OF AI

APPENDIX B – NATO’S AI INTERESTED PARTIES IN THE DEVELOPMENT AND ADOPTION OF AI

POLITICAL AND POLICY MAKING

• International Staff – Emerging Security Challenges Division (IS‑ESC)

• International Staff – Defence Investment Division (IS‑DI)

• International Staff – Defence Policy and Planning Division (IS‑DPP)

• International Staff – Operations (IS‑OPS)

• International Staff – Executive Management (IS‑EM)

• Joint Intelligence and Security Division (JISD)

• Office of Legal Affairs (OLA)

• NATO Standardization Office (NSO)

MILITARY OPERATORS

• International Military Staff (IMS) and its substructure

• NATO Headquarters Consultation, Command and Control Staff (NHQC3S)

• Allied Command Transformation (ACT)

• Headquarters Supreme Allied Command Transformation (HQ SACT)

• Allied Command Operations (ACO) and its substructure

• Supreme Headquarters Allied Powers Europe (SHAPE)

• NATO Bilateral Strategic Command (Bi‑SC)

PROCUREMENT, ARMAMENT AND C3 COMMUNITY

• NATO Capability Groups (CP)

• C3Board and supporting Staff (iDEA Group, Digital Transformation Task Force, etc)

• NATO Support and Procurement Agency (NSPA)

RESEARCH COMMUNITY

• Science and Technology Board (STB)

• Science and Technology Organization (STO) and its bodies

• NATO Industrial Advisory Group (NIAG)

• NATO Communication and Information Agency (NCIA)

• Defence Against Terrorism Programme of Work (DATPoW)

• Science for Peace and Security (SPS) Programme

OTHER(S)

• Join Operation Centre (JOC)

• Lines of Delivery (LoD)

• Multi‑Domain Operations (MDO)

• Office of the Financial Controller (OFC)

• Office of the Chief Information Officer (OCIO)

APPENDIX C –

ABBREVIATIONS AND ACRONYMS

APPENDIX C – ABBREVIATIONS AND ACRONYMS

AA Advanced Algorithms

AI Artificial Intelligence

AIS Automatic Identification System

AMS Autonomous Military System

ARM Radar Resource Management

ATR Automated Target Recognition

AUV Autonomous Underwater Vehicles

CMBA Content‑Based Multi media Analytics

CI Computational Imaging

CNN Convolutional Neural Networks

CRN Cognitive Radio Network

CS Compressive Sensing

CyEW Cyber Electronic Warfare

C2 Command and Control

C4 Command, Control, Communication, and Computers

DL Deep Learning

EO/IR Electro Optical and Infrared

ELM Ethical, Legal and Moral

ES Electronic Support

ESM Electronic Support Measures

EW Electronic Warfare

GAN Generative Adversarial Networks

HAT Human Autonomy Teaming

HBM Human Behaviour Modelling

HMS Human Machine Symbiosis

ISR Intelligence, Surveillance and Reconnaissance

ITS Intelligent Tutoring System

MAS Multi Agent Systems

MCM Mine Countermeasure

MHC Meaningful Human Control

MLM Machine Learning

MSaaS Modelling and Simulation as a Service

MSA Maritime Situational Awareness

NN

Neural Network

RA Reference Architecture

ROM Reduced Order Modelling

SEA Synthetic Environment for Assessment

S&T Science & Technology

UAxS Multi Domain Unmanned/Autonomous Vehicles and Systems

UGV Unmanned Ground Vehicles

UV Unmanned Surface Vessel

UxV Unmanned Vehicle (multi domain)

UXO Unexploded Ordnance

V&A Verification & Validation

VV&A Verification, Validation and Accreditation