Predictive Logistics Summit 2024 - Presentations

ODASD Product Support

North America Opus Suite Conference DASD (PS) Overview April 2024 Ms. LISA P. SMITH, SES

Deputy Assistant Secretary of Defense for Product Support Office of the Under Secretary of Defense for Acquisition and Sustainment

1

Product Support Planning and Execution

Product Support is a key component of weapon system development, implementation, and management.

2

ASD(S) Strategic Plan Strategic Plan – 2023/2024 Deliver

Goal 1: Deliver Sustainable Logistics to Support DoD Mission Requirements

Deliver

Goal 2: Deliver Cost-Effective Materiel Readiness to Meet DoD’s Warfighting Requirements

Enable

Goal 3: Enable Effective, Affordable, and Sustainable Warfighting Capability

Goal 4: Transform Global Logistics to Deliver

Transform Readiness and Lethality

The ASD(S) Strat Plan has 4 strategic goals aligned to the 2022 National Defense Strategy.

3

Goal 3: Enable Effective, Affordable, and Sustainable Warfighting Capability Obj 3.1: Provide the DoD enterprise with policy, processes, and guidance that foster effective product support planning & execution • Revise DoDI 5000.91 to reflect updates in statute and policy • Publish Independent Logistics Assessment Guidebook, Performance Based Logistics Guidebook, PS Business Case Analysis • Partner with the IP Cadre and DAU to develop a strategic messaging plan to disseminate best practices

Obj 3.2: Provide DoD decision makers w/comprehensive, timely, & actionable program sustainment assessments in order to develop affordable PS • LCSPs w/out caveats in FY23, KPPs/KSAs and Service O&S Costs • Signed NC3 Sustainment Metrics Reporting MOA • ASD(S) approved F-35 LCSP • Develop plan to integrate microelectronic capabilities into weapon systems product support strategies • Incorporate language for minimizing microelectronics supply chain vulnerabilities into PS policy and guidebooks

Obj 3.3: Develop and lead civilian logistics workforce policy, training, & recognition

Obj 3.4: Develop & synchronize strategic/operational logistics initiatives w/international allies & partners

• Develop credentials for Digital Product Support, Supply Chain Resiliency, Product Support Data Requirements • Develop plan to identify/address Critical Industrial Skill gaps w/in OIB • Identify and explore recruitment & retention solutions • Publish updated Logistics Human Capital Strategy and updated Logistics Functional Community Competencies • Develop SES Logistics Tier-level training recommendations

• Publish maintenance, repair, and overhaul playbook to expand utilization of allies’ industrial base • Lead bilateral & multi-lateral strategic logistics engagements w/allies & partners • Develop engagement plan to facilitate Korean Ministry of National Defense implementing their Product Support Manager Initiative, and for Singapore Government defining Supply Chain Metrics • In partnership with the IP Cadre, establish an IP Community of Practice to support the Joint Sustainment and Supply Chain Forum

Product Support lines of efforts and metrics enable implementation of the NDS and achieve ASD(S) goal.

4

Modeling & Sim Examples in Product Support • One program used M&S to determine the required manpower numbers and skillsets, along with the inventory and support equipment, needed to support the Sortie Generation Rate Key Performance Parameter • Another built its Intellectual Property strategy around 6 use cases (operational, training, maintenance, spare parts procurement, modifications & upgrades, software maintenance & upgrades) using sustainment modeling to determine the right mix of data rights for the government.

• A third program creatively used M&S across all the Product Support elements to determine the correct mix of an organic and commercial workforce for engineering, supply support, maintenance, and the rest.

5

Regional Sustainment Framework • Objectives • Prevail in Contested Logistics Environments • Enhance Military Readiness

REGIONAL SUSTAINMENT FRAMEWORK (RSF)

• Strengthen Regional Partnerships

• Approach • Pathway Projects with 5 Nations • Strategic Prioritization and Alignment • Aligned with NDS and NDIS • Whole of Government Approach (e.g., DoS, DoC) • Aligned with COCOM and MILDEP priorities

• Leverage Existing Resources • Foster Collaboration and Interoperability • Multi-National Collaboration • Interoperable with Partner/Allied Nations

FOCUSED GLOBAL CO- SUSTAINMENT

• Industry & Inter-agency Collaboration • PSM Workshop (Fall 2024)

Roadmap for a Department of Defense Sustainment Paradigm Shift

Summary • Up-front planning ensures effective, affordable, and sustainable life cycle product support for warfighting capability • Sustainable logistics, cost-effective material readiness, effective, affordable warfighting capability, all rely on modeling and sim to facilitate success • Modeling and simulation tools and experts are essential to successful RSF implementation

7

Questions

8

Expanding Predictive Logistics At Northrop Grumman

Kenney Crooks

NG Fellow 04/16/24

1

Introduction to Northrop Grumman Northrop Grumman Aeronautics Sector develops Advanced Technologies and Concepts for: – Aerospace and Autonomous Systems, and Intelligence, Surveillance and Reconnaissance

RAM Engineering Department: – Responsible for dependability and supportability in the lifecycle management – Ensure Reliability and Maintainability performance for operational availability and mission effectiveness – Provides qualitative and quantitative evidence that systems will not be associated with unacceptable risk – Ensures systems required functions perform under stated conditions for a specified period of operation with limited downtime

Enable the Digital Engineering Ecosystem (DEE) Develop robust Modeling and Simulation capabilities for the entire life cycle Data integrity from design to operations and maintenance Evolve to Model Based and Model Centric approaches with Digital Technologies

3

•

–

Drive efficiency, affordability, and agility

–

Drive innovation for information and insight

–

Established connectivity throughout life-cycle

–

Define governance and traceability

–

Improve operational excellence

–

Increase Customer collaboration

–

Improve user experience

Model Based Sustainment (MBS) –

Identify Supportability impacts early in design

–

Improve availability, supportability, maintainability

–

Enable Cost-effective Maintenance Practices

–

Increase insight into Fielded Performance and Supply Chain

Digital Engineering Ecosystem - Advantages •

Streamline Development Processes – Increase data integrity and reuse – Higher Change Management Visibility

•

MFG O&M

Design

Improved Decision-Making – Optimize Sustainment Strategies – Early Detection of potential failures

DEE

– Enable Predictive Maintenance – Increase response time

•

Enhanced Collaboration – Enable Common Data Models (CDM) – Distributed Analyses and Results

4

Supplier

Customer MBS

DEE Advancements for Sustainment Design Engineering

Data Access & Viz

Manufacturing Service Bus

Customer

DEE

• Real Time KPIs • Maintenance Time Drivers • Model-Centric Collaboration • Spares Availability

Machine Learning • Predictive & Prescriptive • Work Order Generation • Part Replenishment • Quality Control

XR & Mobile Apps Supplier

• Real-time Sensors Alerts • AR Work Instructions • Real Time Quality • Training & Skill Checks

Analytics for Operational Readiness Source

Analysis

Analytics

Readiness

Automation

Historical

Data Management

Conditional

Model Based

Causality

O&M Data

Design

6

Enable Traceability and Data Integrity – Improve upon collection of RAM data such as component reliability and maintainability predictions, cost, mission profile information and utilization.

– Remove disadvantages of key information collected in several different systems of record and frequencies

FRACAS R&M Prediction

Resources

LCN

RAM

LPD

Automating a Solution OPUS repository  Model Configuration auditing  Support access by simultaneous users

• Refine RAM data and Models

• Import RAM and Configuration data

Data Pipeline  Allow data to flow between OPUS and our enterprise repository

Maturation

Automation

Validation

Verification

Define Process  Compare RAM and OPUS data to update accordingly

Establish Technique  Compare data from RAM and OPUS to identify disconnects

Automate Results  Data Visualization and Historical Audits

8

• Report and Compare

• Check Data and Model for Errors

Future State of MBS MFG  Include RAM strategies as early as possible to avoid program roadblocks  Monitor the effectiveness of MBS solutions during production and deployment  Tune predictive measures as data is collected  Refine CDM as the systems are deployed  Utilize advanced machine learning algorithms to assist in complex failure patterns

O&M

Design

CDM

MBS 9

Summary

10

•

Improve total data quality and speed across our systems’ life cycle

•

Promote data reuse from design to operations and maintenance

•

Increase our workforce efficiency through integrated value streams

•

Streamlining to common tools and reduction of overhead costs

•

Increase collaboration and transparency with our customers

•

Reduce efforts on data management and increase focus on analysis and results

•

Higher visibility of causality to operations & maintenance gaps

Focus on Affordability, Speed, and Data Quality

11

2024-04-16

LCM ANALYTICS

Where could we go from here?

Patrik Alfredsson, PhD

Principal Advisor LCM Systecon AB

WHAT IS LIFE CYCLE MANAGEMENT (LCM)?

• the ability and ambition to address and manage the thru-life (life-cycle) perspective of the system/product – –

considering and factoring in impact on future behavior, costs and effectiveness (readiness, productivity, value delivered) over time when taking (strategic) decisions regarding the system/product

• life-cycle perspective entails covering – – – –

availability performance, and its impact on system effectiveness cost for support/maintenance/sustainment of the system/product and its components cost for operations/utilization of the system/product obsolescence – technical/economical life lengths of product components

• this requires –

strategy (directives, process, methodology, …), information and analysis

• analysis – –

information analysis – transform data into information, knowledge and insights decision analysis – assess the impact on cost and effectiveness from different decisions/alternatives – guide/optimize decisions

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 2

”CRASH COURSE”

WHAT IS LIFE CYCLE MANAGEMENT (LCM)?

• of particular importance in early phases – when strategic decisions are taken 200%

total cost (LCC) to reach ambition/satisfy need

with LCM

without LCM

150%

100%

“committed” cost (LCC) 50%

“actual” cost (LCC) 0%

specification

design development

production phase-in

sustainment operations & support

retirement phase-out

• normally, ambition (system effectiveness) takes the hit when costs (LCC) exceed budget/funding

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 3

”CRASH COURSE”

LCM HAS FORMED MY PROFESSIONAL CAREER • 27 years at Systecon – 30+ years in the ”business” – –

Chairman of the board Systecon Group AB Principal Advisor LCM

• academic background – KTH – PhD, 1997 – –

”On the Optimization of Support Systems” 1992 and 1994 Systems Engineering at Virginia Tech

• model development –

models and analysis tools (e.g., Opus Suite, customer-specific, project-related, …)

• projects and consultancy –

analyses, decision support, RAMS, support systems, contracting, …

• education and organizational development

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 4

PATRIK ALFREDSSON

”THE CLASSIC PLAYING FIELD”

OPERATIONAL CONCEPTS (operations/usage) System Effectiveness – Availability 1.0

0.9

TECHNICAL SYSTEM (product)

need of support

capability to support

0.8

0.7

0.6

0

10 000

20 000

30 000

40 000

Cost – LCC

• analyze/assess impact from different alternatives/decisions on – –

availability performance – system effectiveness total cost – LCC

• optimize support (spares dimensioning, maintenance concept (LORA), …)

–

operational concepts and product RAMS properties given as input

50 000

SUPPORT SYSTEM (support)

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 5

FOR OPUS SUITE ANALYTICS

• reliability (and scheduled maintenance) – how long before/between product non-functional/unavailable?

– MTBM/MTBF (mean time between maintenance/failure)

• maintainability (and testability)

OPERATIONAL CONCEPTS (operations/usage)

R

reliability

TECHNICAL SYSTEM (product)

– how easy/fast to perform required maintenance tasks (ideally)?

– MTTM/MTTR (mean time to maintain/repair)

• supportability

A

– what logistic resources are required to support the product

– how long before required logistic resources are available?

– MLDT (mean logistics delay time)

• availability – what fraction of time is the product functional/available?

– A (average availability)

availability

M

S

maintainability

supportability

SUPPORT SYSTEM (support)

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 6

PRODUCT RAMS PROPERTIES

OPTIMIZE SUPPORT THE CORE OF OPUS10

OPUS10

• what quantities of spare components/parts are needed

C as e: J C 1 as e: J 1 Point : Point 117 : 117 C I=37 552 k USD

400

W WT=23, T=23,55 55 Hours Hours

300

J1

200

100

• spares optimization formulation and modeling framework –

23,55 0

0

20 000

37 552 40 000 [kUSD]

first OPUS model in 1970 – METRIC in 1968

• extensions/improvements to OPUS (OPUS10) over the years, e.g., – – – –

unlimited, flexible indenture levels and echelons endurance scenarios supply flexibility – lateral support revised data model (RDM)

• increased design space – LORA – – –

maintenance concept resource dimensioning/requirements minimize down time at minimum LSC

OPERATIONAL CONCEPTS (operations/usage) SUPPORT SYSTEM (support) TECHNICAL SYSTEM (product)

60 000

80 000

Spares Investments

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 8

to limit/minimize impact on unavailability from spares delays/shortages by providing safety stock at workshops and nodes in the supply chain at minimum cost (investment, bound capital)?

Spares Waiting Time 500

[Hours]

– – –

C/E-Curve Diagram

ASSESS AND ANALYZE LCC THE CORE OF CATLOC

• what about other cost elements?

• scope of CATLOC – – – – –

model LCC – cost breakdown – user-defined or from template describe how a cost element is to be assessed – underlying parameters and formulas assess/predict LCC for different alternatives of interest study robustness and sensitivity establish cost drivers Cost [kUSD] 12 000

• classical application –

LCC-based acquisition – tender evaluation with respect to LCC

10 000 8 000 6 000 4 000 2 000

• “unlimited applicability” –

everything is user-defined

0

5 076

4 671

CI , support inv est ment s

984 487

CA , product acquisit ion

3 118

3 798

SupplierA

SupplierB

CN, recurring support cost s

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 9

CATLOC

SIMULATION TO REFLECT SYSTEM EFFECTIVENESS THE CORE OF SIMLOX

• what about system effectiveness considering – – – – – –

time-variability in operational concepts – product availability varying over time shared responsibility across a group of products – a unit different requirements for different missions – quantity-wise different requirements on product state for different modes of operation – functional block diagrams (FBD’s) platform-type products (ships) with on-board maintenance and storage capability etc.

Number of Sy s tems A v ailable Unit s : BA SE, Sy st ems : A / C

– – – –

mission success rate – ability to do mission-related readiness – ability to take on over time etc.

• “easy using simulation” –

much harder using an analytical modeling approach

50 % 40 % 30 % 20 %

48,68 %

A v ailable

10 % 0%

Total ov er all Sy s t ems Total ov er all U nits 16A C_A

Accumul ated Number of Missions

• produce KPI’s that better reflect system effectiveness

Number of Systems

60 %

A c c umulat ed Fix ed M is s ion Q ty Unit s : BA SE, M iss ions : M ISSI ON 120 % 100 %

2, 57 %

80 % 60 % 40 %

97,43 %

20 % 0%

Total ov er all M is s ions Total ov er all U nits 16A C_A

Reques ted St arted Suc c ess f ul

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 10

SIMLOX

OPUS SUITE • powerful analysis capabilities –

“place the point” thru modeling and simulation

• powerful optimization capabilities –

“generate cost-effective support” – sparing, LORA

• strategic questions/decisions – design space – – – – – – –

what products? quantity of products? what support concept (maintenance, supply)? quantity of spare components needed? what maintenance capacity required? quantity of maintenance resources needed? …

• “the strategic platform” –

can be adjusted/modified – but not on a daily, weekly, monthly basis

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 11

SUMMARY

LCM ANALYTICS

• tactical/operational questions/decisions – design space –

allocation of products, equipment, resources –

–

operations and maintenance planning – – –

–

deployed operations based on counters and intervals based on state of the products – predictive maintenance based on state in the support system

…

• what about other strategic decisions?

–

product RAMS properties (LSAR)

• maintenance program for the product?

–

balanced, cost-effective?

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 12

WHERE COULD WE GO FROM HERE?

SCHEDULED MAINTENANCE • scheduled maintenance – what does it entail?

• why?

– –

husbandry of equipment function checks

– –

to keep the equipment in shape to check the functional status of the equipment

– – –

inspections – condition checks preventive tasks service life limits

– – –

to assess the state/condition of the equipment to avoid/reduce risk of failure on the equipment to avoid/reduce risk of failure on the equipment

• scheduled maintenance always cost money –

and often also unavailability/down time

• how often?

–

interval-based

• motivation?

–

ageing effects – varying hazard rate

• when?

–

condition-based

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 13

INTRODUCTION

AGEING EFFECTS • hazard rate – hazard function –

the instantaneous risk (frequency) of failure at ‘age’ t – given survival to up to this ‘age’ hazard rate

‘risk level’ ‘age’

• intervene at a certain ‘risk level’ – –

conduct preventive task, replace component to reduce risk level – ‘push to the left’

• what ‘risk level’ shall we accept?

• assess ‘risk level’ – –

statically – statistically dynamically – from inspections or sensor-values

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 14

HAZARD RATE

•

scheduled maintenance always cost money –

•

all preventive measures must have a proven positive effect on reliability –

•

and sometimes also unavailability/down time

Reliability-Centered Maintenance

what tasks and intervals (‘risk levels’) are cost-effective?

Effective reliability

Inherent reliability

Unscheduled maintenance volumes

Availability (down time)

Scheduled maintenance volumes

Cost

Model

Maintenance program

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 15

RELIABILITY-CENTERED MAINTENANCE (RCM)

• modeling framework defined –

the mathematics in place

• input – –

inherent reliability maintenance program

• output – – –

effective reliability unscheduled and scheduled down time cost of unscheduled and scheduled maintenance

• application – – –

optimization – tune intervals (‘risk levels’) for minimum impact on availability at minimum cost screen a proposed maintenance program in terms of impact on availability and cost …

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 16

”RCM MODEL”

• failure (mode) – inherent characteristics – –

rate shape (e.g., 𝛽𝛽 of a Weibull distribution)

• link preventive task to failures (failures modes) influenced • assess effect of preventive task on specific failure (mode) – –

age or risk level reduction permanent or intermittent

• service life limits yield ‘full reset’

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 17

SOME COMMENTS ON DATA REQUIREMENTS

• ability to screen and set up cost-effective maintenance programs – – – –

sounds exciting? powerful capability? lacking capability? currently relying on OEM recommendations? formed on what basis? cost-effective?

• also applicable for predictive maintenance – –

‘risk level’ rather than ‘age’ – machine learning links sensor-values to ‘risk levels’ what indicated ‘risk levels’ shall we accept before intervening?

• analysis capability outside current scope of Opus Suite –

would you want such a tool – modeling framework productified?

• LCM analytics –

one direction we could go from here

• cost-effective products with well-balanced RAMS properties

LCM Analytics, Patrik Alfredsson, Systecon, 2024-04-16, Slide 18

CONCLUDING REMARKS

Thank you!

INSTITUTE FOR DEFENSE ANALYSES

My Life with Fat-Fingers: Tying Resources to Readiness with a Reproducible Pipeline

Benjamin Ashwell and Kyle Remley, Project Leaders OED Draft April 2024 Approved for public release: distribution is unlimited.

IDA Product ID 3002153

INSTITUTE FOR DEFENSE ANALYSES 730 East Glebe Road Alexandria, Virginia 22305

The Institute for Defense Analyses is a nonprofit corporation that operates three Federally Funded Research and Development Centers. Its mission is to answer the most challenging U.S. security and science policy questions with objective analysis, leveraging extraordinary scientific, technical, and analytic expertise.

About This Publication This work was conducted by the Institute for Defense Analyses (IDA) under contract HQ0034-19-D-0001, Task BA-9-4863, “Navy Sustainment Modeling and Analysis Support,” and Task BA-9-5320, “PEO IWS Readiness Modeling and Analysis Support,” for the Naval Sea Systems Command. The views, opinions, and findings should not be construed as representing the official position of either the Department of Defense or the sponsoring organization.

Acknowledgments The IDA Technical Review Committee was chaired by Dr. V. Bram Lillard and included Dr. Eli Alster and Dr. Megan Gelsinger from the Operational Evaluation Division.

For more information: Dr. Benjamin Ashwell, Project Leader bashwell@ida.org • (703) 845-2046 Dr. V. Bram Lillard, Director, Operational Evaluation Division vlillard@ida.org • (703) 845-2230 Copyright Notice © 2024 Institute for Defense Analyses 730 East Glebe Road, Alexandria, Virginia 22305 • (703) 845-2000 This material may be reproduced by or for the U.S. Government pursuant to the copyright license under the clause at DFARS 252.227-7013 [Feb. 2014].

INSTI T UTE FOR DEFENSE ANALYSES IDA Product ID 3002153

My Life with Fat-Fingers: Tying Resources to Readiness with a Reproducible Pipeline

Benjamin Ashwell and Kyle Remley, Project Leaders

Executive Summary High-fidelity simulations of the U.S. Navy’s sustainment system are powerful tools for understanding how resource decisions (e.g., funding, manpower) influence readiness outcomes. However, there are three major challenges to the widespread adoption of sustainment modeling: the complexity of the model input, the difficulty of obtaining key data elements in a repeatable manner, and a lack of standardization in model assumptions and structure.

In these slides, we show examples of tools, standards, and processes we have built to help address these challenges.

Second, we have developed an end-to-end data processing pipeline for modeling naval aviation. We start by curating the raw Navy data to ensure data integrity and repeatability. Then we process the curated data through our honeybee tool to generate ground truth for modeling in an HRMS format. This automatic pipeline allows us to keep our aviation models up to date with little manual intervention, and it ensures consistency and repeatability in how our model inputs are generated.

Finally, we are developing modeling guidelines and best practices to help the community of analysts standardize model assumptions and structures wherever reasonable. In general, models that serve the same purpose and that are based on the same data should make similar assumptions, and models that make similar assumptions should be structurally similar as well. This level of standardization makes collaboration and model review much more straightforward.

First, we have developed a human-readable model standard (HRMS) that stores all key modeling information in a text-based, non-proprietary format. Separating the modeling data from the specific implementation of a model simplifies the process of building and reviewing models, and the approach is future-proof against changes to specific modeling software. HRMS is also compatible with stinger, an open-source tool IDA has developed that automatically translates HRMS data into OPUS/SIMLOX input, streamlining the process of updating models and reducing the risk of fat-finger errors.

i

My Life with Fat-Fingers: Tying Resources to Readiness with a Reproducible Pipeline Dr. Benjamin Ashwell Dr. Kyle Remley

April 16, 2024

Institute for Defense Analyses 730 East Glebe Road  Alexandria, Virginia 22305

IDA is a non-profit federally funded research and development center (FFRDC), officially sponsored by OUSD A&S Institute for Defense Analyses (IDA) 730 East Glebe Road Alexandria, VA 22305 (703) 845-2000

DC

Because IDA is a non-profit FFRDC: • IDA primarily works for the U.S. government but can also work for state and local governments and non-profits • The U.S. government owns IDA’s deliverables and has unlimited use privileges • IDA has uncommon access to government data (including proprietary and competition sensitive data), beyond that of typical contractors • IDA is free from conflicts of interest

Pentagon VA

MD

IDA

1

The DOD needs tools that answer questions ranging from the tactical to the strategic More Tactical

More Strategic

• How many propellers should NAVSUP purchase?

• What combination of spares will best allow an aircraft carrier to operate in a resupply-denied environment?

• How can DLA best pre-position spares to support cross-service operations?

• How can we most efficiently boost availability by adjusting the balance of maintainer ratings/specialty codes/MOSs in a unit?

• What long-term improvements (e.g., depot throughput increase, failure rate reduction) should the JPO pursue to ensure the long-term health of the F-35 fleet?

• What readiness shortfalls are likely to exist in 202X for a given OPLAN, and what are the root causes behind those shortfalls?

2

DLA – Defense Logistics Agency; JPO – Joint Program Office; MOS – Military Occupational Specialty; NAVSUP – Naval Supply Systems Command; OPLAN – Operation Plan

There are three major challenges to the widespread adoption of sustainment modeling

Building and updating models is challenging.

Even a simple SIMLOX model can have thousands of rows of input, discouraging manual updates and increasing the risk of fat-finger errors.

This barrier to adjusting models discourages analysts from using the models to answer “what if” questions.

It is hard to generate ground truth for modeling.

Simply obtaining key data elements (e.g., configuration data, demand rates, stock levels) is challenging.

Mismatches between “authoritative” data sources must often be disentangled manually, which takes time and leads to a lack of consistency.

It is hard for a community of practice to grow without standardization.

A lack of standardization of structures (i.e., models don’t “look” the same) makes review and collaboration difficult.

A lack of standardization of assumptions (e.g., how to implement a single-echelon model for COSAL optimization) makes cross-model optimization an apples-to-oranges affair.

We have developed tools, standards, and processes to address these challenges.

COSAL – Coordinated Shipboard Allowance List

3

We have built a large-scale data processing pipeline that creates an unequaled view of the complete sustainment system

2. Curate raw data for processing

3. Process raw data into item-level metrics

4. Translate human-readable files into models

Demand Rate

1. Collect raw data

Our data workflow is a force multiplier that allows analysts to pursue a wide range of different analyses

Flight Hours

Cross-cutting data analyses

Correlative studies

End-to-end simulations

We aim to use data everyone has to generate insights no one has.

4

Building and updating models is challenging.

Even a simple SIMLOX model can have thousands of rows of input, discouraging manual updates and increasing the risk of fat-finger errors.

The Human-Readable Model Standard (HRMS) and stinger

This barrier to adjusting models discourages analysts from using the models to answer “what if” questions.

Maritime example

5

The DOD should store its “ground truth for modeling” separately from any specific model implementation

Database 1

Database 2

Automated Processing + Human Review

Specific Implementation

6

The DOD should store its “ground truth for modeling” separately from any specific model implementation

Database 1

Database 2

Automated Processing + Human Review

Human-Readable Ground Truth for Modeling

Specific Implementation

• This process breaks one complex step into two simpler steps • Analysts do not require knowledge of detailed model-isms to read or update the ground truth • A text-based format is future-proof against software changes and is compatible with version control tools

7

The human-readable model standard (HRMS), coupled with automation, lowers the barrier to entry for building good models in a repeatable manner The Navy can leverage the HRMS’s specification to automate translation between raw data sources, HRMS, and OPUS/SIMLOX models

Navy Data Environment RBD Configuration

Transport Delay

Data Reconciliation Tool

HRMS

Model Translation Tool

OPUS/ SIMLOX Model

Part Demand

• OPTEMPO

Constraints

HRMS clarifies the “constellation” of minimum information needed from the Navy’s data environment OPTEMPO – Operational Tempo; RBD – Reliability Block Diagram

•

An open, text-based standard is the definitive storage format of models HRMS is where the analyst makes changes and checks for errors in the model

8

HRMS consolidates the data needed to build OPUS/SIMLOX models into seven Excel-readable input tables Unit Data Station Data OPTEMPO Data Initial Stock Data Reliability Data Wholesale Item Properties Position Data

HRMS ~3,000 rows (VLS model) Seven tables

Model Translation Tool

OPUS/SIMLOX model ~30,000 rows (VLS model) Dozens of tables

9

HRMS – Human-Readable Model Standard; OPTEMPO – Operational Tempo; VLS – Vertical Launch System

Once a model is in the HRMS format, it is fast and easy to run hundreds of “what if?” scenarios with stinger 2 hulls, expensive spares packages, short resupply time 2 hulls, modest spares package, short resupply time Original Navy Model

HRMS Navy Model

2 hulls, cheap spares package, short resupply time Variation Instructions

⋮

~250 other permutations 30 hulls, expensive spares packages, long supply time

After initial HRMS conversion, this pipeline takes just hours to get meaningful answers to hard modeling questions.

10

HRMS – Human-Readable Model Standard

It is hard to generate ground truth for modeling.

Simply obtaining key data elements (e.g., configuration data, demand rates, stock levels) is challenging.

Reproducible Pipeline for Model Construction

Mismatches between “authoritative” data sources must often be disentangled manually, which takes time and leads to a lack of consistency.

Aviation example

11

The link between authoritative data and model inputs should be clear, and it should be automated

No matter how good an initial model is, the data within it will eventually grow stale.

If the model is to remain relevant, it must be easy to update with new data!

Additionally, a set of documented, repeatable procedures for processing raw data into model inputs improves confidence in the models.

12

One of our earliest projects was building a Super Hornet readiness model

Initial discussions with the Navy implied four major databases should provide all the parts and maintenance data needed ΔΕΧΚΠΛΑΤΕ (Ο ανδ Ι λεϖελ μαιντενανχε αχτιονσ)

ςΕΧΤΟΡ

ΛΟΓΧΕΛΛ

ΝΑςΣΥΠ δαταβασε

ΣΑΣΟΡ

ΔΛΑ ινϖεντορψ

13

In reality, there is no “one-stop-shop” for all databases in the DoD Μαριτιμε ρεθυισιτιονσ φολλοω αν εντιρελψ διφφερεντ πατη

Ναϖψ ΕΡΠ

ΔΛΑΤΣ/ΔΑ ΑΣ

(Εντερπρισε Ρεσουρχε Πλαννινγ)

Ονε Τουχη Συππλψ

ΡΣυππλψ

? Αιρχρα φτ φαιλσ

ΙΝΦΟΡΜ21

ΑΦΑΣΤ

Φινανχιαλ Τραχκινγ Δαταβασε

ΝΑςΣΥΠ Οπτιμιζεδ Σπαρινγ Ρεταιλ

Ωηολεσαλε

Isolated data sources

Ωεβ Ιντερφαχε

ΦΧΟΕ

Other databases

?

Ε2Ε

?

ΝΑςΑΙΡ δαταβασε

Ηαδοοπ ΙΔΕ

ΒΟΜ

ςΕΧΤΟΡ

(βιλλ οφ ματεριαλσ)

Δοχυμεντ μαιντενανχε αχτιον

ΛΟΓΧΕΛΛ

ΝΑςΣΥΠ Δαταβασε

Χολλατεδ Δατα φορ Ενδ Υσερ

ΔΕΧΚΠΛΑΤΕ Ο ανδ Ι λεϖελ μαιντενανχε αχτιονσ

Βυφφερ Μαναγεμεντ Τοολ (Ι Λεϖελ Ρεχορδσ)

Raw Data

ΔΛΑ Οπτιμιζεδ Σπαρινγ Ωηολεσαλε

ΔΛΑ Ρεταιλ Στορεσ

ΣΑΣΟΡ

ΔΛΑ Ωηολεσαλε Ινϖεντορψ

Need to contact organizations directly

? ΑΜΣΡΡ Πρε−Εξπενδεδ Ματεριελ Αιρχραφτ Αρμαμεντ Σψστεμσ (ε.γ., βομβ ραχκσ)

ΧΑΔ/ΠΑΔ (εξπλοσιϖεσ) ΤΨΧΟΜ Χοντρολλεδ Ινϖεντορψ

Processed Data

Even knowledge is stovepiped. Many databases are cryptic and poorly documented, and knowledge is often passed down by word of mouth.

Having the data isn’t enough; you have to be able to interpret that data.

14

A tremendous amount of information is required to populate a multi-echelon model!

Operations

Flight hours

Units perform missions Units “ready”

Deployment, testing, and training

Operational-Level Maintenance Simple maintenance with quick turnaround

Parts-level failure rates, criticality, and manpower

Retail stock

Intermediate-Level Maintenance More time-consuming and specialized maintenance

I-level repair capability (BCM fraction), I-level repair turnaround time …

Multi-indenture repair

Support equipment

Wholesale Stock delivery to “retail” sites (= bases) Stock levels

Repair, procurement, reorder, shipping times

Depot-Level Maintenance Extensive repair, scheduled upgrade, or maintenance performed off-site

Part names (NIINHOFs), base allowances, current base stock inventories, time to remove/install part …

Current wholesale stock inventory, wholesale and retail stock deliveries, transportation delays between stations …

Depot repair cycle time, procurement lead time, condemnation rate …

Government/OEM depots

Scheduled maintenance/upgrades/etc.

15

BCM – Beyond Capability of Maintenance; NIINHOF – National Item Identification Number, Head of Family; OEM – Original Equipment Manufacturer

We have developed a reproducible data management procedure that allows us to interpret sustainment data and generate our “ground truth for analysis” Additional tools then consolidate data from Navy and Air Force systems and calculate sustainment metrics by platform

Database 1 Database 2

Database 3

We start by pre-processing raw sustainment data (e.g., ILS-S) into a common, readable format Other Sources/ Expert Input

Supply System Properties: • Inventories/allowances by site • Stock posture health • Projected wholesale due-ins • Etc. Platform Properties: • System deployments • Sortie durations • Allocated sortie rates Component Properties: • Etc. • Failure rates by item by platform • Demand criticality • Local repair fraction • Maintenance turnaround times • Item indenture structure • Repair/procurement lead times • Etc.

Updating all of these metrics without automation is difficult and prone to fat-finger errors.

A well-documented, community-reviewed, repeatable workflow is key to building confidence that the model is up to date and has been fed with good information.

16

A standardized workflow makes updating existing models largely automatic

1. Collect raw data

2. Curate raw data for processing

3. Process raw data into item-level metrics

4. Translate human-readable files into models

Once the baseline model has been built for a given system, our tools make updating the underlying data (e.g., demand rates, stock levels) largely a “hands off” process.

17

It is hard for a community of practice to grow without standardization.

A lack of standardization of structures (i.e., models don’t “look” the same) makes review and collaboration difficult.

A lack of standardization of assumptions (e.g., how to implement a single-echelon model for COSAL optimization) makes cross-model optimization an apples-to-oranges affair.

Standardizing Structures and Assumptions Aviation and Maritime example

18

There should be one -- and preferably only one -- obvious way to do it.

--- The Zen of Python

1) Models that are built on the same raw data sources and that make similar assumptions should “look” similar unless there’s a good reason for them not to.

2) Models that are built on the same raw data sources, and that have the same intended use (e.g., COSAL optimization), should make similar assumptions unless there’s a good reason for them not to.

Standardization of structure makes collaboration and review easier, and standardization of assumptions makes outputs from different models easier to compare.

19

COSAL – Coordinated Shipboard Allowance List

Consistent implementation of assumptions improves the readability of models for the community of analysts Model A

Model B

Wholesale: Give wholesale 1,000 of all items, and treat reorders and repairs as effectively instantaneous

Wholesale Sends fleet orders

Transport Delay: Assume instant transport between wholesale and home port, but assume 15 days from home port to ships

Item Properties: Treat items as a mix of consumables and repairables

Wholesale: No stock; set all reorder lead times to 15 days

Hull 1

Receives wholesale stock

Home Port

Hull 2

⋯

Hull N

Transport Delay: Assume instant transport

Item Properties: Treat all items as consumables

These models make similar assumptions but they implement those assumptions very differently, making direct comparisons difficult.

20

Models built for similar purposes should, generally, be built on the same fundamental assumptions Single-Echelon Model

Multi-Echelon Model

Wholesale w/ Unlimited Stock

Air Base 1

Air Base 2

The fundamentally different assumptions built into these two models make apples-to-apples comparisons nearly impossible.

21

There are three major challenges to the widespread adoption of sustainment modeling

Building and updating models is challenging.

Even a simple SIMLOX model can have thousands of rows of input, discouraging manual updates and increasing the risk of fat-finger errors.

This barrier to adjusting models discourages analysts from using the models to answer “what if” questions.

It is hard to generate ground truth for modeling.

Simply obtaining key data elements (e.g., configuration data, demand rates, stock levels) is challenging.

Mismatches between “authoritative” data sources must often be disentangled manually, which takes time and leads to a lack of consistency.

It is hard for a community of practice to grow without standardization.

A lack of standardization of structures (i.e., models don’t “look” the same) makes review and collaboration difficult.

A lack of standardization of assumptions (e.g., how to implement a single-echelon model for COSAL optimization) makes cross-model optimization an apples-to-oranges affair.

22

Backup Slides

23

We verified that the HRMS format retained fidelity to the predicted Ao from the original Navy model

Original Navy OPUS/SIMLOX Model

Reversestinger

HRMS

stinger

Updated OPUS/SIMLOX Model

We verified the HRMS files by ensuring that the predicted Ao for both models is within the expected deviation inherent to discrete event simulations.

Ao – Operational Availability; HRMS – Human-Readable Model Standard

24

Form Approved OMB No. 0704-0188

REPORT DOCUMENTATION PAGE

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

2. REPORT TYPE

xx-03-2024

Final

3. DATES COVERED (From - To)

4. TITLE AND SUBTITLE

5a. CONTRACT NUMBER

My Life With Fat-Fingers: Tying Resources to Readiness with a Reproducible Pipeline

HQ0034-19-D-0001

5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S)

5d. PROJECT NUMBER

Benjamin A Ashwell

BA-9- 4863; BA-9-5320 5e. TASK NUMBER 4863; 5320

5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

8. PERFORMING ORGANIZATION REPORT NUMBER

Institute for Defense Analyses 730 East Glebe Road Alexandria, Virginia 22305

3002153

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

10. SPONSOR/MONITOR'S ACRONYM(S)

Robert Stukes PEO IWS Naval Sea Systems Command WNY Bldg 197 4E-1140 Washington, DC

NAVSEA

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release: distribution is unlimited.

13. SUPPLEMENTARY NOTES Project Leader: Benjamin A Ashwell 14. ABSTRACT

High-fidelity simulations of the sustainment system are powerful tools for understanding how resource decisions (e.g., funding, manpower) influence readiness outcomes. However, there are three major challenges to the widespread adoption of sustainment modeling: the complexity of the model input, the difficulty of obtaining key data elements in a repeatable manner, and a lack of standardization in model assumptions and structure. In these slides, we show examples of how we have built tools, standards, and processes that help to address these challenges.

15. SUBJECT TERMS Sustainment, readiness, resource-to-readiness, OPUS, SIMLOX, Systecon

16. SECURITY CLASSIFICATION OF: a. REPORT

b. ABSTRACT

c. THIS PAGE

Unclassified

Unclassified

Unclassified

17. LIMITATION OF ABSTRACT

SAR

18. NUMBER 19a. NAME OF RESPONSIBLE PERSON OF Benjamin A Ashwell PAGES 19b. TELEPHONE NUMBER (Include area code) TBD

703-845-2046 Standard Form 298 (Rev. 8/98) Prescribed by ANSI Std. Z39.18

4/16/2024 Washington D.C

New capabilities in Opus Suite Gustaf Solveling, Ph.D. - Systecon North America Predictive Logistics Summit 2024

Agenda 1 2 3 4

Introduction Capability Results SIMLOX Result Viewer Usability Improvements

Introduction Earlier versions February 2023:

Opus Suite Version 2023.0

New versions September 2023: Opus Suite Version 2023.1 February 2024: Opus Suite Version 2024.0 Future versions Fall 2024:

Opus Suite Version 2024.1

Opus Suite Conference 2023

April 2023

Predictive Logistics Summit 2024

April 2024

Capability Operational Modes What?

• New type of results in SIMLOX – Capability results • Extended input data model to capture capabilities in the simulation • Report to show capability results • Result graphs to show capability results

Figure 1: 220930-N-NO820-1034 by Ensign Andrew J. Miya shiro, 2022, U.S. Na vy Photo Ga llery. Retrieved from https://www.na vy.mil/Resources/PhotoGa llery/igphoto/2003089813/ Copyright 2023 by U.S. Na vy.

Capability Operational Modes Why? Currently, SIMLOX: •

Captures the state for each system

•

Shows mission capability for the missions performed

•

Only allows collection of results due to your actual operations

Requested feature: •

Understand what capability the systems and subsystems have for other types of missions

•

Separate the mission operations from mission capabilities Figure 1: 220930-N-NO820-1034 by Ensign Andrew J. Miya shiro, 2022, U.S. Na vy Photo Ga llery. Retrieved from https://www.na vy.mil/Resources/PhotoGa llery/igphoto/2003089813/ Copyright 2023 by U.S. Na vy.

Capability Operational Modes Live Demo • Ship example • Missions • Cruise • Engage • Capabilities • Air defense • Surface defense • Missile defense • …

SIMLOX Result Viewer Application to present SIMLOX Results • Modern technology stack • Customizable • Sales/decision makers in addition to analysts.

Usability Improvements

Coming soon…

Additional features

New in version 2024.1

• Enhanced performance for LORA-XT problems

• OPUS10 Result Viewer

• Filling gaps on the C/E-curve • Filter functions in input tables • Generate items

• Downtime drivers • …

Questions?

Thank you.

UNCLASSIFIED

OPUS Suite Utilization in Maritime Readiness Based Sparing Presented to:

Systecon OPUS Conference Presented by:

Justin Hietsch: NAVSEALOGCEN Reliability Engineering Division POC: Riley Wingert riley.p.wingert.civ@us.navy.mil 16 April 2024 UNCLASSIFIED

Agenda

• Readiness Based Sparing (RBS)

– Policy and Objectives ACCESS LEVELS – Navy Common Readiness Model (NCRM) – RBS System Modeling – Modeling Data – Supply/Provisioning Data – NAVSEALOGCEN OPUS Suite Usage – MARRC-RB – Commonly Encountered Issues

2

RBS Policy and Objectives OPNAVINST 4442.5B

• Provide the range, depth, and location of spares:

‒ Support required readiness objectives at least cost ‒ Given the reliability and maintainability characteristics of a system/equipment

• “Operational Availability (Ao) is the primary measure of material readiness for weapon systems and equipment“ ‒ ‒ ‒ ‒

Historical and/or predicted failure information Many simulated deployments Repair time and logistics delay Models complex situations with multiple variables

OPNAVINST 3000.12 B

• Ao is defined as: UPTIME UPTIME + DOWNTIME

=

MTBF .

MTBF + MTTR + MLDT (Design-Driven)

(Support-Driven)

3

Navy Common Readiness Model (NCRM)

• A decision support capability to optimize resources ($) and readiness • A maintenance, supply, and manpower resource optimization model to dynamically meet mission readiness requirements • Will be developed and utilized by Chief of Naval Operations (OPNAV), System Commands (SYSCOMs), and the Fleet ‒ Establish early program performance requirements ‒ Analyze Ship/Submarine design for readiness during the Systems Engineering Process ‒ Establish readiness budgets for Systems/Ships/Strike Groups/Fleet

4

Navy Common Readiness Model (NCRM) cont.

• Model will reflect the requirements and detailed design (LRUs, redundancy, criticality, tie to mission, etc.) for all Systems, Ships, and Submarines • Planned to leverage COTS software, including OPUS Suite and Windchill Risk & Reliability

5

RBS System Modeling • Critical list of systems selected by community ‒ Redline systems ‒ New acquisition systems ‒ Troubled systems

• Prioritized by program

‒ System criticality ‒ Data availability and accuracy ‒ Readiness impacts

6

RBS Modeling Data • In Service Engineering Agent (ISEA) coordination for validation ‒ Design Reference Mission (DRM) ‒ Operational profile ‒ Reliability Block Diagram (RBD) ‒ System boundaries ‒ Provisioning data ‒ RBS candidates ‒ RBS results and part selection

The foundation for valid model results is proper Provisioning.

7

Supply/Provisioning Data • Part Data:

‒ Part number / National Stock Number ‒ Quantity per Assembly (QPA) ‒ Failure rate ‒ Part Military Essentiality Code (MEC) ‒ Unit price ‒ Additional part descriptors

• Configuration/LRU data mapped to RBD • Default Logistical Delay Times

8

NAVSEALOGCEN OPUS Suite Usage • Only use OPUS10 v2023 and SIMLOX v2023 • Typical Model Constraints used include:

‒ Scenario Type: Steady-State ‒ Spares Problem Type: Either Initial or Replenishment ‒ Aggregation Type: Hardware and Subsystem ‒ Task Type: Use only Replace ‒ Lead Time: No delay at depot upon ship requisition request

• Single variant modeling: one ship configuration used as baseline after reviewing available configuration data • 15 day default used for logistics delay time • 2 hour default used for part retrieval and MTTR • Mission profile can be dependent on ship class (e.g.

Littoral Combat Ships) or the system being modeled • Single maintenance level 9

MARRC-RB

NAVSEAINST 4441.12A/ NAVSUP 4441.37A

• MARRC-RB

‒ Maritime Allowance Readiness Based Sparing Reprovisioning and Change Review Board ‒ Joint NAVSEA/NAVSUP instruction ‒ Purpose to enforce the Maritime Allowance Policy ‒ Review all RBS packages, ACRs, and AERs • Reviews and approves all maritime RBS packages ‒ Packages can be submitted both internally and externally ‒ Can return packages to the modeler for correction

10

MARRC-RB cont.

• MARRC-RB Review and Approval Process

‒ Model is submitted via the MARRC-RB Repository ‒ Model is assigned a validator to review model for adherence to RBS policy and accuracy ‒ Validator reports back to submitter of any required changes or notifies MARRC-RB Gatekeeper model meets requirements ‒ Gatekeeper confirms funding requirements and submits RBS packages to MARRC-RB co-chairs ‒ Co-chairs ensure all instructions have been adhered to; Signing or sending back all reviewed packages ‒ Approved allowance overrides can be loaded to the SAF using unique ship/system/part information 11

Endurance Supply (ES) • OPNAVINST 4442.5B amendment (NAVADMIN 06/23) adds Es ‒ New readiness threshold ‒ ES measures the days a system remains fully operational without an off-ship requisition ‒ Target 85% confidence level that a system remains available for 90 days without replenishment ‒ ES can only be used for the systems listed in the amendment

12

In Review • RBS policy instruction overview • NCRM and data sourcing/input • OPUS usage and assumptions • Model results review via MARRC-RB • Policy and OPUS modeling work together to ensure accurate, reliable results

14

Questions?

15

Systecon Predictive Modeling and Big Data 2024

01 History

Content

02 Current Challenges

03 Way Forward

Size Silos

Terms & Definitions Ownership

Future Data

Management

History • “Big Data” has been around in some form since

early man • The term “Big Data” began to appear in the 1990’s • Current form of “Big Data” has been around for

about a decade • Data collection has always been a part of logistics

management • With the introduction of scalable sensors Big Data is Ishango Bone circa 18,000 BCE discovered in 1960. Which may be an early method used to track trading activity or supplies.

going to get bigger

This Panel was designed in the 1970’s and utilized in the 1980’s onward. Able to perform on-going BIT functions to monitor the major systems of an aircraft, as well as to provide output feedback on a select number of recordable characteristics.

Programs have to ask (1) what data should be collected (2) who can access to it?

Current Challenges SIZE

SILOS

Collection

Who owns the Data

Storage

Who is allowed to access the Data

Manipulate (sort, transform, …)

FUTURE DATA The Unknown Overcoming the Signal to Noise Ratio ALIS (Autonomic Logistics Information System) Each unit is over 200lbs, incorporating operations, maintenance, prognostics, supply chain, customer support services, training and technical data.

Current Challenges - Size Metrics: What kind of results are you looking for?

Data: What is important in the model?

Complexity: How many systems are you trying to model?

• These questions determine how big your model will be • To obtain relevant data, may have to merge data from multiple sources • As you build your model you may end up with hundreds of thousands of lines of input data, requiring large processing power to compute results

Current Challenges - Silos Considerations: •

Data repository management and intended use

•

Control and timely access to data is crucial to modeling

•

Understanding data requirements, rights, and management for contracting

Current Challenges - Future The Unknown •

With sensors technology constantly

◦ It’s smart : must have context

improving the type and amount of data

◦ It’s clean : be free of outside

collected will only continue to grow •

3 Qualities of Good Data

Leads to “Change for Change Sake”

Increase “signal to noise ratio” •

Signal – important useful data

•

Noise – distracting information

signals ◦ It’s purposeful : must be accurate and complete Scott Thomson, The Dangers of Too Much Data

Way Forward Terms and Definitions Don’t Re-invent the Wheel •

Existing Standards MIL-HDBK-260

There is a tendency with new means/methods

Reference Only

to rethink and overthink how we approach new

Combines Definitions of Maintenance

concepts that haven’t been proven.

• IF NEW techniques and PROCEDURES are developed, these MUST be proven and AGREED to prior to implementation.

Metrics from the DOD DEC 2001 Maintenance Metrics U.S. Air Force Air Force specific document Definitions and Metrics to consider

Way Forward Data Ownership PROS Owner becomes GATEKEEPER Open access for use CONS High cost Data Centers Data Managers Increased risk of SILOING

Way Forward FUNCTIONS

New career field Growing in importance Must understand modeling/simulation Must have authority to operate and make decisions

̶

̶

̶

̶

̶

̶

̶

DATA MANAGERS

1960 Data Science termed 2008 Data Scientist becomes mainstream Early 2010’s Data Manager become a career

̶

̶

̶

̶

̶

Develop a data-oriented system Create and enforce policies on data management Create techniques for quality data collection Implement secure and efficient procedures Assist in data related tasks

Thank you.

U.S. Naval Aviation Predictive Analytics

Trigger to Production

Dr. Roy Lancaster Naval Air Systems Command, Sustainment Group NAE Chief Analytics & Data Officer

“Data will talk to you if you’re willing to listen.” — Jim Bergeson NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited.

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited.

Predictive Analytics •Predictive analytics is a form of advanced analytics that examines data or content to answer the question, "what will happen?" or, more precisely, "what is likely to happen?" Predictive analytics is characterized by techniques such as regression analysis, cluster analysis, multivariate statistics, pattern matching, predictive modeling and forecasting.

•Predictive analytics allow business users to focus on execution, rather than manually mine data. Today, there are many successful examples of the use of predictive analytics, spanning demand forecasting, supplier risk monitoring, price variances and spend category opportunity assessment

Only 37% of Predictive Analytics projects making it into production and 15% are producing measurable benefits

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

3

Governance NAAC Analysis Software Analytical Talent

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

4

Governance: “Broadly Control NAE Reporting to the North Stars The purpose of NAAC is to establish the structure, governance, responsibilities, and authorities of NAE Analytics groups and to work as an unified team to strategically build analytical products that provide authoritative answers to analytical North Star questions.

• The NAAC will incorporate analytics groups into a single consortium to: •

Optimally develop and refine analytical products with metrics that effectively analyze Capability, Affordability, Availability, Safety, Training

•

Establish an enterprise Center for Analytics (CfA)

•

Advance naval aviation’s data science and advanced analytics skills across naval aviation analysts

•

Establish and maintain a centrally reporting data environment

•

Prioritize gaps in analytical products to maximize desired readiness and cost outcomes

•

Endorse and govern naval aviation analytical metrics – creating standardization

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

5

Enterprise Analytics Naval Aviation Analytics Consortium

Naval Aviation Center for Analytics Summer 2024 – NAVAIR HQ

The Naval Aviation Enterprise chartered the (NAAC) in August 2019 as an authoritative analytics governance organization to align the currently independent analytics efforts across the enterprise and deliver authoritative, data-driven information for critical questions The NAAC/CADO is designed as the governing body to “broadly control NAE reporting to the North Stars” • • • • •

Center for Analytics Official Publication of Metrics Live Governed Dashboards Eliminate Waste Ensure every action is driving desired outcomes

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

6

Readiness / Cost Impact

Prioritization Framework Do First

Do Last

Feasibility – Likelihood to Deploy NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

7

Multidisciplinar y Investigation

William S. Cleveland (2001) coined the term data science in the context of enlarging the major areas of technical work in the field of statistics. Cleveland described a major altering of the analysis occupation to the point a new field shall emerge and will be called “data science”

Models & Methods

Theory

Data Sciences Tool Evaluation

Computing with Data

Pedagogy

Big Data, Data Science & the U.S. Department of Defense

Data Science: Observed Limitations NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

8

Excel on both domains

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

9

Business Decision Data Life Cycle v1.1 NAVAIR/COMFRC Wayne Jiang - 2024

3 •

2 • •

1 • •

Software Engineer

Develop software applications to capture business transactions

IT Architect

Transaction Data Domain

Develop IT (HW/SW) requirements to capture business transactions Balance IT portfolio

Business Leader

Make business decisions Sponsor operations research studies to inform business decisions

Business Domain

Analysis Domain

• •

Data Engineer

Curate and consolidate data Enable automated data flow to data consumers

5

• •

6

• •

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited.

4

Business Analyst

Translate business data gaps into actionable data calls Synthesize study results to inform business COAs

<UNCLASSIFIED>

Data Scientist

Develop and conduct studies using advanced models and simulation to predict business outcomes Provide scientific evidence for data relationships and identify primary effect drivers OPUS

10

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited.

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited.

Production Options • Determine whether our analytics capabilities will be delivered by the procurement platform, an external analytics platform, or a combination of both.

• Build or Buy, Service oriented architecture?

• Predictive software options?

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

13

14

Legacy Applications

Example Pipeline?

Time to get better !

OOMA/ NALCOMIS SHARP

OPUS

ASM FLTMPS ONETOUCH

…

Data Engineers

Data Stewards Time Spent on Manual Recurring Tasks

Vis, Modeling, AI/ML Product & Service Teams

100% 0%

DevSecOps Software Team

Self Service

NAVAIR Public Release SPR# 2024-0222. Distribution Statement A Approved for public release; Distribution is unlimited. <UNCLASSIFIED>

AI/ML Powered, Agile Software Applications 14

Thank you.