Presentation for transportation forum dr xu liang hdfr

2017 Transportation Forum

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation during Severe Weather Events by Fusing and Assimilating Diverse Data Sources Xu Liang Department of Civil and Environmental Engineering University of Pittsburgh

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Acknowledgement Felipe Hernรกndez and Daniel Luna (PITT) Yao Liang and Rui Wang (IUPUI) William Teng (NASA/ADNET) Jason Norville, James Long, Matthew Delp, Charles Martin, Vince Mazzocchi, and Nevin Myers (PennDOT) 2

zeroto60mph.com

nce.co.uk

topspeed.com

Hydrologic threats to the transportation infrastructure

nydailynews.com

3

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Real-time hydrologic disaster forecasting and response for severe weather events Automated process for forecasting hydrologic disasters and proposing response procedures:

Assimilate data

Fuse data

Run hydrologic model to compute critical variables

Assess the severity of the estimated events

Identify potentially affected areas

Retrieve real-time hydrologic data and weather forecasts

4

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

How to monitor all the threatened bridges at present?

Pennsylvania’s scour-critical bridges Category A Category B Category C

3448 watersheds

5

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

PennDOT’s policy for monitoring threatened bridges

• Tr < 10 years: no bridges require inspection • 10 years ≤ Tr < 50 years: inspect cat. A bridges • 50 years ≤ Tr < 100 years: inspect cat. A and B bridges • 100 years ≤ Tr: inspect cat.

A, B and C bridges

2

50 year return period

24-hour accumulated rainfall2 (mm)

Match bridges’ SCBI category with the return period of the storm:

10 year return period

100 year return period

NOAA’s Precipitation Frequency Data Server: http://dipper.nws.noaa.gov/hdsc/pfds/

6

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Current practice at PennDOT for determining post-storm bridge inspections

1

erh.noaa.gov, water.weather.gov

Gather precipitation data from the NWS for the previous 24/48 hours

7

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Current practice at PennDOT for determining post-storm bridge inspections

2 Compute the average precipitation on the watershed/drainage area corresponding to each bridge

8

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Current practice at PennDOT for determining post-storm bridge inspections

3 Manually determine the most severe duration of the storm from rain gauges nearby

9

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

4

Determine which region each watershed belongs to and estimate the return period with the corresponding IDF curve

Rain intensity

Current practice at PennDOT for determining post-storm bridge inspections 100 y

50 y Mean intensity

Rain return period (T)

25 y 10 y

Duration

Rain duration

10

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Current practice at PennDOT for determining post-storm bridge inspections

Compare return periods with category thresholds to determine which bridges require inspection

5 11

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Hydrologic Disaster Forecast and Response (HDFR) system • … automatically retrieve, process, and fuse data from heterogeneous/diverse data sources in near real-time • … automatically compute critical variables to access and forecast hydrologic disasters • … significantly reduce forecast uncertainties through state-of-the-art parameter calibration and data assimilation algorithms using modern hydrological models

• … achieve near real-time forecasts for hazards management and responses

12

Data

Fuser

Modeling

Assimilation

Severity

Components of the Hydrologic Disaster Forecast and Response (HDFR) system

13

Data

Fuser

Modeling

Assimilation

Severity

Data modules Status: 12 completed, 2 pending • NASA NLDAS-2 (gridded land observations) and TRMM (precipitation) • NOAA RFC (GRASS) multi-sensor precipitation • NOAA MADIS METAR, HYDRO: meteorological stations • NOAA NWS GFS (GRASS), NAM: global and continental forecasts • USGS hydrometeorology • NASA MODIS (GRASS): satellite snow cover • NOAA SNODAS (GRASS): snow cover and snow water equivalent • NASA LPRM: satellite soil moisture • NASA GPM (satellite precipitation) and SMAP (satellite soil moisture) • NASA SSW (multiple) 14

Data

Fuser

Modeling

Assimilation

Severity

NOAA RFC radar precipitation: Completed

1 hour accumulated rainfall (mm)

This data source includes quality-verified gauge-validated radar precipitation estimates •

Coverage: Most of the United States’ territory

•

4 km HRAP grid, hourly, 1 hour latency

•

Variables: precipitation only

15

Data

Fuser

Modeling

Assimilation

Severity

NWS Global Forecast System: Completed

Surface temperature 1/21/15 (k)

This data source includes gridded hydrometeorological forecasts computed using numerical models •

Coverage: Entire planet

•

0.25°, 0.5° and 1° grid, 3-hourly, 240-hour forecast every 6 hours

•

Variables: 358, including values for the precipitation, soil, planet surface, and several layers of the atmosphere

16

Data

Fuser

Modeling

Assimilation

Severity

USGS hydrometeorology: Completed

This data source includes measurements from USGS rain gauge and river monitoring stations •

Coverage: 1.5 million sites throughout the United States

•

Point data on select locations, temporal resolution and latency varies, typically 1 hour for both

•

Variables: primarily runoff, gage height, and precipitation

17

Data

Fuser

Modeling

Assimilation

Severity

NASA MODIS: Completed

This data source contains snow cover estimates based on the MODIS instrument on the Terra and Aqua satellites •

Coverage: Global

•

0.05° grid, daily, 1 day latency

•

Variables: snow cover and albedo

18

Data

Fuser

Modeling

Assimilation

Severity

NOAA SNODAS: Completed

Assimilates satellite, airborne, and ground observations into a model to provide combined estimates •

Coverage: Contiguous United States

•

30 arc-sec grid, daily, 1 day latency

•

8 variables: including snow water equivalent, depth, and temperature

19

Data

Fuser

Modeling

Assimilation

Severity

GPM: Completed

This data source contains precipitation estimates based on observations from a satellite constellation •

Coverage: Global (60°S - 60°N)

•

0.1° grid, 30 minute, 6 hour latency

•

Variables: precipitation

20

Data

Fuser

Modeling

Assimilation

Severity

Integration with GRASS GIS GRASS GIS

PyGrass

C Library (compiled)

Python Library

New Python module

21

Data

Fuser

Modeling

Assimilation

Severity

Multiscale Kalman Smoother based framework*

1

Initialization

// Compute total mean scaleSum ← 0 scaleCount ← 0 foreach scale in scales do if scale has observations then mean ← compute observations’ mean scaleSum ← scaleSum + mean end if end foreach totalMean ← scaleSum/scaleCount // Initialize values foreach scale in scales do assign values for P, Q, and R scale.fallibility ← get initial fallibility foreach cell in scale do cell.value ← totalMean end foreach end foreach

3

Downward sweep

2

Upward sweep

grid ← get grid with finest scale foreach cell in grid do fuse observations (Eq. 4, 6) end foreach grid ← get next coarser scale repeat newFall ← propagate fallibility (Eq. 1) foreach cell in scale do propagate from children (Eq. 2) fuse observations (Eq. 4, 6) end foreach grid.fallibility ← newFall grid ← get next coarser scale until scale doesn’t exist

scale ← get coarsest scale repeat foreach cell in scale do smooth from parent (Eq. 3) end foreach scale.fallibility ← update fallibility (Eq. 5) scale ← get next finer scale until scale doesn’t exist

*

Parada, L. M. & Liang, X. (2004), “Optimal multiscale Kalman filter for assimilation of near-surface soil moisture into land surface models,” Journal of Geophysical Research, AGU, 109(D24).

22

Data

Fuser

Modeling

Assimilation

Severity

MKS fuser improvements • 2 datasets with different spatial resolutions and different projections • Arbitrary target projection • Presence of “no-data” values (extensible to point measurements) • Spatiotemporal interpolation module

• Removal of the database dependency • Integration with GRASS 23

Data

Fuser

Modeling

Assimilation

Severity

Precipitation fusion NLDAS-2

NWS precipitation

24-hour accumulated rainfall 1/10/10 8:00 (mm)

24

Data

Fuser

Modeling

Assimilation

Severity

Components of the Hydrologic Disaster Forecast and Response (HDFR) system

25

Data

Fuser

Modeling

Assimilation

Severity

Juniata watershed â&#x20AC;&#x201C; storm 1 20 18

Return Period (years)

16 14 12 10 8 6 4 2 0 9/4/04

9/14/04 TQ Measured

TP - Watershed

9/24/04 TP - HDFR

10/4/04 TP - PennDOT

26

Data

Fuser

Modeling

Assimilation

Severity

West Branch Susquehanna watershed â&#x20AC;&#x201C; storm 1 35

Return Period (years)

30 25 20 15 10 5 0 9/4/04

9/14/04 TQ Measured

TP - Watershed

9/24/04 TP - HDFR

10/4/04 TP - PennDOT

27

Data

Fuser

Modeling

Assimilation

Severity

Juniata watershed â&#x20AC;&#x201C; storm 2 5.0 4.5

Return Period (years)

4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 2/26/11

3/8/11 TQ Measured

TP - Watershed

3/18/11 TP - HDFR

TP - PennDOT

3/28/11

28

Data

Fuser

Modeling

Assimilation

Severity

West Branch Susquehanna watershed â&#x20AC;&#x201C; storm 2 3.5

Return Period (years)

3.0 2.5 2.0 1.5 1.0 0.5

0.0 2/26/11

3/8/11

TQ Measured

TP - Watershed

3/18/11

TP - HDFR

3/28/11

TP - PennDOT

29

Data

Fuser

Modeling

Assimilation

Severity

Watersheds of scour-critical bridges Pennsylvaniaâ&#x20AC;&#x2122;s scour-critical bridges Category A

Category B Category C

3448 watersheds

30

Data

Fuser

Modeling

Assimilation

Severity

Distribution of watershed size 10000

Small-scale modeling

1000

Area (km2)

100

10

1

Large-scale modeling

0.1

0.01

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percentage of watersheds

31

Data

Fuser

Modeling

Assimilation

Severity

Distributed Hydrology Soil Vegetation Model Bug fixes: • Misplaced initial state on previous time step • Re-initialization of stream channel storage • Absence of water table calculations in the initial time step • …

Lettenmaier, Dennis et al., “Distributed Hydrology Soil Vegetation Model,” University of Washington, Pacific Northwest National Laboratory, available online at: http://www.hydro.washington.edu/Lettenmaier/Models/DHSVM/

32

Data

Fuser

Modeling

Assimilation

Severity

Indiantown Run watershed

Calibration DHSVM model 100 24 1,472 4 21 33,455 1

m resolution variables/cell cells soil layers reaches state variables h time step

Sensitivity analysis: 7 parameters per soil type 3 parameters for forest cells Manningâ&#x20AC;&#x2122;s n for streams 22 months (+ 10 months spin-up) Multi-objective: NSE, MARE, bias Ensemble optimization: GA, CMA-ES, MetroACO, GD 33 2531 evaluations

Data

Fuser

Modeling

Assimilation

Severity

Calibrated model performance 100000

0

10

10000

20

1000

40

50

100

60

Precipitation (mm/h)

Discharge (l/s)

30

70

10

80

90

1

100

Measured

Modeled

Precipitation

34

Data

Fuser

Modeling

Assimilation

Severity

Distribution of watershed size 10000

Large/robust bridges

Small-scale modeling

1000

Area (km2)

100

10

High threat!

Low threat

1

Large-scale modeling

Small/vulnerable bridges

0.1

0.01

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percentage of watersheds

35

Data

Fuser

Modeling

Assimilation

Severity

DHSVM-GRASS integration Elevation

Stream definition

Stream width estimation

36

Data

Fuser

Modeling

Assimilation

Severity

OPTIMISTS (Optimized PareTo Inverse Modeling through Integrated Stochastic Search)

Root samples

Front 3

Observations

Simulation

Front 2

Error

Streamflow

Random samples

Assimilation time step

Front 1 (Pareto front)

Time

Initial distribution of state variables

Likelihood

Target distribution of state variables

Evolutionary optimization

HernĂĄndez, F. and Liang, X.: Hybridizing sequential and variational data assimilation for robust high-resolution hydrologic forecasting, Hydrol. Earth Syst. Sci. Discuss., (September), 1â&#x20AC;&#x201C;25, doi:10.5194/hess-2016-454, 2016.

37

Data

Fuser

Modeling

Assimilation

Severity

OPTIMISTS performance â&#x20AC;&#x201C; Indiantown Run (DHSVM) Scenario 2 Scenario 1 Scenario 2 Assimi. period Forecast period 0

Streamflow (l/s)

10000

1000

0

10000 10

10

20

20

30

30

1000 40

40

50

50

60

60

100 70

70

100

0.022 / -0.003 / 54.3% 0.301 / 0.394 / 41.3% 10

80

-0.428 / -0.733 / 58.3%

90

0.451 / 0.212 / 23.2%

100 10

Observations

Default model

Precipitation (mm/h)

Scenario 1 Assimi. period Forecast period

80 90 100

OPTIMISTS

Precipitation

38

Data

Fuser

Modeling

Assimilation

Severity

Components of the Hydrologic Disaster Forecast and Response (HDFR) system

39

Data

Fuser

Modeling

Assimilation

Severity

Storm severity module â&#x20AC;&#x201C; single duration

PennDOTâ&#x20AC;&#x2122;s IDF Maps

Precipitation [mm]

Return Period Time Series [years]

40

Data

Fuser

Modeling

Assimilation

Severity

Storm severity module â&#x20AC;&#x201C; multi-duration Duration 1

Maximum Return Period [years]

Duration 2

Associated Aggregation [Dt]

Time Stamp 41

Data

Fuser

Modeling

Assimilation

Severity

Severity curves for other variables Lackawanna River at Old Forge, PA 25000

20000

Pearson Type III

Streamflow (cfs)

Gumbel 2 years

15000

5 years

10 years 25 years 50 years

10000

100 years 200 years 500 years 1000 years

5000

0 1

Log-Normal

10

Duration (days)

100

42

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Screenshots of the HDFR under GRASS. Left: the HDFR module menu.

Right: examples of data sets either downloaded using the data modules or computed using analysis modules

43

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Severity calculation using HDFR where precipitation information is downloaded, fused, and the return period is estimated. This precipitation event shows a return period of 45 years

44

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Severity calculation using HDFR with forecasted precipitation GFS NAM

MKS Forecast: 3:00-9:00 am, March 25th, 2017

Severity RETURN PERIOD

DURATION

TIME STAMP

45

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Severity calculation using HDFR with forecasted precipitation (video)

46

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Pending activities • SMAP data at the SSW; module completion • LPRM soil-moisture module development

• Testing and integration of data modules into GRASS • Testing of OPTIMISTS on high-resolution models

• Improvements to the fast flood severity estimation module

47

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Acknowledgement This work is supported in part by US DOT under the Award # OASRTRS-14H-PIT, and in part by PennDOT under the Award PITT WO #009 Disclaimer: The views, opinions, findings and conclusions reflected in this presentation are solely those of the authors and do not represent the official policy or position of the USDOT/OST-R, or any State or other entity. USDOT/OST-R does not endorse any third party products or services that may be included in this presentation or associated materials.

48

Improving Hydrologic Disaster Forecasting and Response (HDFR) for Transportation by Fusing and Assimilating Diverse Data Sources

Thank You! Any Questions?

49