Grid-Stat: Verifying Probability Forecasts

In this section, we'll use Grid-Stat to verify a probability of precipitation forecast using a gridded precipitation analysis field. All of the methods for verifying probabilities are also implemented in the Point-Stat tool. To verify probabilities in MET, you choose a set of probability thresholds between 0 and 1 to apply to your probabilistic forecast. You then choose a single threshold to apply to your deterministic observation field. These thresholds are used to define an Nx2 contingency table (as opposed to the traditional 2x2 table), from which several statistics are derived.

Start by making a copy of a Grid-Stat configuration file for probabilities:

cp scripts/config/GridStatConfig_POP_12 tutorial/config/GridStatConfig_tutorial_POP_12

Open up the file tutorial/config/GridStatConfig_tutorial_POP_12 and edit it as follows:

• Keep these set as fcst_field[] = [ "POP/L0/PROB" ]; and obs_field[] = [ "APCP_12(*,*)" ];
  To select a probability of precipitation forecast and the 12-hour accumulated precipitation observation.
• Set fcst_thresh[] = [ "ge0.00 ge0.25 ge0.50 ge0.75 ge1.0" ];
  To define the probability thresholds between 0 and 1. Note, these must be set as greater-than-or-equal-to.
• Set obs_thresh[] = [ "gt0.0 ge2.54" ];
  To define thresholds for accumulated rain as strictly non-zero or >= 2.54 mm (0.1 inch).
• Keep this set as output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 0, 0, 0, 1 ];
  To enable the output of the probability lines types, PCT, PSTD, PJC, and PRC.

Next, run Grid-Stat to verify the probabilities by executing the following command:

bin/grid_stat \
data/sample_fcst/2005080312/pop5km_2005080312F096.grib_G212 \
out/pcp_combine/sample_obs_2005080800V_12A.nc \
tutorial/config/GridStatConfig_tutorial_POP_12 \
-outdir tutorial/out/grid_stat -v 2

This command creates six output files in the tutorial/out/grid_stat output directory, 1 STAT file, 1 NetCDF matched pairs file, and 4 ASCII files - one for each output line type. Execute the following command to view the NetCDF matched pairs file:

ncview tutorial/out/grid_stat/grid_stat_POP_12_1080000L_20050808_000000V_pairs.nc&

Click through the variables to see the values of the forecast and observation fields for each masking region.

One of the probabilistic line types generated is the Probability Contingency Table (PCT) line, which contains the counts for the Nx2 contingency table. We'll now run 2 Stat-Analysis jobs to first, aggregate together the Nx2 contingency table counts across the two masking regions, and second, to derive statistics from the counts (PSTD). Execute the following Stat-Analysis jobs:

bin/stat_analysis \
-lookin tutorial/out/grid_stat/grid_stat_POP_12_1080000L_20050808_000000V.stat -v 2 \
-job aggregate -line_type PCT \
-vx_mask DTC165 -vx_mask DTC166 -obs_thresh ">0.000"

bin/stat_analysis \
-lookin tutorial/out/grid_stat/grid_stat_POP_12_1080000L_20050808_000000V.stat -v 2 \
-job aggregate_stat -line_type PCT -out_line_type PSTD \
-vx_mask DTC165 -vx_mask DTC166 -obs_thresh ">0.000"

Next, run the same jobs listed above but use the higher threshold -obs_thresh ">=2.540". Does the Brier Score get better or worse (0 is a perfect Brier Score)? Does the forecast perform better for the threshold of >0.000 or >=2.540?