Testing and Evaluation

Testing and evaluation undertaken by the developers of new NWP techniques from the research community is generally focused on case studies. Extensive testing and evaluation must be performed to ensure that these new techniques are indeed ready for operational consideration. Testing and evaluation by the DTC generally focuses on extended retrospective time periods. The cases selected for these retrospective tests encapsulate a broad range of weather regimes ranging from null, to weak and strong events. The exact periods chosen vary based on the type of phenomenon that is the focus of the test. For some test activities, these cases will be chosen from all four seasons (e.g., extra-tropical for general predictions), whereas for others the cases will come from a particular season (e.g., hurricane season, convective season). Before testing begins, a design document is created in consultation with the developers, relevant area experts, and verification experts, detailing the specifications of the testing to be done. The DTC's evaluation of these retrospective forecasts includes standard verification techniques, as well as new verification techniques when appropriate. All verification statistics also undergo a statistical significance assessment.

By conducting carefully controlled, rigorous testing, including the generation of objective verification statistics, the DTC is able to provide the operational community with guidance for selecting new NWP technologies with potential value for operational implementation. DTC testing also provides the research community with baselines against which the impacts of new techniques can be evaluated. The statistical results may also aid researchers in selecting model configurations to use for their projects.

Bernardet, L., P. Bogenschutz, J. Snook, A. Loughe, 2005. WRF Forecasts over the Southeasr United States: Does a larger domain lead to better results? WRF/MMM5 User’s Workshop, Boulder, CO.

Bernardet, L., J. Wolff, L. Nance, E. Gilleland, Betsy Weatherhead, C. Harrop, M. Govett, and A. Loughe, 2008: Objective Verification Results from Forecasts Generated with the ARW and NMM Dynamic Cores of the WRF Model. 9th WRF Users' Workshop, Boulder, CO.

Bernardet, L., L. Nance, M. Demitras, S. Koch, T. Fowler, A. Loughe, J. L. Mahoney, H. Chuang, M. Pyle, and R. Gall, 2008: The Developmental Testbed Center and its winter forecasting experiment. Bull. Amer. Meteor. Soc. 5, 611-627.

Bernardet, L.R., L. Nance, H.-Y. Chuang, A. Loughe, and S.E. Koch, 2004: Verification statistics for the NCEP WRF pre-implementation test. Part 1: Deterministic verification of ensemble members. First Joint WRF/MM5 User's Workshop, Boulder, CO, 229-232.

Bernardet, L.R., L. Nance, H.-Y. Chuang, A. Loughe, M. Demirtas, S. Koch, and R. Gall, 2005: The Developmental Testbed Center Winter Forecasting Experiment. 21st Conf. on Weather Analysis and Forecasting, Washington, D.C., Amer. Meteor. Soc., CD-ROM, 7.1.

Brown, J. M., S. Benjamin, T. G. Smirnova, G. A. Grell, L. R. Bernardet, L. B. Nance, R. S. Collander, and C. W. Harrop, 2007: Rapid Refresh Core Test: aspects of WRF-ARW and WRF-NMM performance relevant to the Rapid Refresh application. 22nd Weather Analysis and Forecasting Conf., Park City, Utah, AMS.

Davis, C., L. Nance, L. Bernardet, M. Pyle, and H. Chuang, 2004: WRF forecasts of recent significant weather events: A comparison of ARW and NMM cores. First Joint WRF/MM5 User's Workshop, Boulder, CO, 243-246.

Demirtas, M., L.B. Nance, L.R. Bernardet, Y. Lin, A. Loughe, R. Gall, and S. E. Koch, 2005: Quantitative precipitation forecasts (QPF) verification of DWFE. 17th Conf. on Numerical Weather Prediction, Washington, D.C., Amer. Meteor. Soc., CD-ROM, 7.2.

Nance, L., L. Bernardet, H. Chuang, G. DiMego, M. Demirtas, R. Gall, S. Koch, Y. Lin, A. Loughe, J. Mahoney, and M. Pyle, 2005: The WRF Developmental Testbed Center: A status report. WRF/MM5 User's Workshop, Boulder, CO, Amer. Meteor. Soc., CD-ROM, 2.8.

Seaman, N., R. Gall, L. Nance, S. Koch, L. Bernardet, G. DiMego, J. Powers, and F. Olson, 2004: The WRF process: Streamlining the transition of new science from research into operations. First Joint WRF/MM5 User's Workshop, Boulder, CO, 173-176.

Koch, S.E., R. Gall, G. DiMego, E. Szoke, J. Waldstreicher, P. Manousos, B. Meisner, N. Seaman, M. Jackson, R. Graham, A. Edman, and D. Nietfeld, 2005: Lessons learned from the DTC Winter Forecast Experiment. 17th Conf. on Numerical Weather Prediction, Washington, D.C., Amer. Meteor. Soc., CD-ROM, 7.6.

Szoke, E., S. Koch, and D. Novak, 2005: An examination of the performance of two high-resolution numerical models for forecasting extended snow bands during the DTC Winter Forecast Experiment. 21st Conf. on Weather Analysis and Forecasting, Washington, D.C., Amer. Meteor. Soc., CD-ROM, 7.3.

Primary Contacts

NANCE, Louisa: | DTC DEPUTY DIRECTOR | ph: 303.497.8921 | email: nanceucar.edu
BERNARDET, Ligia