Verification and validation activities are critical to the success of modeling and prediction efforts ongoing at organizations around the world.  Having reproducible results via a consistent framework is equally important for model developers and users alike.  The Model Evaluation Tools (MET) was developed over a decade ago by the Developmental Testbed Center (DTC) and expanded to the METplus framework with a view towards providing a consistent platform delivering reproducible results.

The METplus system is an umbrella verification, validation, and diagnostic tool at the core of the DTC testing and evaluation capability (Brown et al. 2020).  It is also supported by the community of thousands of users from both U.S. and international organizations. These tools are designed to be highly flexible to allow for quick adaptation to meet evaluation and diagnostic needs.  A suite of python wrappers has been implemented to facilitate a fast set-up and implementation of the system, and to enhance the pre-existing plotting capabilities.

Over the past few years, METplus has been driven by the needs of the Unified Forecast System (UFS) community, the U.S. Air Force, and the National Center for Atmospheric Research (NCAR) laboratories and collaborators.  Many organizations across the community have joined  with the DTC core partners in contributing to METplus development, fostering a more robust and dynamic framework for the entire Earth-system modeling community to use. During the past year, several leading organizations in verification and statistics research have joined the METplus contributor’s community.

The Naval Research Lab (NRL) started transitioning their capability to METplus during 2019 and began directly collaborating with METplus developers in late July 2020. They intend to contribute methods for the data assimilation and ensemble communities. Dr. Elizabeth Satterfield, a research scientist and the METplus transition coordinator at NRL, stated “By leveraging community-based tools, we can build a unified verification framework which is consistent across the suite of Navy atmospheric models and also be consistent with our operational partners. Such a framework allows us to make use of more modern verification metrics, including feature or process-based metrics, that can assist with identification and diagnosis of specific sources of model error. In addition, this framework will allow NRL to better tailor our verification products to inform data assimilation and model development, as well as the needs of the end user.  Finally, a consistent model verification framework aids in collaboration with other U.S. partners (e.g. NCEP, JEDI) who are employing the same community tools.”

Similarly, after months of discussion and project development activities, the Met Office in the United Kingdom has also begun transitioning their verification and diagnostics capability to MET.  The project is called NG-Ver, for Next Generation Verification. Adapting MET to work with unstructured grids is a key requirement and good reason for collaborating. Beyond the operational implementation, the collaboration will eventually focus on integrating innovative methods for verification and diagnostics while generalizing METplus support of file formats.  Dr. Marion Mittermaier, the manager of the model diagnostics and novel verification methods section at the Met Office, shared in a brief to her staff that stated “METplus offers a wide variety of highly configurable open source verification tools, which the Met Office can contribute to in the fullness of time. Given the open source nature of MET/METplus, it also enables the wider UM partnership to contribute more readily to getting common verification tools established across the partnership for the evaluation of model releases, especially regional configurations.” She also added “I am particularly excited to see the collaboration formalized at last and look forward to working with DTC scientists and developers. We have many common areas of interest and having a joint framework for leveraging new tools will accelerate the availability of these to the user community.”

Finally, the DTC has been coordinating with NCAR’s unified Earth-system modeling initiative, called System for Integrated Modeling of the Atmosphere (SIMA), for integrating METplus into the SIMA framework as a verification, validation, and diagnostics tool. The National Science Foundation (NSF) community is the target for the framework, and the activities directly support the NCAR/NOAA Memorandum of Agreement, signed in 2019, to work collaboratively on developing a common framework for Earth-system modeling. METplus was recently demonstrated to provide verification and diagnostics for weather-scale prediction (~1 deg) from the SIMA Community Atmosphere Model (CAM). Andrew Gettelman, the SIMA climate lead, said "METplus can provide new ways for us to look at cross scale models and do weather verification even on climate scale model output to help us improve processes important for extreme weather events."

Ultimately, these additions to the METplus community have substantially boosted the DTC’s ability to provide testing and evaluation capability to make Research-to-Operations more efficient and provide evidence-based decisions. The DTC looks forward to continuing to expand METplus capability through these collaborations.

Brown et al, 2020: The Model Evaluation Tools (MET): More than a decade of community-supported forecast verification. Bull. Amer. Meteor. Soc. Early Online Release. DOI: https://doi.org/10.1175/BAMS-D-19-0093.1

The U.S. Air Force (USAF) partnership with the DTC has provided superb value to the USAF over the years, as this collaboration dates back as far as 2003.  Early in the relationship, the DTC benefited the USAF-NCAR partnership with WRF development and transition to USAF operations.  In recent years, the DTC has focused primarily on USAF needs for model evaluation through continued development of the Model Evaluation Tools (MET), along with project efforts related to model testing and evaluation (T&E).  Throughout this period, the DTC and the other DTC partners have been valuable teammates, exemplifying the true model (pun intended) for interagency cooperation.

Moving into the next decade, the science and application of environmental modeling face numerous evolving challenges and opportunities, and the DTC will play a central role in several of them.  One challenge is the proliferation of emergent commercial weather data (CWD) sources available for Research and Development (R&D), as well as for data assimilation (DA) with operational weather forecast models.  These new data sources are being released under much shorter timelines than those from traditional government observing systems, necessitating that we incorporate the data more rapidly into our model DA systems, which will require us to test at a faster pace and evaluate model performance (enhanced or degraded) for each new data type. The DTC has a golden opportunity to partner with the Joint Center for Satellite Data Assimilation (JCSDA), a peer organization working on next-generation DA architectures and science improvements, to enhance our collective ability to test more rapidly and evaluate the utility of new data types for DA in the sponsoring agencies’ modeling systems.

In addition to standard verification statistics, which are the expected output of model evaluation studies (e.g., Observing System Experiments/OSEs), the sponsoring agencies will also need to measure the value added by each new data type in monetary terms.  Given our limited budgets, which are unlikely to significantly increase, it is highly unlikely we will be able to purchase new data without a trade-off from somewhere else.  In other words, in order to purchase new data, we will probably have to draw the budget away from another valid requirement. But how do we decide which element is more important and is the best use of our limited funds?  One objective measure would be to select the options that provide the most value per dollar.  Because most of the sponsor agencies will face decisions of this type, building and operating a model T&E capability would be a promising opportunity for the DTC to support their interagency partners to manage the flood of CWD headed our way.

Another pending opportunity for the DTC is to take the lead in model T&E for the future of interagency model collaboration envisioned by the nascent Environmental Prediction Innovation Center (EPIC). The initial NOAA-managed contract for EPIC is still in the early stages and the first increment focuses on software engineering, the software development environment, and related tools and applications.  As such, EPIC will need advanced model T&E capabilities and tools to incorporate into the new national collaborative environment, and who better than the DTC to lead the way in that area?  One clear example is MET, a capability in which the USAF is already heavily invested.  By continuing to expand the capabilities of MET to meet the needs of multiple partner agencies, as well as working to include MET in EPIC’s new software development environment, the DTC can expand its role by becoming one of the cornerstone, essential partners in the U.S. modeling community.

The USAF can point to several successful DTC-partnered projects that have benefited both USAF operations as well as contributed to improved capabilities for our interagency partners.  However, even when considering those past successes, we feel the DTC’s best days are ahead, and we in the USAF are looking forward to being a part of the new efforts yet to come.

Keith Searight joined the DTC in June as NOAA GSL’s co-lead for the Model Evaluation Tools (METplus) verification system, which is becoming the unified verification, validation, and diagnostics software for the Unified Forecast System (UFS). He also performs DTC-wide project management duties for GSL, such as project tracking, budgets, and reporting. Previous to this job, he was NOAA’s technical lead for Science On a Sphere, and continues to do so part time.   

Keith grew up in a small college town in central Illinois surrounded by corn fields. After attending college at the University of Tulsa, double majoring in geosciences and computer science, he attended Stanford for an MS in geophysics.  Although he started his career as an applied geoscientist in the energy business, he decided to switch to a more computer-focused career,  studying computer science part time at CU Boulder.  After earning his MS degree, his first software job was a research programmer in the Dept. of Atmospheric Sciences at University of Illinois writing gridded data management and visualization software.  Later, he became a commercial software developer, working for several small startup companies.  He pivoted his career again toward scientific research labs by joining NCAR’s Research Applications Laboratory in 2008, where he led the software engineering of weather forecasting systems for military sponsors.  After NCAR, he worked for a period at the National Renewable Energy Laboratory (NREL) in Golden, then he joined Colorado State University’s Cooperative Institute for Research in the Atmosphere (CSU/CIRA) at NOAA GSL in Boulder seven years ago.  

The variety of his work is what inspires him, as his role encompasses software coding, technical writing, and managing people and projects all in the name of improving forecasting skill. On any given day, he might work on budget analysis and tracking, project management and reporting, software testing, application and script coding, and documentation writing. 

The study of the earth’s atmosphere and simulating its properties requires programming computer software run on supercomputers.  In simplified terms, the focus of his work is developing the software that gauges how close the forecasts matched what actually happened and then generates graphics that help analyze their accuracy.  Those results are used to develop and test out new ideas to make our weather forecasts even better.  

His first job, however, was a far cry from his current profession. Growing up in the Midwest, summer jobs in the agricultural arena were the most plentiful options for teens.  So one summer, he worked in the fields detasseling corn.

Thankfully, he had greater dreams to fulfill. When he was a kid, he aspired to be an astronomer and study the solar system.  He had his own telescope and avidly followed the new discoveries and wondrous imagery captured by the spacecraft flying near the planets and moons.  Although his eventual career ended up being a little more “grounded,” it just means his focus has been more specifically dedicated to studying just one important planet, our own. 

When asked what he loves about his job here, he replied, “I really enjoy the variety of science, technology, and management activities in which I’m involved.  I’m inspired by DTC’s mission to partner with NOAA, NCAR, and other organizations to make weather forecasting as accurate as it can be. I also appreciate all the community connections and collaborations that we have across government and academic institutions in the US and around the world.” 

Keith has lived in Colorado for decades now. He’s married with two adult sons in college studying nuclear engineering and cyber security, respectively. 

Favorite quote? Well, these are strange times, and the following quote seems to fit. 

“The optimist proclaims we live in the best of all possible worlds; and the pessimist fears this is true.   ~James Branch Cabell

The Hurricane Weather Research and Forecast system (HWRF), which is one of NOAA’s operational models used to predict the track, intensity, and structure of tropical cyclones, undergoes an upgrade cycle that is generally conducted on an annual basis.  Through the code management and developer support framework provided by the DTC, innovations from the research community that have been added to branches within the HWRF code repository serve as candidates for testing as part of these upgrade cycles. Scientists at NOAA’s Environmental Modeling Center (EMC) and the Developmental Testbed Center (DTC) frequently collaborate when testing and evaluating (T&E) changes to the HWRF physics schemes or data assimilation system in hopes of improving HWRF predictions.  For the 2020 HWRF upgrade cycle, the DTC focused on T&E of two potential upgrades to model physics stemming from the DTC visitor projects: 1) upgrades to the cloud-overlap scheme used in the Rapid Radiative Transfer Model for General Circulation Models (RRTMG), made available by John Henderson and Michael Iacono of Atmospheric and Environmental Research (AER) and 2) the Mellor-Yamada-Nakanishi-Niino (MYNN) Planetary Boundary Layer, based on work conducted by Dr. Robert Fovell (SUNY Albany) and a collaboration with Dr. Joseph Olson (NOAA Global Systems Laboratory).

Testing these potential upgrades focused on thirteen tropical cyclones in the North Atlantic ocean from the past three years that provided a mixture of storm characteristics and previous operational model performance. Preliminary results for the cloud-overlap upgrades (four of the thirteen storms) suggested that the results would not be sufficient to warrant operational implementation. The DTC communicated this feedback to AER, and worked with them to further analyze the results. The analysis suggested that a different configuration of the cloud-overlap upgrades might perform better. After coordinating with EMC and AER, the DTC tested the revised cloud-overlap configuration. This second test demonstrated up to 4% improvement in the 3–5 day hurricane-track forecast, which was sufficient for EMC to transition the cloud-overlap changes into the 2020 operational HWRF. This process illustrates the important role of iterative testing and development when transitioning research to operations. Results from the MYNN experiment indicated the scheme was not yet ready for operational implementation. However, the results are informing additional changes to the code by the developers, which may further enhance the performance of the MYNN PBL in high-wind conditions for potential application within NOAA’s Unified Forecast System (UFS).

Since these tests were conducted, the 2020 configuration of HWRF was finalized and implemented in operations. During the summer of 2020, DTC and EMC worked together to merge the final version of the code back to the trunk of the HWRF repository. This step enables researchers to add further innovations to the latest version of the code, ensuring that any scientific results are directly applicable to the operational HWRF, and positioning the community to contribute to the next HWRF implementation in early 2021.

As part of the Unified Forecast System (UFS) effort, a Limited Area Model (LAM) is under development, based on the non-hydrostatic Finite­ Volume Cubed-Sphere Dynamical Core (FV3), to achieve high-resolution forecasts. This project leverages the available environment provided by current UFS developments to investigate the Rapid Refresh Forecast System (RRFS) skill in representing the structure of convection generated by squall lines over the Great Plains of the United States. 

Squall lines occur with a high frequency in this region, causing severe weather events and much of the precipitation that falls there throughout the year. A pre-frontal squall line over Oklahoma on 4 May 2020 is the case under study. Convective initiation was observed over northeastern Oklahoma at 20Z and around 22Z a line of storms evolved that extended across the state (Fig. 1), resulting in several high-wind and large-hail events.

The Gridpoint Statistical Interpolation (GSI)  data assimilation (DA) system was used for the analysis and the Model Evaluation Tools (MET) for verification. The GSI 3-dimensional variational (3DVAR) DA capability was tested, along with two physics suites: GFS-based physics and the suite developed at NOAA’s Global Systems Laboratory (GSD SAR). Hourly cycles with 18-hour forecasts starting at 00Z on May 4th through 06Z, May 5th were performed. 

Results from the 2-h forecast initialized at 18Z suggest that the GSD SAR suite with DA experiment was able to represent the initial convection over northeast Oklahoma (see black circles in Fig. 1-A and 2-B), but forecast cycles prior to that initialized at 17Z failed to capture the initial convection over Oklahoma at 20Z (not shown). Experiments using the GFS suite performed poorly for the convective initiation over Oklahoma up to the 20Z valid hour (Fig. 2-D).

Although more convection developed over other states, a slightly better representation of the squall-line structure is shown after 21Z in the experiments using the GFS suite when compared to the GSD SAR experiments, especially when using DA (Fig. 3-D). The experiments using the GSD SAR suite do not reproduce the evolution of the squall line over Oklahoma and instead, a line of storms developed northward of the observed one (Fig. 3-A and 3-B). After 22Z, the GFS experiments continue to do a better job capturing the squall line structure over Oklahoma while the GSD SAR experiments miss this system until the last cycle. Nevertheless, the convection over Arkansas, Missouri and Illinois is better captured in the experiment using the GSD SAR suite with DA.

The forecasted 2-m temperature provides further insight into strengths and weaknesses of the configurations in each experiment. Larger bias is observed with longer forecast length when using the GSD SAR suite (Fig. 4-A). After the 5-h forecast, the 2-m temperature forecasts are cooler than the observations by up to 1.74 K. A high positive DA impact is observed for RMSE results up to the 8-hour forecast, after which the impact becomes negative (Fig. 4-B). The forecasts generated using the GFS physics suite have a RMSE and bias that are generally smaller than those associated with the GSD physics suite. The inclusion of DA reduces the bias for all lead times and reduces the RMSE but only during the first 4 hours of the forecast.

To understand the elements that are limiting the initiation and overproduction of convection in GFS experiments and the convection evolution in the GSD SAR experiments, future work will test the GSI hybrid (3DEnVar) analysis and adjust parameters related to cloud analysis and surface-data analysis enhancements. 

 This visitor project forms part of my doctoral research at CPTEC/INPE (Brazil) and has been underway since March 2020. I am grateful to the DTC for this opportunity, especially to Louisa Nance, Ming Hu, Guoqing Ge, Will Mayfield, Eric Gilleland, Jacob Carley, Daryl Kleist, and my academic advisor Luiz Fernando Sapucci, for their guidance and support during this time. 

Jamie Wolff, the lead of the software containers and cloud computing task in the DTC, contacted me in the Spring of 2019 about a unique opportunity to collaborate on a course that would offer interested students at Metropolitan State University an opportunity to run and experiment with an end-to-end numerical weather prediction (NWP) system utilizing cloud computing.

The motivation behind this collaboration was to introduce undergraduates of a bachelor’s degree Meteorology program to NWP in a stress-free environment in which students would not spend unnecessary time configuring, compiling, and optimizing the code and libraries for the NWP system.  The DTC containers would streamline running the NWP system so the students would be able to grasp how beneficial NWP could be in their toolkit. The overarching goal was to empower the students to use and understand NWP models on a fundamental level and prepare them for post-undergraduate positions in the NWP arena.

It was decided that an active learning approach would be the ideal format to present the information and disseminate the material taught as a series of workshops/seminars.

During the 2019 Fall semester, I taught a Forecasting Lab course that met once a week for 2 hours.  The extended class time allowed the DTC team to teach the attendees and demonstrate how to compile and run the Weather Research and Forecasting (WRF) Model on the Amazon Web Service (AWS) Cloud Server. The DTC team met with my students for a total of eight hours (six face-to-face hours and two hours via Google Meet).

The initial design of the WRF Cloud Computing (WCC) course was somewhat challenging because the setup and configuration of the AWS on our UNIX-based workstations faced some computing and financial hurdles. Nevertheless, the DTC team created Docker container images for each component of the end-to-end WRF-based NWP system on the AWS Cloud. Docker makes it easier to create, deploy, and run applications by using “containers.” A container is a software tool that packages code and all of its dependencies, so the application runs quickly and reliably from one computing environment to another (Docker Website). The end-to-end system containers were built and configured by the DTC team ahead of classes. With the accessibility of the components in containers, it was possible to efficiently and effectively create a full end-to-end NWP system teaching toolkit for use by the students.

Next, the DTC team provided step-by-step instructions on how to use those images to access the WRF data and ecosystem to run several case studies through the end-to-end NWP system from pre-processing, to running the model, then post-processing, visualization, and verification.  The DTC team also recommended installing Docker on the MSU Denver Weather Lab network so the students could run the end-to-end NWP container system locally on our in-house desktops as well.

The Developmental Testbed Center has developed a set of instructional training videos. Two videos are now available to familiarize the NWP development community with the UFS code and the Medium-Range Weather (MRW) App v1.0.

• UFS Code Repositories describes the overall structure of the Medium-Range Weather Application ("MRW App") source code repositories, directory structures, and where to find further documentation.
• Raw Initial Conditions for the UFS Medium-Range Weather Application v1.0 describes the formats of raw initial conditions that can be used to initialize the MRW App v1.0 and how to stage them on disk to use with the App.

Three videos are available to familiarize the community with the Common Community Physics Package (CCPP).

• CCPP Suites introduces the components of Common Community Physics Package (CCPP) suites and explains how to add to, modify, or build a new suite.
• CCPP-Compliant Parameterizations describes how to make a physics parameterization CCPP-compliant.
• CCPP: Adding a Variable to a Scheme explains how to add a new variable to an existing CCPP-compliant scheme on the physics-scheme side, as well as on the host side (if using the UFS or CCPP Single Column Model).