GRIB2 Migration
A few days ago I started having a problem with the previous GFS forecast data source I was using for Seasonality forecasts. Just as a bit of background, Seasonality will download forecasts for U.S. locations from the NWS’s NDFD service. For locations outside the U.S., I have to get the data from somewhere else, and the best data source I’ve been able to find is 0.5 degree GFS model output, also available from the NWS. Because the GFS model data is in GRIB format and very large, I first download it to a “forecast” server that I host, and throw it all into a database. Then Seasonality will contact my forecast server to get the forecast for its configured locations.
With the old GFS data source offline, I had to find a new one. I came across a production server at NCEP that hosted 0.5 degree GFS model output, but the data was in GRIB2 format, which presented a problem because up until this point, all of my forecast data was in GRIB (or GRIB1) format.
Fortunately, there were very few modifications because of the format change, but there are some adjustments that needed to be made. I am using the wgrib and wgrib2 tools to convert the GRIB data to ASCII format first, so it’s easier to parse when importing it into the database. With wgrib, I would use a command like this to convert the data to ASCII:
wgrib -s <grib_file> | egrep \":<variable>:<level>\" | wgrib -i -grib <grib_file> -text -o <txt_file>
The authors of wgrib2 changed things a bit, and actually made the command a little bit easier:
wgrib2 -s <grib_file> | egrep \":<variable>:<level>\" | wgrib2 -i <grib_file> -text <txt_file>
The first two commands in the string don’t change at all, but the third one does. The -grib and -o flags are no longer needed. wgrib2 assumes that the input file is in GRIB2 format, and if you specify -text, the next argument is assumed to be the output file. The problem was that my output data wasn’t right. For some reason, my forecasts were just all screwed up, even though the ASCII files themselves looked okay. I ended up finding some documentation on the -text flag, and it seems that the default output order has been changed from the original North->South to South->North, so all of my data was being inverted. This was somewhat problematic, since I don’t really want to return a forecast for the southern hemisphere when someone asks for a location in the northern hemisphere. The fix is fairly simple, just pass -order raw to the wgrib2 command…
wgrib2 -s <grib_file> | egrep \":<variable>:<level>\" | wgrib2 -i <grib_file> -order raw -text <txt_file>
In general, it looks like most of the NWS datasets will only be available in GRIB2 in the near future, so I hope this information saves other wgrib users some time when doing their own GRIB -> GRIB2 migrations.
Hi Mike,
I’m actually writing to seek some help on using wgrib2. I’m new to the grib2 files. I’ve got wgrib2 installed on the unix server, and I’m able to call wgrib2 (which lists all the available options). However, I’m having a hard time to find any sort of manual for wgrib2 besides the cpc.ncep.noaa.gov site. I’d like to know how to use the options (how to call the command) to get lat/lon information as well as temperature, moisture…If I could just see some examples, that would be most helpful. Thanks.
Unfortunately I can’t be of too much help here because I don’t have much experience with GRIB2 yet. I basically convert the data to ASCII and then massage it for my application. Wesley over at CPC put together a couple of scripts to simplify downloading partial GRIB datasets that I found helpful when working on my own project. You might want to read through them to learn possible ways to interact with the GRIB data. Here’s the URL:
http://www.cpc.ncep.noaa.gov/products/wesley/fast_downloading_grib.html
Beyond that, I welcome anyone else to chime in here with some examples. Good luck!
Looks like everyone has to go thru the pain of conversion. The documentation that is available assumes an in-depth knowledge of GRIB. I was hoping that wgrib2 would be backwardly compatible with GRIB1. But, no such luck. Some of my customers still get some data in GRIB1 format. Hence, I have to support two different methodologies. Also, the fact that wgrib and wgrib2 has vastly different option sets, does not help either.