Convert GRIB2 format weather data that cannot be opened with pygrib to netCDF and visualize it

Introduction

The "GRIB2" format is the industry standard for the format of meteorological data with a three-dimensional spread. There is a detailed introduction by @e_toyoda about the GRIB2 format. About the WMO (World Meteorological Organization) grid data format GRIB2

There are various tools for reading data from GRIB2 format files. For python, you can use a module called pygrib.

However, since the national synthetic radar and estimated meteorological distribution are data at 1km intervals nationwide, the amount of data is large, and unlike GSM and MSM, the compression format specified by the Japan Meteorological Agency is used. .. This doesn't seem to be supported by pygrib and there is some data that can't be opened.

The national synthetic radar is explained below. 1km mesh national synthetic radar GPV

I want to make a Deep Learning program using this data with Keras, so I want to be able to handle it with python. Therefore, I converted it to the netCDF format that can be handled by python using a tool called wgrib2 that supports the Japan Meteorological Agency format.

National synthetic radar that cannot be opened with pygrib

The National Synthetic Radar was downloaded from Research Institute for Sustainable Humanosphere, Kyoto University. When I open this data file with pygrib, I get the following error:

>>> import pygrib
>>> grbs=pygrib.open("Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin")
ECCODES ERROR   :  Unable to find template productDefinition from grib2/template.4.50008.def 
ECCODES ERROR   :  Unable to find template productDefinition from grib2/template.4.50008.def

This is probably because the definition used by the Japan Meteorological Agency does not exist in pygrib.

Conversion using wgrib2

wgrib2

For wgrib2, @ysomei has summarized it below. It is a tool provided by the US National Oceanic and Atmospheric Administration (NOAA). I've just analyzed the meteorological data

There are various options and you can do various things. For example, you can output the contents of the GRIB file to CSV as follows. Except for the created CSV file, var0_1_201 and surface are included in the label that specifies the data. The last three elements are longitude, latitude, and data (“data representative value”).

`python`


> wgrib2 -V GRIB_formatted_file -csv tmp.csv
> cat tmp.csv
Abbreviation
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.919,22.6875,7.25
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.931,22.6875,16.5
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.944,22.6875,9.25
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.956,22.6875,0
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.969,22.6875,0
Omission

Combined radar data format (data representative value and level value)

The "data representative value" of the synthetic radar is in mm / h, which is the rainfall intensity. The original GRIB2 format file does not have a direct rainfall intensity value set, but a "level value" from 0 to 251. For example, a "level value" of "37" means that the rainfall intensity is "7.0 mm / h or more and less than 7.5 mm / h". The "data representative value" is "7.75 mm / h".

In other words, there is a conversion table from "level value" to "data representative value", but wgrib2 seems to do that too (although I did not decipher wgrib2).

Convert to NetCDF format

Use the following options to convert from GRIB2 to netCDF format using wgrib2.

`wgrib.sh`


wgrib2 GRIB_formatted_file -netcdf GRIB_formatted_file.nc

This will create a netCDF format file. However, please note that the file size will increase dramatically, probably because the compression is released.

`ls`


235702   Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin
34454836 Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin.nc

Reading data using netCDF4 with python

This file can be handled as follows using the python module netCDF4.

`netcdf4_open.py`


from netCDF4 import Dataset
(Omitted)

nc = Dataset( fn , 'r' )
xx = nc.variables['longitude'][:] #
yy = nc.variables['latitude' ][:] #
data = nc.variables['var0_1_201_surface'][:]
lons, lats = np.meshgrid( xx , yy )
        
_save_file = savefilenamelist[ptr_sfile]

drawmap( data[0,:,:] , levels_rai , precip_colormap , _save_file ,_flag_show , "RAIN" , norm)

We have specified var0_1_201_surface to identify the data contained in the file. When I referred to the contents of the GRIB file with wgrib2, it seemed to be divided into var0_1_201 and surface, but when I converted it to a NetCDF file, it seems to be converted to the label var0_1_201_surface. When you output the information of the netcdf file as shown below, this label appears at the end of variables.

`DumpNetCDF`


>>> from netCDF4 import Dataset
>>> nc=Dataset("Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin.nc",'r')
>>> nc
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format NETCDF3):
    Conventions: COARDS
    History: created by wgrib2
    GRIB2_grid_template: 0
    dimensions(sizes): latitude(3360), longitude(2560), time(1)
    variables(dimensions): float64 latitude(latitude), float64 longitude(longitude), 
    float64 time(time), float32 var0_1_201_surface(time,latitude,longitude)
    groups:

Visualization

The python source to visualize. I am using matplotlib.

Although it is a front miso, this and that of visualization is [try to draw a "weather map-like front" by machine learning based on weather data (2) Story of weather data visualization as input](https: // qiita. com / m-taque / items / 988b08185097dca5f5b5 "Drawing a" weather map-like front "based on meteorological data by machine learning (2) The story of weather data visualization as input"). Since the color map to be visualized has a white range, the map itself is slightly colored in the visualization below. The order of overlaying the colored map and contour map is specified by zorder.

`draw_routine`


def drawmap( values , levels , cmap , _save_filename , flag_show , name ,norm) :
        fig,ax = plt.subplots(figsize=(5,5))
        plt.subplots_adjust(left=0.02, right=0.98, top=0.98, bottom=0.02)
     
        m = Basemap(projection='stere', llcrnrlat=25, urcrnrlat=47, llcrnrlon=125, urcrnrlon=150, lat_0=60,  lon_0=140, resolution='i' )
        
        m.drawparallels(np.arange(-80.,81.,10.))
        m.drawmeridians(np.arange(-180.,181.,10.))
        m.drawcoastlines()
        m.drawmapboundary(fill_color='#eeeeee' , zorder=-1)
        m.fillcontinents(color='#ceceee' , zorder=-1)
        
        x , y = m(lons, lats)

        m.contourf( x , y , values , levels , cmap=cmap , zorder=0 , norm=norm )

        fig.savefig(_save_filename)
        if flag_show == 'yes' :
                plt.show()
        plt.close()

You can create an image like the one shown below (UTC data at 15:00 on September 8, 2019).

gsrd.v5.201909081500.png

Summary

I described how to convert a file that is GRIB2 format but cannot be handled by pygrib to netCDF format so that it can be handled from python.

[PYTHON] Convert GRIB2 format weather data that cannot be opened with pygrib to netCDF and visualize it