Introduction

Deep Learning beginners tried the weather forecast. The reasons for choosing the theme of weather forecast are (1) the image of the meteorological satellite "Himawari" is available, the burden of data acquisition is small, and (2) the problem setting is simple, so it is easy for beginners to work on. Because I thought it was.

I chose Keras + (Tensorflow) for the Deep Learning library. There are many deep learning libraries out there, but looking at the sample code on the web, it seemed to be the easiest to use Keras, so I decided to try it with Keras.

Environmental setting

The PC I usually use has Windows 7. First of all, I introduced Anaconda3 to enable Python.

After the introduction of Anaconda3, you will be able to run Python by launching "Anaconda Prompt" from "All Programs" ➔ "Anaconda3". As of May 24, 2017, it seems that Python 3.6 can be used in the default environment. It seems that Tensorflow does not support Python 3.6 at the moment, so first set up Anaconda Prompt to use Python 3.5.

(C:\Users\IBM_ADMIN\Anaconda3) C:\Users\IBM_ADMIN>conda create -n py35 python=3.5 anaconda

(C:\Users\IBM_ADMIN\Anaconda3) C:\Users\IBM_ADMIN>activate py35

(py35) C:\Users\IBM_ADMIN>python --version
Python 3.5.2 :: Anaconda 4.3.1 (64-bit)

You are now ready to use Python 3.5. Next, we will introduce Keras and Tensorflow.

(py35) C:\Users\IBM_ADMIN>pip install tensorflow

(py35) C:\Users\IBM_ADMIN>pip install keras

Since the backend is not tensorflow by the default setting of Keras, rewrite the value of "backend" of ".keras / keras.json" in the root directory of Anaconda Prompt as follows.

{
    "floatx": "float32",
    "image_data_format": "channels_last",
    "backend": "tensorflow",
    "epsilon": 1e-07
}

Now you are ready to use Keras.

Data preparation

Images of the meteorological satellite "Himawari" are available for download at here. Meteorological satellite images from July 2015 to the present are available.

When performing a weather forecast using an image as an input, the image will be pitch black unless it is a daytime image, and the weather forecast will not be established. In addition, the Japan Meteorological Agency publish weather observation data every 3 hours from 0:00. Therefore, I downloaded the images at 9, 12, and 15:00 evenly from each season, and acquired 1095 images as training data and 90 images as test data. For example, one day the meteorological satellite image of the Japanese archipelago is as follows.

Next, create a training dataset and a test dataset. In order to make a weather forecast using the image of a meteorological satellite, it is necessary to create an application that outputs the future weather when the image at a certain point is input. So, when I was given a meteorological satellite image of the Japanese archipelago at a certain point, I decided to create an application that would predict the weather in Tokyo 24 hours later, and for the meteorological satellite images of each date and time. I decided to give the weather observation data 24 hours later as a label. Also, I thought it would be difficult to distinguish between "cloudy" and "rainy" from meteorological satellite images, so I decided to predict "sunny" and "other than that" from the images.

Let's assume that the meteorological satellite image from 9 o'clock on April 1, 2017 is obtained. On the other hand, the label I prepared is a CSV file in the following format. "1.0" is sunny and "0.0" is other weather.

Date and time,Tokyo weather
2017/4/2 9:00,1.0
2017/4/2 12:00,1.0
2017/4/2 15:00,1.0
2017/4/3 9:00,0.0
2017/4/3 12:00,0.0

After preparing the training data and its correct label, and the test data and its correct label in this way, the training data set and the test data set were created using the program below. When you run this program, the training and test datasets will be created as a file named "data.pkl.gz" under the directory where you saved the correct labels for the training data.

import os
import numpy
import pandas
import pickle
from PIL import Image

train_data_src_dir = "Absolute path of the directory where the training data is saved"
train_label_csv = "Absolute path of correct label of learning data in CSV format"
test_data_src_dir = "Absolute path to the directory where you saved the test data"
test_label_csv = "Absolute path of correct label of test data in CSV format"

def img2nparray(file):
    img = Image.open(file, "r")
    array = numpy.asarray(img, dtype="uint8")
    array = array.reshape(array.shape[0], array.shape[1], 1)
    return array

def get_label_set(file):
    labels = pandas.read_csv(file, encoding="shift-jis")
    labels = labels[labels["Tokyo weather"].notnull()]
    
    return labels["Tokyo weather"].as_matrix()
    

def generate_dataset():
    print("Generating train data set")
    master_dataset = []
    files = os.listdir(train_data_src_dir)
    for file in files:
        master_dataset.append(img2nparray(train_data_src_dir + file))
        
    master_dataset = numpy.array(master_dataset)
    train_label_set = get_label_set(train_label_csv)
    train_set = master_dataset, train_label_set
    
    print("Generating test data set")
    test_dataset = []
    files = os.listdir(test_data_src_dir)
    for file in files:
        test_dataset.append(img2nparray(test_data_src_dir + file))
        
    test_dataset = numpy.array(test_dataset)
    test_label_set = get_label_set(test_label_csv)
    test_set = test_dataset, test_label_set
    
    return (master_dataset, train_label_set), (test_dataset, test_label_set)

if __name__ == '__main__':
    dataset = generate_dataset()
    
    print("Creating pickle file")
    f = open(os.path.dirname(train_label_csv) + os.sep + 'data.pkl.gz', 'wb')
    binary = pickle.dump(dataset, f, protocol=2)
    f.close()
    print("Created")

This completes the data preparation.

Practice of weather forecast by meteorological satellite image

Now let's run the weather forecast using the prepared data. Since the input data is an image and I wanted to try Deep Learning easily, the learning and weather usage is here I used the program for distinguishing handwritten characters almost as it is. In part, the size of the image is changed according to the data I prepared. We also set the number of classes to 2 so that we can expect binary values of "sunny" and "other than that". I've also made some modifications to read the dataset I created instead of the MNIST dataset.

We will acquire learning data evenly from each season, train using the program linked above, and practice the weather forecast for the test images at 9, 12, 15:00 from April 1st to 30th, 2017. As a result, the correct answer rate was about 65.5% as shown below.

1095 train samples
90 test samples
Train on 1095 samples, validate on 90 samples
Epoch 1/12
1095/1095 [==============================] - 112s - loss: 1.8521 - acc: 0.4922 - val_loss: 0.6720 - val_acc: 0.6000
Epoch 2/12
1095/1095 [==============================] - 110s - loss: 0.6840 - acc: 0.5096 - val_loss: 0.6604 - val_acc: 0.6000
Epoch 3/12
1095/1095 [==============================] - 109s - loss: 0.6809 - acc: 0.5187 - val_loss: 0.6559 - val_acc: 0.6000
Epoch 4/12
1095/1095 [==============================] - 107s - loss: 0.6748 - acc: 0.5187 - val_loss: 0.6839 - val_acc: 0.7222
Epoch 5/12
1095/1095 [==============================] - 112s - loss: 0.6731 - acc: 0.5836 - val_loss: 0.6461 - val_acc: 0.6000
Epoch 6/12
1095/1095 [==============================] - 108s - loss: 0.6705 - acc: 0.5845 - val_loss: 0.6390 - val_acc: 0.6000
Epoch 7/12
1095/1095 [==============================] - 108s - loss: 0.6705 - acc: 0.5817 - val_loss: 0.6478 - val_acc: 0.6000
Epoch 8/12
1095/1095 [==============================] - 110s - loss: 0.6646 - acc: 0.6027 - val_loss: 0.6317 - val_acc: 0.6000
Epoch 9/12
1095/1095 [==============================] - 113s - loss: 0.6744 - acc: 0.5863 - val_loss: 0.6483 - val_acc: 0.7778
Epoch 10/12
1095/1095 [==============================] - 118s - loss: 0.6617 - acc: 0.6192 - val_loss: 0.6183 - val_acc: 0.6222
Epoch 11/12
1095/1095 [==============================] - 114s - loss: 0.6630 - acc: 0.6046 - val_loss: 0.6149 - val_acc: 0.6222
Epoch 12/12
1095/1095 [==============================] - 104s - loss: 0.6549 - acc: 0.6046 - val_loss: 0.6170 - val_acc: 0.6556
Test loss: 0.616963325606
Test accuracy: 0.655555557542

in conclusion

This time, I tried to summarize how deep learning beginners prepare an execution environment for deep learning and try the weather forecast. The correct answer rate is a delicate result of 65.5%, and there seems to be room for improvement. For example, it may be possible to improve the accuracy by inputting the most recent image as input instead of inputting only the image 24 hours ago.

The weather forecast using meteorological satellite images has been practiced by others [http://qiita.com/wbh/items/1984b582071ef60c92f3), so I would like to get hints for improving the correct answer rate from this information in the future. I will.

[PYTHON] Deep Learning beginners tried weather forecasting from meteorological satellite images using Keras

Introduction

Environmental setting

Data preparation

Practice of weather forecast by meteorological satellite image

in conclusion