[PYTHON] Datensatzvorbereitung für PyTorch
Python-Skript zum Vorbereiten eines Datensatzes (Bild) für tiefes Lernen
Verwenden Sie train_test_split usw. von scikit-learn, wenn Sie den Datensatz in train-val aufteilen? Wenn Sie einen Ordner für Lern- / Auswertungsdaten im Voraus vorbereiten und die Daten daraus abrufen möchten (obwohl ich denke, dass sie fast nicht vorhanden sind), können Sie die Methode in diesem Artikel verwenden. Klicken Sie hier für das Programm GitHub-moriitkys/PrepareDataSet Die Spezifikationen des in diesem Artikel vorgestellten Programms lauten wie folgt.
- Für DateLoader von PyTorch (Keras ist ebenfalls verfügbar)
- Es ist möglich, Daten für Deep Learning mit der Benutzeroberfläche von TKinter festzulegen (CreatePanel () von mylib / create_panel.py wird instanziiert und die Verarbeitung wird ausgeführt).
- Wahl des Lernens oder Erraten
- Modell-Backbone-Auswahl (ResNet, Mobilenet, MyNet)
- Auswahl der Teilungsausführung von Trainings- / Bewertungsdaten (unabhängig davon, ob sie beim letzten Mal geteilt wurden oder nicht).
- Trainingsdatenverhältnis (Zug: Wert)
- Auswahl der Datenerweiterung
- Gesamtzahl der Epochen --Wenn Sie eine Datenerweiterung durchführen, wird MakeDataSetRGB () in mylib / makedataset_rgb.py instanziiert, die Daten im Dataset-Ordner und im Dataset_val werden aufgefüllt und in Dataset_aug und Dataset_val_aug generiert.
- Durch die geteilte Ausführung wird ein Ordner wie dataset_val in derselben Hierarchie für den Dataset-Ordner erstellt und alle Auswertungsdaten werden verschoben.
- Es wird angenommen, dass die Hierarchie Ordner für jede Klasse enthält, z. B. / dataset / 1 / ..., / dataset / 2 / ... (Klassennamen können alphabetisch sein)
- Es gibt auch ein Programm, das vom geteilten Zustand (Dataset_Val existiert) in den Zustand vor dem Split (Dataset) (revert_dataset_val ()) zurückkehrt.
Wie man die Benutzeroberfläche tatsächlich bedient, um Variablen zu bestimmen und Daten zu generieren / vorzubereiten
Das Obige ist die Einstellung, wenn ・ Zugmodus ・ ResNet ・ Zug-Wert-Split-Ausführung ・ Zug: Wert = 0,65: 0,35 ・ Erweiterungsausführung ・ Epochen = 40. Schließlich werden die folgende Verzeichnisstruktur und Ausgabe vorgenommen.
 |
 |
 |
|---|---|---|
| Figure 1-a.Standardverzeichnisstatus | Figure 1-b.Verzeichnis nach Aufteilung und Erweiterung des Datensatzes | Figure 1-c.Ausgabe, wenn das Programm erfolgreich abgeschlossen wurde |
Unterschiede zwischen Keras und Pytorch
In beiden Fällen sind die vom Panel (Tkinter) erhaltenen Variablen wie flag_train und total_epochs gleich. Der Unterschied liegt in der Vorbereitung des Datensatzes.
Keras-Datensatzvorbereitung
Keras bereitet die Daten als Array von Numpy vor. Ich konnte mir keinen guten Weg vorstellen, also habe ich List mit numpy.array konvertiert, in npy gespeichert und beim Lernen von npy gelesen. Ich beabsichtige, den Speicherverbrauch durch Leeren der Liste zu reduzieren, möchte diesen Teil jedoch ordentlich umschreiben.
Keras-Dataset-Vorbereitung (klicken Sie hier, um das Programm anzuzeigen)
#Settings and prepare your dataset
import glob
import os
import sys
import keras
from keras import layers, models, optimizers
from keras.utils import np_utils
import keras.backend as K
import keras.layers as KL
import tensorflow as tf
from keras.preprocessing.image import load_img, img_to_array, array_to_img
from keras.preprocessing.image import random_rotation, random_shift, random_zoom
import numpy as np
import random
import matplotlib.pyplot as plt
import PIL
from PIL import Image
import cv2
from pathlib import Path
import shutil
from sklearn.model_selection import train_test_split
import mylib.makedataset_rgb as mkdataset
import mylib.create_panel as create_panel
import mylib.utils as myutils
# ------ Setting panels ------
import tkinter
from tkinter import messagebox
img_size_mynet = [28,28]# You can change input image size(Pay attention to network shape)
setting_panel = create_panel.CreatePanel(img_size_mynet = img_size_mynet)
setting_panel.create_buttons()#If you push "start", exit this line.
# ------ set params and preparing dataset ------
flag_train = setting_panel.flag_train
flag_aug = setting_panel.flag_aug
flag_split = setting_panel.flag_split
ratio_train = float(setting_panel.var_sp.get())#0.0 ~ 1.0
total_epochs = int(setting_panel.var_sp_epochs.get())
type_backbone = setting_panel.type_backbone#ex) ResNet, Mobilenet, MyNet
layer_name_gradcam = setting_panel.layer_name_gradcam# Don't use
img_size = setting_panel.img_size#ex) ResNet:[224,224], Mobilenet:[192,192], MyNet:[28,28]
print(type_backbone)
print("img_size=" + str(img_size))
#How many classes are in "dataset" folder
categories = [i for i in os.listdir(os.getcwd().replace("/mylib", "") + "/dataset")]
categories_idx = {}#ex) HookWrench:0, SpannerWrench:1
for i, name in enumerate(categories):
categories_idx[name] = i
nb_classes = len(categories)#ex) nb_classes=2
dirname_dataset = "dataset"# dataset folder
dirname_dataset_val = dirname_dataset + "_val"
output_folder = "outputs_keras/"+type_backbone
x_train, y_train, x_val, y_val = [],[],[],[]
def aug_dataset(dirname_dataset_1, dirname_dataset_val_1):
'''
This function returns updated dataset dirname
Contain MakeDataSetRGB() (mylib/makedataset_rgb.py)
Argument1: Foldername (String), Argument2: Foldername (String)
Usage:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
'''
dirname_dataset_aug = dirname_dataset_1 + "_aug"
dirname_dataset_val_aug = dirname_dataset_val_1 + "_aug"
make_dataset = mkdataset.MakeDataSetRGB()
if os.path.exists(dirname_dataset_aug ) == True \
or os.path.exists(dirname_dataset_val_aug ) == True:
#https://pythonbasics.org/tkinter-messagebox/
tki2 = tkinter.Tk()
tki2.withdraw()
ret = messagebox.askyesno('Bestätigung', '_Es gibt einen Aug-Ordner._Möchten Sie den aug-Ordner wirklich löschen?')
if ret == True:
if os.path.exists(dirname_dataset_aug ) == True:
shutil.rmtree(dirname_dataset_aug)
if os.path.exists(dirname_dataset_val_aug ) == True:
shutil.rmtree(dirname_dataset_val_aug)
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
tki2.destroy()
else:
tki2.destroy()
tki2.mainloop()
else:
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
dirname_dataset_2 = dirname_dataset_1 + "_aug"
dirname_dataset_val_2 = dirname_dataset_val_1 + "_aug"
return dirname_dataset_2, dirname_dataset_val_2
def prepare_dataset(dirname_dataset, dirname_dataset_val):
label = 0
for j in categories:# Prepare Training Dataset
files = glob.glob(dirname_dataset + "\\" + str(j) + "/*")
for imgfile in files:
img = load_img(imgfile, target_size=(img_size[0], img_size[1]))
array = img_to_array(img) / 255
x_train.append(array)
y_train.append(label)
label += 1
label = 0
for j in categories:# Prepare Validation Dataset
files = glob.glob(dirname_dataset_val + "\\" + str(j) + "/*")
for imgfile in files:
img = load_img(imgfile, target_size=(img_size[0], img_size[1]))
array = img_to_array(img) / 255
x_val.append(array)
y_val.append(label)
label += 1
if flag_train == True:
print("train mode")
print("total epochs = " + str(total_epochs))
if flag_split == True:
revert_dataset_val()
prepare_dataset_val()
print("splitting complete")
if flag_split == False and os.path.exists(dirname_dataset_val) == False:
prepare_dataset_val()
print("You have not splitted dataset, so splitteing automatically done")
if flag_aug == True:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
print("dataset source is " + dirname_dataset + "&" + dirname_dataset_val)
elif flag_aug == False:
dirname_dataset_aug = dirname_dataset + "_aug"
dirname_dataset_val_aug = dirname_dataset_val + "_aug"
make_dataset = mkdataset.MakeDataSetRGB()
if os.path.exists(dirname_dataset_aug ) == True \
and os.path.exists(dirname_dataset_val_aug ) == True:
dirname_dataset = dirname_dataset_aug
dirname_dataset_val = dirname_dataset_val_aug
prepare_dataset(dirname_dataset, dirname_dataset_val)
# make directory (weights_folder, outputs)
if os.path.exists("weights_pytorch/"+type_backbone) == False:
os.makedirs("weights_pytorch/"+type_backbone)
if os.path.exists("outputs_pytorch/"+type_backbone) == False:
os.makedirs("outputs_pytorch/"+type_backbone)
if os.path.exists(output_folder) == False:
os.makedirs(output_folder)
# In Keras, use numpy array for NN model
if os.path.exists("tmp_npy") == False:
os.makedirs("tmp_npy")
x_train, y_train, x_val, y_val = np.array(x_train), np.array(y_train), np.array(x_val), np.array(y_val)
np.save("tmp_npy/x_train.npy", x_train)
np.save("tmp_npy/y_train.npy", y_train)
np.save("tmp_npy/x_test.npy", x_val)
np.save("tmp_npy/y_test.npy", y_val)
x_train, y_train, x_val, y_val = [],[],[],[]
print("Complete")
#Settings and prepare your dataset
import glob
import os
import sys
import keras
from keras import layers, models, optimizers
from keras.utils import np_utils
import keras.backend as K
import keras.layers as KL
import tensorflow as tf
from keras.preprocessing.image import load_img, img_to_array, array_to_img
from keras.preprocessing.image import random_rotation, random_shift, random_zoom
import numpy as np
import random
import matplotlib.pyplot as plt
import PIL
from PIL import Image
import cv2
from pathlib import Path
import shutil
from sklearn.model_selection import train_test_split
import mylib.makedataset_rgb as mkdataset
import mylib.create_panel as create_panel
import mylib.utils as myutils
# ------ Setting panels ------
import tkinter
from tkinter import messagebox
img_size_mynet = [28,28]# You can change input image size(Pay attention to network shape)
setting_panel = create_panel.CreatePanel(img_size_mynet = img_size_mynet)
setting_panel.create_buttons()#If you push "start", exit this line.
# ------ set params and preparing dataset ------
flag_train = setting_panel.flag_train
flag_aug = setting_panel.flag_aug
flag_split = setting_panel.flag_split
ratio_train = float(setting_panel.var_sp.get())#0.0 ~ 1.0
total_epochs = int(setting_panel.var_sp_epochs.get())
type_backbone = setting_panel.type_backbone#ex) ResNet, Mobilenet, MyNet
layer_name_gradcam = setting_panel.layer_name_gradcam# Don't use
img_size = setting_panel.img_size#ex) ResNet:[224,224], Mobilenet:[192,192], MyNet:[28,28]
print(type_backbone)
print("img_size=" + str(img_size))
#How many classes are in "dataset" folder
categories = [i for i in os.listdir(os.getcwd().replace("/mylib", "") + "/dataset")]
categories_idx = {}#ex) HookWrench:0, SpannerWrench:1
for i, name in enumerate(categories):
categories_idx[name] = i
nb_classes = len(categories)#ex) nb_classes=2
dirname_dataset = "dataset"# dataset folder
dirname_dataset_val = dirname_dataset + "_val"
output_folder = "outputs_keras/"+type_backbone
x_train, y_train, x_val, y_val = [],[],[],[]
def aug_dataset(dirname_dataset_1, dirname_dataset_val_1):
'''
This function returns updated dataset dirname
Contain MakeDataSetRGB() (mylib/makedataset_rgb.py)
Argument1: Foldername (String), Argument2: Foldername (String)
Usage:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
'''
dirname_dataset_aug = dirname_dataset_1 + "_aug"
dirname_dataset_val_aug = dirname_dataset_val_1 + "_aug"
make_dataset = mkdataset.MakeDataSetRGB()
if os.path.exists(dirname_dataset_aug ) == True \
or os.path.exists(dirname_dataset_val_aug ) == True:
#https://pythonbasics.org/tkinter-messagebox/
tki2 = tkinter.Tk()
tki2.withdraw()
ret = messagebox.askyesno('Bestätigung', '_Es gibt einen Aug-Ordner._Möchten Sie den aug-Ordner wirklich löschen?')
if ret == True:
if os.path.exists(dirname_dataset_aug ) == True:
shutil.rmtree(dirname_dataset_aug)
if os.path.exists(dirname_dataset_val_aug ) == True:
shutil.rmtree(dirname_dataset_val_aug)
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
tki2.destroy()
else:
tki2.destroy()
tki2.mainloop()
else:
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
dirname_dataset_2 = dirname_dataset_1 + "_aug"
dirname_dataset_val_2 = dirname_dataset_val_1 + "_aug"
return dirname_dataset_2, dirname_dataset_val_2
def prepare_dataset(dirname_dataset, dirname_dataset_val):
label = 0
for j in categories:# Prepare Training Dataset
files = glob.glob(dirname_dataset + "\\" + str(j) + "/*")
for imgfile in files:
img = load_img(imgfile, target_size=(img_size[0], img_size[1]))
array = img_to_array(img) / 255
x_train.append(array)
y_train.append(label)
label += 1
label = 0
for j in categories:# Prepare Validation Dataset
files = glob.glob(dirname_dataset_val + "\\" + str(j) + "/*")
for imgfile in files:
img = load_img(imgfile, target_size=(img_size[0], img_size[1]))
array = img_to_array(img) / 255
x_val.append(array)
y_val.append(label)
label += 1
if flag_train == True:
print("train mode")
print("total epochs = " + str(total_epochs))
if flag_split == True:
revert_dataset_val()
prepare_dataset_val()
print("splitting complete")
if flag_split == False and os.path.exists(dirname_dataset_val) == False:
prepare_dataset_val()
print("You have not splitted dataset, so splitteing automatically done")
if flag_aug == True:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
print("dataset source is " + dirname_dataset + "&" + dirname_dataset_val)
elif flag_aug == False:
dirname_dataset_aug = dirname_dataset + "_aug"
dirname_dataset_val_aug = dirname_dataset_val + "_aug"
make_dataset = mkdataset.MakeDataSetRGB()
if os.path.exists(dirname_dataset_aug ) == True \
and os.path.exists(dirname_dataset_val_aug ) == True:
dirname_dataset = dirname_dataset_aug
dirname_dataset_val = dirname_dataset_val_aug
prepare_dataset(dirname_dataset, dirname_dataset_val)
# make directory (weights_folder, outputs)
if os.path.exists("weights_pytorch/"+type_backbone) == False:
os.makedirs("weights_pytorch/"+type_backbone)
if os.path.exists("outputs_pytorch/"+type_backbone) == False:
os.makedirs("outputs_pytorch/"+type_backbone)
if os.path.exists(output_folder) == False:
os.makedirs(output_folder)
# In Keras, use numpy array for NN model
if os.path.exists("tmp_npy") == False:
os.makedirs("tmp_npy")
x_train, y_train, x_val, y_val = np.array(x_train), np.array(y_train), np.array(x_val), np.array(y_val)
np.save("tmp_npy/x_train.npy", x_train)
np.save("tmp_npy/y_train.npy", y_train)
np.save("tmp_npy/x_test.npy", x_val)
np.save("tmp_npy/y_test.npy", y_val)
x_train, y_train, x_val, y_val = [],[],[],[]
print("Complete")
Im Folgenden sind insbesondere die Hauptteile des Datenlesens aufgeführt.
for imgfile in files:
img = load_img(imgfile, target_size=(img_size[0], img_size[1]))
array = img_to_array(img) / 255
x_train.append(array)
y_train.append(label)
# ~ x_val, y_Gleiches gilt für val
So laden Sie die Daten in das Modell. Verwenden Sie beim Klassifizieren np_utils.to_categorical, um die Form des Arrays zu transformieren (z. B. One Hot Label oder 1ofK-Vektor).
y_train1=np_utils.to_categorical(y_train,nb_classes)
y_val1=np_utils.to_categorical(y_val,nb_classes)
history = model.fit(x_train,y_train1,epochs=total_epochs, callbacks = [cp_callback],batch_size=32,validation_data=(x_val,y_val1))
PyTorch-Datensatzvorbereitung
In PyTorch ist das Lesen eines Datensatzes mit ImageFolder usw., das Teilen des Zugwerts mit train_test_split usw. von scicit-learn und das Verwenden von DataLoader zum Sammeln von Trainingsdaten und Etikettenpaaren in Stapeleinheiten eine der Datensatzvorbereitungen. Es gibt zwei Möglichkeiten.
PyTorch-Dataset-Vorbereitung (klicken Sie hier, um das Programm anzuzeigen)
#Settings and prepare your dataset
import glob
import os
import sys
import numpy as np
import random
import matplotlib.pyplot as plt
import PIL
from PIL import Image
import cv2
import torch
import torchvision.transforms as transforms
from pathlib import Path
from torch.utils.data import DataLoader, Dataset
from torchvision.datasets import ImageFolder
from sklearn.model_selection import train_test_split
import shutil
import mylib.makedataset_rgb as mkdataset
import mylib.create_panel as create_panel
import mylib.utils as myutils
# ----- Setting buttons -----
import tkinter
from tkinter import messagebox
img_size_mynet = [28,28]# You can change input image size(Pay attention to network shape)
setting_panel = create_panel.CreatePanel(img_size_mynet = img_size_mynet)
setting_panel.create_buttons()#If you push "start", exit this line.
# ----- set params and preparing dataset -----
flag_train = setting_panel.flag_train
flag_aug = setting_panel.flag_aug
flag_split = setting_panel.flag_split
ratio_train = float(setting_panel.var_sp.get())#0.0 ~ 1.0
total_epochs = int(setting_panel.var_sp_epochs.get())
type_backbone = setting_panel.type_backbone
layer_name_gradcam = setting_panel.layer_name_gradcam
img_size = setting_panel.img_size
print(type_backbone)
print("img_size=" + str(img_size))
#How many classes are in "dataset" folder
categories = [i for i in os.listdir(os.getcwd().replace("/mylib", "") + "/dataset")]
nb_classes = len(categories)#ex) nb_classes=2
dirname_dataset = "dataset"# dataset folder
dirname_dataset_val = dirname_dataset + "_val"
output_folder = "outputs_pytorch/"+type_backbone
def aug_dataset(dirname_dataset_1, dirname_dataset_val_1):
'''
This function returns updated dataset dirname
Contain MakeDataSetRGB() (mylib/makedataset_rgb.py)
Argument1: Foldername (String), Argument2: Foldername (String)
Usage:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
'''
dirname_dataset_aug = dirname_dataset_1 + "_aug"
dirname_dataset_val_aug = dirname_dataset_val_1 + "_aug"
make_dataset = mkdataset.MakeDataSetRGB(do_reverse=True,
do_gamma_correction=True,
do_add_noise=True,
do_cut_out=True,
do_deformation=True )
if os.path.exists(dirname_dataset_aug ) == True \
or os.path.exists(dirname_dataset_val_aug ) == True:
#https://pythonbasics.org/tkinter-messagebox/
tki2 = tkinter.Tk()
tki2.withdraw()
ret = messagebox.askyesno('Bestätigung', '_Es gibt einen Aug-Ordner._Möchten Sie den aug-Ordner wirklich löschen?')
if ret == True:
if os.path.exists(dirname_dataset_aug ) == True:
shutil.rmtree(dirname_dataset_aug)
if os.path.exists(dirname_dataset_val_aug ) == True:
shutil.rmtree(dirname_dataset_val_aug)
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
tki2.destroy()
else:
tki2.destroy()
tki2.mainloop()
else:
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
dirname_dataset_2 = dirname_dataset_1 + "_aug"
dirname_dataset_val_2 = dirname_dataset_val_1 + "_aug"
return dirname_dataset_2, dirname_dataset_val_2
def prepare_dataset_val():
for j in categories:
if os.path.exists(dirname_dataset_val + "\\" + str(j) ) == False:
os.makedirs(dirname_dataset_val + "\\" + str(j))
files = glob.glob(dirname_dataset + "\\" + str(j) + "/*")
for imgfile in files:# move some data from "dataset" to "dataset_val"
if myutils.train_or_val(ratio_train) == "val":
shutil.move(imgfile, dirname_dataset_val+"\\" + str(j) + "/")
def revert_dataset_val():
'''
Revert Dataset
This function revert splitted validation dataset directory to dataset directory
'''
for j in categories:
if os.path.exists(dirname_dataset_val + "\\" + str(j) ) == True:
files = glob.glob(dirname_dataset_val + "\\" + str(j) + "/*")
for imgfile in files:#Move all images in "dataset_val" to "dataset"
shutil.move(imgfile, dirname_dataset + "\\" + str(j))
if os.path.exists(dirname_dataset_val) == True:
shutil.rmtree(dirname_dataset_val)#Delete "dataset_val" folder
transform = transforms.Compose([transforms.Resize((img_size[0], img_size[1])), transforms.ToTensor()])
train_loader = []
test_loader = []
def prepare_dataset(transform, dirname_dataset, dirname_dataset_val):
dataset = ImageFolder(dirname_dataset, transform)# Prepare Training Dataset
dataset_val = ImageFolder(dirname_dataset_val, transform)# Prepare Validation Dataset
print(dataset.class_to_idx)
return dataset, dataset_val
batch_size_train = 32
batch_size_val = 16
def get_device(gpu_id=-1):
global batch_size_train, batch_size_val
if gpu_id >= 0 and torch.cuda.is_available():
print("GPU mode")
batch_size_train = 32
batch_size_val = 16
return torch.device("cuda", gpu_id)
else:
return torch.device("cpu")
device = get_device(gpu_id=0)
if flag_train == True:
print("train mode")
print("total epochs = " + str(total_epochs))
if flag_split == True:
revert_dataset_val()
prepare_dataset_val()
print("splitting complete")
elif flag_split == False and os.path.exists(dirname_dataset_val) == False:
prepare_dataset_val()
print("You have not splitted dataset, so splitteing automatically done")
if flag_aug == True:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
print("dataset source is " + dirname_dataset + "&" + dirname_dataset_val)
elif flag_aug == False:
dirname_dataset_aug = dirname_dataset + "_aug"
dirname_dataset_val_aug = dirname_dataset_val + "_aug"
make_dataset = mkdataset.MakeDataSetRGB()
if os.path.exists(dirname_dataset_aug ) == True \
and os.path.exists(dirname_dataset_val_aug ) == True:
dirname_dataset = dirname_dataset_aug
dirname_dataset_val = dirname_dataset_val_aug
#prepare_dataset()
train_data, test_data = prepare_dataset(transform, dirname_dataset, dirname_dataset_val)
# make directory (weights_folder, outputs)
if os.path.exists("weights_pytorch/"+type_backbone) == False:
os.makedirs("weights_pytorch/"+type_backbone)
if os.path.exists("outputs_pytorch/"+type_backbone) == False:
os.makedirs("outputs_pytorch/"+type_backbone)
# In PyTorch, use DataLoader for NN model
train_loader = DataLoader(train_data, batch_size=batch_size_train, shuffle=True)
test_loader = DataLoader(test_data, batch_size=batch_size_val, shuffle=True)
if os.path.exists(output_folder) == False:
os.makedirs(output_folder)
print("Complete")
#Settings and prepare your dataset
import glob
import os
import sys
import numpy as np
import random
import matplotlib.pyplot as plt
import PIL
from PIL import Image
import cv2
import torch
import torchvision.transforms as transforms
from pathlib import Path
from torch.utils.data import DataLoader, Dataset
from torchvision.datasets import ImageFolder
from sklearn.model_selection import train_test_split
import shutil
import mylib.makedataset_rgb as mkdataset
import mylib.create_panel as create_panel
import mylib.utils as myutils
# ----- Setting buttons -----
import tkinter
from tkinter import messagebox
img_size_mynet = [28,28]# You can change input image size(Pay attention to network shape)
setting_panel = create_panel.CreatePanel(img_size_mynet = img_size_mynet)
setting_panel.create_buttons()#If you push "start", exit this line.
# ----- set params and preparing dataset -----
flag_train = setting_panel.flag_train
flag_aug = setting_panel.flag_aug
flag_split = setting_panel.flag_split
ratio_train = float(setting_panel.var_sp.get())#0.0 ~ 1.0
total_epochs = int(setting_panel.var_sp_epochs.get())
type_backbone = setting_panel.type_backbone
layer_name_gradcam = setting_panel.layer_name_gradcam
img_size = setting_panel.img_size
print(type_backbone)
print("img_size=" + str(img_size))
#How many classes are in "dataset" folder
categories = [i for i in os.listdir(os.getcwd().replace("/mylib", "") + "/dataset")]
nb_classes = len(categories)#ex) nb_classes=2
dirname_dataset = "dataset"# dataset folder
dirname_dataset_val = dirname_dataset + "_val"
output_folder = "outputs_pytorch/"+type_backbone
def aug_dataset(dirname_dataset_1, dirname_dataset_val_1):
'''
This function returns updated dataset dirname
Contain MakeDataSetRGB() (mylib/makedataset_rgb.py)
Argument1: Foldername (String), Argument2: Foldername (String)
Usage:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
'''
dirname_dataset_aug = dirname_dataset_1 + "_aug"
dirname_dataset_val_aug = dirname_dataset_val_1 + "_aug"
make_dataset = mkdataset.MakeDataSetRGB(do_reverse=True,
do_gamma_correction=True,
do_add_noise=True,
do_cut_out=True,
do_deformation=True )
if os.path.exists(dirname_dataset_aug ) == True \
or os.path.exists(dirname_dataset_val_aug ) == True:
#https://pythonbasics.org/tkinter-messagebox/
tki2 = tkinter.Tk()
tki2.withdraw()
ret = messagebox.askyesno('Bestätigung', '_Es gibt einen Aug-Ordner._Möchten Sie den aug-Ordner wirklich löschen?')
if ret == True:
if os.path.exists(dirname_dataset_aug ) == True:
shutil.rmtree(dirname_dataset_aug)
if os.path.exists(dirname_dataset_val_aug ) == True:
shutil.rmtree(dirname_dataset_val_aug)
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
tki2.destroy()
else:
tki2.destroy()
tki2.mainloop()
else:
make_dataset.do_augmentation(dataset_folder_name = "dataset")
make_dataset.do_augmentation(dataset_folder_name = "dataset_val")
dirname_dataset_2 = dirname_dataset_1 + "_aug"
dirname_dataset_val_2 = dirname_dataset_val_1 + "_aug"
return dirname_dataset_2, dirname_dataset_val_2
def prepare_dataset_val():
for j in categories:
if os.path.exists(dirname_dataset_val + "\\" + str(j) ) == False:
os.makedirs(dirname_dataset_val + "\\" + str(j))
files = glob.glob(dirname_dataset + "\\" + str(j) + "/*")
for imgfile in files:# move some data from "dataset" to "dataset_val"
if myutils.train_or_val(ratio_train) == "val":
shutil.move(imgfile, dirname_dataset_val+"\\" + str(j) + "/")
def revert_dataset_val():
'''
Revert Dataset
This function revert splitted validation dataset directory to dataset directory
'''
for j in categories:
if os.path.exists(dirname_dataset_val + "\\" + str(j) ) == True:
files = glob.glob(dirname_dataset_val + "\\" + str(j) + "/*")
for imgfile in files:#Move all images in "dataset_val" to "dataset"
shutil.move(imgfile, dirname_dataset + "\\" + str(j))
if os.path.exists(dirname_dataset_val) == True:
shutil.rmtree(dirname_dataset_val)#Delete "dataset_val" folder
transform = transforms.Compose([transforms.Resize((img_size[0], img_size[1])), transforms.ToTensor()])
train_loader = []
test_loader = []
def prepare_dataset(transform, dirname_dataset, dirname_dataset_val):
dataset = ImageFolder(dirname_dataset, transform)# Prepare Training Dataset
dataset_val = ImageFolder(dirname_dataset_val, transform)# Prepare Validation Dataset
print(dataset.class_to_idx)
return dataset, dataset_val
batch_size_train = 32
batch_size_val = 16
def get_device(gpu_id=-1):
global batch_size_train, batch_size_val
if gpu_id >= 0 and torch.cuda.is_available():
print("GPU mode")
batch_size_train = 32
batch_size_val = 16
return torch.device("cuda", gpu_id)
else:
return torch.device("cpu")
device = get_device(gpu_id=0)
if flag_train == True:
print("train mode")
print("total epochs = " + str(total_epochs))
if flag_split == True:
revert_dataset_val()
prepare_dataset_val()
print("splitting complete")
elif flag_split == False and os.path.exists(dirname_dataset_val) == False:
prepare_dataset_val()
print("You have not splitted dataset, so splitteing automatically done")
if flag_aug == True:
dirname_dataset, dirname_dataset_val = aug_dataset(dirname_dataset, dirname_dataset_val)
print("dataset source is " + dirname_dataset + "&" + dirname_dataset_val)
elif flag_aug == False:
dirname_dataset_aug = dirname_dataset + "_aug"
dirname_dataset_val_aug = dirname_dataset_val + "_aug"
make_dataset = mkdataset.MakeDataSetRGB()
if os.path.exists(dirname_dataset_aug ) == True \
and os.path.exists(dirname_dataset_val_aug ) == True:
dirname_dataset = dirname_dataset_aug
dirname_dataset_val = dirname_dataset_val_aug
#prepare_dataset()
train_data, test_data = prepare_dataset(transform, dirname_dataset, dirname_dataset_val)
# make directory (weights_folder, outputs)
if os.path.exists("weights_pytorch/"+type_backbone) == False:
os.makedirs("weights_pytorch/"+type_backbone)
if os.path.exists("outputs_pytorch/"+type_backbone) == False:
os.makedirs("outputs_pytorch/"+type_backbone)
# In PyTorch, use DataLoader for NN model
train_loader = DataLoader(train_data, batch_size=batch_size_train, shuffle=True)
test_loader = DataLoader(test_data, batch_size=batch_size_val, shuffle=True)
if os.path.exists(output_folder) == False:
os.makedirs(output_folder)
print("Complete")
--DataSet: Gibt ein Datenpaar und die richtige Bezeichnung zurück --DataLoader: Ermöglicht das Abrufen von Daten in einem Mini-Batch
def prepare_dataset(transform, dirname_dataset, dirname_dataset_val):
dataset = ImageFolder(dirname_dataset, transform)# Prepare Training Dataset
dataset_val = ImageFolder(dirname_dataset_val, transform)# Prepare Validation Dataset
print(dataset.class_to_idx)
return dataset, dataset_val
#~Abkürzung~
train_data, test_data = prepare_dataset(transform, dirname_dataset, dirname_dataset_val)
#~Abkürzung~
# In PyTorch, use DataLoader for NN model
train_loader = DataLoader(train_data, batch_size=batch_size_train, shuffle=True)
test_loader = DataLoader(test_data, batch_size=batch_size_val, shuffle=True)
In Zukunft werde ich einen Artikel veröffentlichen, in dem das Lernen damit detailliert beschrieben wird. Die Methode zum Einfügen von Daten in das Modell ist jedoch wie folgt.
for batch_idx, (image, label) in enumerate(train_loader):
#image, label = Variable(image), Variable(label)#cpu
image, label = Variable(image).cuda(), Variable(label).cuda()
optimizer.zero_grad()
output = model(image)
Über die Anwendung
Dieses Programm wird in einem späteren Artikel über Keras VS PyTorch verwendet. Das Datenerweiterungsprogramm (makedataset_rgb.py) und das UI-Panel-Programm (create_panel.py) sind in mylib unterteilt, daher hoffe ich, dass es hilft.
Referenz
https://pytorch.org/docs/stable/data.html https://qiita.com/mathlive/items/2a512831878b8018db02 https://qiita.com/takurooo/items/e4c91c5d78059f92e76d
Recommended Posts