Precautions when inputting from CSV with Python and outputting to json to make it an exe

As the title says, I will write a memorandum of notes when inputting from CSV with Python and outputting to json to make it an exe. The implementation example is a completely personal tool, so I'll just give you an overview. The important thing is the error message and the remedy.

CSV causing a decoding error

First, load csv. Prepare the data in csv called ʻinput.csv` and read it.

import csv

csv_file = open("./input.csv", "r")
c = csv.DictReader(csv_file)

Suddenly, when I try to look at the contents of csv here, I get the following error (probably).

Exception has occurred: UnicodeDecodeError
'cp932' codec can't decode byte 0xef in position 0: illegal multibyte sequence
  File "C:\Users\hoge\fuga.py", line 19, in <module>
    for row in c:

You will be scolded for not being able to decode. This is because python tries to make the ʻutf-8fileshift-jis`. It needs to be modified as follows.

csv_file = open("input.csv", "r",encoding="utf-8-sig")

All the same data json

The part that was the most annoying this time.

#Excerpt of only parameters from csv
params = []
for row in c:
    points.append(row)
n = len(params)

#Read json and secure the parameter frame
import json
json_open = open('sample.json', 'r')
j = json.load(json_open)
for i in range(n-1):
    j["Movements"].append(j["Movements"][0])

I did something awkward in the bottom row, which was the source of the trouble, but sample.json has a data template, which is extended to the number of parameter lines n. Therefore, I took the method of copying the template n-1 times. As a result, n frames have been successfully created in the Movements key of the json data j, so after that, we will just turn the for statement and insert the data.

…… But after I finished putting in the data, I was troubled by the mysterious result that ** for some reason all the rows have the same data **.

After a bit of debugging, I found that every input rewrites every row, which means that ** every row references the same data **. Even if the apparent data is in the n line, it only refers to one data, and if you modify any one of them, all the data will be affected.

Why did this happen?

For types like ʻintandstr, even if you copy a variable with = , it will be considered as" another variable "and assigned an ID when you make any changes, but list If you edit the variable after copying the variable with = (actually, it could not be copied), the original data will be edited. I knew it as knowledge, but so was dict. The only solution is to copy the variables using a technique called hard copy.

#Excerpt of only parameters from csv
params = []
for row in c:
    points.append(row)
n = len(params)

#Read json and secure the parameter frame
import json
import copy
json_open = open('sample.json', 'r')
j = json.load(json_open)
for i in range(n-1):
    new_j = copy.deepcopy(j["Movements"][0])
    j["Movements"].append(new_j)

In this case, it had to be deepcopy (deep copy) instead of justcopy (shallow copy).

Exe that doesn't work

There is a module called pyinstaller that makes it an exe so that even people who do not have Python can run it. When I tried to pip this, I couldn't install it due to an error, and if I went around, I should downgrade pip (20 → 18), so I succeeded. When I tried to reproduce it now, for some reason even the latest pip succeeded, so the error cannot be reproduced. I'm sorry.

So I made it into an exe using pyinstaller safely, but even if I execute it, it ends immediately. It doesn't even spit out the first print that I put in for debugging, so it seems that there is a problem at the time of ʻimport`. So, I will hunt witches one by one for the following modules that I actually put in.

import json
import pprint
import csv
import cmath
import math
import numpy as np
import copy
import os

As a result, it was caused by numpy. Fortunately, I didn't use the part that needed numpy in the program (I used np.pi, but I don't need it anymore), so I just deleted the module. If you have to exe a program that uses numpy to calculate rigorously, think again ...

Summary

I'm sorry that there are many contents that "I managed to do it, but I didn't understand", but I will forget more and more if I do not accumulate such things, so I wrote it for myself. I've been afraid to skip this kind of thing until now, so from now on I'll actively write program records like this diary.

Code

It's a completely personal tool, so I don't think it's useful to anyone, but for the time being.

import json
import pprint
import csv
import cmath
import math
import copy
import os

json_open = open('./sample.json', 'r')
j = json.load(json_open)

print("sample.loaded json")

csv_file = open("./input.csv", "r",encoding="utf-8-sig")
c = csv.DictReader(csv_file)

points = []

for row in c:
    points.append(row)

n = len(points)

for i in range(n-1):
    new_j = copy.deepcopy(j["Movements"][0])
    j["Movements"].append(new_j)

for i,p in enumerate(points):
    x = float(p['X'])
    y = float(p['Y'])
    z = float(p['Z'])
    duration = float(p['Duration'])
    j["Movements"][i]["Duration"] = duration
    j["Movements"][i]["StartPos"]["x"] = x
    j["Movements"][i]["StartPos"]["y"] = y
    j["Movements"][i]["StartPos"]["z"] = z
    j["Movements"][i-1]["EndPos"]["x"] = x
    j["Movements"][i-1]["EndPos"]["y"] = y
    j["Movements"][i-1]["EndPos"]["z"] = z
    y -= 1.5
    c = complex(-x,-z)
    rad = cmath.phase(c)
    deg = -math.degrees(rad)+90
    #print(deg)
    j["Movements"][i]["StartRot"]["y"] = deg 
    j["Movements"][i-1]["EndRot"]["y"] = deg 
    c2 = complex(abs(c),y)
    rad2 = cmath.phase(c2)
    deg2 = math.degrees(rad2)
    #print(-deg2)
    j["Movements"][i]["StartRot"]["x"] = deg2 
    j["Movements"][i-1]["EndRot"]["x"] = deg2

pprint.pprint(j)

with open('output.json', 'w') as f:
    json.dump(j, f, indent=4)

print("Completed the work")
end = input()