After successfully reading each of the three files, combine the three files measureddata and samples so that they can be used later in the process. At this time, it is convenient to use the Pandas concat function. If the parameter ignore_index is set to True, the column numbers of the Pandas DataFrame ^{5 </ sup> to be joined will be renumbered. </ p>}

data_file1 = "measureddata_20191004033314.tsv"    #Change the string entered in the variable to the tsv file name of your measured data
data_file2 = "measureddata_20191004033325.tsv"    #Change the string entered in the variable to the tsv file name of your measured data
data_file3 = "measureddata_20191004033338.tsv"    #Change the string entered in the variable to the tsv file name of your measured data

data1 = pd.read_csv(data_file1, delimiter="\t")
data2 = pd.read_csv(data_file2, delimiter="\t")
data3 = pd.read_csv(data_file3, delimiter="\t")

data = pd.concat([data1, data2, data3], ignore_index=True)    #Combine with Pandas concat function
data

9779 rows × 13 columns

sample_file1 = "samples_20191004033311.tsv"    #Change the string entered in the variable to the tsv file name of your samples
sample_file2 = "samples_20191004033323.tsv"    #Change the string entered in the variable to the tsv file name of your samples
sample_file3 = "samples_20191004033334.tsv"    #Change the string entered in the variable to the tsv file name of your samples

sample1 = pd.read_csv(sample_file1, delimiter="\t")    
sample2 = pd.read_csv(sample_file2, delimiter="\t")
sample3 = pd.read_csv(sample_file3, delimiter="\t")

sample = pd.concat([sample1, sample2, sample3], ignore_index=True)    #Combine with Pandas concat function
sample

317 rows × 66 columns

Chap.3 Data preparation

After reading the data, the next step is to prepare the data so that it can be analyzed. First, start by extracting the data to be used this time from measured data.
This time, we will use 5 kinds of chemical substances (ΣPCBs, ΣDDTs, ΣPBDEs, ΣCHLs, ΣHCHs) for 3 kinds of finless porpoise, striped dolphin, and melon-headed whale.

First, extract the data whose unit is [ng / g lipid]. This is because, as mentioned in Part 1, data with different units cannot be simply compared.
Next, extract the data whose chemical substance name is one of the above five types, and use the reset_index method to renumber the lines.

data_lipid = data[(data["Unit"] == "ng/g lipid")]
data_lipid = data_lipid[(data_lipid["ChemicalName"] == "ΣPCBs") | (data_lipid["ChemicalName"] == "ΣDDTs") |
                        (data_lipid["ChemicalName"] == "ΣPBDEs") |  (data_lipid["ChemicalName"] == "ΣCHLs") |
                        (data_lipid["ChemicalName"] == "ΣHCHs")]
data_lipid = data_lipid.reset_index(drop=True)
data_lipid[data_lipid["SampleID"] == "SAA001941"]    #Data list with Sample ID SAA001941

After extracting the measured data data, the next step is to integrate the samples and the measured data Data Frame.
In Part1 and Part2, sample information (scientific name, collection site, etc.) was added to the measured data of each chemical substance, but this time, the measured value of the chemical substance is given to each sample, so be careful that the processing is different from before. ..

#Create a DataFrame with only SampleID and scientific name
df = sample[["SampleID", "ScientificName"]]
for chem in ["ΣCHLs", "ΣDDTs", "ΣHCHs", "ΣPBDEs", "ΣPCBs"]: #Loop processing by chemical substance name
    #Create a DataFrame for only the chemicals in the chemth list
    data_lipid_chem = data_lipid[data_lipid["ChemicalName"] == chem]
    #Column name, leaving only the SampleID and measurements"MeasuredValue"Changed to chemical substance name
    data_lipid_chem = data_lipid_chem[["SampleID", "MeasuredValue"]].rename(columns={"MeasuredValue": chem})
    data_lipid_chem = data_lipid_chem.drop_duplicates(subset='SampleID')
    #Df with SampleID(ID+Scientific name)And data_lipid_chem(ID+measured value)Merge
    df = pd.merge(df, data_lipid_chem, on="SampleID")
df = df.dropna(how="any") #Remove rows containing NaN
df = df.drop("SampleID", axis=1)
df

94 rows × 6 columns

As shown above

, the scientific name is stored in the DataFrame as it is in the current state. Since the species name will be used for visualization in the future, a variable representing the scientific name will be prepared for convenience. First, create a dictionary that associates each scientific name with a value (finless porpoise: 0, striped dolphin: 1, melon-headed whale: 2). </ p>

species = df["ScientificName"].unique().tolist()    #Output a list of scientific names in df
class_dic = {}
for i in range(len(species)):
    class_dic[species[i]] = i
class_dic

{'Neophocaena phocaenoides': 0,
 'Stenella coeruleoalba': 1,
 'Peponocephala electra': 2}

Next, add the Class column to df and enter the value corresponding to the scientific name in each row in the Class column. ```python df["Class"] = 0 for irow in range(len(df)): df.at[irow, "Class"] = class_dic[df.at[irow, "ScientificName"]] df = df.loc[:, ["Class", "ScientificName", "ΣPCBs", "ΣDDTs", "ΣPBDEs", "ΣCHLs", "ΣHCHs"]] df ```

94 rows × 7 columns

Since it was easier to use the pivot table, modify the data_lipid and below as follows.

df = pd.pivot_table(data_lipid, index=['ScientificName', 'SampleID'], columns='ChemicalName', values=['MeasuredValue'])
df = df.dropna(how="any") #Remove rows containing NaN
df

94 rows × 5 columns

From this point onward, analysis is performed using the data frame of the pivot table.

Chap.4 Visualization of data overview

Now that the data preparation is complete, let's visualize and check the entire data for the time being. The data handled this time is five-dimensional (there are five explanatory variables). Since humans cannot grasp the five-dimensional space, it is impossible to visualize the distribution of all the five-dimensional data as it is. Here, a distribution map of $ {} _ 5 C_2 = 10 $ is generated.

It is convenient to use pandas' pandas.plotting.scatter_matrix () function to visualize the scatter plot of multiple variables. In the scatter_matrix () function, specify the range to be visualized with the frame parameter. This time, all rows and 2nd column of df ^{6 and after.}

pd.plotting.scatter_matrix(
    frame=df["MeasuredValue"],
    c=list(df["MeasuredValue"].iloc[:, 0]),
    alpha=0.5)
plt.show()

Chap.5 Principal component analysis

In

Chap.4, a distribution map of $ {} _ 5 C_2 = 10 $ was generated as an overview of the entire data. However, with this, only the distribution of data for each combination of two types of chemical substances can be confirmed. In other words, it is not possible to get an overview of the entire 5-dimensional data. Therefore, the five-dimensional data is summarized and visualized by principal component analysis. </ p>

First, since the length of each axis is different as it is, standardize ^{7 for each variable and set the length of all axes in 5 dimensions. Unify.}

dfs = df.apply(lambda x: (x-x.mean())/x.std(), axis=0)
dfs

94 rows × 5 columns

Alternatively, you can use the Standard Scaler for standardization.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(df)
scaler.transform(df)
dfs = pd.DataFrame(scaler.transform(df), columns=df.columns)
dfs

94 rows × 5 columns

After standardization, perform principal component analysis. Here, the result of Standard Scaler is used.
For principal component analysis, scikit-learn's sklearn.decomposition.PCA () is used.
In principal component analysis, first, a synthetic variable (first principal component) is created so that the axis is drawn in the direction that maximizes the variation in the distribution of the data, and then the second principal component, which complements the data lost by the composite variable. Create the third principal component and synthetic variables. After that, if the data is transferred to the space centered on each principal component, the rest is only interpreted.

pca = PCA()
feature = pca.fit(dfs)    #Principal component analysis
feature = pca.transform(dfs)   #Transfer data to principal component space

pd.DataFrame(feature, columns=["PC{}".format(x + 1) for x in range(len(dfs.columns))])

94 rows × 5 columns

Let's also find the eigenvectors. The eigenvector is like the "force to tilt the axis" applied when transferring the original data to the principal component axis space.

Calculation of eigenvectors in Python is implemented in components_ included in PCA of scikit-learn.

pd.DataFrame(pca.components_, columns=df.columns[2:], index=["PC{}".format(x + 1) for x in range(len(dfs.columns))])

Now that we know the eigenvectors, we will find the eigenvalues of each principal component. The eigenvalue is a coefficient associated with the eigenvector and is an index indicating the size of each principal component axis.

In Python, the contribution rate can be calculated by using explained_variance_ included in the PCA of scikit-learn.

pd.DataFrame(pca.explained_variance_, index=["PC{}".format(x + 1) for x in range(len(dfs.columns))])

Next, find the contribution rate. The contribution rate is an index showing how much information each principal component axis explains in the data. In other words, the main component with a contribution rate of 60% can explain 60% of the total data, and 40% is lost.

In Python, the contribution rate can be calculated by using explained_variance_ratio_ included in the PCA of scikit-learn.

pd.DataFrame(pca.explained_variance_ratio_, index=["PC{}".format(x + 1) for x in range(len(dfs.columns))])

Here, the cumulative contribution rate is also calculated. The reason why the cumulative contribution rate is necessary is to see "what percentage of data is lost". In other words, if you look at the main components that have a cumulative contribution rate above a certain level, you can reduce the number of dimensions and get an overview of the entire data.

Here, the cumulative total value is calculated using the cumsum function of numpy.

plt.plot([0] + list( np.cumsum(pca.explained_variance_ratio_)), "-o")
plt.xlabel("Number of principal components")
plt.ylabel("Cumulative contribution rate")
plt.grid()
plt.show()

Regarding each generated principal component axis, I'm not sure what it means as it is. However, by visualizing the contribution of each variable in the principal component, the meaning of each principal component can be seen. Here, we plot the contribution of each variable in the first and second principal components on a scatter plot. ```python plt.figure() for x, y, name in zip(pca.components_[0], pca.components_[1], df["MeasuredValue"].columns[:]): plt.text(x, y, name) plt.scatter(pca.components_[0], pca.components_[1], alpha=0.8) plt.grid() plt.xlabel("PC1") plt.ylabel("PC2") plt.show() ```

Finally, visualize the result of principal component analysis. Generally, the eigenvalues of 1 or more or the cumulative contribution rate of 80% or more are adopted, and the distribution is examined by the combination of the adopted principal components. In this case, it is assumed that up to the second main component is adopted and visualized in the graph. As with a general scatter plot, you can use the scatter function of matplotlib and specify the first principal component (0th column of feature) and the second principal component (1st column of feature) for each of the parameters X and Y.

plt.figure()
plt.scatter(x=feature[:, 0], y=feature[:, 1], alpha=0.8, c=list(df.iloc[:, 0])) #Describe plot color coding in c
plt.grid()
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()

footnote

^{1 An explanatory variable is a variable that explains the objective variable that is the desired result. In other words, for the objective variable that indicates the result, the variable that causes it is the explanatory variable. Also an independent variable.}

^{2 A variable created by combining multiple variables. In principal component analysis, each principal component is a synthetic variable.}

^{3 Software or libraries whose source code, which is the blueprint of a program, is widely disclosed to the general public. As long as it is within the range permitted by the license, everyone can use it free of charge. Also, since anyone can develop it, it is easy to update from the user's perspective and fix bugs that are not noticed by a small number of people.
Python, Jupyter Notebook, and Anaconda are all open source software.}

^{4 Both are methods used in machine learning. Classification is image classification, etc., and regression is used for future forecasts such as stock prices. Clustering is used to group similar data based on data trends.}

^{5 A data type that stores tabular data in Pandas. Manipulate data by specifying cells, rows, and columns as if you read a CSV or TSV format file in Excel.}

^{6 In python, numbers start with "0".}

^{7 The operation of subtracting the average value from the measured value and dividing by the standard deviation.}