[PYTHON] 100 Language Processing Knock 2020 Chapter 7: Word Vector

The other day, 100 Language Processing Knock 2020 was released. I myself have only been in natural language processing for a year, and I don't know the details, but I will solve all the problems and publish them in order to improve my technical skills.

All shall be executed on jupyter notebook, and the restrictions of the problem statement may be broken conveniently. The source code is also on github. Yes.

Chapter 6 is here.

The environment is Python 3.8.2 and Ubuntu 18.04.

Chapter 7: Word Vector

Create a program that performs the following processing for a word vector (word embedding) that expresses the meaning of a word as a real vector.

Please download the required dataset from here.

The downloaded file shall be placed under data.

60. Reading and displaying word vectors

Download the learned word vector (3 million words / phrases, 300 dimensions) in the Google News dataset (about 100 billion words) and display the word vector of "United States". However, note that "United States" is internally referred to as "United_States".

Use gensim.

code


from gensim.models import KeyedVectors

code


model = KeyedVectors.load_word2vec_format('data/GoogleNews-vectors-negative300.bin.gz', binary=True)

code


model["United_States"]

output


array([-3.61328125e-02, -4.83398438e-02,  2.35351562e-01,  1.74804688e-01,
       -1.46484375e-01, -7.42187500e-02, -1.01562500e-01, -7.71484375e-02,
        1.09375000e-01, -5.71289062e-02, -1.48437500e-01, -6.00585938e-02,
        1.74804688e-01, -7.71484375e-02,  2.58789062e-02, -7.66601562e-02,
       -3.80859375e-02,  1.35742188e-01,  3.75976562e-02, -4.19921875e-02,
       -3.56445312e-02,  5.34667969e-02,  3.68118286e-04, -1.66992188e-01,
       -1.17187500e-01,  1.41601562e-01, -1.69921875e-01, -6.49414062e-02,
       -1.66992188e-01,  1.00585938e-01,  1.15722656e-01, -2.18750000e-01,
       -9.86328125e-02, -2.56347656e-02,  1.23046875e-01, -3.54003906e-02,
       -1.58203125e-01, -1.60156250e-01,  2.94189453e-02,  8.15429688e-02,
        6.88476562e-02,  1.87500000e-01,  6.49414062e-02,  1.15234375e-01,
       -2.27050781e-02,  3.32031250e-01, -3.27148438e-02,  1.77734375e-01,
       -2.08007812e-01,  4.54101562e-02, -1.23901367e-02,  1.19628906e-01,
        7.44628906e-03, -9.03320312e-03,  1.14257812e-01,  1.69921875e-01,
       -2.38281250e-01, -2.79541016e-02, -1.21093750e-01,  2.47802734e-02,
        7.71484375e-02, -2.81982422e-02, -4.71191406e-02,  1.78222656e-02,
       -1.23046875e-01, -5.32226562e-02,  2.68554688e-02, -3.11279297e-02,
       -5.59082031e-02, -5.00488281e-02, -3.73535156e-02,  1.25976562e-01,
        5.61523438e-02,  1.51367188e-01,  4.29687500e-02, -2.08007812e-01,
       -4.78515625e-02,  2.78320312e-02,  1.81640625e-01,  2.20703125e-01,
       -3.61328125e-02, -8.39843750e-02, -3.69548798e-05, -9.52148438e-02,
       -1.25000000e-01, -1.95312500e-01, -1.50390625e-01, -4.15039062e-02,
        1.31835938e-01,  1.17675781e-01,  1.91650391e-02,  5.51757812e-02,
       -9.42382812e-02, -1.08886719e-01,  7.32421875e-02, -1.15234375e-01,
        8.93554688e-02, -1.40625000e-01,  1.45507812e-01,  4.49218750e-02,
       -1.10473633e-02, -1.62353516e-02,  4.05883789e-03,  3.75976562e-02,
       -6.98242188e-02, -5.46875000e-02,  2.17285156e-02, -9.47265625e-02,
        4.24804688e-02,  1.81884766e-02, -1.73339844e-02,  4.63867188e-02,
       -1.42578125e-01,  1.99218750e-01,  1.10839844e-01,  2.58789062e-02,
       -7.08007812e-02, -5.54199219e-02,  3.45703125e-01,  1.61132812e-01,
       -2.44140625e-01, -2.59765625e-01, -9.71679688e-02,  8.00781250e-02,
       -8.78906250e-02, -7.22656250e-02,  1.42578125e-01, -8.54492188e-02,
       -3.18359375e-01,  8.30078125e-02,  6.34765625e-02,  1.64062500e-01,
       -1.92382812e-01, -1.17675781e-01, -5.41992188e-02, -1.56250000e-01,
       -1.21582031e-01, -4.95605469e-02,  1.20117188e-01, -3.83300781e-02,
        5.51757812e-02, -8.97216797e-03,  4.32128906e-02,  6.93359375e-02,
        8.93554688e-02,  2.53906250e-01,  1.65039062e-01,  1.64062500e-01,
       -1.41601562e-01,  4.58984375e-02,  1.97265625e-01, -8.98437500e-02,
        3.90625000e-02, -1.51367188e-01, -8.60595703e-03, -1.17675781e-01,
       -1.97265625e-01, -1.12792969e-01,  1.29882812e-01,  1.96289062e-01,
        1.56402588e-03,  3.93066406e-02,  2.17773438e-01, -1.43554688e-01,
        6.03027344e-02, -1.35742188e-01,  1.16210938e-01, -1.59912109e-02,
        2.79296875e-01,  1.46484375e-01, -1.19628906e-01,  1.76757812e-01,
        1.28906250e-01, -1.49414062e-01,  6.93359375e-02, -1.72851562e-01,
        9.22851562e-02,  1.33056641e-02, -2.00195312e-01, -9.76562500e-02,
       -1.65039062e-01, -2.46093750e-01, -2.35595703e-02, -2.11914062e-01,
        1.84570312e-01, -1.85546875e-02,  2.16796875e-01,  5.05371094e-02,
        2.02636719e-02,  4.25781250e-01,  1.28906250e-01, -2.77099609e-02,
        1.29882812e-01, -1.15722656e-01, -2.05078125e-02,  1.49414062e-01,
        7.81250000e-03, -2.05078125e-01, -8.05664062e-02, -2.67578125e-01,
       -2.29492188e-02, -8.20312500e-02,  8.64257812e-02,  7.61718750e-02,
       -3.66210938e-02,  5.22460938e-02, -1.22070312e-01, -1.44042969e-02,
       -2.69531250e-01,  8.44726562e-02, -2.52685547e-02, -2.96630859e-02,
       -1.68945312e-01,  1.93359375e-01, -1.08398438e-01,  1.94091797e-02,
       -1.80664062e-01,  1.93359375e-01, -7.08007812e-02,  5.85937500e-02,
       -1.01562500e-01, -1.31835938e-01,  7.51953125e-02, -7.66601562e-02,
        3.37219238e-03, -8.59375000e-02,  1.25000000e-01,  2.92968750e-02,
        1.70898438e-01, -9.37500000e-02, -1.09375000e-01, -2.50244141e-02,
        2.11914062e-01, -4.44335938e-02,  6.12792969e-02,  2.62451172e-02,
       -1.77734375e-01,  1.23046875e-01, -7.42187500e-02, -1.67968750e-01,
       -1.08886719e-01, -9.04083252e-04, -7.37304688e-02,  5.49316406e-02,
        6.03027344e-02,  8.39843750e-02,  9.17968750e-02, -1.32812500e-01,
        1.22070312e-01, -8.78906250e-03,  1.19140625e-01, -1.94335938e-01,
       -6.64062500e-02, -2.07031250e-01,  7.37304688e-02,  8.93554688e-02,
        1.81884766e-02, -1.20605469e-01, -2.61230469e-02,  2.67333984e-02,
        7.76367188e-02, -8.30078125e-02,  6.78710938e-02, -3.54003906e-02,
        3.10546875e-01, -2.42919922e-02, -1.41601562e-01, -2.08007812e-01,
       -4.57763672e-03, -6.54296875e-02, -4.95605469e-02,  2.22656250e-01,
        1.53320312e-01, -1.38671875e-01, -5.24902344e-02,  4.24804688e-02,
       -2.38281250e-01,  1.56250000e-01,  5.83648682e-04, -1.20605469e-01,
       -9.22851562e-02, -4.44335938e-02,  3.61328125e-02, -1.86767578e-02,
       -8.25195312e-02, -8.25195312e-02, -4.05273438e-02,  1.19018555e-02,
        1.69921875e-01, -2.80761719e-02,  3.03649902e-03,  9.32617188e-02,
       -8.49609375e-02,  1.57470703e-02,  7.03125000e-02,  1.62353516e-02,
       -2.27050781e-02,  3.51562500e-02,  2.47070312e-01, -2.67333984e-02],
      dtype=float32)

61. Word similarity

Calculate the cosine similarity between "United States" and "U.S.".

code


model.similarity("United_States", "U.S.")

output


0.73107743

62. 10 words with high similarity

Output 10 words with high cosine similarity to “United States” and their similarity.

code


import numpy as np
import pandas as pd

code


simularities = model.most_similar("United_States")
pd.DataFrame(
    simularities,
    columns = ['word', 'Degree of similarity'],
    index = np.arange(len(simularities)) + 1
)

63. Analogy by additive construct

Subtract the "Madrid" vector from the "Spain" word vector, calculate the vector by adding the "Athens" vector, and output 10 words with high similarity to that vector and their similarity.

code


simularities = model.most_similar(positive=['Spain', 'Athens'], negative=['Madrid'])
pd.DataFrame(
    simularities,
    columns = ['word', 'Degree of similarity'],
    index = np.arange(len(simularities)) + 1
)

64. Experiments with analogy data

Download the evaluation data of the word analogy, calculate vec (word in the second column) --vec (word in the first column) + vec (word in the third column), and find the word with the highest similarity to the vector. , Find the similarity. Add the obtained word and similarity to the end of each case.

code


with open('data/questions-words.txt') as f:
    lines = f.read().splitlines()

dataset = []
category = None
for line in lines:
    if line.startswith(':'):
        category = line[2:]
    else:
        lst = [category] + line.split(' ')
        dataset.append(lst)

code


pd.DataFrame(dataset[:10])

code


from tqdm import tqdm

code


for i, lst in enumerate(tqdm(dataset)):
    pred, prob = model.most_similar(positive = lst[2:4], negative = lst[1:2], topn = 1)[0]
    dataset[i].append(pred)

code


pd.DataFrame(dataset[:10])

65. Correct answer rate in analogy tasks

Measure the accuracy rate of the semantic analogy and the syntactic analogy using the execution results of> 64.

The theory that evaluate_word_analogies () should be used is widely known.

code


semantic_analogy = [lst[-2:] for lst in dataset if not lst[0].startswith('gram')]
syntactic_analogy = [lst[-2:] for lst in dataset if lst[0].startswith('gram')]

code


acc = np.mean([true == pred for true, pred in semantic_analogy])
print('Semantic analogy correct answer rate:', acc)

output


Semantic analogy correct answer rate: 0.7308602999210734

code


acc = np.mean([true == pred for true, pred in syntactic_analogy])
print('Literary analogy correct answer rate:', acc)

output


Literary analogy correct answer rate: 0.7400468384074942

66. Evaluation by WordSimilarity-353

Download the evaluation data of The WordSimilarity-353 Test Collection and calculate the Spearman correlation coefficient between the ranking of similarity calculated by the word vector and the ranking of human similarity judgment.

code


import zipfile

code


#Read from zip file
with zipfile.ZipFile('data/wordsim353.zip') as f:
    with f.open('combined.csv') as g:
        data = g.read()

#Decode byte sequence
data = data.decode('UTF-8').splitlines()
data = data[1:]

#Tab delimited
data = [line.split(',') for line in data]
len(data)

output


353

code


for i, lst in enumerate(data):
    sim = model.similarity(lst[0], lst[1])
    data[i].append(sim)

code


pd.DataFrame(
    data[:10],
    columns = ['Word 1', 'Word 2', 'Human', 'vector']
)

code


from scipy.stats import spearmanr

It is possible to apply argsort twice to get the ranking, but it may be useless in terms of calculation amount.

code


def rank(x):
    args = np.argsort(-np.array(x))
    rank = np.empty_like(args)
    rank[args] = np.arange(len(x))
    return rank

code


human = [float(lst[2]) for lst in data]
w2v = [lst[3] for lst in data]
human_rank = rank(human)
w2v_rank = rank(w2v)
rho, p_value = spearmanr(human_rank, w2v_rank)

output


Rank correlation coefficient: 0.700313895424209
p-value: 2.4846350292113526e-53

code


print('Rank correlation coefficient:', rho)
print('p-value:', p_value)

code


import matplotlib.pyplot as plt

code


plt.scatter(human_rank, w2v_rank)
plt.show()

67. k-means clustering

Extract the word vector related to the country name and execute k-means clustering with the number of clusters k = 5.

I'm not sure where to get the country name from, but the analogy dataset does a good job

code


countries = {
    country
    for lst in dataset
    for country in [lst[2], lst[4]]
    if lst[0] in {'capital-common-countries', 'capital-world'}
} | {
    country
    for lst in dataset
    for country in [lst[1], lst[3]]
    if lst[0] in {'currency', 'gram6-nationality-adjective'}
}
countries = list(countries)
len(countries)

output


129

code


country_vectors = [model[country] for country in countries]

code


from sklearn.cluster import KMeans

code


kmeans = KMeans(n_clusters=5)
kmeans.fit(country_vectors)

output


KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,
       n_clusters=5, n_init=10, n_jobs=None, precompute_distances='auto',
       random_state=None, tol=0.0001, verbose=0)

code


for i in range(5):
    cluster = np.where(kmeans.labels_ == i)[0]
    print('class', i)
    print(', '.join([countries[k] for k in cluster]))

output


Class 0
Suriname, Honduras, Tuvalu, Guyana, Venezuela, Peru, Cuba, Ecuador, Nicaragua, Dominica, Colombia, Belize, Mexico, Bahamas, Jamaica, Chile
Class 1
Netherlands, Egypt, France, Syria, Finland, Germany, Uruguay, Switzerland, Greenland, Italy, Lebanon, Malta, Algeria, Europe, Tunisia, Brazil, Ireland, England, Libya, Spain, Argentina, Liechtenstein, Iran, Jordan, USA, Iceland, Sweden, Norway, Qatar, Portugal, Denmark, Canada, Israel, Belgium, Morocco, Austria
Class 2
Kazakhstan, Lithuania, Turkmenistan, Serbia, Croatia, Greece, Uzbekistan, Armenia, Latvia, Albania, Slovenia, Cyprus, Ukraine, Georgia, Belarus, Bulgaria, Kyrgyzstan, Macedonia, Estonia, Montenegro, Turkey, Azerbaijan, Tajikistan, Poland, Russia, Romania, Hungary, Slovakia, Moldova
Class 3
Ghana, Senegal, Zambia, Sudan, Somalia, Zimbabwe, Gabon, Madagascar, Angola, Liberia, Gambia, Niger, Uganda, Mauritania, Namibia, Eritrea, Botswana, Malawi, Mozambique, Guinea, Kenya, Nigeria, Burundi, Mali, Rwanda
Class 4
Japan, China, Pakistan, Samoa, Bahrain, Fiji, Australia, India, Laos, Bhutan, Malaysia, Taiwan, Cambodia, Nepal, Korea, Oman, Thailand, Bangladesh, Indonesia, Iraq, Vietnam, Afghanistan, Philippines

They are divided.

68. Ward's method clustering

Execute hierarchical clustering by Ward's method for word vectors related to country names. Furthermore, visualize the clustering result as a dendrogram.

code


from scipy.cluster.hierarchy import dendrogram, linkage

code


plt.figure(figsize=(16, 9), dpi=200)
Z = linkage(country_vectors, method='ward')
dendrogram(Z, labels = countries)
plt.show()

69. Visualization by t-SNE

Visualize the vector space of word vectors related to country names with t-SNE.

code


from sklearn.manifold import TSNE

code


tsne = TSNE()
tsne.fit(country_vectors)

output


TSNE(angle=0.5, early_exaggeration=12.0, init='random', learning_rate=200.0,
     method='barnes_hut', metric='euclidean', min_grad_norm=1e-07,
     n_components=2, n_iter=1000, n_iter_without_progress=300, n_jobs=None,
     perplexity=30.0, random_state=None, verbose=0)

code


plt.figure(figsize=(15, 15), dpi=300)
plt.scatter(tsne.embedding_[:, 0], tsne.embedding_[:, 1])
for (x, y), name in zip(tsne.embedding_, countries):
    plt.annotate(name, (x, y))
plt.show()

Next is Chapter 8

Language processing 100 knocks 2020 Chapter 8: Neural network

Recommended Posts

100 Language Processing Knock 2020 Chapter 7: Word Vector
Language processing 100 knock-86: Word vector display
[Language processing 100 knocks 2020] Chapter 7: Word vector
100 Language Processing Knock 2020 Chapter 1
100 Language Processing Knock Chapter 1
100 Language Processing Knock 2020 Chapter 3
100 Language Processing Knock 2020 Chapter 2
100 Language Processing Knock Chapter 1 (Python)
100 Language Processing Knock Chapter 2 (Python)
100 Language Processing Knock-87: Word Similarity
100 Language Processing Knock with Python (Chapter 1)
100 Language Processing Knock Chapter 1 in Python
100 Language Processing Knock 2020 Chapter 4: Morphological Analysis
100 Language Processing Knock 2020 Chapter 9: RNN, CNN
100 Language Processing Knock (2020): 28
I tried 100 language processing knock 2020: Chapter 3
100 Language Processing Knock-82 (Context Word): Context Extraction
100 Language Processing Knock: Chapter 1 Preparatory Movement
100 Language Processing Knock 2020 Chapter 6: Machine Learning
100 Language Processing Knock Chapter 4: Morphological Analysis
100 Language Processing Knock 2020 Chapter 10: Machine Translation (90-98)
100 Language Processing Knock 2020 Chapter 8: Neural Net
100 Language Processing Knock (2020): 38
I tried 100 language processing knock 2020: Chapter 1
100 language processing knock 00 ~ 02
100 Language Processing Knock 2020 Chapter 1: Preparatory Movement
100 Language Processing Knock Chapter 1 by Python
100 Language Processing Knock 2020 Chapter 3: Regular Expressions
100 Language Processing Knock 2015 Chapter 4 Morphological Analysis (30-39)
I tried 100 language processing knock 2020: Chapter 2
I tried 100 language processing knock 2020: Chapter 4
100 Language Processing Knock Chapter 10 Vector Space Method (II) + Overall Summary
100 Language Processing Knock 2020 with GiNZA v3.1 Chapter 4
100 Language Processing Knock with Python (Chapter 2, Part 2)
[Programmer newcomer "100 language processing knock 2020"] Solve Chapter 1
100 language processing knock 2020 [00 ~ 39 answer]
100 language processing knock 2020 [00-79 answer]
100 language processing knock 2020 [00 ~ 69 answer]
100 Amateur Language Processing Knock: 17
100 language processing knock 2020 [00 ~ 49 answer]
100 language processing knocks ~ Chapter 1
100 Amateur Language Processing Knock: 07
100 Amateur Language Processing Knock: 09
100 Amateur Language Processing Knock: 47
100 Language Processing Knock-53: Tokenization
100 Amateur Language Processing Knock: 97
100 language processing knock 2020 [00 ~ 59 answer]
100 Amateur Language Processing Knock: 67
100 Language Processing Knock UNIX Commands Learned in Chapter 2
100 Language Processing Knock Regular Expressions Learned in Chapter 3
100 Language Processing with Python Knock 2015
100 Language Processing Knock-58: Tuple Extraction
100 Language Processing Knock-57: Dependency Analysis
100 language processing knock-50: sentence break
Natural language processing 3 Word continuity
100 Language Processing Knock-25: Template Extraction
I tried 100 language processing knock 2020
100 language processing knock-56: co-reference analysis
Solving 100 Language Processing Knock 2020 (01. "Patatokukashi")
Natural language processing 2 Word similarity
100 Amateur Language Processing Knock: Summary