It is a challenge record of Language processing 100 knock 2015. The environment is Ubuntu 16.04 LTS + Python 3.5.2 : : Anaconda 4.1.1 (64-bit). Click here for a list of past knocks (http://qiita.com/segavvy/items/fb50ba8097d59475f760).
In Chapter 10, we will continue to study word vectors from the previous chapter.
Using the data created in> 94, calculate the Spearman correlation coefficient between the ranking of similarity output by each model and the ranking of human similarity judgment.
main.py
# coding: utf-8
import numpy as np
fname_input = 'combined_out.tab'
with open(fname_input, 'rt') as data_file:
#Array of similarity
human_score = []
my_score = []
N = 0
for line in data_file:
cols = line.split('\t')
human_score.append(float(cols[2]))
my_score.append(float(cols[3]))
N += 1
#sort
human_index_sorted = np.argsort(human_score)
my_index_sorted = np.argsort(my_score)
#Arrangement of rank
human_order = [0] * N
my_order = [0] * N
for i in range(N):
human_order[human_index_sorted[i]] = i
my_order[my_index_sorted[i]] = i
#Spearman correlation coefficient calculation
total = 0
for i in range(N):
total += pow(human_order[i] - my_order[i], 2)
result = 1 - (6 * total) / (pow(N, 3) - N)
print(result)
Results for question 85 word vector
0.22645511508225769
Results for question 90 word vector
0.5013384068756902
Spearman's correlation coefficient is a value that indicates how much there is a correlation between two rankings like this one. Details will come out a lot if you google with "Spearman's rank correlation coefficient", so I will omit it, but you can find it with the following formula.
Where $ D $ is the difference between the two rankings for each piece of data, and $ N $ is the number of data. The maximum is 1, and the higher the value, the more correlated. In this problem, the larger the value, the closer the ranking is to human judgment.
To find $ D $, first for each line of "combined_out.tab" created in Problem 94, the ranking in human similarity and You need to find the rank in the word vector. If you simply sort, you will not know the original line and you will not be able to find the ranking difference, so I used it in Problem 75 Numpy. Find the sorted index with argsort () and use it to create a new one. I tried to make an array of ranks for each row.
The result is a word2vec overwhelming victory. word2vec is amazing.
**. Below is the corrected code. Thank you! *main2.py (version advised by shiracamus)
# coding: utf-8
fname_input = 'combined_out.tab'
class Data:
def __init__(self, human_score, my_score):
self.human_score = human_score
self.my_score = my_score
def __repr__(self):
return 'Data%s' % repr(self.__dict__)
#Data array creation
with open(fname_input) as data_file:
def read_data():
for line in data_file:
word1, word2, human_score, my_score = line.split('\t')
yield Data(float(human_score), float(my_score))
data = list(read_data())
#Ranking
data_sorted_by_human_score = sorted(data, key=lambda data: data.human_score)
for order, d in enumerate(data_sorted_by_human_score):
d.human_order = order
data_sorted_by_my_score = sorted(data, key=lambda data: data.my_score)
for order, d in enumerate(data_sorted_by_my_score):
d.my_order = order
#Spearman correlation coefficient calculation
N = len(data)
total = sum((d.human_order - d.my_order) ** 2 for d in data)
result = 1 - (6 * total) / (N ** 3 - N)
print(result)
That's all for the 96th knock. If you have any mistakes, I would appreciate it if you could point them out.
Recommended Posts