[PYTHON] Format summary of formats that can be serialized with gensim

gensim

A library of topic models implemented in Python. The details of the function are not covered here. This time, I will summarize the formats of various formats that can be converted when converting a character string to the BoW format with gensim.

Execution code

Output as Official Reference.

from gensim import corpora
from collections import defaultdict
from pprint import pprint

documents = ["Human machine interface for lab abc computer applications",
             "A survey of user opinion of computer system response time",
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",
             "Relation of user perceived response time to error measurement",
             "The generation of random binary unordered trees",
             "The intersection graph of paths in trees",
             "Graph minors IV Widths of trees and well quasi ordering",
             "Graph minors A survey"]
stoplist = set('for a of the and to in'.split())
texts = [[word for word in document.lower().split() if word not in stoplist]
         for document in documents]

frequency = defaultdict(int)
for text in texts:
    for token in text:
        frequency[token] += 1

texts = [[token for token in text if frequency[token] > 1]
         for text in texts]
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

corpora.MmCorpus.serialize("./corpus.mm", corpus)
corpora.BleiCorpus.serialize("./corpus.blei", corpus)
corpora.LowCorpus.serialize("./corpus.low", corpus)
corpora.SvmLightCorpus.serialize("./corpus.svmlight", corpus)
corpora.UciCorpus.serialize("./corpus.low", corpus)

pprint(texts)
print("\n")
pprint(dictionary.token2id)
print("\n")
pprint(corpus)

Output

[['human', 'interface', 'computer'],
 ['survey', 'user', 'computer', 'system', 'response', 'time'],
 ['eps', 'user', 'interface', 'system'],
 ['system', 'human', 'system', 'eps'],
 ['user', 'response', 'time'],
 ['trees'],
 ['graph', 'trees'],
 ['graph', 'minors', 'trees'],
 ['graph', 'minors', 'survey']]


{'computer': 1,
 'eps': 8,
 'graph': 10,
 'human': 2,
 'interface': 0,
 'minors': 11,
 'response': 6,
 'survey': 4,
 'system': 5,
 'time': 7,
 'trees': 9,
 'user': 3}


[[(0, 1), (1, 1), (2, 1)],
 [(1, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)],
 [(0, 1), (3, 1), (5, 1), (8, 1)],
 [(2, 1), (5, 2), (8, 1)],
 [(3, 1), (6, 1), (7, 1)],
 [(9, 1)],
 [(9, 1), (10, 1)],
 [(9, 1), (10, 1), (11, 1)],
 [(4, 1), (10, 1), (11, 1)]]

Matrix Market format

corpus.mm

%%MatrixMarket matrix coordinate real general
9 12 28                                           
1 1 1
1 2 1
1 3 1
2 2 1
2 4 1
2 5 1
2 6 1
2 7 1
2 8 1
3 1 1
3 4 1
3 6 1
3 9 1
4 3 1
4 6 2
4 9 1
5 4 1
5 7 1
5 8 1
6 10 1
7 10 1
7 11 1
8 10 1
8 11 1
8 12 1
9 5 1
9 11 1
9 12 1

• Comment out and use%
• Numbers start at 1
• First, the number of rows and columns, the number of non-zero elements, is expressed as an integer.
• When this line is output by gensim, there are a lot of spaces at the end of the sentence for some reason.
• The rest is indicated by i j v, which indicates that the j-th (column) word has appeared v times in the i-th (row) sentence.
• corpus.mm.index is generated but not a readable file

Blei format

corpus.blei

3 0:1 1:1 2:1
6 1:1 3:1 4:1 5:1 6:1 7:1
4 0:1 3:1 5:1 8:1
3 2:1 5:2 8:1
3 3:1 6:1 7:1
1 9:1
2 9:1 10:1
3 9:1 10:1 11:1
3 4:1 10:1 11:1

text:corpus.blei.vocab

0
1
2
3
4
5
6
7
8
9
10
11

• Format by Blei, the original paper of LDA
• Numbers start at 0
• Each line corresponds to each sentence
• The number of words in the text is indicated at the beginning of the line
• Represented by j: v, indicating that the jth element of that column is v
• Corpus.blei.index and corpus.blei.vocab are generated at the same time when serializing
• index is not a readable file
• vocab seems to indicate how many BoW features exist at the maximum.

UCI format

corpus.uci

9                   
12                  
28                  
1 1 1
1 2 1
1 3 1
2 2 1
2 4 1
2 5 1
2 6 1
2 7 1
2 8 1
3 1 1
3 4 1
3 6 1
3 9 1
4 3 1
4 6 2
4 9 1
5 4 1
5 7 1
5 8 1
6 10 1
7 10 1
7 11 1
8 10 1
8 11 1
8 12 1
9 5 1
9 11 1
9 12 1

text:corpus.uci.vocab

0
1
2
3
4
5
6
7
8
9
10
11

• BoW format by the University of California, Irvine
• Numbers start at 1
• Similar to Mm format, except for the following:
• No comment out
• The summary of the first line spans multiple lines
• At the same time, corpus.low.index and corpus.low.vocab are output.
• index is not a readable file
• vocab seems to indicate how many BoW features exist at the maximum. But for some reason the numbers start at 0

Low format

corpus.low

9
0 1 2
1 3 4 5 6 7
0 3 5 8
2 5 5 8
3 6 7
9
9 10
9 10 11
4 10 11

text:corpus.low.vocab

0
1
2
3
4
5
6
7
8
9
10
11

• Format by Phan et al. Who implemented LDA in C / C ++
• This implementation seems to infer LDA with MCMC by Gibbs sampling instead of the original variational Bayes by Blei.
• Numbers start at 0
• The number of sentences is shown on the first line
• Each line and each sentence after that correspond, and only the jth word indicates j.

SvmLight format

corpus.svmlight

0 1:1 2:1 3:1
0 2:1 4:1 5:1 6:1 7:1 8:1
0 1:1 4:1 6:1 9:1
0 3:1 6:2 9:1
0 4:1 7:1 8:1
0 10:1
0 10:1 11:1
0 10:1 11:1 12:1
0 5:1 11:1 12:1

• Format of Joachims et al. With C implementation of SVM
• Numbers start at 1
• The 0 at the beginning of each line is a remnant of using -1/1/0 when representing positive and negative examples in SVM.
• The above two points are different from the Blei format.
• Each line corresponds to each sentence
• Represented by j: v, indicating that the jth element of that column is v
• corpus.mm.index is generated but not a readable file

reference

• Sparse matrix representation method 1 Matrix Market
• gensim official