[PYTHON] Machine learning meeting information for HRTech

TL; DR

We will create a learning model to create something like this by utilizing meeting data.

The entire article is here.

1. Premise

• Reading here
• On top of that, the following data must be prepared
• Morphologically parsed conference data
  • filterred_mecabdata.pkl
• Detailed member data
  • memberdict.pkl
• However, this file is not used in this article

The following is the meeting data based on the data of the parliamentary minutes.

2. Machine learning

Calculate the sentences that are characteristic of each member and the similarity between each member. Let's think about what makes the sentences similar, but here we will evaluate the degree of similarity by the following two methods.

• How to use TF-IDF score
• How to use doc2vec

Hey there you! You thought ** TF-IDF is not machine learning **! !!

Yes, that's right. But people in the street don't know the difference, so if the result looks good, you won't notice it. So

"AI worked hard! (Smiley)"

That said, there should be almost no problem. There are as many images of AI as there are people (Kiri!

2.1 Calculation of similarity between people using TF-IDF

Please see here. If you use the chan scipy library, even sparse matrices that tend to be large can be processed at a stress-free speed.

2.2 Calculation of similarity between people with doc2vec

Please see here. It's easy to learn with the gensim package.

3. Network analysis

Please see here.

Create a Network Graph and use a technique called the louvain method to cluster people with similar remarks from the Netowrk Graph.

When visualized, you will get the following result.

When this Netowrk Graph is written in Json, the specific contents are as follows.

{
  "directed": false,
  "multigraph": false,
  "graph": {},
  "nodes": [
    {
      "size": 3,
      "cluster": 1,
      "id": "Tomomi Inada"
    },
    {
      "size": 54,
      "cluster": 3,
      "id": "Taro Aso"
    },
    {
      "size": 142,
      "cluster": 0,
      "id": "Hiroshige Seko"
    },
    {
      "size": 39,
      "cluster": 4,
      "id": "Yasuhisa Shiozaki"
    },
    {
      "size": 30,
      "cluster": 1,
      "id": "Sanae Takaichi"
    },
    {
      "size": 95,
      "cluster": 1,
      "id": "Shinzo Abe"
    }
  ],
  "links": [
    {
      "weight": 0.5984722375869751,
      "source": "Tomomi Inada",
      "target": "Hiroshige Seko"
    },
    {
      "weight": 0.9666371941566467,
      "source": "Tomomi Inada",
      "target": "Shinzo Abe"
    },
    {
      "weight": 0.48173508048057556,
      "source": "Tomomi Inada",
      "target": "Yasuhisa Shiozaki"
    },
    {
      "weight": 0.4896692633628845,
      "source": "Tomomi Inada",
      "target": "Sanae Takaichi"
    },
    {
      "weight": 0.7263149619102478,
      "source": "Taro Aso",
      "target": "Hiroshige Seko"
    },
    {
      "weight": 0.6178034543991089,
      "source": "Taro Aso",
      "target": "Shinzo Abe"
    },
    {
      "weight": 0.46518972516059875,
      "source": "Taro Aso",
      "target": "Yasuhisa Shiozaki"
    },
    {
      "weight": 0.8961162567138672,
      "source": "Hiroshige Seko",
      "target": "Yasuhisa Shiozaki"
    },
    {
      "weight": 1.2007122039794922,
      "source": "Hiroshige Seko",
      "target": "Shinzo Abe"
    },
    {
      "weight": 0.945235550403595,
      "source": "Hiroshige Seko",
      "target": "Sanae Takaichi"
    },
    {
      "weight": 0.9955565333366394,
      "source": "Yasuhisa Shiozaki",
      "target": "Shinzo Abe"
    },
    {
      "weight": 0.9067516922950745,
      "source": "Yasuhisa Shiozaki",
      "target": "Sanae Takaichi"
    },
    {
      "weight": 1.053189754486084,
      "source": "Sanae Takaichi",
      "target": "Shinzo Abe"
    }
  ]
}