I searched for the skills needed to become a web engineer in Python

Introduction

I scraped a job change site with Python and briefly investigated "necessary skills". Since I am aiming to be an engineer, I was wondering what kind of requirements are actually set up on the job change site.

Ideas and implementations <a href="https://qiita.com/kakiuchis/items/2c9b327cadf1e8dbdf6e" rel=”nofollow noopener” target="_blank"> Since I studied scraping, how to make an inexperienced person a web director I have taken the article of , which I considered in a data-driven manner, as a reference.

Please point out mistakes and small mistakes!

Required Skills Conclusion

For those who want to see only the results

--Common --Team development and communication skills --Git / GitHub

• front end --html / css / javascript is compulsory ――It would be nice if you could do either vue or React, then Angular --Knowledge of design skills UI / UX --Server-side languages such as PHP, Ruby, Java --Skills to handle Photoshop / Illustrator
• webpack
• etc……

--Server side --Learning languages such as Ruby, PHP, Python, C (C #?), Java --Cloud knowledge such as AWS / GCP --Knowledge of server infrastructure in general --Web application frameworks such as Rails, Laravel, and Django --Database in general --General network --Front-end language --Docker container

• CI/CD
• etc……

I don't mind getting messy from the middle.

Is it the expected result?

Then to the contents.

Environment & build order

--Environment

• MacOS Mojave 10.14.6

• Python3.8.1

• Environment --Install Homebrew (included from the beginning) --Mysql installation (included from the beginning) --Install pyenv with homebrew --Install python with pyenv --Create a python environment under the directory with venv --Install libraries such as bs4, MeCab, jupyter notebook, pandas

--Reference

-<a href="https://qiita.com/sakaeda11/items/3832472b5eb923e0128f" rel=”nofollow noopener” target="_blank"> Python environment construction on Mac

-<a href="https://qiita.com/fiftystorm36/items/b2fd47cf32c7694adc2e" rel=”nofollow noopener” target="_blank"> venv: Python virtual environment management -<a href="https://qiita.com/kakiuchis/items/2c9b327cadf1e8dbdf6e" rel=”nofollow noopener” target="_blank"> Since I studied scraping, I learned how to become a web director by data-driven. I considered it.

Collecting information from job change sites

Scraping → The implementation of morphological analysis is generally as <a href="https://qiita.com/kakiuchis/items/2c9b327cadf1e8dbdf6e" rel=”nofollow noopener” target="_blank"> reference article , so it is omitted. The flow is as follows: Get the job URL for each job → Open the URL and get the required skill part → Calculate the frequency for each noun by morphological analysis. <a href="https://qiita.com/itkr/items/513318a9b5b92bd56185" rel=”nofollow noopener” target="_blank"> Scraping with Python and Beautiful Soup and <a href="https:/ /www.crummy.com/software/BeautifulSoup/bs4/doc/ "rel =” nofollow noopener ”target =" _ blank "> Official document was also referred to.

Especially scraping ・ Put sleep before opening the page ・ Check robots.txt and check disallow Let's be careful.

The number of job offers acquired is Front end: 208 cases Server side: 181 cases Games: 175 was

analysis

Click here for the code used for the analysis.

analysis_job_search.py

import MeCab
import pandas as pd
import numpy as np
import mysql.connector as mydb
import pandas.io.sql as psql
import collections

pd.set_option('display.max_rows', 500)

#Method to extract nouns from text
def devide_by_mecab(text):
    tagger = MeCab.Tagger("-Ochasen")
    node = tagger.parseToNode(text)
    word_list = []
    while node:
        pos = node.feature.split(",")[0]
        if pos in ["noun"]:
            word = node.surface
            word_list.append(word)
        node = node.next
    return "  ".join(word_list)

#Connect to MySQL Personal Settings Here
connection = mydb.connect(
  host = '',
  port = '',
  user = '',
  password = '',
  database = ''
)

#Get data from DB
df_frontend = psql.read_sql("SELECT * FROM table WHERE search_word = 'front end'",connection)
df_serverside = psql.read_sql("SELECT * FROM table WHERE search_word = 'Server side'",connection)
df_game = psql.read_sql("SELECT * FROM table WHERE search_word = 'Game programmer'",connection)

#need_A method that decomposes skills into nouns and returns them
def get_all_words(df):
    all_words = []
    for index, row in df.iterrows():
        words = devide_by_mecab(row['need_skills']).split()
        all_words.extend(words)
    return all_words

count_of_words_frontend = collections.Counter(get_all_words(df_frontend))
count_of_words_serverside = collections.Counter(get_all_words(df_serverside))
count_of_words_game = collections.Counter(get_all_words(df_game))

#See all nouns in order of frequency
count_of_words_frontend.most_common()
count_of_words_serverside.most_common()
count_of_words_game.most_common()

#Arrangement of words related to skill(count >= 6)
frontend_top_words = ['JavaScript', 'CSS', 'HTML', 'js', 'Vue', 'React', 'design', 'team', 'Git', 'UI', 'PHP', 'UX', 'Angular', 'Ruby', 'Javascript', 'jQuery', 'Photoshop', 'communication', 'API', 'TypeScript', 'SPA', 'Java', 'Sass', 'designer', 'Illustrator', 'JS', 'test', 'server', 'webpack', 'GitHub', 'AWS', 'AngularJS', 'WordPress', 'Webpack', 'Rails', 'iOS', 'CMS', 'Python', 'Redux', 'MySQL', 'Gulp', 'Android', 'gulp', 'C', 'SCSS', 'git', 'DB', 'Linux', 'Babel', 'Docker', 'CI']
serverside_top_words = ['Ruby', 'PHP', 'AWS', 'Python', 'C', 'Java', 'server', 'Rails', 'team', 'infrastructure', 'js', 'Git', 'server', 'Android', 'JavaScript', 'Go', 'Linux', 'Perl', 'HTML', 'MySQL', 'RDBMS', 'CSS', 'Kura', 'Udo', 'API', 'front', 'management', 'GitHub', 'iOS', 'DB', 'GCP', 'React', 'Vue', 'network', 'Node', 'HTTP', 'Swift', 'CI', 'Objective', 'Docker', 'Security', 'Javascript', 'Azure', 'native', 'PostgreSQL', 'architecture', 'SQL', 'test', '#', 'smart', 'phone', 'UI', 'MVC', 'communication', 'git', 'Scala', 'Kotlin' , 'CD', 'Database', 'TypeScript', 'Apache', 'LAMP', 'designer', 'container', 'RDB', 'Laravel']
game_top_words = ['C', '3', 'D', 'Unity', 'Java', 'PHP', 'design', '++', 'server', '++、', 'network', 'JavaScript', 'Android', 'management', '#、', 'Objective', 'Photoshop', 'Maya', 'team', 'designer', 'Linux', 'MySQL', 'Ruby', 'Python', 'infrastructure', '#', 'Graphics', 'server', 'Excel', 'graphic', 'communication', 'Unreal', 'DCG', 'AWS', 'Perl', 'Illustrator', 'Engine', 'planner', 'Word', 'native', 'motion', 'director', 'HTML', 'UI', 'Flash', 'effect', 'VB', 'sound', 'DS', 'OpenGL', 'iOS', 'DirectX']

#Method to create DataFrame of word and number of occurrences
def get_top_word_df(top_words,count_of_words):
    df = pd.DataFrame({})
    for i,word in enumerate(top_words):
        word_data = pd.Series([word,count_of_words[word]], index=['word','count'], name=i)
        df = df.append(word_data)
    return df

df_frontend_top_words =  get_top_word_df(frontend_top_words,count_of_words_frontend)
df_serverside_top_words =  get_top_word_df(serverside_top_words,count_of_words_serverside)
df_game_top_words =  get_top_word_df(game_top_words,count_of_words_game)

for df in [df_frontend_top_words,df_serverside_top_words,df_game_top_words]:
    df['rank'] = df['count'].rank(ascending = False, method = 'min').astype(int)
    df['count'] = df['count'].astype(int)

df_frontend_top_words[['rank','word','count']]
df_serverside_top_words[['rank','word','count']]
df_game_top_words[['rank','word','count']]

First, all the words are output in order of frequency, and the words that do not seem to be related to the skill are manually removed and output again. The game programmer was personally interested, so I added it.

The result is as follows.

front end

Server side

Game programmer

I feel that the results are almost as expected.

Originally, the same words such as'javascript',' Javascript', and'js' should be named properly, but it seemed to be difficult because there were many words, so I was frustrated.

Conclusion

Front-end engineer: It seems that html, css, javascript are outstanding and indispensable. In addition, the second step is to be able to use frameworks such as Vue and React while strengthening the design, UI / UX, and it seems good to deepen the understanding on the server side after that.

Server-side engineer: The cloud is divided into'Kura'and'Udo'! …… Quiet talk. It seems that Ruby, PHP, Python, and Java are the main languages (I don't know which one is C). In addition, cloud is also a compulsory subject. I also want the understanding of the database, network, and front end side. There seems to be a lot of studying.

Game programmer: After all the direction is a little different and it is interesting. C ++, C # and Unity, 3D are the main fields, and it seems that it is necessary to learn around graphics while programming.

I have little knowledge, so I can only say something really rough ... orz. I would like to study based on this result!

Task

――Since the word is a little strict, the number of acquisitions is small (around 200)

-Since it is a survey of only one site, there is a bias

--Name identification such as lowercase letters, abbreviations, typographical errors, etc.

the end

I just looked at the frequency briefly, but I'm glad that it also served as a guide for the fields I should study.

It might be more interesting to compare on multiple sites or try on a cross-sectional site. If you are interested, please check it out with your own eyes!