[PYTHON] Qiita Job I tried to analyze the job offer

table of contents

--Background --Data acquisition --I looked at the data --Impressions and future plans

background

While I was changing jobs, I was caught up in the phrase "Congratulations on changing jobs for 300,000 yen" and registered with Qiita Jobs, but when I tried using it, I thought that the functions were a little lacking.

I couldn't narrow down the work location, or I didn't know the number of pages.

Fortunately, I have most of the information I need in the job listings, and the total number of pages is only about 20 (obtained manually), so I decided to get it in Python.

Data acquisition

The general flow is like this.

--Get the content as HTML in the URL of the job list in Python --Create a beautiful soup with the acquired HTML, analyze tags, etc. and cut out the necessary parts. --The cut out part is converted to JSON and stored in Firebase cloud firestore. --On the other hand, save it locally as a csv file for analysis of takeover.

We got a total of 294 jobs. (Is this probably all?)

I looked at the data

The tools I used were Pandas and Tableau.

One thing I would like to say first is that the content from now on is created based on the content of the Qiita jobs job offer, so it will not reflect the job offer of the entire IT industry, but ** for your reference **.

The first thing I was most interested in was the number of job openings by work location.

I see, the number of job vacancies in Tokyo has won overwhelmingly. Isn't it necessary to narrow down the work location? However, it seems a little inconvenient for those who are looking for a local job such as U / I turn.

Next, let's look at job vacancies by job type.

TOP5 was a web application, other engineer, backend, frontend, and infrastructure engineer. I feel that the amount of work on the Web is more in demand than the work of smartphone development.

Also, to see what other engineers are looking for, I compared the back-end engineers and front-end engineers who have similar job openings to other engineers.

As a result, for other engineers, there are a lot of technical tags such as JAVA, C ++, C #, games, animation, etc., probably embedded systems, game development, etc.

By the way, if you summarize the techniques used for all job offers, you will get the following hit map.

Finally, let's check the average annual income by job type, but it should be noted here that the "average annual income" is simply the average of the upper and lower limits of annual income.

Looking at this figure, you can see that the salary range of the web application engineer is large.

You can also see that the back-end engineer has a higher average annual salary than the front-end and back-end.

Impressions and future plans

This is the first time I've used Tableau, so I may still be unfamiliar with creating visualization diagrams, but I'd love to use it again if I have the opportunity.

I'm sure there's more you can do with the data you get from Qiita Jobs. I just searched the contents of the data today, but in the future

1. I would like to use AWS services to create something that will see if new jobs are coming to the specified job category.
2. This time, I used only the information that can be obtained from the list, but I think that it can also be used for machine learning because there is a lot of further information such as essential skills and working conditions in each job details.

In addition, you can check this drawing at the following URL. Tableau Public

Thank you for watching until the end.