intel Python was too fast (under certain conditions)
In conclusion, sciktlearn is much faster.
Looking at SystemMonitor, it doesn't mean that it uses all the CPU cores as some rumors say, but it seems that code optimization saves unnecessary processing of the CPU. This area is unknown unless you read the technical book carefully, but it turned out that there is a process that is overwhelmingly faster than Vanila Python.
Model Name: iMac
Model Identifier: iMac19,1
Processor Name: Intel Core i5
Processor Speed: 3.7 GHz
Number of Processors: 1
Total Number of Cores: 6
L2 Cache (per Core): 256 KB
L3 Cache: 9 MB
Memory: 24 GB
It is recommended to install intel python in Anaconda, but I personally don't like Anaconda, so I built it like this. (When I became a jupyter due to an unexpected trap, I ran on base instead of idp, so I had a lot of trouble setting it.)
.
pyenv
└── anaconda3-2019.10
└── intel_python
In both cases, the measurement was started 10 minutes after the network was shut down and restarted. Now the result.
Since the Statistics Bureau had interesting data organized by type of non-regular employment in a questionnaire for non-regular employees, I looked at the rate on a heat map. The data is from 2017.
data
Bureau of Statistics "Gender, education, reasons for incumbent employment, incumbent employment status / population by employment type (non-regular staff / employees) -nationwide, national cities, prefectures, prefectures and cities, Ordinance-designated city "
Look at uniqueness
def show_columns(dfl, num=10):
for i in dfl.columns:
print('----------------')
print(i)
print(dfl[i].nunique(), 'Unique')
print(dfl[i].unique()[:num])
df["value"] = pd.to_numeric(df["value"], errors='coerce')
show_columns(df, 117)
'''
tab_code
1 unique
['001-2017']
----------------
Table item
1 unique
['population']
----------------
cat01_code
1 unique
[0]
----------------
Men and women
1 unique
['Total number']
----------------
cat02_code
9 unique
[ 0 1 11 12 13 14 15 16 17]
----------------
education
9 unique
['Total number' 'Graduates' '小学・中学（Graduates）' '高校・旧制中（Graduates）' '専門学校（２年未満）（Graduates）'
'Vocational school (less than 2-4 years) (graduate)' 'Vocational school (4 years or more) (graduate)' 'Junior college (graduate)' 'Technical college (graduate)']
----------------
cat03_code
15 unique
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14]
----------------
Reason for hiring
15 unique
['Total number' 'Mainly because I want to work at my own convenient time' 'Mainly because I want to get household subsidies and tuition fees' 'Mainly because it is easy to balance with housework, childcare, long-term care, etc.'
'Mainly because commuting time is short' 'Mainly because I can make use of my specialized skills' 'Mainly because there is no regular staff / employee job' 'Mainly other'
'I want to work at my own convenient time' 'Because I want to get household subsidies, school fees, etc.' 'Because it is easy to balance with housework, childcare, long-term care, etc.' 'Because the commuting time is short'
'Because you can make use of your specialized skills' 'Because there is no regular staff / employee job' 'Other']
----------------
cat04_code
7 unique
[0 1 2 3 4 5 6]
----------------
Incumbent status / employment form
7 unique
['Total number' 'part' 'part time job' 'Temporary staff at a worker dispatching office' 'Contract employee' 'Commission' 'Other']
----------------
area_code
117 unique
[ 0 1 1000 1001 1100 2000 2001 3000 3001 4000 4001 4100
5000 5001 6000 6001 7000 7001 8000 8001 9000 9001 10000 10001
11000 11001 11100 12000 12001 12100 13000 13001 13100 14000 14001 14100
14130 14150 15000 15001 15100 16000 16001 17000 17001 18000 18001 19000
19001 20000 20001 21000 21001 22000 22001 22100 22130 23000 23001 23100
24000 24001 25000 25001 26000 26001 26100 27000 27001 27100 27140 28000
28001 28100 29000 29001 30000 30001 31000 31001 32000 32001 33000 33001
33100 34000 34001 34100 35000 35001 36000 36001 37000 37001 38000 38001
39000 39001 40000 40001 40100 40130 41000 41001 42000 42001 43000 43001
43100 44000 44001 45000 45001 46000 46001 47000 47001]
----------------
Area classification
117 unique
['Nationwide' 'Nationwide市部' 'Hokkaido' 'Hokkaido市部' 'Sapporo' 'Aomori Prefecture' 'Aomori Prefecture市部' 'Iwate Prefecture' 'Iwate Prefecture市部' 'Miyagi Prefecture' 'Miyagi Prefecture市部'
'Sendai city' 'Akita' 'Akita市部' 'Yamagata Prefecture' 'Yamagata Prefecture市部' 'Fukushima Prefecture' 'Fukushima Prefecture市部' 'Ibaraki Prefecture' 'Ibaraki Prefecture市部' 'Tochigi Prefecture'
'Tochigi Prefecture City' 'Gunma Prefecture' 'Gunma Prefecture市部' 'Saitama' 'Saitama市部' 'Saitama City' 'Chiba' 'Chiba市部' 'Chiba' 'Tokyo'
'Tokyo city area' 'Special wards' 'Kanagawa Prefecture' 'Kanagawa Prefecture市部' 'Yokohama City' 'Kawasaki City' 'Sagamihara City' 'Niigata Prefecture' 'Niigata Prefecture市部' 'Niigata City'
'Toyama Prefecture' 'Toyama Prefecture市部' 'Ishikawa Prefecture' 'Ishikawa Prefecture市部' 'Fukui prefecture' 'Fukui prefecture市部' 'Yamanashi Prefecture' 'Yamanashi Prefecture市部' 'Nagano Prefecture' 'Nagano Prefecture市部'
'Gifu Prefecture' 'Gifu Prefecture市部' 'Shizuoka Prefecture' 'Shizuoka Prefecture市部' 'Shizuoka City' 'Hamamatsu city' 'Aichi prefecture' 'Aichi prefecture市部' 'Nagoya city' 'Mie Prefecture'
'Mie Prefecture City' 'Shiga Prefecture' 'Shiga Prefecture市部' 'Kyoto' 'Kyoto市部' 'Kyoto City' 'Osaka' 'Osaka市部' 'Osaka City' 'Sakai City' 'Hyogo prefecture'
'Hyogo Prefecture City' 'Kobe City' 'Nara Prefecture' 'Nara Prefecture市部' 'Wakayama Prefecture' 'Wakayama Prefecture市部' 'Tottori prefecture' 'Tottori prefecture市部' 'Shimane Prefecture' 'Shimane Prefecture市部'
'Okayama Prefecture' 'Okayama Prefecture市部' 'Okayama City' 'Hiroshima Prefecture' 'Hiroshima Prefecture市部' 'Hiroshima city' 'Yamaguchi Prefecture' 'Yamaguchi Prefecture市部' 'Tokushima Prefecture' 'Tokushima Prefecture市部' 'Kagawa Prefecture'
'Kagawa Prefecture City' 'Ehime Prefecture' 'Ehime Prefecture市部' 'Kochi Prefecture' 'Kochi Prefecture市部' 'Fukuoka Prefecture' 'Fukuoka Prefecture市部' 'Kitakyushu' 'Fukuoka City' 'Saga Prefecture'
'Saga Prefecture City' 'Nagasaki Prefecture' 'Nagasaki Prefecture市部' 'Kumamoto Prefecture' 'Kumamoto Prefecture市部' 'Kumamoto City' 'Oita Prefecture' 'Oita Prefecture市部' 'Miyazaki prefecture' 'Miyazaki prefecture市部'
'Kagoshima prefecture' 'Kagoshima prefecture市部' 'Okinawa Prefecture' 'Okinawa Prefecture市部']
----------------
time_code
1 unique
[2017000000]
----------------
Time axis(Annual)
1 unique
['2017']
----------------
unit
1 unique
['Man']
----------------
value
2737 unique
[21325700. 19530900. 892700. 735200. 334200. 182900. 143800.
189800. 156600. 380400. 320200. 181500. 146500. 133100.
150500. 121100. 276400. 226300. 488800. 441000. 336700.
287200. 339300. 286000. 1353100. 1236400. 232200. 1114300.
1067300. 165600. 2330100. 2319600. 1569400. 1710700. 1642400.
697600. 266100. 147000. 336200. 325700. 125200. 157200.
144800. 181200. 160200. 121800. 106500. 144700. 122700.
339200. 272400. 339000. 288800. 641000. 603900. 120600.
131400. 1329600. 1257400. 395200. 310400. 272900. 259300.
245500. 469500. 446000. 267500. 1535900. 1505000. 474000.
139100. 918600. 876700. 261900. 220100. 175100. 141400.
111500. 83700. 61400. 103100. 93500. 282100. 265900.
108900. 459200. 430400. 204500. 213200. 204400. 92400.
69200. 139900. 117900. 197000. 179500. 98400. 80700.
869100. 759500. 148300. 288600. 123500. 103600. 207600.
186600. 264800. 216200. 118900. 169800. 162800. 168400.
141500. 262400. 233200. 253800. 193400.]
'''
The reasons for employment are described for each educational background, motive, and region. Remove unnecessary data.
df_01 = df[(df['education'] == 'Total number')
& (df['Reason for hiring'] != 'Total number')
& (df['education'] == 'Total number')
& (df['Area classification'] == 'Nationwide')
& (df['Incumbent status / employment form'] != 'Total number')]
Try to output a heat map by specifying xy
df_pivot = pd.pivot(data=df_01, columns='Incumbent status / employment form', index='Reason for hiring', values='value')
s = sns.heatmap(df_pivot, annot=True, cmap='Blues',
fmt="0", linewidths=1, linecolor="white",)
plt.savefig('heat.png')
plt.close('all')
Part-time workers should be legally the same form of employment, but for some reason they are separated. According to the Statistics Bureau, the definition of part is as follows.
16A-Q07 Number of part-timers and part-time workers Q & A about labor force survey (answer)
However, even if I read it, I can't find any reason to divide the part-time job.
There are many reasons for choosing a part-time job because I want to use my spare time and I want to'subsidize' my household and tuition fees.
The ratio of part-time workers is too large to see other maps, so normalize. Pandas comes standard with means and standard deviations. At times like this, Pandas is a lot of fun, and above all, fun.
df_pivot=(df_pivot-df_pivot.mean())/df_pivot.std()
And plot
Contractors are often selected because they require specialized skills. From the map, part-time workers and part-time workers work for relatively positive reasons, but temporary and contract employees tend to have many negative reasons because they could not become regular employees. Even with the problem of non-regular employment, it is clear that contract employees and temporary employees in non-regular employment are particularly dissatisfied. On the contrary, as far as the part is concerned, it can be seen that there is not much dissatisfaction.