[PYTHON] Did the number of store closures increase due to the impact of the new coronavirus?

Recently, store sales have been sluggish due to the new coronavirus, and the store is in danger of closing. .. .. I see a lot of news articles like that, so I tried to find out what it was actually like! This is a series of articles I tried, so please close your eyes for small gabber. .. ..

The outline is like this. --Data acquisition source: https://kaiten-heiten.com/ --Method: Tag the data and plot the number of closed stores for each tag. ――What to see: How many stores are closed in April 2020 compared to the usual month? --Evaluation period: January 1, 2019 to April 28, 2020 --Understood (1) The number of rental cars and hotels closed in April 2020 is higher than usual (2) The number of restaurants and retail stores closed every month is large, and April 2020 is not particularly large. ③ The number of stores opened is decreasing overall

Data organization

Data acquisition

The data was obtained from the store opening and closing.com (https://kaiten-heiten.com/).

Tagging

The following four types of tags are attached to the tags.

--Industry (for example, restaurants, retail stores, etc.) --Small industry (grilled meat, Genghis Khan, etc. at restaurants) --Region (Hokkaido, Tohoku, etc.) --Prefectures (Kyoto, Kagoshima, etc.)

Below are some screenshots of the opening and closing .com articles. If you look at the bottom of the image, you can see that there are four types of tags that you want to get this time. Use this by scraping.

Information such as opening and closing is taken from the article title. In the image above, it will be "closed". In addition, the relationship between industry-small industry and the relationship between region-prefecture are created from the {Open / Close} information- {Industry / Region} page in Open / Close.com. The screenshots on those pages are below, and you can see the relationship by looking at them.

Tagging problems and precautions

There were some issues with tagging. Note that this solution is arbitrary and can distort the results.

There is a small industry tag in the article tag, but there is no industry tag

For example, "Shoe store" is attached as a small industry as shown below, but the industry is not classified.

To solve this, create a dictionary of {industry: small industry group} from the page that describes the industry-small industry relationship introduced in the previous section image. Using this, I adopted the industry to which the tagged small industry belongs as the industry for the article.

The method of creating a dictionary was created by creating a dictionary for each of store opening information-industry and store closing information-industry, and creating a union of the elements. The reason for taking the union is that there is a difference in the display on the {opening, closing} information page.

#Create a dictionary from the url of the opening / closing page
def get_group(url):
    html = urlopen(url)
    bsObj = BeautifulSoup(html, 'html.parser')
    
    body = bsObj.find('div', attrs={'class': 'post_body'})
    titles = bsObj.find('div', attrs={'class': 'post_body'}).find_all('h3')
    group = [title.text for title in titles]
    elems = bsObj.find('div', attrs={'class': 'post_body'}).find_all('p')
    small_group = [[a.text for a in elem.find_all('a')] for elem in elems if elem.find('a')]
    dict_group_to_small_group = {g: set(sg) for g, sg in zip(group, small_group)}
    return dict_group_to_small_group

g_kaiten = get_group('https://kaiten-heiten.com/kaiten/kaiten-gyousyubetsu/')
g_heiten = get_group('https://kaiten-heiten.com/heiten/heiten-gyousyubetsu/')

#Create a dictionary with the sum of the elements of the created dictionary
group_to_small_group = dict(g_kaiten)
for key in g_heiten.keys():
    if key in group_to_small_group.keys():group_to_small_group[key] |= g_heiten[key]
    else: group_to_small_group[key] = g_heiten[key]

{Opening, closing} Information-There are a lot of small industry tags that are not by industry

There are a lot of tags that I forgot to type and tags that are not on the industry page. For example, the tag "Doujin Shop" is not listed in the {Open, Closed} Information-Industry page. However, there are actually articles with the "Doujin Shop" tag. If you search for "doujin shop" tag, many articles have retail store as an industry tag.

Therefore, in the case of Doujinshi, add the industry "Retailer" to the ** {Industry: Small Industry Group} dictionary by hand as follows.

group_to_small_group['retail store'] |= set(['Doujin shop'])

We did this for ** almost all tags ** that are not in the {industry: small industry group} dictionary. The only small industry tag I didn't do was "blog". This is because it was unclear what kind of industry it belongs to, and the small industry tag "blog" is ** ignored ** this time.

I haven't done it this time, but as a smarter method, I think it would have been better to put off those without an industry tag and attach the industry tag with the largest number of cases with the desired small industry tag. .. ..

The article tag has a prefecture tag but no region tag

For example, the following screenshots have the prefecture "Osaka" tag, but do not have the desired area tag "Kinki".

This can be done by creating and using a {region: prefecture group} dictionary in the same way as "the article tag has a small industry tag but no industry tag".

g_kaiten = get_group('https://kaiten-heiten.com/kaiten/area-open/')
g_heiten = get_group('https://kaiten-heiten.com/heiten/area-close/')

region_to_pref = dict(g_kaiten)
for key in g_heiten.keys():
    if key in region_to_pref.keys():region_to_pref[key] |= g_heiten[key]
    else: region_to_pref[key] = g_heiten[key]

There is a region tag that is not on the region page

This happens frequently with tags in Hokkaido. Just like in the case of "Article tag has prefecture tag but no region tag", you can add it to the dictionary every time you find it. For example, you can do as follows.

region_to_area['Hokkaido'] |= set(['Ashibetsu'])

• Although we are making a {region, prefecture} dictionary, only Hokkaido is {Hokkaido, place names in Hokkaido}. .. ..

Data visualization

important point

In this analysis, the date the article was created is the date it was {opened, closed}. However, as a matter of fact, some of the articles posted on Open / Close.com are created by providing information, so the date when the article was created does not necessarily correspond to the {Open / Close} date. Please try to grasp the outline.

The data period is from January 1, 2019 to April 28, 2020.

DataFrame I summarized the obtained data in a DataFrame of pandas. Some of them are below.

--Date: The date the article was created --Group: i: i-th industry (because some industries have multiple industries) --SmallGroup: i: i-th small industry (because some industries have multiple small industries) --Name: Store name --Region: i: i-th region (in the case of simultaneous closing of multiple stores, i = 2 and 3 also have values) --Prefecture: i: i-th prefecture (i = 1, ..., 5 is the same as the area and has a value when multiple stores are closed at the same time) --State: Opening, closing, closing, etc.

• URL：URL --Year / Month: Date with only the year and month extracted

As you can see from the above, the data may be tagged with multiple industries. Therefore, here, when classifying by tag, it is assumed that elements with multiple tags belong to all of those multiple tags. For example, if a store has multiple industry tags (retailers, restaurants), it belongs to both the DataFrame, which is a collection of retail stores only, and the DataFrame, which is a collection of only restaurants. This function is provided by the following function.

def compile_columns_to_one_column(df, columns={'Group:{}'.format(i) for i in range(1, 4+1)}, result_column_name='Group'):
    #Create a set of elements of the target column
    groups = set([])
    for col in columns:
        groups |= set(df[col].dropna())

    #Create a df for each element of the created set and concat
    group_df = pd.DataFrame()
    for g in groups:
        group_df = pd.concat([group_df] + [df[g == df[col]].assign(tmp=g) for col in columns])
    
    return group_df.rename(columns={'tmp': result_column_name})

#DataFrame by industry and small industry
group_df = compile_columns_to_one_column(base_df, {'Group:{}'.format(i) for i in range(1, 4+1)}, result_column_name='Group')
sgroup_df = compile_columns_to_one_column(group_df, {'SmallGroup:{}'.format(i) for i in range(1, 4+1)}, result_column_name='SmallGroup')
group_df = group_df.loc[:, ['Date', 'Year/Month', 'Group', 'Name', 'State', 'URL']]
sgroup_df = sgroup_df.loc[:, ['Date', 'Year/Month', 'Group', 'SmallGroup', 'Name', 'State', 'URL']]

#Create DataFrame for each region and prefecture
region_df = compile_columns_to_one_column(base_df, {'Region:{}'.format(i) for i in range(1, 3+1)}, result_column_name='Region')
pref_df = compile_columns_to_one_column(region_df, {'Prefecture:{}'.format(i) for i in range(1, 5+1)}, result_column_name='Prefecture')
region_df = region_df.loc[:, ['Date', 'Year/Month', 'Region', 'Name', 'State', 'URL']]
pref_df = pref_df.loc[:, ['Date', 'Year/Month', 'Region', 'Prefecture', 'Name', 'State', 'URL']]

For example, the elements of group_df that summarizes the industries are as follows.

Number of stores closed

By industry

The list of industries is as follows. --Service --Sports

• rental --Public facilities / transportation / finance
• retail store --Beauty / Health / Welfare --Sightseeing / Accommodation / Travel / Leisure --Car / Real estate
• restaurant

Whole industry

The code and results counted and plotted by industry are as follows.

tmp = group_df[group_df['State'] == 'closed']
tmp = tmp.groupby(['Year/Month', 'Group']).size().to_frame('Number of stores closed').reset_index()
plt.figure(figsize=(15, 10))
sns.barplot(x='Year/Month', y='Number of stores closed', data=tmp, hue='Group')
plt.legend(ncol=3)
plt.show()

Looking at this, we can see that:

――The number of restaurants and retail stores closed was originally very large, and March and April are not yet large. ――It seems that the number of rental, sightseeing / accommodation / travel / leisure closures has increased.

Based on this, we will look at small rental and tourism industries.

Small industry

I will not write because there are many small industries. Please confirm that the rental and travel small industries are written in the figure below.

First, make a similar plot on the rental system.

rentals = sgroup_df[(sgroup_df['State'] == 'closed') & (sgroup_df['Group'] == 'rental')]
rentals_count = rentals.groupby(['Year/Month', 'SmallGroup']).size().to_frame('Number of stores closed').reset_index()

max_count = rentals_count['Number of stores closed'].max()
min_ym, max_ym = rentals_count['Year/Month'].min(), rentals_count['Year/Month'].max()

plt.figure(figsize=(15, 10))
plt.xlim(min_ym, max_ym)
plt.ylim(0, max_count*1.1)
sns.barplot(x='Year/Month', y='Number of stores closed', data=rentals_count, hue='SmallGroup')
plt.legend(ncol=2)
plt.show()

Looking at this, you can see that "rent-a-car / car-sharing" is increasing very much. Looking at the data based on this, we found that a large number of Toyota Rent-A-Cars closed on April 18. Below is the entire code and raw data.

rentals_rentalcar = rentals[(rentals['SmallGroup'] == 'Car rental / car sharing') & (rentals.Date >= datetime.date(2020, 4, 1))]
writer = pytablewriter.MarkdownTableWriter()
writer.from_dataframe(rentals_rentalcar.loc[:, ['Date', 'Name', 'URL']])
writer.write_table()

Next, let's look at the travel system.

travels = sgroup_df[(sgroup_df['State'] == 'closed') & (sgroup_df['Group'] == 'Sightseeing / Accommodation / Travel / Leisure')]
travels_count = travels.groupby(['Year/Month', 'SmallGroup']).size().to_frame('Count').reset_index()

max_count = travels_count.Count.max()
min_ym, max_ym = rentals_count['Year/Month'].min(), rentals_count['Year/Month'].max()

#Since there are many types, plot by size
size = 9
small_groups = travels_count['SmallGroup'].unique()
plot_num = int(len(small_groups)/size)+1
for i in range(plot_num):
    target_groups = small_groups[i*size:(i+1)*size]
    
    plt.figure(figsize=(15, 5))
    plt.xlim(min_ym, max_ym)
    plt.ylim(0, max_count*1.1)
    sns.barplot(x='Year/Month', y='Count', data=travels_count[travels_count['SmallGroup'].isin(target_groups)], hue='SmallGroup')
    plt.legend(ncol=3)
    plt.show()

Looking at this, we can see that the number of "hotels, business hotels" and "capsule hotels" closed in April has increased compared to the usual month. Similarly, when the raw data is output, it is as follows. The hotel is conspicuously closed in the first cabin. At other hotels, the same vendors are not closed together, and it seems that tourist destination hotels nationwide are closed. Is it a sharp decrease in tourists? .. ..

By region

Next, let's look at each region. However, I could not find such a difference in each region. .. ..

tmp = region_df[region_df['State'] == 'closed']
tmp = tmp.groupby(['Year/Month', 'Region']).size().to_frame('Number of stores closed').reset_index()
plt.figure(figsize=(15, 5))
sns.barplot(x='Year/Month', y='Number of stores closed', data=tmp, hue='Region')
plt.legend(ncol=4)
plt.show()

Number of stores opened

By industry

The type of business is the same as the one handled at the closing. In April, the number of stores opened in all industries seems to be smaller than usual.

tmp = group_df[group_df['State'] == 'Opening']
tmp = tmp.groupby(['Year/Month', 'Group']).size().to_frame('Number of stores opened').reset_index()
plt.figure(figsize=(15, 5))
sns.barplot(x='Year/Month', y='Number of stores opened', data=tmp, hue='Group')
plt.legend(ncol=3)
plt.show()

By region

The areas and prefectures are the same as those handled when the store was closed. As with each industry, the number of stores opened in all regions seems to be lower than usual.

tmp = region_df[region_df['State'] == 'Opening']
tmp = tmp.groupby(['Year/Month', 'Region']).size().to_frame('Number of stores opened').reset_index()
plt.figure(figsize=(15, 5))
sns.barplot(x='Year/Month', y='Number of stores opened', data=tmp, hue='Region')
plt.legend(ncol=4)
plt.show()

Summary and impressions

• Data organization was difficult (mainly tagging)
• Eating and drinking establishments have a large number of closures, and April is not a peculiar number of closures.
• Did many hotels and rental car stores close because the number of people traveling has decreased due to the new coronavirus?
• As expected, the number of new stores is decreasing

What i remembered --Japanese display on seaborn!

Other things I was interested in and investigated

The main thing I investigated was the sudden changes other than April.

June 2019 "Public Facilities / Transportation / Finance" Tag

Looking at the breakdown of the "Public Facilities / Transportation / Finance" tags for this month, it looks like the result of Nomura Securities closing many branches.

The reason why only Nomura Securities appears significantly is that Daiwa Securities has not consolidated or abolished large-scale branches, and other bank-affiliated securities have joint stores with banks, so they are not recognized as closed. Or is it simply because Nomura Securities has more branches than other securities and there are many consolidations and abolitions?

"Karaoke" tag from January to May 2019

Many karaoke tags were closed at this time, so I took a look at the raw data. Below is the raw data from January to May, and it seems that each karaoke company tends to close multiple stores on the same day.

This seems to be a company other than karaoke. For example, closing unprofitable franchise management all at once?