☆ Professor Anzai… !! I want to analyze the data …… Part 1 Data preparation ☆ Let's analyze the NBA player stats (results) with Python. basketball

Everyone across the country of love basketball, hello. My name is Hikoichi Aida. I usually work as a manager and data scientist on a high school basketball team, analyzing various data.

This time, I would like to analyze the player stats (results) of the NBA, which is a professional basketball league in the United States. Analysis is easy, but please keep in touch.

The first is about scraping & preprocessing because it is data preparation. I don't know when the second and subsequent sessions will be, please forgive me. It may not be forever.

environment

I used Google Colaboratory. The process introduced this time can be operated only with the pre-installed library. It's very convenient.

Data collection

Scraping

When I was searching for what to do with the data collection part, I found the following blog article.

• Web Scraping NBA Stats - Towards Data Science

Most of the content is the same as this article, but I thought that the height and weight of the players might be scraped, so I included the URL of the player's personal page as a scraping target. Since the record representing the column name is inserted at regular intervals, that record is skipped.

data = pd.DataFrame()
years = [i for i in range(2000, 2002)]
for year in years:
    url = "https://www.basketball-reference.com/leagues/NBA_{}_per_game.html".format(year)
    # this is the HTML from the given URL
    html = urlopen(url)
    soup = BeautifulSoup(html)

    soup.findAll('tr', limit=2)
    # use getText()to extract the text we need into a list
    headers = [th.getText() for th in soup.findAll('tr', limit=2)[0].findAll('th')]
    # exclude the first column as we will not need the ranking order from Basketball Reference for the analysis
    headers = ['URL'] + headers[1:] + ['Year']

    rows = soup.findAll('tr')[1:]
    player_stats = [[rows[i].a.get('href')] + [td.getText() for td in rows[i].findAll('td')] for i in range(len(rows)) if (rows[i].findAll('td')) and (rows[i].a)]
    stats = pd.DataFrame(player_stats)
    stats['Year'] = str(year)
    stats.columns = headers
    data = pd.concat([data, stats])
data = data.dropna()

Here is an example of the page to be scraped.

• 2018-19 NBA Player Stats: Per Game | Basketball-Reference.com

All the stats of the players who participated in that season are contained in the one-page site, so I think that you can collect enough data even if you drag and copy and paste it into spreadsheet software such as Excel.

This time, we targeted data for 20 years (2000-2019). The scraping result looks like this.

Preprocessing

Missing value

The stats that represent the probability such as FG% (field goal success rate = shoot success rate) seem to be empty if the number of trial shots is 0. Replace with NaN.

data = data.replace(r'^\s*$', np.NaN, regex=True)

Data type change

The data type is a character string. Convert the data you want to treat as numbers to float. There are some data with only integers, but it's troublesome, so I'll make them all float. Before the change, the grade expressed as a percentage is written as .XXX, and it cannot be converted to a number as it is, so add 0 at the beginning.

add_zero_cols = [col for col in data.columns if '%' in col]
num_cols = ['Age'] + list(data.columns[5:-1])

for col in add_zero_cols:
    data[col] = '0' + data[col]
for col in num_cols:
    data[col] = data[col].astype(float)

Let's check. Shows the top 10 average scores.

data.sort_values('PTS', ascending=False)[['Player', 'PTS', 'Year']].head(10)

There seems to be no problem. There are many well-known superstars in Japan such as James Harden, Allen Iverson, and Kobe Bryant.

I took a quick look at the data source page to see what the * mark after the name represents, but I wasn't sure: sweat_smile: It may represent a player in the Hall of Fame.

Additional data Height / weight

We also collected data on height and weight from the URL of the personal page that was additionally included in the scraping item. (Since it can be collected by slightly changing the code at the beginning, the code for additional data is omitted.)

Finally, such data is ready. (Weight and Height columns have been added to the far right)

By the way, the top 10 heights in the last 20 years

df.groupby(['Player']).max().reset_index().sort_values('Height', ascending=False)[['Player', 'Year', 'Height', 'Weight']].head(10)

The unit is m (meter). Third place is that Yao Min, also known as the Great Wall of China. The height is too big.

end

Now that the data is ready, I would like to visualize it next time.

bonus

Average Assist Top 10

data.sort_values('AST', ascending=False)[['Player', 'AST', 'Year']].head(10)

Average Rebound Top 10

data.sort_values('TRB', ascending=False)[['Player', 'TRB', 'Year']].head(10)

It seems that players who have not reached the specified number of games must be excluded.