Introduction

** * This article is a sequel to the previous article "Automatically aggregate JCG deck distribution with Python". ** ** Please also refer to the part that reuses the analysis method and code used in the previous article.

In addition, this article assumes the following knowledge.

・ Basic grammar of Python ・ Web scraping ・ Creating a table in Pandas ・ Data visualization with matplotlib

Since the first half focuses on the outline of analysis, I think that you can get a certain image without knowledge of Python. The second half will focus on the actual coding in Python.

This article contains programming code, so ** we recommend viewing it in a PC browser **.

Purpose

We will aggregate the decks that have advanced to the JCG final tournament and perform a ** comparative analysis of the number of cards used.

In Previous article, we analyzed the deck distribution of JCG, but in this article, we will analyze the deck list in detail, so the table below Is automatically created.

** Example: JCG Shadowverse Friday, August 28th Rotation Tournament Final Tournament **

Previous content: Analysis of archetype distribution

** Contents of this time **: Comparative analysis of recruitment cards Example: Spell W スペルW_2389.png

approach

Follow the steps below to proceed with the analysis.

Get the JCG JSON file
Archetype determination
Get the deck list from the Shadowverse portal (hereafter referred to as the portal)
Graph the aggregated data

There are many overlaps with the previous contents regarding steps 1 and 2, so please refer to here for a more detailed explanation.

Get JSON file

Similar to Previous article, ** Get the JSON file from the JCG site **. An example of the acquired JSON file is shown below.

`example.json`


{
    "result": "success",
    "participants": [
        {
            "id": "42155", 
            "chk": 1, #Check-in status
            "nm":  "somey", #Registered name
            "dk": [ #Deck information
                {
                    "cl": 3, #Class 1
                    "hs": "3.3.6so2A.6so2A.6so2A.5-gkQ.5-gkQ.5-gkQ.6_X7Q.6_X7Q.6_X7Q.6_djc.6_djc.6_djc.5-gka.5-gka.5-gka.6turQ.6turQ.6turQ.6xnsw.6xnsw.6xnsw.6_ZZg.6_ZZg.6_ZZg.gPWp2.gPWp2.gPWp2.6q8sC.6q8sC.6q8sC.6twYy.6twYy.6twYy.6xpaI.5-glM.5-glM.5-glM.6t_RI.6t_RI.6t_RI"
                }, #Deck 1 card list (to be exact, part of the portal URL)
                {
                    "cl": 8, #Class 2
                    "hs": "3.8.71TeA.71TeA.71TeA.6zemS.6zemS.6zemS.71QTC.71QTC.71QTC.6zd2w.6zd2w.6zd2w.6s2wi.6zcK2.6zcK2.6zcK2.6s65g.6zcKC.6zcKC.6zcKC.6s5My.6s5My.6s5My.6zhCY.6zhxQ.6zhxQ.6zhxQ.6-UTo.6-UTo.71Xo6.71Xo6.71Xo6.6oEnY.6oEnY.6oEnY.6oHDo.6oHDo.6vvW6.6vvW6.6vvW6"
                } #Deck 2 card list (to be exact, part of the portal URL)
            ],
            "en": 1527921725,
            "te": 1, #Winning information
            "pr": 1,
            "cu": 0
        },

Among them, a list of 40 alphanumeric characters below `" hs ": ``` (example: `" 3.3.6so2A. ‥ Omitted ‥ .6t_RI "```) and a portal Represents part of the URL of.

Use this information to ** archetype determine ** and ** get deck information from the portal **.

Archetype determination

The archetype is determined using the list of 40 cards obtained from the JSON file. The logic used for the judgment is the same as Previous article, and the judgment is made by ** whether the key card is included **.

** Example: Includes 3 magic tool specialty stores → Specialty store witch **

Get the portal URL

Perform ** web scraping ** to collect card name and number information from the portal. The URL of the portal has the following structure.

** (Example: Spell Witch) ** https://shadowverse-portal.com/deckbuilder/create/3?hash=3.3.6so2A.6so2A.6so2A.5-gkQ.5-gkQ.5-gkQ.6_X7Q.6_X7Q.6_X7Q.6_djc.6_djc.6_djc.5-gka.5-gka.5-gka.6turQ.6turQ.6turQ.6xnsw.6xnsw.6xnsw.6_ZZg.6_ZZg.6_ZZg.gPWp2.gPWp2.gPWp2.6q8sC.6q8sC.6q8sC.6twYy.6twYy.6twYy.6xpaI.5-glM.5-glM.5-glM.6t_RI.6t_RI.6t_RI?lang=ja

If you compare it with the value stored in ```" hs "`` of the JSON file, you can see that it represents the following of the portal URL ``hash represents`. Use this to generate the URL of the portal.

Get deck information from the portal

Request the HTML file of the portal from the generated URL, and perform structural analysis of HTML with Beautiful Soup. ** Beautiful Soup is a library that extracts information from HTML files **.

Regarding web scraping of the portal, I referred to this article.

Get Shadowverse deck data using BeautifulSoup and Twitter API in Python

Let's take a closer look at the portal HTML file. [Here](https://shadowverse-portal.com/deck/3.3.6so2A.6so2A.6so2A.5-gkQ.5-gkQ.5-gkQ.6_X7Q.6_X7Q.6_X7Q.6-Ntw.6_djc.6_djc. 6_djc.6turQ.6turQ.6turQ.6xnsw.6xnsw.6xnsw.6_ZZg.6_ZZg.6_ZZg.6q8sC.6q8sC.6q8sC.6twYy.6twYy.6twYy.6t_RI.6t_RI.6t_RI.5- The deck of gPWp2.gPWp2.gPWp2.5-glM.5-glM.5-glM?lang = ja) will be explained as an example. First, display the HTML source code with the browser function (in the case of Chorme, right-click → display the page source)

As you scroll through the pages, you will find information on the card name and number of cards.

`example.html`


<p class="el-card-list-info-name">
<span class="el-card-list-info-name-text">Light of wisdom</span>
</p>
<p class="el-card-list-info-count">×3</p>
<p class="el-card-list-link">
<a
class="el-icon-search is-small tooltipify"
href="/card/100314010"
target="_blank"
data-card-tribe-name="-"
data-card-atk="0"
data-card-evo-atk="0"
data-card-life="0"
data-card-evo-life="0"
data-card-name="Light of wisdom"
data-card-skill-disc="Draw a card."

Looking at other cards in the same way, the card name is el-card-list-info-name-text and the number is ```el-card-list-info-count``. You can see that the class attribute is given.

By using the select function of Beautiful soup, ** elements of specified class attributes and tags can be extracted together **.

Graphing aggregated data

The last aggregated data is tabulated with Pandas and graphed with matplotlib.

coding

Library import

First import the required libraries.

`import.py`


import requests
import bs4
import json
import pandas as pd
import sys

Archetype decision function

Defines a function that determines the archetype. Pass ** class information ** (`` sv_class```) and ** 40 card list ** ( `sv_deck```) as arguments, and sort archetypes according to the defined judgment logic. I will. As a return value, the archetype name of the judgment result is returned.

`archetype.py`


#List of archetypes
arche_dict = {"E":["Reno Seus E", "Amatsu E", "Other E"],"R": ["Evolution R", "Cooperation R", "Other R"],"W": ["Spell W", "Specialty store W", "Arcane W", "Other W"],"D": ["Discard D", "Whale D", "Other D"],"Nc": ["Netherworld Nc", "Funeral Nc",  "Other Nc"],"V": ["Control V", "Frenzy V", "Other V"],"B": ["Eira B", "Control B", "Other B"],"Nm": ["AFNm", "Other Nm"]}

#Class, deck type analysis
def deck_arche_analysis(sv_deck, sv_class):

    if sv_class == 1: #Elf
        if sv_deck.count("6lZu2") == 3:
            return arche_dict["E"][0]
        elif sv_deck.count("6pQTI") == 3:
            return arche_dict["E"][1]
        else:
            return arche_dict["E"][2]
    elif sv_class == 2: #Royal
        if sv_deck.count("6td16") > 1:
            return arche_dict["R"][0]
        elif sv_deck.count("6_B9A") == 3:
            return arche_dict["R"][1]
        else:
            return arche_dict["R"][2]
    elif sv_class == 3: #Witch
        if sv_deck.count("6_djc") == 3:
            return arche_dict["W"][0]
        elif sv_deck.count("6q95g") == 3:
            return arche_dict["W"][1]
        elif sv_deck.count("6t_Rc") == 3:
            return arche_dict["W"][2]
        else:
            return arche_dict["W"][3]
    elif sv_class == 4: #Dragon
        if sv_deck.count("6yB-y") == 3:
            return arche_dict["D"][0]
        elif sv_deck.count("6_zhY") == 3:
            return arche_dict["D"][1]
        else:
            return arche_dict["D"][2]
    elif sv_class == 5: #Necromancer
        if sv_deck.count("6n7-I") > 1:
            return arche_dict["Nc"][0]
        elif sv_deck.count("70OYI") == 3:
            return arche_dict["Nc"][1]
        else:
            return arche_dict["Nc"][2]
    elif sv_class == 6: #Vampire
        if sv_deck.count("6rGOA") == 3:
            return arche_dict["V"][0]
        elif sv_deck.count("6v1MC") ==3:
            return arche_dict["V"][1]
        else:
            return arche_dict["V"][2]
    elif sv_class == 7: #Bishop
        if sv_deck.count("6nupS") == 3:
            return arche_dict["B"][0]
        elif sv_deck.count("6nsN2") == 3:
            return arche_dict["B"][1]
        else:
            return arche_dict["B"][2]
    elif sv_class == 8: #Nemesis
        if sv_deck.count("6zcK2") == 3:
            return arche_dict["Nm"][0]
        else:
            return arche_dict["Nm"][1]

Get the card name and number of cards from the portal

From the URL of the portal, create a function that outputs the combination of the name and number of cards used. As a return value, all pairs of card name and number of cards are returned as dictionary variables.

`get_card-list.py`


#Get the deck from the portal
def get_deck(url):
    #Request html file
    response = requests.get(url)
    #encoding
    response.encoding = response.apparent_encoding
    #html analysis
    soup = bs4.BeautifulSoup(response.text, "html.parser")
    #Get a list of card names
    names = soup.select(r".el-card-list-info-name-text")
    #Get a list of sheets
    counts = soup.select(r".el-card-list-info-count")
    #{card name:Number of sheets}Create a dictionary of
    deck = {name.text: int(count.text[1:]) for name, count in zip(names, counts)}
    return deck

I referred to this article for the function to get the deck list from the portal.

Get Shadowverse deck data using BeautifulSoup and Twitter API in Python

Get JSON file

Specify the JCG tournament you want to check and get the JSON file. Also, define arche_search as the archetype name you want to look up. This is compared with the result of the archetype judgment function and used to check whether it is the desired archetype.

`json.py`


#Enter the JCG tournament number
compe_num = input("Please enter the JCG tournament number you want to look up")
#Enter the archetype you want to look up
arche_search = input("Enter the archetype you want to look up")
#URL of the tournament json file
jcg_url = "https://sv.j-cg.com/compe/view/entrylist/" +  str(compe_num) + "/json"
#Request json file
res_jcg = requests.get(jcg_url)
#Save as json format text
j_txt = json.loads(res_jcg.text)

#If not the final tournament, end the program
if not len(j_txt["participants"]) == 32:
    sys.exit()

The last part determines if the tournament number you entered is the final tournament. If it is a group qualifying, the program will be interrupted.

Collection of card name and number of cards

`analysis.py`


#Dictionary type variable to store user name, card name, number of sheets
arche_summary = {}

#Header part of the portal
dbsp_header = "https://shadowverse-portal.com/deck/"

#Get the number of classes
for i in range(len(j_txt["participants"])):
    #Confirmation of winning information
    if j_txt["participants"][i]["te"] == 0: #Lost
        continue
    elif j_txt["participants"][i]["te"] == 1: #Winning
        for j in range(2):
            #Get class information
            class_ij = j_txt["participants"][i]["dk"][j]["cl"]
            #Acquisition of card information
            deck_ij = j_txt["participants"][i]["dk"][j]["hs"]
            #Determine archetype from class card information
            archetype = deck_arche_analysis(deck_ij, class_ij)

            #When the judgment result matches the archetype you want to check
            if archetype == arche_search:
                #Generate portal URL
                dbsp_url = dbsp_header+deck_ij
                #Get the combination of card name and number of cards from the portal
                deck_dict = get_deck(dbsp_url)
                #arche_Write information in summary
                arche_summary[j_txt["participants"][i]["nm"]] = deck_dict
            else:
                continue
    else:
        continue

Graphing

Finally, convert the aggregated result to Pandas DataFrame format and graph it with matplotlib.

** * If you want to execute the following code as it is, you need to set the Japanese font in matplotlib. ** ** If you want to execute it as it is, download the font referring to the following article and place it in the same directory as this python file.

Graph of matplotlib on Heroku

`graphs.py`


#Import matplotlib
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
#The following two lines are for Japanese setting, font download is required in advance
from matplotlib.font_manager import FontProperties
fontprop = FontProperties(fname="ipaexg.ttf")

#Convert aggregated results to DataFrame format
df_arche_summary = pd.DataFrame(arche_summary)
df_arche_summary = df_arche_summary.fillna(0).astype("int")

#Advanced table settings
fig, ax = plt.subplots()
ax.axis("off")
ax.axis("tight")
tb = ax.table(cellText=df_arche_summary.values,colLabels=df_arche_summary.columns,rowLabels=df_arche_summary.index,colWidths=[0.15]*len(df_arche_summary.columns),loc='center',bbox=[0,0,1,1], cellLoc="center", rowLoc="right")
tb.auto_set_font_size(False)
tb.set_fontsize(8)
for i in range(len(df_arche_summary.columns)):
    tb[0,i].set_text_props(font_properties=fontprop, weight='bold', color="w")
    tb[0,i].set_facecolor('#2b333b')
for k in range(1,len(df_arche_summary.index)+1):
    tb[k,-1].set_text_props(font_properties=fontprop,weight='bold', color="w")
    tb[k,-1].set_facecolor('#2b333b')
plt.savefig(arche_search + "_" + compe_num + ".png ",bbox_inches="tight")

Execution result

** Command line **

`python`


$ python JCG_decklist.py
Enter the JCG tournament number you want to look up: 2389
Enter the archetype you want to look up: Spell W

Execution result スペルW_2389.png

in conclusion

In this article, I introduced how to get the card list of the JCG final tournament by web scraping. The code from the previous article and the code from this time have a lot in common, so you can combine both functions into one program.

As an aside, the most time-consuming part of coding is the layout of the table. The number of columns and rows changed each time, and I had a lot of trouble with adjustment parameters.

If you have any corrections or mistakes regarding the code or analysis method, please let us know in the comments.

[Python] Automatically create a deck list comparison table for the JCG final

Introduction

Purpose

approach

Get JSON file

example.json

Archetype determination

Get the portal URL

Get deck information from the portal

example.html

Graphing aggregated data

coding

Library import

import.py

Archetype decision function

archetype.py

Get the card name and number of cards from the portal

get_card-list.py

Get JSON file

json.py

Collection of card name and number of cards

analysis.py

Graphing

graphs.py

Execution result

python

in conclusion

`example.json`

`example.html`

`import.py`

`archetype.py`

`get_card-list.py`

`json.py`

`analysis.py`

`graphs.py`

`python`