[PYTHON] You can see the transition of points in the J League on the graph!

Overview

Article summary

While introducing the service created this time, I will introduce the development flow and stumbling points.

App overview

――You can see the transition of the points of the J League team in the graph. -You can jump to the service from here. --Scraping from the J League data site, data is collected, distributed as an API, and displayed from the front.

Reason for creating the app

――Because I wanted to see the overwhelming results of Kawasaki Frontale visually.

Overall composition

The source code for Front is on GitHub.

Diagram

Technology used

• Python
• AWS(Lambda,APIGateway,DaynamoDB)
• JavaScript(Nuxt.js,Chart.js)

Creation flow & stumbling point

flow

1. Scraping J League Data Site with AWS Lambda and storing it in AWS Daynamo DB
2. Create an API for AWS Daynamo DB data with API Gateway & Lambda
3. Receive API with Nuxt.js and display it with Chart.js
4. Deploy on GitHub pages

Flow details

I will explain the flow of development while introducing the sites that I referred to.

1. Scraping with Lambda

Scraping uses requests and beautifulsoup. To use a third party library with Lambda, I referred to the following site

Use Docker image for scraping with Lambda [Python]

To send scraped data to Daynamo DB, you can use the official library called Python Boto3.

·reference Official Document I tried working with DynamoDB from AWS Lambda using Python Boto 3

2. Create API with Lambda + API Gateway

Create a Lambda function and API Gateway that fetches data from DaynamoDB. It's pretty easy. It was difficult to sort and get the data from DaynamoDB, so I decided to sort again with Lambda. Sorting the list of dictionaries is also easy with the library.

·reference API Gateway + Lambda + DynamoDB Sort the list of Python dictionaries

3.Nuxt.js ✖️　Chart.js I referred to the following to handle chart.js in Nuxt. There is an official sample to handle the API with chart.js, so refer to it! ·reference

Easy graph creation using vue-chart.js with Nuxt.js Official

Impressions

Write your impressions lazily.

――After all Lambda is fun and easy to develop and convenient ~ ――It's nice to be able to easily display graphs with Chart.js! ――I want to improve JS power!

Future outlook

I will list what I want to do in the future.

--Strengthening security (such as inserting an API key) --API test --API documentation --UI improvements

• CICD