[PYTHON] Use AppSync on the front and back ends
Use AppSync on the front and back ends
This article is a hands-on article for reviewing and fixing the knowledge gained by developing Serverless Web App Mosaic. It is one of w2or3w / items / 87b57dfdbcf218de91e2).
It would be nice to read this article after looking at the following.
- Creating a new project with Vue CLI and implementing common templates
- Image processing with Lambda + OpenCV (gray image creation)
Introduction
An API called AppSync is used for data management of uploaded images and processed images, and for data transfer with the client side. DynamoDB is used as the data source for AppSync. AppSync can also be set up with the Amplify CLI, but I didn't use the Amplify CLI because it seemed that I couldn't specify the partition key or sort key for DynanoDB. Build DynamoDB and AppSync in the AWS console. Request from Vue on the front end using Amplify, and from Lambda (Python) on the back end by HTTP.
content
AppSync setup
I can set up AppSync with the Amplify CLI, but it seems that I can't specify the partition key or sort key of DynamoDB. I may be able to do it, but I didn't know how to do it. If anyone knows, please let me know. So, it's a little more troublesome than the command line, but let's make it with the AWS console.
Creating a DynamoDB table
Create DynamoDB as the data source for AppSync first.
AWS Console> DynamoDB> Create Table
Create a table with the following settings.
Table name: sample_appsync_table
Partition key: group (string)
Sort key: path (string)
Creating the AppSync API
Once DynamoDB is created, it's time to create AppSync. AWS Console> AppSync> Create API
Step1: Getting started
Select "Import DynamoDB table" and press the "Start" button. 
Step2: Create a model
Select the table you created earlier (sample_appsync_table).
Select "New role" in "Create or use an existing role".
Press the "Import" button.
Just press the "Create" button.
Step3: Create resource
Set the API name and press the "Create" button.
Get schema.json
AWS Console> AppSync> sample_appsync_table> From the Schema menu
Download schema.json. This file is used in the web application.
Obtaining credentials
AWS Console> AppSync> sample_appsync_table> From the Settings menu
Check the information of ʻAPI URL and ʻAPI KEY in API Detail. This information will be used in the app.
Leave the authentication mode as "API Key". The API key is valid for 7 days by default, but you can extend it up to 365 days by editing it. I will write another article about how to use the authentication mode as a Cognito user like Storage.
Subscription on the front end (Vue web app)
After setting up AppSync, we will continue to update the web application.
Add graphql definition file
Create a src / graphql folder and add 3 files.
src/graphql/schema.json
Add the file you downloaded earlier to the project as it is.
src/graphql/queries.js
export const listSampleAppsyncTables = `query listSampleAppsyncTables($group: String) {
listSampleAppsyncTables(
limit: 1000000
filter: {
group: {eq:$group}
}
)
{
items
{
group
path
}
}
}
`;
It is a query to get the record list by specifying the group of the partition key. This is a mystery, but if you don't specify the limit, you won't be able to get the data. I think it's an AppSync specification, not graphql, but what about it? I've specified a reasonably large number of 1000000, but honestly it's too subtle. If anyone knows a better way to write, please let me know.
src/graphql/subscriptions.js
export const onCreateSampleAppsyncTable = `subscription OnCreateSampleAppsyncTable($group: String) {
onCreateSampleAppsyncTable(group : $group) {
group
path
}
}
`;
This is a subscription for specifying the partition key group and notifying you with the information when a record is inserted. I wrote "when a record is inserted", but inserting a record directly into DynamoDB doesn't work and it must be inserted by AppSync create.
Added AppSync information to aws-exports.js
Add the information required to access AppSync to src / aws-exports.js.
src/aws-exports.js
const awsmobile = {
"aws_project_region": "ap-northeast-1",
"aws_cognito_identity_pool_id": "ap-northeast-1:********-****-****-****-************",
"aws_cognito_region": "ap-northeast-1",
"aws_user_pools_id": "ap-northeast-1_*********",
"aws_user_pools_web_client_id": "**************************",
"oauth": {},
"aws_user_files_s3_bucket": "sample-vue-project-bucket-work",
"aws_user_files_s3_bucket_region": "ap-northeast-1",
"aws_appsync_graphqlEndpoint": "https://**************************.appsync-api.ap-northeast-1.amazonaws.com/graphql",
"aws_appsync_region": "ap-northeast-1",
"aws_appsync_authenticationType": "API_KEY",
"aws_appsync_apiKey": "da2-**************************"
};
export default awsmobile;
The information contained in this file is important, so please handle it with care so as not to leak it.
Web application implementation
After selecting an image in Home and uploading it, implement it to list the information of the image uploaded or monochrome processed image at the page transition destination.
Add a page called List. Router settings for that.
src/router.js
import Vue from 'vue';
import Router from 'vue-router';
import Home from './views/Home.vue';
import About from './views/About.vue';
import List from './views/List.vue';
Vue.use(Router);
export default new Router({
routes: [
{
path: '/',
name: 'home',
component: Home,
},
{
path: '/about',
name: 'about',
component: About,
},
{
path: '/list',
name: 'list',
component: List,
},
]
});
List page view
src/views/List.vue
<template>
<List />
</template>
<script>
import List from '../components/List'
export default {
components: {
List
}
}
</script>
List page components
src/components/List.vue
<template>
<v-container>
<p>list</p>
<router-link to="/" >link to Home</router-link>
<hr>
<v-list>
<v-list-item v-for="data in this.dataList" :key="data.path">
<v-list-item-content>
<a :href="data.image" target=”_blank”>
<v-list-item-title v-text="data.path"></v-list-item-title>
</a>
</v-list-item-content>
<v-list-item-avatar>
<v-img :src="data.image"></v-img>
</v-list-item-avatar>
</v-list-item>
</v-list>
</v-container>
</template>
<script>
import { API, graphqlOperation, Storage } from 'aws-amplify';
import { listSampleAppsyncTables } from "../graphql/queries";
import { onCreateSampleAppsyncTable } from "../graphql/subscriptions";
const dataExpireSeconds = (30 * 60);
export default {
name: 'List',
data: () => ({
group: null,
dataList: [],
}),
mounted: async function() {
this.getList();
},
methods:{
async getList() {
this.group = this.$route.query.group;
console.log("group : " + this.group);
if(!this.group){
return;
}
let apiResult = await API.graphql(graphqlOperation(listSampleAppsyncTables, { group : this.group }));
let listAll = apiResult.data.listSampleAppsyncTables.items;
for(let data of listAll) {
let tmp = { path : data.path, image : "" };
let list = [...this.dataList, tmp];
this.dataList = list;
console.log("path : " + data.path);
Storage.get(data.path.replace('public/', ''), { expires: dataExpireSeconds }).then(result => {
tmp.image = result;
console.log("image : " + result);
}).catch(err => console.log(err));
}
API.graphql(
graphqlOperation(onCreateSampleAppsyncTable, { group : this.group } )
).subscribe({
next: (eventData) => {
let data = eventData.value.data.onCreateSampleAppsyncTable;
let tmp = { path : data.path, image : "" };
let list = [...this.dataList, tmp];
this.dataList = list;
console.log("path : " + data.path);
Storage.get(data.path.replace('public/', ''), { expires: dataExpireSeconds }).then(result => {
tmp.image = result;
console.log("image : " + result);
}).catch(err => console.log(err));
}
});
},
}
}
</script>
Get the group with the query parameter. With mounted before the screen is displayed, record data is acquired by specifying a group, or record data is acquired when an insert event is received. The acquired record data is held in a member variable array called dataList and displayed side by side in v-list on the screen. In v-list, the path and image of record data are displayed. The image is accessed by getting the address with the expiration date (30 minutes) with Strage.
src/components/Home.vue
<template>
<v-container>
<p>home</p>
<router-link to="about" >link to About</router-link>
<hr>
<v-btn @click="selectFile">
SELECT A FILE !!
</v-btn>
<input style="display: none"
ref="input" type="file"
@change="uploadSelectedFile()">
</v-container>
</template>
<script>
import Vue from 'vue'
import { Auth, Storage } from 'aws-amplify';
export default {
name: 'Home',
data: () => ({
loginid: "sample-vue-project-user",
loginpw: "sample-vue-project-user",
}),
mounted: async function() {
this.login();
},
methods:{
login() {
console.log("login.");
Auth.signIn(this.loginid, this.loginpw)
.then((data) => {
if(data.challengeName == "NEW_PASSWORD_REQUIRED"){
console.log("new password required.");
data.completeNewPasswordChallenge(this.loginpw, {},
{
onSuccess(result) {
console.log("onSuccess");
console.log(result);
},
onFailure(err) {
console.log("onFailure");
console.log(err);
}
}
);
}
console.log("login successfully.");
}).catch((err) => {
console.log("login failed.");
console.log(err);
});
},
selectFile() {
if(this.$refs.input != undefined){
this.$refs.input.click();
}
},
uploadSelectedFile() {
let file = this.$refs.input.files[0];
if(file == undefined){
return;
}
console.log(file);
let dt = new Date();
let dirName = this.getDirString(dt);
let filePath = dirName + "/" + file.name;
Storage.put(filePath, file).then(result => {
console.log(result);
}).catch(err => console.log(err));
this.$router.push({ path: 'list', query: { group: dirName }});
},
getDirString(date){
let random = date.getTime() + Math.floor(100000 * Math.random());
random = Math.random() * random;
random = Math.floor(random).toString(16);
return "" +
("00" + date.getUTCFullYear()).slice(-2) +
("00" + (date.getMonth() + 1)).slice(-2) +
("00" + date.getUTCDate()).slice(-2) +
("00" + date.getUTCHours()).slice(-2) +
("00" + date.getUTCMinutes()).slice(-2) +
("00" + date.getUTCSeconds()).slice(-2) +
"-" + random;
},
}
}
</script>
Use uploadSelectedFile to move to the List page after uploading the file. At that time, a query parameter called group is attached.
This completes the repair of the front end (web application), but the operation check is done after the back end side is completed.
Hit from the backend (Lambda Python)
We will implement a record insertion via AppSync for the file uploaded from the web application and the path (S3 Key) of the monochrome image generated and uploaded by Lambda.
Install gql.
pip install gql -t .
Update lambda_function.py as follows:
lambda_function.py
# coding: UTF-8
import boto3
import os
from urllib.parse import unquote_plus
import numpy as np
import cv2
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
s3 = boto3.client("s3")
from gql import gql, Client
from gql.transport.requests import RequestsHTTPTransport
ENDPOINT = "https://**************************.appsync-api.ap-northeast-1.amazonaws.com/graphql"
API_KEY = "da2-**************************"
_headers = {
"Content-Type": "application/graphql",
"x-api-key": API_KEY,
}
_transport = RequestsHTTPTransport(
headers = _headers,
url = ENDPOINT,
use_json = True,
)
_client = Client(
transport = _transport,
fetch_schema_from_transport = True,
)
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
key = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
logger.info("Function Start (deploy from S3) : Bucket={0}, Key={1}" .format(bucket, key))
fileName = os.path.basename(key)
dirPath = os.path.dirname(key)
dirName = os.path.basename(dirPath)
orgFilePath = u'/tmp/' + fileName
processedFilePath = u'/tmp/processed-' + fileName
if (key.startswith("public/processed/")):
logger.info("not start with public")
return
apiCreateTable(dirName, key)
keyOut = key.replace("public", "public/processed", 1)
logger.info("Output local path = {0}".format(processedFilePath))
try:
s3.download_file(Bucket=bucket, Key=key, Filename=orgFilePath)
orgImage = cv2.imread(orgFilePath)
grayImage = cv2.cvtColor(orgImage, cv2.COLOR_RGB2GRAY)
cv2.imwrite(processedFilePath, grayImage)
s3.upload_file(Filename=processedFilePath, Bucket=bucket, Key=keyOut)
apiCreateTable(dirName, keyOut)
logger.info("Function Completed : processed key = {0}".format(keyOut))
except Exception as e:
print(e)
raise e
finally:
if os.path.exists(orgFilePath):
os.remove(orgFilePath)
if os.path.exists(processedFilePath):
os.remove(processedFilePath)
def apiCreateTable(group, path):
logger.info("group={0}, path={1}".format(group, path))
try:
query = gql("""
mutation create {{
createSampleAppsyncTable(input:{{
group: \"{0}\"
path: \"{1}\"
}}){{
group path
}}
}}
""".format(group, path))
_client.execute(query)
except Exception as e:
print(e)
For ʻENDPOINT and ʻAPI_KEY, refer to the API settings you created earlier in AppSync.
Zip it up, upload it to S3 and deploy it to Lambda.
Operation check
When you run the web app and upload an image, Lambda hits AppSync, detects it and lists it on the web app side. Even if I hit the URL with the query parameter directly, I get the list from AppSync and list it.
The web application (Vue) project is as follows. https://github.com/ww2or3ww/sample_vue_project/tree/work5
The Lambda project is below. https://github.com/ww2or3ww/sample_lambda_py_project/tree/work3
Afterword
This was the first topic I asked aloud after participating in JAWS UG Hamamatsu. Isn't it possible to specify the DynamoDB partition key or sort key in the Amplify API? You don't often use DynamoDB without key settings, right? I don't know WebSocket in detail, but is it something like long polling? I remember speaking with excitement.
By the way, networks are difficult, aren't they? I respect those who can call themselves network engineers. To be honest, I don't really understand even if it is called MQTT over WebSocket. Please tell me in an easy-to-understand manner.
AppSync samples are often talked about as a set with the Amplify CLI, so they are only used from the front end. Chat app, TODO app. DynamoDB is also a full scan. I think that DynamoDB tends to increase the number of records, or it tends to be used for that purpose, and in that sense, scanning all records is not good.
Recommended Posts