I decided to use datastore as a service, and tried and errored how to test in the development environment and CI. : thinking: Since there is less information compared to AWS, I summarized it. : grinning:
--Create a container for the flask app and Cloud Datastore Emulator --Linker-compose services --Initial data input of data store --Operate Cloud Datastore Emulator from the flask app
.
├── app
│ ├── Dockerfile
│ └── src
│ ├── main.py
│ └── requirements.txt
├── datastore
│ ├── Dockerfile
│ ├── entrypoint
│ └── import
│ ├── 2020-01-21.overall_export_metadata
│ ├── default_namespace
│ │ └── kind_test_data
│ │ ├── default_namespace_kind_test_data.export_metadata
│ │ └── output-0
│ └── run.sh
└── docker-compose.yaml
datastore/Dockerfile
Minimal creation from the official SDK image. Grant execution authority to the emulator startup shell and data input shell.
FROM google/cloud-sdk:alpine
RUN apk add --update --no-cache openjdk8-jre \
&& gcloud components install cloud-datastore-emulator beta --quiet
COPY . /datastore/
WORKDIR /datastore
RUN chmod +x ./entrypoint
RUN chmod +x ./import/run.sh
ENTRYPOINT ["./entrypoint"]
datastore/entrypoint
Even when docker-compose down is done, the data is stored in the/ datastore / .data /directory to maintain the data.
If you start it without options, you can only access it inside the container, so start it as --host-port = 0.0.0.0:8081.
From the environment variable, pour it along with the project name.
#!/usr/bin/env bash
gcloud config set project ${DATASTORE_PROJECT_ID}
gcloud beta emulators datastore start \
--data-dir=/datastore/.data \
--host-port=${DATASTORE_LISTEN_ADDRESS}
After starting the server, you can import the data by throwing the save path of the dumped data to the following endpoint.
export DATASTORE_PROJECT_ID
curl -X POST localhost:8081/v1/projects/${DATASTORE_PROJECT_ID}:import \
-H 'Content-Type: application/json' \
-d '{"input_url":"/datastore/import/2020-01-21.overall_export_metadata"}'
This time, from the gcp console, I brought the entire directory dumped to gcs under datastore / import.
You may generate sdk data directly.
Build a fluent app for the sample.
app/Dockerfile
FROM python:3.7-slim-buster
ENV HOME /api/
ADD ./ ${HOME}
WORKDIR ${HOME}
RUN pip install --upgrade pip \
&& pip install --no-cache-dir -r ${HOME}src/requirements.txt
ENTRYPOINT ["python", "src/main.py"]
app/main.py
I think it's a little too suitable, but I created an endpoint that just saves and retrieves data. : thinking:
The authentication information bites the dummy authentication.
from flask import Flask, jsonify
from google.auth.credentials import AnonymousCredentials
from google.cloud import datastore
from os import getenv
client = datastore.Client(
credentials=AnonymousCredentials(),
project=getenv('PROJECT_ID')
)
app = Flask(__name__)
app.config['JSON_AS_ASCII'] = False
@app.route('/')
def index():
key = client.key('EntityKind', 1234)
entity = datastore.Entity(key=key)
entity.update({
'foo': u'bar'
})
client.put(entity)
result = client.get(key)
return jsonify(result)
if __name__ == '__main__':
app.run(host='0.0.0.0')
app/requirements.txt
Flask==1.1.1
google-auth==1.6.2
google-cloud-datastore==1.8.0
docker-compose.yaml
If you open the datastore port on the host side as well, you can see it with a GUI tool, which is convenient.
https://github.com/GabiAxel/google-cloud-gui
version: '3.7'
x-custom:
gcp:
- &gcp_project_id "dummy"
services:
app:
build: "./app/"
volumes:
- "./app/:/app/"
environment:
FLASK_APP: dev
DATASTORE_HOST: "http://datastore:8081"
DATASTORE_EMULATOR_HOST: "datastore:8081"
PROJECT_ID: *gcp_project_id
TZ: Asia/Tokyo
ports:
- "5000:5000"
depends_on:
- datastore
datastore:
build: "./datastore"
volumes:
- "./datastore/.data:/datastore/.data"
environment:
DATASTORE_PROJECT_ID: *gcp_project_id
DATASTORE_LISTEN_ADDRESS: 0.0.0.0:8081
ports:
- "18081:8081"
docker-compose up
If you access http: // localhost: 5000 with your browser or curl, you should see {"foo": "bar"}.
You can see that the data is stored below
/datastore/.data/WEB-INF/appengine-generated/local_db.bin
datastore_1 | Updated property [core/project].
datastore_1 | WARNING: Reusing existing data in [/datastore/.data].
datastore_1 | Executing: /google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=0.0.0.0 --port=8081 --store_on_disk=True --consistency=0.9 --allow_remote_shutdown /datastore/.data
app_1 | * Serving Flask app "main" (lazy loading)
app_1 | * Environment: production
app_1 | WARNING: This is a development server. Do not use it in a production deployment.
app_1 | Use a production WSGI server instead.
app_1 | * Debug mode: off
app_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
datastore_1 | [datastore] Jan 22, 2020 6:22:00 AM com.google.cloud.datastore.emulator.CloudDatastore$FakeDatastoreAction$9 apply
datastore_1 | [datastore] INFO: Provided --allow_remote_shutdown to start command which is no longer necessary.
datastore_1 | [datastore] Jan 22, 2020 6:22:01 AM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub <init>
datastore_1 | [datastore] INFO: Local Datastore initialized:
datastore_1 | [datastore] Type: High Replication
datastore_1 | [datastore] Storage: /datastore/.data/WEB-INF/appengine-generated/local_db.bin
datastore_1 | [datastore] Jan 22, 2020 6:22:02 AM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub load
datastore_1 | [datastore] INFO: Time to load datastore: 218 ms
datastore_1 | [datastore] API endpoint: http://0.0.0.0:8081
datastore_1 | [datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
datastore_1 | [datastore]
datastore_1 | [datastore] export DATASTORE_EMULATOR_HOST=0.0.0.0:8081
datastore_1 | [datastore]
datastore_1 | [datastore] Dev App Server is now running.
datastore_1 | [datastore]
datastore_1 | [datastore] The previous line was printed for backwards compatibility only.
datastore_1 | [datastore] If your tests rely on it to confirm emulator startup,
datastore_1 | [datastore] please migrate to the emulator health check endpoint (/). Thank you!
datastore_1 | [datastore] The health check endpoint for this emulator instance is http://0.0.0.0:8081/Jan 22, 2020 6:22:11 AM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
datastore_1 | [datastore] INFO: Adding handler(s) to newly registered Channel.
datastore_1 | [datastore] Jan 22, 2020 6:22:11 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
datastore_1 | [datastore] INFO: Detected HTTP/2 connection.
You can also import dump data by sending a request to the emulator endpoint.
docker-compose exec datastore bash ./import/run.sh
I referred to the following site.
--Connect and use Cloud Datastore Emulator from Node.js between Docker containers --Qiita
Recommended Posts