[PYTHON] Use Tabpy with Cloud Run (on GKE)

I've never touched Tabpy, so A memo when I tried running Tabpy on Cloud Run (on GKE) while also studying Cloud Run

What is Tabpy

A mechanism that enables Tableau to visualize the results of statistical processing and machine learning in Python by linking Tableau and Python. Install the Python module with pip on any PC or server, start the Tabpy service, and access Tabpy using Tableau's external service connection.

What is Cloud Run?

https://cloud.google.com/run/?hl=ja Fully managed service for running containers Charged according to request processing time It seems to be an auto scale with a nice feeling, so it requires less infrastructure than GKE, but there are various restrictions (time limit, port designation, etc.) Cloud Functions-like image that allows individuals to move anything

Difference from on GKE https://tomokazu-kozuma.com/the-difference-between-cloud-run-and-cloud-run-on-gke/

Why Cloud Run on GKE

I couldn't connect to Tabpy with Cloud Run. .. I'm not sure if the inconsistency between the SSL certificate automatically issued by Cloud Run and the SSL certificate used for SSL conversion of Tabpy is bad. .. .. (Certificates automatically issued by Cloud Run cannot be used)

Development environment

Premise to use Cloud Shell

Deploy Tabpy to Cloud Run

Dockerfile

Create a GKE cluster (sample)

Set up referring to the following https://cloud.google.com/run/docs/gke/setup

Create sample cluster

gcloud beta container clusters create <Cluster name> \
   --machine-type=g1-small \
   --preemptible \
   --num-nodes=3 \
   --disk-size=10 \
   --zone=us-central1-a \
   --addons=HorizontalPodAutoscaling,HttpLoadBalancing,Istio,CloudRun \
   --cluster-version=latest \
   --enable-stackdriver-kubernetes \
   --enable-ip-alias \
   --scopes cloud-platform

To enable Cloud Run for an already established GKE cluster, enable the blue circle items in the capture below in order from the top.

Creating a Dockerfile

Add libraries to install as needed Just install tabpy with pip install and start tabpy, it's basically OK

Dockerfile

from continuumio/miniconda3:latest

ARG config_dir=/tmp/
ADD file.conf $config_dir

WORKDIR $config_dir

RUN pip install --upgrade pip && \
        pip install numpy pandas scikit-learn scipy textblob nltk vaderSentiment && \
        pip install reverse_geocoder geopy && \
        pip install tabpy==1.0.0

RUN tabpy-user add -u <username> -p <password> -f pwd.txt

ENV PORT 8080
EXPOSE 8080

ENTRYPOINT tabpy --config=file.conf

file.conf

#Create this file in the same directory as the Dockerfile
[TabPy]
TABPY_PORT = 8080
TABPY_PWD_FILE = /tmp/pwd.txt

Creating a Docker image

Execute the following command in the directory where the Dockerfile is stored. An image called tabpy-sample is created.

python

docker image build -t asia.gcr.io/<Project ID>/tabpy-sample:latest .

PUSH of Docker image

Push to Container Registry

python

docker push asia.gcr.io/<Project ID>/tabpy-sample:latest

Deploy to Cloud Run

CloudRunOnGKE

gcloud beta run deploy tabpy-sample \
   --image asia.gcr.io/<Project ID>/tabpy-sample \
   --platform gke \
   --cluster <GKE cluster name> \
   --cluster-location <Location>

When the platform option is managed, it will be deployed to CloudRun

(reference)CloudRun

gcloud beta run deploy tabpy-sample \
   --image asia.gcr.io/<Project ID>/tabpy-sample \
   --platform managed \
   --region <region>

Custom domain mapping

https://cloud.google.com/run/docs/mapping-custom-domains?hl=ja

When deployed to Cloudrun on GKE, By default you can't access the service without specifying the host in the header via curl Since Tableau's external connection service cannot specify headers, it is necessary to map and access a custom domain.

Confirmation of domain ownership

python

gcloud domains verify [DOMAIN]

Map domain to service

python

gcloud beta run domain-mappings create --service [SERVICE] --domain [DOMAIN]

Get DNS record data

python

gcloud beta run domain-mappings describe --domain [DOMAIN]

Fixed IP Reservation (if using Cloudrun on GKE)

python

gcloud compute addresses create [IP-NAME] --addresses [EXTERNAL-IP] --region [REGION]

EXTERNAL-IP specifies the IP address of the A record acquired in the procedure for acquiring DNS record data.

Add a DNS record with your domain registrar

https://cloud.google.com/run/docs/mapping-custom-domains?hl=ja#dns_update

Addictive point

Dockerfile I made a Dockerfile with reference to the following (plagiarized), but it seems that it was a little old. .. https://github.com/erichannell/TabPy-docker Currently, pip install seems to complete the tabpy installation.

Modify Dockerfile by referring to the official Tabpy documentation https://github.com/tableau/TabPy/blob/master/docs/server-install.md#starting-tabpy

Port when deploying Cloudrun

Cloudrun must specify listen PORT to 8080 https://stackoverflow.com/questions/55662222/container-failed-to-start-failed-to-start-and-then-listen-on-the-port-defined-b https://cloud.google.com/run/docs/reference/container-contract?hl=ja#port

Tabpy has a default port of 9004, so you need to specify the port 8080 using the config file. https://github.com/tableau/TabPy/blob/master/docs/server-config.md

Access Tabpy on Cloud Run with Tableau

Connect from Tableau Desktop

Click Manage External Service Connections from the top help menu

Server: Custom domain set Port: 80 Username: User set in Dockerfile Password: User set in Dockerfile

Connect from Tableau Server

Allow External Service Connections in Tableau Server

https://help.tableau.com/current/server-linux/ja-jp/config_r_tabpy.htm https://help.tableau.com/current/server-linux/ja-jp/cli_security_tsm.htm#tsm_security_vizql-extsvc-ssl-enable

Run the following on the Tableau Server command line

Enable external services on tableau server

tsm security vizql-extsvc-ssl enable --connection-type tabpy --extsvc-host <hostname> --extsvc-port <PORT> --extsvc-username <Tabpy username> --extsvc-password <Tabpy password>

tsm pending-changes apply

At the moment (January 2020), it seems that it cannot be used from Tableau Online. .. ..

orz https://help.tableau.com/current/pro/desktop/ja-jp/r_connection_manage.htm

Workbooks containing external service scripts cannot be published to Tableau Online.

Pages that I referred to in various ways

[Let's run Tabpy in a container using Docker! ](Http://lovedata.main.jp/2018/11/20/docker%E3%82%92%E4%BD%BF%E3%81%A3%E3%81%A6mac%E3%81%A7% E3% 82% 82tabpy% E3% 82% 92% E5% 8B% 95% E3% 81% 8B% E3% 81% 9D% E3% 81% 86% EF% BC% 81 /)

Think about Google Cloud Run ports

[Knative in Cloud Run on GKE](https://qiita.com/toshi0607/items/eeeabe81b1beac343b6b#cloud-run-on-gke%E3%81%AE%E3%83%87%E3%83%97% E3% 83% AD% E3% 82% A4)