[GCP] A memorandum when running a Python program on Cloud Functions

I tried to run Python with Cloud Functions and it melted for days, so I will leave it so that my future self will not melt any more time.

log

When the process ends normally, you can also check the log output by print from Cloud Logging, When an exception occurs, print in the except clause does not print details to Cloud Logging, leaving only the information crash. With this, there is no hint, and I have no idea what kind of error occurred in which process. If you use the logging module, the log will be output firmly where you want to output, and the icon for each row of Cloud Logging will change according to the log level, so it will be easier to use.

import logging
import traceback

def main():
    try:
        #processing
        logger.info("output by logger")
        print("output by print") #Output when processing ends normally
    except:
        traceback.print_exc() #Not output to Cloud Logging
        logger.error("error by logger: ", exc_info=True)
        raise

Around authentication with other services of GCP / Google

With the ADC (https://cloud.google.com/docs/authentication/production?hl=ja), you can get the credential information with google.auth.default (). Depending on the service to be authenticated, the scope information of Credential is given, and the role to the execution service account is set as necessary.

When developing locally, save the credential information in a json file, and just specify the path of the json file in the environment variable GOOGLE_APPLICATION_CREDENTIALS to pass the authentication ofgoogle.auth.default (), so local and Cloud There is no need to branch the processing in Functions.

import google.auth
from google.cloud import kms_v1
import gspread

def spreadsheet(url):
    credentials, _ = google.auth.default(scopes=['https://spreadsheets.google.com/feeds','https://www.googleapis.com/auth/drive'])
    gc = gspread.authorize(credentials)
    return gc.open_by_url(url)

def kms_client():
    credentials, _ = google.auth.default()
    return kms_v1.KeyManagementServiceClient(credentials=credentials)

Investigate the cause of 400 Bad Request error when creating a new from Cloud Functions API

It's a bit off the Python program Tips, but I'm addicted to deploying functions to Cloud Functions using terraform, so I'll leave that as well.

I don't think you'll hit this problem when creating a function from the GCP console on the web, but if you apply it even though you can see the difference in plan in terraform.

400 Bad Request INVALID_ARGUMENT The request has errors

↑ Only a message like this is output, and I don't know which item is defective.

For terraform, adding TF_LOG = debug to the environment variable does not change much of the information you get. However, make a note of the request body information output by this TF_LOG = debug.

In the cloud functions api Reference Page (https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/create?hl=ja) ** Try this API ** If you enter the request body output by TF_LOG = debug and execute EXECUTE, the field name with the defect and the details of the defect will be output to message in addition to 400 Bad Request INVALID_ARGUMENT The request has errors. think.

Predefined environment variables

Since I print it out with print every time and check it, I will leave the output as it is as a memorandum.

environ({
  'PORT': '8080',
  'X_GOOGLE_WORKER_PORT': '8091',
  'NODE_ENV': 'production',
  'X_GOOGLE_ENTRY_POINT': 'entrypoint',
  'FUNCTION_TRIGGER_TYPE': 'HTTP_TRIGGER',
  'GCLOUD_PROJECT': 'project_id',
  'DEBIAN_FRONTEND': 'noninteractive',
  'X_GOOGLE_FUNCTION_MEMORY_MB': '256',
  'FUNCTION_TIMEOUT_SEC': '60',
  'SUPERVISOR_INTERNAL_PORT': '8081',
  'ENTRY_POINT': 'entrypoint',
  'X_GOOGLE_LOAD_ON_START': 'false',
  'X_GOOGLE_FUNCTION_REGION': 'asia-northeast1',
  'X_GOOGLE_FUNCTION_VERSION': '1',
  'WORKER_PORT': '8091',
  'VIRTUAL_ENV': '/env',
  'X_GOOGLE_GCP_PROJECT': 'project_id',
  'CODE_LOCATION': '/user_code',
  'PWD': '/user_code',
  'X_GOOGLE_CONTAINER_LOGGING_ENABLED': 'false',
  'FUNCTION_NAME': 'dummy_function',
  'X_GOOGLE_CODE_LOCATION': '/user_code',
  'FUNCTION_MEMORY_MB': '256',
  'X_GOOGLE_FUNCTION_IDENTITY': 'dummy_service_account@project_id.iam.gserviceaccount.com',
  'FUNCTION_IDENTITY': 'dummy_service_account@project_id.iam.gserviceaccount.com',
  'FUNCTION_REGION': 'asia-northeast1',
  'GCP_PROJECT': 'project_id',
  'X_GOOGLE_FUNCTION_NAME': 'dummy_function',
  'X_GOOGLE_SUPERVISOR_HOSTNAME': 'supervisor',
  'HOME': '/tmp',
  'SUPERVISOR_HOSTNAME': 'supervisor',
  'PATH': '/env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
  'X_GOOGLE_GCLOUD_PROJECT': 'project_id',
  'X_GOOGLE_FUNCTION_TRIGGER_TYPE': 'HTTP_TRIGGER',
  'X_GOOGLE_SUPERVISOR_INTERNAL_PORT': '8081',
  'X_GOOGLE_FUNCTION_TIMEOUT_SEC': '60',
  'LC_CTYPE': 'C.UTF-8',
})

Summary

If useful information can be output to the log, defect investigation will be easier. Authentication is finely controlled by the role setting of the service account, relying on google.auth.

I have only a feeling that there is still a point of harmony, but I feel that I have grasped the point.