[PYTHON] Automatically generate Google Cloud Build status badges
Introduction
People who have used GitHub Actions or CircleCI think about Google Cloud Build. ** I can't get a status badge! !! ** **
When I looked it up, it seems that some people think that it is similar, and I found some things like "automatically create badges from build results" in OSS, but somehow I got a hand in the itchy place. It's hard to use because it doesn't reach. : sweat_drops: So I made it myself.
You can display the badge in the README like this.
The following is a summary of how to build a series of systems. It's quite long, so I don't care about the story, so I'd like to give it a try! If you have any questions, please go to here.
What is Google Cloud Build?
It is a managed service to realize CI / CD (If you are wondering what CI / CD is, please google ...). GitHub Actions, AWS CodeBuild, CircleCI .com / ja /) Something is a relative. Not to mention Jenkins and GitLab CI / CD, which are not managed services. ... If you know the names around here, you can think of them as Google versions.
So, for example, in the case of GitHub Actions [like this](https://docs.github.com/ja/actions/configuring-and-managing-workflows/configuring-a-workflow#referencing-an-action-in-the- It will generate a build status badge on the same-repository-where-a-workflow-file-uses-the-action), but it seems that Google Cloud Build doesn't have this feature (I found it though I searched the document). I couldn't ...).
It's nice to be able to automate builds, tests, and deployments with CI / CD, but it's annoying that developers don't get feedback in the end. Of course, you can check the result from the console, but opening the console one by one is dull, and I think it's natural to want to put it as a badge in the README of the repository that you often see while working. I will.
If you don't have a badge, you can make one!
Before that, I searched for it
If you are a programmer, you can make your own! It leads to thinking. : sweat_smile: But before that, when I searched for something similar, I found something like that.
It seems that I can achieve what I want to do, but there are some problems.
** Problem 1: Only fixed badges can be generated **
I know that the message part of the badge is fixed depending on the build status, but do you want to change the label part dynamically?
For example[test|success]Or[deploy|success]It's like that.
If it is an existing one, the label part of the badge will be fixed, so I feel that it is actually difficult to use.
** Problem 2: Can't issue multiple badges for a branch ** In view of the above problem, if it is an existing one, badges are saved in units of repository branches, so if you are running a build for testing and deployment, for example, in a branch for development environment, those two badges Will not be able to save (the existing one can only issue one badge to a branch, so it's likely that the label was fixed). Besides, it is quite possible to set various builds for branches such as static analysis and vulnerability testing, and in fact Cloud Build allows such settings, so this limitation is quite painful. ・ ・.
** Problem 3: The work of preparing and arranging the badge as a template is troublesome ** It's possible to prepare it in a script, but I would like to avoid it if possible. I want to keep the work for implementation as simple as possible.
Requirement definition
I decided that it would be difficult to use the existing one, so I aimed for something like this based on the above (bold is the improvement part).
- The ** label part of the badge can be changed dynamically by setting **.
- The message part of the badge should be switched according to the build status.
- Allow ** generate and save multiple badges for repositories / branches **.
- ** Badges are generated completely dynamically **, eliminating the need for advance preparation such as templates.
- Use managed services to build the system so that maintenance is as unnecessary as possible.
- Make it as inexpensive as possible (← important).
System configuration
For the overall flow, I referred to existing products. : pray: On GCP, Cloud Source Repository, Cloud Build, Cloud Pub / Sub, Cloud Functions, and Cloud Storage are linked (the Cloud Source Repository part can be GitHub instead).
When you push to the repository, Cloud Build will be executed with that as a trigger. A message will be sent to Cloud Pub / Sub as the Cloud Build runs. The flow is that Cloud Functions fires triggered by a Cloud Pub / Sub message, a badge image is generated in Cloud Functions, and it is saved in Cloud Storage.
All of them use managed services, so maintenance is not required, and each service has a free tier and only charges are used, so it is easy on your wallet.
I will actually make it
Generating badges (SVG images) with Cloud Functions
It's Cloud Functions, but since I'm Pythonista, it's a definite matter to make it with Python. So, I want to dynamically generate badges in Cloud Functions, so I searched for various libraries that can create images in SVG format, but in the end it is even more dangerous pybadges I found something called google / pybadges). As the name implies, this is a library that can generate badges, and it's exactly what you wanted to do! Thank you for using it. : pray:
It's very simple to use. As expected, Python is too easy to be stupid ...
$ pip install pybadges
from pybadges import badge
#Generate a badge, that's it ...
svg_image = badge(left_text="build", right_text="success", right_color="green")
print(svg_image[:64]) # '<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.'
That's the solution for the dynamic badge generation part!
Save badges from Cloud Functions
You need to work with Cloud Storage from Cloud Functions to store your badge. This has an Official Client Library, so you can use it in an instant.
$ pip install google-cloud-storage
from google.cloud import storage
#It is not necessary to specify authentication information etc. for normal execution on Cloud Functions.
gcs_client = storage.Client()
bucket = gcs_client.bucket("your-bucket-name")
blob = bucket.blob("any/path/badge.svg")
#If you do not explicitly set the cache, public content will be cached for 1 hour ...
blob.cache_control = "max-age=60, s-maxage=60"
#It seems that the character string can be saved as it is even if it is not a file.
#Note that if you do not specify the content type, it will be just a text file instead of an image when accessed from the outside!
blob.upload_from_string(svg_image, content_type="image/svg+xml")
There seems to be no problem with saving badges.
Repository connection and Cloud Build settings
Now that we have a plan for the main processing part, I would like to prepare around the system settings.
For the time being, if you are using an external service such as GitHub, you need to connect the repository and set up Cloud Build, but I will omit this part. : sweat_drops: I've included an explanation about Cloud Build at the beginning, but since it is intended for those who have used Cloud Build (or GCP users who can use it by looking at the document without touching it), it is already connected to the repository and build. Proceed on the assumption that you have set.
When building a system, what you need to check here is the trigger of Cloud Build.
Cloud Build trigger
Cloud Build sets up triggers to automate builds. This is an excerpt of the trigger setting screen.
One configuration file (so-called cloudbuild.yaml) that defines the build process is set in the trigger.
Generates the result of this build process as a badge.
Triggers can be set for the repository, but multiple triggers can be set for the repository (one-to-many relationship). This means that you can configure multiple builds, such as test and deploy.
Example: Relationship between repository and trigger
Repository
|
+--Test trigger
|
+--Load test trigger
|
+--Deployment trigger
Then, the target branch when the trigger is executed is also set at the same time, but as you can see from the regular expression, multiple branches can be set for the trigger (one-to-many relationship). In other words, you can set builds for different environments such as development environment and production environment with one trigger.
Example: Relationship between repository, trigger and branch
Repository
|
+--Test trigger
| |
| +--master branch
| |
| +--develop branch
|
+--Load test trigger
| |
| +--develop branch
|
+--Deployment trigger
|
+--master branch
|
+--develop branch
If you understand this specification, you will naturally be able to see in what hierarchy the build result badge should be saved.
Cloud Pub / Sub topics
It would be nice if Cloud Functions could be executed triggered by the execution of Cloud Build, but that is not possible, so let's go through Cloud Pub / Sub (GCP services are like that). That's why I need to send a message to Cloud Pub / Sub triggered by the execution of Cloud Build, but this does not require any special work! As you can see in the Documentation, the notification from Cloud Build is actually at the stage when Cloud Pub / Sub is enabled. It is set arbitrarily on the GCP side.
The Pub / Sub topic that Cloud Build publishes build update messages to is
cloud-builds. When you enable the Pub / Sub API, thecloud-buildstopic is automatically created.
So the topic cloud-builds is already available, so let's set it as a trigger for Cloud Functions firing.
Cloud Storage bucket creation
Have a bucket where you want to store your badge. You can click on the GUI, but you can make it quickly with commands.
$ export BUCKET_NAME='your-bucket-name'
$ gsutil mb -c standard -l us-central1 gs://${BUCKET_NAME}
$ gsutil iam ch allUsers:objectViewer gs://${BUCKET_NAME}
The region can be ʻasia-northeast1, but if you set it to ʻus-central1 free tier Applies!
Recommended for those who want to make it even a little cheaper.
The third command grants all users read permission on the bucket. If you don't do this, you won't be able to see it from the outside even if you put the badge on the README.
That's all for preparing the bucket.
Creating Cloud Functions
Create a Cloud Functions that fires on the cloud-builds topic.
Messages received from Cloud Pub / Sub can be retrieved in JSON format, so I'll do my best to extract information about Cloud Build from there.
... and here is the finished product. About 1/3 is comment and log output, so it should not be as complicated as the amount of code.
requirements.txt
google-cloud-storage
pybadges
main.py
"""Cloud Build Badge
Generate a badge from Cloud Build information and save it to GCS.
Create a GCS bucket in advance and create environment variables`_CLOUD_BUILD_BADGE_BUCKET`Set the bucket name in.
"""
import base64
from collections import defaultdict
import dataclasses
import json
import logging
import os
import sys
from typing import List, Optional, overload
from google.cloud import storage
import pybadges
@dataclasses.dataclass(frozen=True)
class Build:
"""Build information.
Parameters
----------
status : str
status.
trigger : str
Trigger ID.
repository : str, optional
Repository name.
branch : str, optional
Branch name.
"""
status: str
trigger: str
repository: Optional[str] = None
branch: Optional[str] = None
@dataclasses.dataclass(frozen=True)
class Badge:
"""Badge object.
Parameters
----------
label : str
label.
message : str
message.
color : str
Hexagon color code.
logo : str, optional
Logo image in Data URI format.
"""
label: str
message: str
color: str
logo: Optional[str] = None
def to_svg(self) -> str:
"""Generate SVG format badges.
Returns
-------
str
Image data in SVG format.
"""
return pybadges.badge(
logo=self.logo, left_text=self.label,
right_text=self.message, right_color=self.color
)
def entry_point(event, context):
"""Entry point."""
try:
return execute(event, context)
except Exception as e:
logging.error(e)
sys.exit(1)
def execute(event, context):
"""Main processing."""
# Pub/Get Sub messages.
pubsub_msg = base64.b64decode(event["data"]).decode("utf-8")
pubsub_msg_dict = json.loads(pubsub_msg)
#Exit if the status is irrelevant.
build = parse_build_info(pubsub_msg_dict)
if build.status not in {
"WORKING", "SUCCESS", "FAILURE", "CANCELLED", "TIMEOUT", "FAILED"
}:
return
#Output the log.
if not build.repository:
logging.info("Unknown repository.")
if not build.branch:
logging.info("Unknown branch.")
#Make sure the destination GCS bucket is set.
bucket_name = get_config("_CLOUD_BUILD_BADGE_BUCKET", pubsub_msg_dict)
if not bucket_name:
RuntimeError(
"Bucket name is not set. Set the value to the environment variable '_CLOUD_BUILD_BADGE_BUCKET'."
)
#Generate a badge and save it to GCS.
badge = create_badge(pubsub_msg_dict)
exported_badges = export_badge_to_gcs(badge, bucket_name, build)
#Output the log.
for url in exported_badges:
logging.info(f"Uploaded badge to '{url}'.")
def parse_build_info(msg: dict) -> Build:
"""Extract the required data from the Cloud Build information.
Parameters
----------
msg : dict
Pub/The message received from Sub.
Returns
-------
Build
Pub/Data that parses Sub messages.
"""
status = msg["status"]
trigger = msg["buildTriggerId"]
#Information may not be obtained, such as when the build definition file itself is corrupted.
repository, branch = None, None
if "substitutions" in msg:
repository = msg["substitutions"].get("REPO_NAME")
branch = msg["substitutions"].get("BRANCH_NAME")
return Build(
status=status, trigger=trigger, repository=repository, branch=branch
)
def create_badge(msg: dict) -> Badge:
"""Generate a badge.
Parameters
----------
msg : dict
Pub/The message received from Sub.
Returns
-------
Badge
A badge that shows the status of the build.
"""
status = msg["status"]
label = get_config("_CLOUD_BUILD_BADGE_LABEL", msg, default="build")
logo = get_config("_CLOUD_BUILD_BADGE_LOGO", msg)
status_to_color = defaultdict(lambda: "#9f9f9f")
status_to_color["WORKING"] = "#dfb317"
status_to_color["SUCCESS"] = "#44cc11"
status_to_color["FAILURE"] = "#e05d44"
return Badge(
label=label, message=status.lower(),
color=status_to_color[status], logo=logo
)
def export_badge_to_gcs(badge: Badge, bucket_name: str, build: Build) -> List[str]:
"""Save the badge to GCS.
Parameters
----------
badge : Badge
badge.
bucket_name : str
Bucket name.
build : Build
Build information.
Returns
-------
list of str
A list containing the URLs of badges saved in GCS.
"""
def upload(path: str) -> None:
bucket = gcs_client.get_bucket(bucket_name)
blob = bucket.blob(path)
blob.cache_control = "max-age=60, s-maxage=60"
blob.upload_from_string(badge.to_svg(), content_type="image/svg+xml")
def to_url(path: str) -> str:
return f"https://storage.googleapis.com/{bucket_name}/{path}"
uploaded = []
gcs_client = storage.Client()
location = f"triggers/{build.trigger}/badge.svg"
upload(location)
uploaded.append(to_url(location))
if not build.repository or not build.branch:
return uploaded
branch = build.branch.replace("/", "_") #Slashes cannot be used, so replace them.
location = f"repositories/{build.repository}/triggers/{build.trigger}/branches/{branch}/badge.svg"
upload(location)
uploaded.append(to_url(location))
return uploaded
@overload
def get_config(key: str, msg: dict) -> Optional[str]: ...
@overload
def get_config(key: str, msg: dict, default: None) -> Optional[str]: ...
@overload
def get_config(key: str, msg: dict, default: str) -> str: ...
def get_config(key, msg, default=None):
"""Get the set value.
Parameters
----------
key : str
Key for getting the setting value.
msg : dict
Pub/The message received from Sub.
default : str, optional
The default value when the setting value does not exist.
Returns
-------
str
Set value (if it does not exist`None`Or`default`The value specified for).
"""
value = None
if "substitutions" in msg:
value = msg["substitutions"].get(key)
if not value:
value = os.getenv(key)
if not value:
value = default
return value
You can also deploy with a command here. In addition, the region is ʻus-central1` is a little consideration that it is more efficient if it is closer to the bucket of the storage destination.
$ export FUNCTION_NAME='any-function-name'
$ export BUCKET_NAME='your-bucket-name'
$ gcloud functions deploy ${FUNCTION_NAME} \
--runtime python37 \
--entry-point entry_point \
--trigger-topic cloud-builds \
--region us-central1 \
--set-env-vars _CLOUD_BUILD_BADGE_BUCKET=${BUCKET_NAME}
The preparation is complete. After that, run Cloud Build and the badge will be generated automatically.
About how to use
Specifying the target build
You do not need! Badges are automatically created for all builds running in your project (in other words, you can't exclude specific builds ...). Let's turn the build steadily and make a pounding badge! : muscle:
Badge type
This badge will be generated in near real time according to the Build Status.
There are a total of 6 types for each status.
- Building: working
- Successful build: success
- Build failure: failure
- Cancel build: canceled
- Build timeout: timeout
- Step timeout: failed
Badge reference
The same badge is saved in two places, and you can refer to each at the following URL.
Badge reference
https://storage.googleapis.com/BUCKET/triggers/TRIGGER/badge.svg
https://storage.googleapis.com/BUCKET/repositories/REPOSITORY/triggers/TRIGGER/branches/BRANCH/badge.svg
BUCKET: Cloud Storage bucket name.TRIGGER: Cloud Build trigger ID.REPOSITORY: The name of the repository to be built.BRANCH: The name of the branch to build.
At first, I thought that the longer save destination below would be enough, but depending on the failure of the build, I could not get the repository name or branch name in Cloud Functions (in the build configuration file itself). Syntax error). Therefore, I saved the above pattern as it is a place where I can get the minimum information and show a unique build. I think that there are actually many GCP projects prepared for each environment, such as huge projects (the project of the team I participate in is like that), so if you save it in units of build triggers I feel that the practicality will not be impaired. : sweat_smile:
Example: Repository-trigger-branch relationship (for large projects)
Project for production environment
|
+--Repository
|
+--Test trigger
| |
| +--master branch
|
+--Deployment trigger
|
+--master branch
Project for development environment
|
+--Repository
|
+--Test trigger
| |
| +--develop branch
|
+--Load test trigger
| |
| +--develop branch
|
+--Deployment trigger
|
+--develop branch
So, if you have only one branch set for a trigger, see here.
https://storage.googleapis.com/BUCKET/triggers/TRIGGER/badge.svg
On the other hand, if multiple branches are set for the trigger, please refer to here (depending on how the build is turned, the repository name / branch name may not be fetched, so it may not be updated ...).
https://storage.googleapis.com/BUCKET/repositories/REPOSITORY/triggers/TRIGGER/branches/BRANCH/badge.svg
Label customization
I would say it myself, but I personally find it quite useful. You can customize the label part of the badge freely and easily. : v:
For example, if you want to make "cloud build" the default instead of "build", set the Cloud Functions environment variable to _CLOUD_BUILD_BADGE_LABEL.
In addition, if Cloud Build has explicit tasks such as testing and deploying and you want it to be labeled "test" or "deploy", set the Cloud Build trigger environment variable to _CLOUD_BUILD_BADGE_LABEL.
Alternatively, you can set it in the substitutions clause in cloudbuild.yaml.
cloudbuild.yaml
substitutions:
_CLOUD_BUILD_BADGE_LABEL: deploy
... Furthermore, if you want to put a logo or something, you can set an image in Data URI format in _CLOUD_BUILD_BADGE_LOGO!
Actually, I wanted to put the Cloud Build logo by default, but there seems to be various problems related to rights, so if you want to put out the logo, please customize it yourself. : sweat_smile: ~~ By the way, although it is for the architecture diagram, the official icon set has been issued. ~~
So, if you customize it, it will look like this. This is no longer an official badge ...
Summary
I was able to build a system that automatically generates Cloud Build status badges using GCP services. I will put it on GitHub, so if you want to stick the Cloud Build badge in the same way, please use it. Let's have a good CI / CD life!
Recommended Posts