A Python script that lists execution history in an environment running many AWS Glue Jobs

I've previously blogged how to view Glue Job execution history in a command line one-liner.

-One-liner to see the execution history at once in the environment where many AWS Glue Jobs are running

However, the date and time display was not good and it was difficult to use, so I rewrote it with a Python script.

import boto3

#If you want to use a profile other than default, specify it here
profile = "default"

session = boto3.session.Session(profile_name = profile)

client = session.client("glue")

jobs = client.get_jobs()

header = [
    "started",
    "completed",
    "executionTime",
    "status",
    "name",
    "allocatedCapacity",
    "maxCapacity",
    "glueVersion",
    "errorMessage",
]

result = []

for job in jobs["Jobs"]:
    name = job["Name"]
    history = client.get_job_runs(JobName = name)
    for run in history["JobRuns"]:
        started = run["StartedOn"].strftime("%Y-%m-%d %H:%M:%S")
        if "CompletedOn" in run:
            completed = run["CompletedOn"].strftime("%Y-%m-%d %H:%M:%S")
        else:
            completed = ""
        executionTime = str(run["ExecutionTime"])
        if executionTime == "0":
            executionTime = ""
        status = run["JobRunState"]
        if "ErrorMessage" in run:
            errorMessage = run["ErrorMessage"]
        else:
            errorMessage = ""
        allocatedCapacity = str(run["AllocatedCapacity"])
        maxCapacity = str(run["MaxCapacity"])
        glueVersion = str(run["GlueVersion"])
        result.append([
            started,
            completed,
            executionTime,
            status,
            name,
            allocatedCapacity,
            maxCapacity,
            glueVersion,
            errorMessage,
        ])

#Sort by startup time
result.sort(key = lambda r: r[0])

#Output tab-delimited
print("\t".join(header))
for r in result:
    print("\t".join(r))

It displays the start time and end time, but it seems to be displayed in local time depending on the environment variable TZ.

Recommended Posts

A Python script that lists execution history in an environment running many AWS Glue Jobs
What's in that variable (when running a Python script)
Building an environment that uses Python in Eclipse
Create an exe file that works in a Windows environment without Python with PyInstaller
Try running python in a Django environment created with pipenv
Let's create a script that registers with Ideone.com in Python.
A set of script files that do wordcloud in Python3
A script that retrieves tweets with Python, saves them in an external file, and performs morphological analysis.
The eval () function that calculates a string as an expression in python
[Django] sqlite version error when running python manage.py in aws cloud9 environment
Run the Python interpreter in a script
Creating a virtual environment in an Anaconda environment
A complete guidebook to using pyenv, pip and python in an offline environment
Create a Vim + Python test environment in 1 minute
Ubuntu18.04.05 Creating a python virtual environment in LTS
A memo that I wrote a quicksort in Python
A script that just gets an RSS feed
Simply build a Python 3 execution environment on Windows
Create a virtual environment with conda in Python
A program that removes duplicate statements in Python
Rewriting elements in a loop of lists (Python)
Install the python package in an offline environment
"Python Kit" that calls a Python script from Swift
Think about building a Python 3 environment in a Mac environment
[Python] Create an asynchronous task execution environment + monitoring environment
Work in a virtual environment with Python virtualenv.
Call a Python script from Embedded Python in C ++ / C ++
Introduction to Python "Re" 1 Building an execution environment
Launch a shell while a Python script is running
Create a Python execution environment on IBM i
Insert an object inside a string in Python
[Docker] Create a jupyterLab (python) environment in 3 minutes!
Build a python execution environment with VS Code
A Python script that crawls RSS in Azure Status and posts it to Hipchat
A python script that gets the number of jobs for a specified condition from indeed.com
Try running a Schedule to start and stop an instance on AWS Lambda (Python)
Script to easily create a client device environment for AWS IoT (Python v2 version)
[AWS] Development environment version that tried to build a Python environment with eb [Elastic Beanstalk]
A note that runs an external program in Python and parses the resulting line