Made good with Python and xonsh so that you can see EC2 instances just by entering the shell

Introduction

Are you all studying Terraform? (Intimidating)

I'm just getting started and now desperately [this book](https://www.amazon.co.jp/ Practice Terraform-System Design and Best Practices on AWS-Technology Fountain Series (NextPublishing)-Nomura-Tomonori-ebook / I'm reading dp / B07XT7LJLC) It's a lot of fun, and I'm hoping that if I can freely touch the AWS environment, I'll be able to study infrastructure, so I'm happy about this time, but I was dissatisfied with the study environment.

• Practice Terraform AWS System Design and Best Practices(Technology Fountain Series (NextPublishing)) |Nomura Tomonori|engineering|Kindle Store| Amazon

It's too annoying to see the AWS console one by one! !! !!

Although it is written in the book (details are scary of copyright, I will avoid making a statement) Basically, to check the instance of AWS, I think that I will check from the console by Web UI

That's usually fine, but anyway "Create an instance with Terraform ... Then? Delete next? Yes Yes ... Then? Console again ..." It was quite painful to do it while watching the Kindle

Moreover, I'm not interested in instances I don't know, and I don't want to touch them, so I just want the function to display them. You can use awc cli, but I want it to look like a Web UI without biting the filter.

For the time being, only EC2 instances are set up at the moment, so it's easy to list them, so make it! It's like

Preparation

Repository creation

First of all, as a preparation, create a repository for Terraform Due to the nature of Terraform, repository creation is based on source code control, so it may be natural. Then I think it would be easier to embed a Python instance display script in that repository.

In my case, it's a repository I made for practice, so I embedded the script in the repository. However, the repository cannot be published because it is related to the copyright of the book. Thank you for your understanding

Manage AWS tokens with direnv and dotenv!

direnv

direnv is a nice guy who says "just enter the directory and the alias and environment will be loaded without permission!"

-Let's use direnv --Qiita -[Build a development environment for each project in direnv · lifeisegg2k / pub --- rails-gem --- study Wiki](https://github.com/lifeisegg2k/pub --- rails-gem --- study / wiki Build a development environment for each project in / direnv)

dotenv

I always use dotenv with direnv I feel like I put it in from npm, so it's a bit annoying, but I think that Mac users can easily enter because there are methods to put npm with brew

-Install Node.js on Mac-- Qiita -Use .env file instead of environment variables (dotenv) | Makumaku Node.js notes

Keep AWS IAM information in dotenv

Those who have read this far are probably those who have touched AWS IAM information. Or maybe you're reading a Terraform book So of the IAM information

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION

I won't explain what that means If you want to know more about the meaning of these tokens, please check the official page.

--AWS CLI Settings --AWS Command Line Interface

Speaking of the practical Terraform book that I got the token information or wrote at the beginning, I think that it will be smooth if you can proceed from the person who entered Chapter 2 to the next.

Actually describe the token information using dotenv Create an .env file directly under the repository repo

As a caveat, the .env file will be uploaded on git as it is, so please reflect it properly in .gitignore.

repo/.gitignore

.env

repo/.env

AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAAAA
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
AWS_DEFAULT_REGION=ap-northeast-1

--Environment variables that set the AWS CLI --AWS Command Line Interface

How to check if it works properly

-Summary of authentication information and option setting method that can be used with AWS CLI | Developers.IO

Use dotenv with direnv

Next is how to load dotenv with direnv and automatically reflect the settings under the working directory. direnv reads information in .envrc

repo/.envrc

dotenv
xonsh --rc xonshrc.xsh

The dotenv on the first line is set to automatically load .env with direnv.

I will omit the explanation later, but I will use a shell called xonsh. This is because Python scripts can be easily run on the terminal, so if you write an AWS script in Python, you can run it in a shell quite conveniently.

Summary

With the above, it is a setting that you can easily connect to AWS just by entering the directory There is no inconvenience even if you use it everyday, and the development of Terraform will be much easier.

Let's use the EC2 instance list display script in Python!

After setting direnv and dotenv, let's run the script that displays the instance in Python immediately.

repo/describe_instances.py

import os
import boto3
import pandas as pd


def create_session():
    client = boto3.session.Session(
        aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
        aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
        region_name=os.environ["AWS_DEFAULT_REGION"]
    ).client("ec2")

    return client


def to_pandas():
    response = create_session().describe_instances()
    df = pd.io.json.json_normalize(
        pd.DataFrame(
            pd.DataFrame(
                response["Reservations"]
            ) \
                .apply(lambda r: r.str[0])
                .Instances \
                .values
        )[0]
    )
    df.loc[df["Tags"].isnull(), "Tags"] = df.loc[df["Tags"].isnull(), "Tags"].apply(lambda x: [])
    df.Tags = df.Tags.apply(lambda r: {t["Key"]: t["Value"] for t in r} if len(r) > 0 else {})
    df = pd.concat([df.drop(["Tags"], axis=1), df["Tags"].apply(pd.Series)], axis=1)
    df["SecurityGroups"] = df["SecurityGroups"].apply(lambda r: ", ".join([i["GroupName"] for i in r]))

    return df[["Name",
               "ImageId",
               "InstanceType",
               "Placement.AvailabilityZone",
               "State.Name",
               "PublicDnsName",
               "PublicIpAddress",
               "KeyName",
               "Monitoring.State",
               "LaunchTime",
               "SecurityGroups",
               "Owner",
               ]].sort_values(["LaunchTime",
                               "State.Name"],
                              ascending=[False,
                                         True])


if __name__ == "__main__":
    print(to_pandas())

Save the python script

python describe_instances.py

Try to run Instances are displayed in Dataframe and can be listed.

Let's use xonsh temporarily with direnv!

xonsh is a shell that allows you to customize settings in Python It is often used by Python users, and it is recommended because it is quite convenient even with standard settings. (By the way, I don't usually use it. Somehow.)

It's easy to deploy, just use the package manager pip used by Python.

pip install xonsh

There was a description of xonsh in .envrc earlier, but let's temporarily switch the shell to xonsh and run the instance list display script on xonsh! Is an attempt

• Documentation Index — xonsh 0.9.18 documentation

Why use xonsh temporarily

The reason for using xonsh temporarily is that the overhead of calling Python on the shell is too large, to put it straightforwardly. It is very inefficient to connect to AWS, process the list, and display each time from the shell call, so I chose xonsh, which can store even variables in the shell.

xonsh settings

xonsh can be conveniently customized by creating an xonshrc file Here we'll make some changes to the official xonshrc to make it easier for you to run your own instance listing script.

repo/xonshrc.xsh

# adjust some paths
$PATH.append('/home/scopatz/sandbox/bin')
$LD_LIBRARY_PATH = ['/home/scopatz/.local/lib', '/home/scopatz/miniconda3/lib', '']

# alias to quit AwesomeWM from the terminal
def _quit_awesome(args, stdin=None):
    lines = $(ps ux | grep "gnome-session --session=awesome").splitlines()
    pids = [l.split()[1] for l in lines]
    for pid in pids:
        kill @(pid)

aliases['qa'] = _quit_awesome

# some customization options, see https://xon.sh/envvars.html for details
$MULTILINE_PROMPT = '`·.,¸,.·*¯`·.,¸,.·*¯'
$XONSH_SHOW_TRACEBACK = True
$XONSH_STORE_STDOUT = True
$XONSH_HISTORY_MATCH_ANYWHERE = True
$COMPLETIONS_CONFIRM = True
$XONSH_AUTOPAIR = True


#From here the original xonshrc

import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))

#Import your own function
import describe_instances
_df = describe_instances.to_pandas()
#Display columns 0 to 5
print(_df[_df.columns[0:5]])

• Run Control File — xonsh 0.9.18 documentation

Tips

Now, just move to the directory and the instance list will be displayed automatically, and even more.

describe_instances.to_pandas()

You can get a list of Dataframes just by pouring

It's convenient, so please use it ~~~