[PYTHON] [End of 2020] A memo to start using AWS CLI (Version 2)

I decided to participate in a project that uses AWS for the first time in a few years. As a memo, it was necessary to update the information such as the release of Version 2 of the AWS CLI.

For AWS CLI Version 2, see here.

What is explained in this article

--Installing and configuring AWS CLI Version 2 --How to import csv file for authentication and explanation of existing problems --Other --About the Osaka region that will be updated in 2021 --About the column structure of AWS authentication csv --How to install AWS CLI Version 2 in individual environment of pyenv

Work environment

cat /etc/issue
> Ubuntu 18.04.5 LTS

python -V
> Python 3.8.6

pip --version
> pip 20.2.1 from /home/dev-user/.pyenv/versions/3.8.6/envs/sandbox-awscli/lib/python3.8/site-packages/pip (python 3.8)

(Reference: Construction procedure with Pyenv)

The work done in pyenv to create the above environment on Ubuntu 18.04 is as follows.

# Python3.Get 8 environment
pyenv install 3.8.6

#3 for work.8 Make a copy of the environment
pyenv virtualenv 3.8.6 sandbox-awscli

#Apply the environment copied to the current directory
pyenv local sandbox-awscli

AWS CLI (Version 2) installation

A simple pip install awscli will install the old version 1. Install with Official Method.

#Download the installer
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

#Answer the installer
unzip awscliv2.zip

#Perform installation
# (/usr/local/Will be installed in bin)
sudo ./aws/install

aws --version
> aws-cli/2.1.14 Python/3.7.3 Linux/5.4.0-58-generic exe/x86_64.ubuntu.18 prompt/off

(Reference) Install by specifying the copy destination, such as for general users

As an example of not using sudo in the README, the following command describes how to install and add the path (~/.local/bin) to the environment variable.

# (Option description)
#   -i, --install-dir <path>Specify the copy destination directory of the executable file.(default: /usr/local/aws-cli
#   -b, --bin-dir <path>Specify the directory where you want to create the symlink to the executable.(default: /usr/loca/bin)
./install -i ~/.local/aws-cli -b ~/.local/bin

(Reference) Install in individual environment of pyenv

When installing in the individual environment of pyenv (sandbox-awscli in this case), it will be as follows.

./aws/install -i ~/.pyenv/versions/sandbox-awscli/bin/aws-cli -b ~/.pyenv/versions/sandbox-awscli/bin

In my environment, the aws command didn't work unless I reread the path in source after execution.

AWS user verification

Check the information of the user to use on the AWS console. ・ Is the group policy suitable for the purpose?

Check the region

Check the recent situation.

Region list


Tokyo region

Basically this

area Region name
Asia-Pacific(Tokyo) ap-northeast-1

About Osaka Region (ap-northeast-3)

Extending AWS Osaka Local Region to Full Region

By early 2021, Osaka's local region has expanded to a full AWS region with three Availability Zones. Like all other AWS Regions, Availability Zones are separated by their own power, cooling system, and physical security. (As of December 28, 2020)

Prepare the authentication file (.csv) for configuration

As a new feature from AWS CLI Version 2, you can read and set the authentication information from the csv file (because it is a workflow familiar to GCP users, private is a nice update for GCP main body)

However, it is good when creating a new user and using it, but when using an existing user, it took a lot of work, so I will explain it later.

For new users

You can download it from the csv download button that is normally displayed on the creation completion screen.

The csv file has the following column structure.

User name Password Access key ID Secret access key Console login link
USERNAME XXX YYY https://ZZZ.signin.aws.amazon.com/console

The contents of the file (new_user_credentials.csv) are as follows:

User name,Password,Access key ID,Secret access key,Console login link
USER NAME,,ACCESS KEY,SECRET KEY,https://USER ID.signin.aws.amazon.com/console

For existing users

*** The csv file downloaded from the "Create Access Key" completion screen as an existing user does not include columns such as User Name required at the time of import. *** ***

Therefore, you need to add it manually after downloading it by following the steps below.

Download csv for authentication

*** "AWS Console"-> "IAM"-> "Authentication"-> "Create Access Key" *** And it is displayed on the creation completion screen *** "Download .csv file" *** Please download from.

The csv file has the following column structure.

Access key ID Secret access key

If you try to import as it is, the following error will occur.

aws configure import --csv file://yuuki-sandbox_accessKeys.csv --profile-prefix prof-
> Expected header "User Name" not found

Add column

I couldn't find a good way, so if I want to use import as an existing user, I decided to copy the contents of csv of the new user mentioned above and rewrite the necessary parts.

*** If you know a good way, I would appreciate it if you could tell me! *** ***

AWS CLI settings

Import csv for authentication.

*** When imported without options, a profile is automatically created with the username stored in the User Name column of csv and given the credentials. *** ***

--profile can also be set in the import command, but be aware that the above will be prioritized and ignored.

--profile-prefix There is an option called , which can be a profile named .

# --profile-According to the prefix option, prof-<USER NAME>A profile named is created and the credentials are imported here.
aws configure import --csv file://new_credentials.csv --profile-prefix prof-
> Successfully imported 1 profile(s)

#Confirmation of profile addition
aws configure list-profiles
> <Profile name>

Other settings

Set the rules.

Region settings

aws configure set region ap-northeast-1 --profile <Profile name>

Output type setting

aws configure set output <Output type> --profile <Profile name>

The output types that can be selected are as follows (as of December 28, 2020)

Output type Explanation
json It is output in JSON string format.
yaml It is output in YAML string format. (Available only with AWS CLI version 2.)
yaml-stream The output is streamed and formatted as a YAML string. Streaming can speed up the processing of large data types. (Available only with AWS CLI version 2.)
text It is output in the form of a multi-line tab-delimited string value. This is useful for passing output to a text processor such as grep, sed, or awk.
table The string that forms the cell border+|-Is output in tabular format using. Information is usually displayed in a "easy-to-understand" format that is easier to read than other formats, but it is not useful programmatically.

Change default profile

#For Bourne shell
export AWS_DEFAULT_PROFILE=<Profile name>

#For fish shell
set -x AWS_DEFAULT_PROFILE <Profile name>

If necessary, add it to .bashrc etc.


I will briefly display the information to confirm that the authentication has passed.

Try to display the bucket list of s3

Required access policy example: AmazonS3ReadOnlyAccess

aws s3 ls
> 2020-12-28 17:36:44 yuuki-sample-bucket

Try to display the IAM user list

Required access policy example: IAMReadOnlyAccess

aws iam list-users
> ~~
> - Arn: arn:aws:iam::XXX:user/yuuki-sandbox
>   CreateDate: '2020-12-28T02:16:22+00:00'
>   Path: /
>   UserId: YYY
>   UserName: yuuki-sandbox


Please point out any mistakes or missing work.

Recommended Posts

[End of 2020] A memo to start using AWS CLI (Version 2)
A memo connected to HiveServer2 of EMR with python
A memo of how to use AIST supercomputer ABCI
A memo to visually understand the axis of pandas.Panel
I want to start a lot of processes from python
Start to Selenium using python
How to develop in a virtual environment of Python [Memo]
[Introduction to AWS] A memorandum of building a web server on AWS
How to insert a specific process at the start and end of spider with scrapy
A memorandum of using eigen3
I tried to make a regular expression of "amount" using Python
I tried to make a regular expression of "date" using Python
[Go] Create a CLI command to change the extension of the image
Convert a large number of PDF files to text files using pdfminer
I tried to get a list of AMI Names using Boto3
How to save only a part of a long video using OpenCV
Output search results of posts to a file using Mattermost API
A memo to move Errbot locally
[AWS SAM] Introduction to Python version
How to test each version of IE using Selenium in modan.IE (VM)
Things to be aware of when building a recommender system using Item2Vec
June 2017 version to build Tensorflow / Keras environment on GPU instance of AWS
Try using n to downgrade the version of Node.js you have installed
Allow Slack to notify you of the end of a time-consuming program process
I tried to perform a cluster analysis of customers using purchasing data
Getting a combination of elements using itertools
How to check the version of Django
A rough understanding of python-fire and a memo
How to draw a graph using Matplotlib
Try to make a kernel of Jupyter
How to install a package using a repository
Regularly post to Twitter using AWS lambda!
A memo explaining the axis specification of axis
[AWS] Summary of CLI installation error countermeasures
Summary of how to write AWS Lambda
[Circuit x Python] How to find the transfer function of a circuit using Lcapy
Free version of DataRobot! ?? Introduction to "PyCaret", a library that automates machine learning
Specify the start and end positions of files to be included with qiitap
Save an array of numpy to a wav file using the wave module
A guide to batch installation of version control tools for major scripting languages
A story of using rembg to make a white background + person video into a black background video
[AWS Lambda] Create a deployment package using the Docker image of Amazon Linux
I made a script to record the active window using win32gui of Python