[PYTHON] Try to make your own AWS-SDK with bash

Overview

I made AWS-SDK with bash. AWS-AWS will be able to handle even PCs that do not have an SDK or programming environment.

When making it, I try to observe the following conditions.

• Use plain bash (this time git bash), don't use anything that isn't in standard git bash
• Do not use programming languages. Solving with awk, sed, bash
• Use the same profile information as AWS (~ / .aws / credentials)

Articles can be opened and closed for each service, so please read only the services you need.

Diagram

The finished product

Since it is not possible to check the operation of all the functions of the SDK, we will only check the operation below in this article.

• Manipulate DynamoDB data
• Run Lambda directly
• Manipulate SQS data
• Get own profile information from STS

• Since S3 has a big difference, this script does not support it. We have not tested anything more than a simple operation check, so please use it at your own risk.

Before use: Prepare profile

When using it, please create profile information in ~ / .aws / credentials. The format is text and no extension is required.

~/.aws/credentials

[default]
aws_access_key_id = AKIA**************************
aws_secret_access_key = *****************************

Operation check: Operate DynamoDB data

./aws-sdk-bash.sh "default" "ap-northeast-1" "dynamodb" "/" "" \
"content-type:application/x-amz-json-1.0;host:@;x-amz-date:@;x-amz-target:DynamoDB_20120810.GetItem" \
"{\"TableName\": \"target_table\", \"Key\": {\"id\": {\"S\": \"key\"}}}"

{"Item":{"entity":{"S":"string_data"},"id":{"S":"key"}}}

from boto3 import Session
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "dynamodb"
).get_item(
    TableName = "target_table",
    Key = {"id" : {"S" : "key"}}
)

#Enter data
./aws-sdk-bash.sh "default" "ap-northeast-1" "dynamodb" "/" "" \
"content-type:application/x-amz-json-1.0;host:@;x-amz-date:@;x-amz-target:DynamoDB_20120810.PutItem" \
"{\"TableName\": \"target_table\", \"Item\": {\"id\": {\"S\": \"newData\"}, \"entity\":{\"S\":\"new_string_data\"}}}"

{}

#After entering the data, check that the number of cases is increasing with Scan
./aws-sdk-bash.sh "default" "ap-northeast-1" "dynamodb" "/" "" \
"content-type:application/x-amz-json-1.0;host:@;x-amz-date:@;x-amz-target:DynamoDB_20120810.Scan" \
"{\"TableName\": \"target_table\"}"

{"Count":2,"Items":[{"entity":{"S":"new_string_data"},"id":{"S":"newData"}},{"entity":{"S":"string_data"},"id":{"S":"key"}}],"ScannedCount":2}

Operation check: Run Lambda directly

./aws-sdk-bash.sh "default" "ap-northeast-1" "lambda" "/2015-03-31/functions/sample_lambda/invocations" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-invocation-type:RequestResponse;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"{\"Message\":\"Hello\"}"

{"statusCode": 200, "body": "\"Hello from Lambda! I am SAMPLE\""}

from boto3 import Session
import json
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "lambda"
).invoke(
    FunctionName = "sample_lambda",
    InvocationType = "RequestResponse",
    Payload = json.dumps({"Message" : "Hello"})
)

Operation check: Manipulate SQS data

#Send message
./aws-sdk-bash.sh "default" "ap-northeast-1" "sqs" "/" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"Action=SendMessage&MessageBody=%7B%22id%22%3A%22NewMessage%22%7D&QueueUrl=https%3A%2F%2Fsqs.ap-northeast-1.amazonaws.com%2F***********************%2Fsqs-send-request-test-0424&Version=2012-11-05"

<?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><SendMessageResult><MessageId>013a849d-f344-4d90-a6be-e37302c6b029</MessageId><MD5OfMessageBody>abc832604cb20908715bca6f197c8945</MD5OfMessageBody></SendMessageResult><ResponseMetadata><RequestId>01e07779-a593-5162-b698-2050d59e1524</RequestId></ResponseMetadata></SendMessageResponse>

#Receive the message you sent
./aws-sdk-bash.sh "default" "ap-northeast-1" "sqs" "/" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"Action=ReceiveMessage&QueueUrl=https%3A%2F%2Fsqs.ap-northeast-1.amazonaws.com%2F*****************%2Fsqs-send-request-test-0424&Version=2012-11-05"

<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult><Message><MessageId>6335c486-d1b6-4478-84c4-40b92f5d655b</MessageId><ReceiptHandle>***********</ReceiptHandle><MD5OfBody>abc832604cb20908715bca6f197c8945</MD5OfBody><Body>{&quot;id&quot;:&quot;NewMessage&quot;}</Body></Message></ReceiveMessageResult><ResponseMetadata><RequestId>9e05f85b-4a27-5d6f-97c4-af723bb68a70</RequestId></ResponseMetadata></ReceiveMessageResponse>

from boto3 import Session
import json
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "sqs"
).send_message(
    QueueUrl = "https://sqs.ap-northeast-1.amazonaws.com/**************/sqs-send-request-test-0424",
    MessageBody = json.dumps({"id": "NewMessage"})
)

Operation check: Get own profile information from STS

#Get your own information
./aws-sdk-bash.sh "default" "ap-northeast-1" "sts" "/" "" \
"host:@;x-amz-content-sha256:@;x-amz-date:@;x-amz-user-agent:aws-sdk-js/2.668.0 callback" \
"Action=GetCallerIdentity&Version=2011-06-15"

<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <GetCallerIdentityResult>
    <Arn>arn:aws:iam::*****************:user/**************</Arn>
    <UserId>*****************</UserId>
    <Account>*****************</Account>
  </GetCallerIdentityResult>
  <ResponseMetadata>
    <RequestId>1703851c-5b55-49b5-8bcf-130ab4dbb6f2</RequestId>
  </ResponseMetadata>
</GetCallerIdentityResponse>

from boto3 import Session
Session(
    profile_name = "default", 
    region_name = "ap-northeast-1"
).client(
    service_name = "sts"
).get_caller_identity()

Source code

If you want to run it, save it in any location with the file name aws-sdk-bash.sh. For the command to be executed, refer to "Operation check".

aws-sdk-bash.sh

#!/bin/bash

#Profile path is fixed
CREDENTIALS_FILE=~/.aws/credentials
#Algorithm is signature v4, hmac SHA256
ALGORITHM='AWS4-HMAC-SHA256'

#Script arguments
_INPUT_PROFILE_NAME=$1
_INPUT_REGION=$2
_INPUT_SERVICE=$3
_INPUT_CANONICAL_URI=$4
_INPUT_CANONICAL_QUERY_STRING=$5
_INPUT_OPTIONAL_HEADERS=$6
_INPUT_PAYLOAD=$7

#Creating an endpoint
METHOD=POST
PROTOCOL=https
HOST_NAME=${_INPUT_SERVICE}.${_INPUT_REGION}.amazonaws.com

# .Get profile information from aws file
#Input profile path, profile name, profile key name, number of lines to read file
get_credentials () {
    _CREDENTIALS_FILE=$1; _PROFILE_NAME=$2; _KEY_NAME=$3; _READ_LENGTH=$4;

    # PROFILE_Get the line number of NAME
    PROFILE_IDX=`nl $_CREDENTIALS_FILE | grep $_PROFILE_NAME | head -n 1 | awk '{print $1}'`
    PROFILE_IDX_END=`expr $PROFILE_IDX + $_READ_LENGTH`

    #Get access key ID
    RESULT=`cat $CREDENTIALS_FILE | sed -n "${PROFILE_IDX},${PROFILE_IDX_END}p" | grep "=" | grep ${_KEY_NAME} | \
    tr -d " " | sed "s/=/ /g" | awk '{print $2}' | \
    head -n 1`

    echo -n $RESULT
}

#Create a hash with SHA256
#Input message: file, key: none
#Output hash: hex format
create_digest_from_file () {
    cat $1 | openssl dgst -sha256 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: text
#Output hash: hex format
sign_from_string () {
    echo -n $2 | openssl dgst -sha256 -hmac $1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: hex format
#Output hash: hex format
sign_from_string_with_hex_key () {
    echo -n $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: file, key: hex format
#Output hash: hex format
sign_from_file_with_hex_key () {
    cat $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

#HMAC the basic information of signature v4 (access key ID, date and time of transmission, region, service name)-Hash with SHA256
get_signature_key () {
    TEMP_DATE=`sign_from_string AWS4$1 $2`
    TEMP_REGION=`sign_from_string_with_hex_key $TEMP_DATE $3`
    TEMP_SERVICE=`sign_from_string_with_hex_key $TEMP_REGION $4`
    sign_from_string_with_hex_key $TEMP_SERVICE 'aws4_request'
}

#Get access key ID
ACCESS_KEY_ID=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_access_key_id 2`

#Get a secret access key
SECRET_ACCESS_KEY=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_secret_access_key 2`

#Get UTC date and time (Format example: 2020/12/31T12:34:At 50 AMZ_DATE:20201231T123450Z DATE_STAMP:20201231）
UTC_DATE=`date -Iseconds -u | sed "s/+/ /g" | awk '{print $1 "Z"}'`
AMZ_DATE=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/://g"`
DATE_STAMP=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/T/ /g" | awk '{print $1}'`

#Create temporary files, delete temporary files when finished
TEMP_HEADERS=`mktemp`
TEMP_CANONICAL_REQUEST=`mktemp`
TEMP_STRING_TO_SIGN=`mktemp`
TEMP_PAYLOAD=`mktemp`
trap "rm -f $TEMP_HEADERS; rm -f $TEMP_CANONICAL_REQUEST; rm -f $TEMP_STRING_TO_SIGN; rm -f $TEMP_PAYLOAD" EXIT

#Copy the payload to a temporary file
echo -n ${_INPUT_PAYLOAD} > $TEMP_PAYLOAD

#Create a SHA256 hash (without key) from the BODY data to be sent
PAYLOAD_HASH=`create_digest_from_file $TEMP_PAYLOAD`

#Set the send header
echo -n "${_INPUT_OPTIONAL_HEADERS}" | sed "s#x-amz-content-sha256:@#x-amz-content-sha256:${PAYLOAD_HASH}#" | sed "s#host:@#host:${HOST_NAME}#" | sed "s#x-amz-date:@#x-amz-date:${AMZ_DATE}#" | xargs -d ";" -r -I @ echo @ >> $TEMP_HEADERS
SIGNED_HEADERS=`echo -n "${_INPUT_OPTIONAL_HEADERS}" | xargs -d ";" -r -I @ echo ";@" | sed 's/:.*//'`
SIGNED_HEADERS=`echo -n $SIGNED_HEADERS | sed 's/ //g' | sed 's/^;//'`

#Make a canonical request
#Standardize request information
#Reference information: Task 1:Make a legitimate request for signature version 4
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-create-canonical-request.html
echo $METHOD > $TEMP_CANONICAL_REQUEST
echo $_INPUT_CANONICAL_URI >> $TEMP_CANONICAL_REQUEST
echo $_INPUT_CANONICAL_QUERY_STRING >> $TEMP_CANONICAL_REQUEST
cat $TEMP_HEADERS >> $TEMP_CANONICAL_REQUEST
echo "" >> $TEMP_CANONICAL_REQUEST
echo $SIGNED_HEADERS >> $TEMP_CANONICAL_REQUEST
echo -n $PAYLOAD_HASH >> $TEMP_CANONICAL_REQUEST

#Create a signature string
#Reference information: Task 2:Create a signature version 4 signature string
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-create-string-to-sign.html
CANONICAL_REQUEST_HASH=`create_digest_from_file $TEMP_CANONICAL_REQUEST`
CREDENTIAL_SCOPE=${DATE_STAMP}/${_INPUT_REGION}/${_INPUT_SERVICE}/"aws4_request"

#Set signature algorithm and signature scope
echo ${ALGORITHM} > $TEMP_STRING_TO_SIGN
echo ${AMZ_DATE} >> $TEMP_STRING_TO_SIGN
echo ${CREDENTIAL_SCOPE} >> $TEMP_STRING_TO_SIGN
echo -n ${CANONICAL_REQUEST_HASH} >> $TEMP_STRING_TO_SIGN

#Calculate Signature for signature
#Reference information: Task 3:Calculate AWS Signature Version 4 Signatures
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-calculate-signature.html
SIGNING_KEY=`get_signature_key ${SECRET_ACCESS_KEY} ${DATE_STAMP} ${_INPUT_REGION} ${_INPUT_SERVICE}`
SIGNATURE=`sign_from_file_with_hex_key ${SIGNING_KEY} ${TEMP_STRING_TO_SIGN}`

#Set the signature in the header of the HTTP request
#Reference information: Task 4:Add signature to HTTP request
# https://docs.aws.amazon.com/ja_jp/general/latest/gr/sigv4-add-signature-to-request.html
AUTHORIZATION_HEADER="${ALGORITHM} Credential=${ACCESS_KEY_ID}/${CREDENTIAL_SCOPE}, SignedHeaders=${SIGNED_HEADERS}, Signature=${SIGNATURE}"

#Send an HTTP request.
QUERY_STRING=${_INPUT_CANONICAL_QUERY_STRING}
if [ ! $QUERY_STRING = "" ]; then
    QUERY_STRING="?${QUERY_STRING}"
fi
curl -s -X POST ${PROTOCOL}://${HOST_NAME}${_INPUT_CANONICAL_URI}${QUERY_STRING} -d @$TEMP_PAYLOAD -H @$TEMP_HEADERS -H "Authorization: ${AUTHORIZATION_HEADER}"

Explanation of source code

The processing flow is as follows. There are different SDKs for different languages, but if you implement these four you can do the same thing as the AWS-SDK.

\def\of{\unicode[serif]{x306E}}
\bbox[8px, border: 2px solid gray]{\rlap{\tt 1.\quad profile information\Get}\hspace{80mm}}

\triangledown

\def\of{\unicode[serif]{x306E}}
\bbox[8px, border: 2px solid gray]{\rlap{\tt 2.\quad parameter\Get and create}\hspace{80mm}}

\triangledown

\bbox[8px, border: 2px solid gray]{\rlap{\tt 3.\Hash quad parameters with SHA256}\hspace{80mm}}

\triangledown

\bbox[8px, border: 2px solid gray]{\rlap{\tt 4.\Send HTTP request with quad curl}\hspace{80mm}}

For version 4 signing, see the official AWS "Completely Signed Version 4 Signing Process Example (Python)" -request-examples.html) Please refer to.

Hereafter, each process will be explained.

1. Acquisition of profile information

This is the process to get profile information from ~ / .aws / credentials.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
command&Purpose\\
\hline 
{\tt nl} &line number+File\Displays the contents of.\\
&line number\It is a cat that comes out.\\
\hdashline 
{\tt sed\;\text{-n}} &Specified line\Get the text of.\\
\hline 
\end{array}

Use "nl + grep profile name" to get the number of lines of the profile name to get. "Sed -n" will take the line after the profile name, so take the access key ID and secret access key from there.

aws-sdk-bash.sh_Acquisition of signature information

# .Get profile information from aws file
#Input profile path, profile name, profile key name, number of lines to read profile
get_credentials () {
    _CREDENTIALS_FILE=$1; _PROFILE_NAME=$2; _KEY_NAME=$3; _READ_LENGTH=$4;

    # PROFILE_Get the line number of NAME
    PROFILE_IDX=`nl $_CREDENTIALS_FILE | grep $_PROFILE_NAME | head -n 1 | awk '{print $1}'`
    PROFILE_IDX_END=`expr $PROFILE_IDX + $_READ_LENGTH`

    #Get access key ID
    RESULT=`cat $CREDENTIALS_FILE | sed -n "${PROFILE_IDX},${PROFILE_IDX_END}p" | grep "=" | grep ${_KEY_NAME} | \
    tr -d " " | sed "s/=/ /g" | awk '{print $2}' | \
    head -n 1`

    echo -n $RESULT
}

aws-sdk-bash.sh_Caller

#Get access key ID
ACCESS_KEY_ID=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_access_key_id 2`

#Get a secret access key
SECRET_ACCESS_KEY=`get_credentials $CREDENTIALS_FILE ${_INPUT_PROFILE_NAME} aws_secret_access_key 2`

2. Get and create parameters

** Get date and time **

Get the date and time in UTC. Create a string in a format that includes hours, minutes, and seconds, and a format that does not include hours, minutes, and seconds.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
value&format\\
\hline 
{\tt \text{AMZ_DATE}} & {\tt YYYYMMDDTHHMMSS}\Specify in the format of.\\
&Behind{\tt UTC}Point to{\tt Z}I will put on.\\
&Colon, hyphen\Do not include symbols such as.\\
\hdashline 
{\tt \text{DATE_STAMP}} & {\tt YYYYMMDD}\Specify in the format of.\\
&Specify only the date.\\
&Colon, hyphen\Do not include symbols such as.\\
\hline 
\end{array}

aws-sdk-bash.sh_Get UTC date and time

#Get UTC date and time (Format example: 2020/12/31T12:34:At 50 AMZ_DATE:20201231T123450Z DATE_STAMP:20201231）
UTC_DATE=`date -Iseconds -u | sed "s/+/ /g" | awk '{print $1 "Z"}'`
AMZ_DATE=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/://g"`
DATE_STAMP=`echo -n $UTC_DATE | sed "s/-//g" | sed "s/T/ /g" | awk '{print $1}'`

** Handle text **

The character string is written to a temporary file and handled. Temporary files are deleted when the process ends.

You can handle it with bash variables, but it is easier to implement if you handle strings in a file. For signature information, blank lines, line breaks at the end, and the order of appearance of data are strictly determined, and if even one shifts, the signature will not pass.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
command&Purpose\\
\hline 
{\tt mktemp} & /Create a temporary file in tmp.\\
&File names are unique and\\
&Access rights are also set appropriately.\\
\hdashline 
{\tt trap} &specific\The command is executed at the timing of.\\
&Deletes temporary files at the end of the process.\\
\hline 
\end{array}

aws-sdk-bash.sh_Creating a temporary file

#Create temporary files, delete temporary files when finished
TEMP_HEADERS=`mktemp`
TEMP_CANONICAL_REQUEST=`mktemp`
TEMP_STRING_TO_SIGN=`mktemp`
TEMP_PAYLOAD=`mktemp`
trap "rm -f $TEMP_HEADERS; rm -f $TEMP_CANONICAL_REQUEST; rm -f $TEMP_STRING_TO_SIGN; rm -f $TEMP_PAYLOAD" EXIT

** Parameter replacement **

There are some parameters that the user is not aware of when making a request. Such parameters are set on the script side by omitting the value with @.

\begin{array}{ll}
User-set parameters& {\tt \text{x-amz-date:@}} \\
Parameters actually sent& {\tt \text{x-amz-date:20200504T145432Z}}\\
& \\
\end{array}

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
header\key of&Value to be set\\
\hline 
{\tt \text{x-amz-content-sha256}} &payload\Hash value of\\
\hdashline 
{\tt host} & {\tt AWS endpoint\URL}\\
\hdashline 
{\tt \text{x-amz-date}}  &Date and time of request\\
\hline 
\end{array}

aws-sdk-bash.sh_Set header

#Create a SHA256 hash (without key) from the BODY data to be sent
PAYLOAD_HASH=`create_digest_from_file $TEMP_PAYLOAD`

#Set the send header
echo -n "${_INPUT_OPTIONAL_HEADERS}" | sed "s#x-amz-content-sha256:@#x-amz-content-sha256:${PAYLOAD_HASH}#" | sed "s#host:@#host:${HOST_NAME}#" | sed "s#x-amz-date:@#x-amz-date:${AMZ_DATE}#" | xargs -d ";" -r -I @ echo @ >> $TEMP_HEADERS
SIGNED_HEADERS=`echo -n "${_INPUT_OPTIONAL_HEADERS}" | xargs -d ";" -r -I @ echo ";@" | sed 's/:.*//'`
SIGNED_HEADERS=`echo -n $SIGNED_HEADERS | sed 's/ //g' | sed 's/^;//'`

3. Hashing

To get HMAC-SHA256, write as follows in Python.

HMAC-Get SHA256

def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(("AWS4" + key).encode("utf-8"), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, "aws4_request")
    return kSigning

To do the same with bash, use openssl.

\def\of{\unicode[serif]{x306E}}
\begin{array}{l|l}
\hline 
command&Purpose\\
\hline 
{\tt openssl\;dgst\;\text{-}sha256} &Text{\tt SHA256}Hash with.\\
&The result will be a hexadecimal number.\\
\hdashline 
{\tt openssl\;dgst\;\text{-}sha256} &Text{\tt SHA256}Hash with.\\
\qquad{\tt \text{-}hmac\;(key)} &Plain text\Specify the key of.\\
&The result will be a hexadecimal number.\\
\hdashline 
{\tt openssl\;dgst\;\text{-}sha256} &Text{\tt SHA256}Hash with.\\
\qquad{\tt \text{-}mac\;hmac} &Hexadecimal\Specify the key of.\\
\qquad{\tt \text{-}macopt\;hexkey:(key)}&The result will be a hexadecimal number.\\
\hline 
\end{array}

aws-sdk-bash.sh_Hashing

#Create a hash with SHA256
#Input message: file, key: none
#Output hash: hex format
create_digest_from_file () {
    cat $1 | openssl dgst -sha256 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: text
#Output hash: hex format
sign_from_string () {
    echo -n $2 | openssl dgst -sha256 -hmac $1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: text, key: hex format
#Output hash: hex format
sign_from_string_with_hex_key () {
    echo -n $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

# HMAC-Create a hash with SHA256
#Input message: file, key: hex format
#Output hash: hex format
sign_from_file_with_hex_key () {
    cat $2 | openssl dgst -sha256 -mac hmac -macopt hexkey:$1 | grep stdin | awk '{print $2}'
}

#HMAC the basic information of signature v4 (access key ID, date and time of transmission, region, service name)-Hash with SHA256
#Same process as getSignatureKey in Python
get_signature_key () {
    TEMP_DATE=`sign_from_string AWS4$1 $2`
    TEMP_REGION=`sign_from_string_with_hex_key $TEMP_DATE $3`
    TEMP_SERVICE=`sign_from_string_with_hex_key $TEMP_REGION $4`
    sign_from_string_with_hex_key $TEMP_SERVICE 'aws4_request'
}

How to check AWS-SDK request

The format of the payload and header thrown by the AWS-SDK varies from service to service. There is no official documentation, so you have to look it up yourself.

How to check with boto3 + Wireshark

If you communicate as it is, it will be encrypted, so I will add a little work.

Turn off encryption and use HTTP communication

import boto3
client = boto3.client("dynamodb", use_ssl = False)

print(client.get_item(TableName = "target_table", Key = {"id" : {"S":"key"}}))

If use_ssl is set to False, it will be sent to port 80. Since it communicates in plain text, it can be read by Wireshark.

If you look at Wireshark, you can see that the data we are sending is as follows.

text

POST / HTTP/1.1
Host: dynamodb.ap-northeast-1.amazonaws.com
Accept-Encoding: identity
X-Amz-Target: DynamoDB_20120810.GetItem
Content-Type: application/x-amz-json-1.0
User-Agent: Boto3/1.12.43 Python/3.8.2 Windows/10 Botocore/1.15.43
X-Amz-Date: 20200501T213154Z
Authorization: AWS4-HMAC-SHA256 Credential=AKIA**********/20200501/ap-northeast-1/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=***********************************************************
Content-Length: 58

{"TableName": "target_table", "Key": {"id": {"S": "key"}}}

It is OK if you set the data so that the same request will fly.

Note that use_ssl cannot be used with some services. For example, Lambda and MQTT. When I try to connect to port 80 with a service that only supports HTTPS, it times out without a response.

• To check whether communication is possible without SSL, ask AWS "Service Endpoint and Quota Table"there is. Communication is possible if the endpoint protocol is HTTP or HTTPS.

How to check with the verification function of the browser

If you want to see the payload on a service that requires SSL, use the javascript version of the SDK.

test.html

<script src="https://sdk.amazonaws.com/js/aws-sdk-2.668.0.min.js"></script>
<script type="text/javascript">
    AWS.config.update({
        accessKeyId : 'AKIA******************',
        secretAccessKey : '**********************************'
    });
    AWS.config.region = 'ap-northeast-1';
    
    let lambda = new AWS.Lambda();
    let params = {
        FunctionName : 'sample_lambda',
        InvocationType : 'RequestResponse',
        Payload : JSON.stringify({
            "Message" : "Hello"
        })
    };
    
    lambda.invoke(params, (err, data) => console.log(JSON.parse(data.Payload)));
</script>

After writing the process you want to check in the HTML file, open it in a browser. With Chrome, you can check network communication data by right-clicking and opening "Verify" in your browser.

There are differences such as browser-specific data included and CORS may be required, but we have all the necessary data. With reference to this, it is OK if you set the same request to fly.