[PYTHON] Issue a signed URL with AWS SQS

AWS S3 has a "signed URL". SQS can do something similar to this feature.

When do you use it?

For example, suppose you want to create a system like this.

• Periodically send the screen status of the device (whether it is in operation or if there are people) to AWS.
• The number of terminals is 10,000, the transmission interval is 5 seconds, AWS stores data in SQS and analyzes it once an hour.
• AWS-SDK does not work on the terminal, and only simple HTTPS communication is possible.

Consider the configuration on AWS.

** <Configuration A: API Gateway + Lambda> **

Lambda + API Gateway has a reputation for being cheap, but it becomes expensive when handling a large number of requests.

It's a rough calculation, but when there are 172.8 million requests a day, the charge for accumulating data is as follows.

Total: $ 1187 per day

With Lambda delay, WAF, CloudFront, authentication, etc., it goes up even more.

** <Configuration B: AWS IoT + Rule Engine + SQS> **

If you have a configuration that transfers from AWS IoT to SQS with a rule engine, the price will go down. But this time ...

Total: $ 337 per day

This time, MQTT is impossible due to the port, and WebSocket is also impossible due to the terminal side.

** **

By the way, if you can send directly from the terminal to SQS, it will cost about 1/20, $ 69 every day. There is a security problem because AWS-SDK cannot be used. Something more than that ...

It's Sorya likely.

** <Configuration D: SQS + signed URL> **

If you think about it carefully, it seems that you can do it well if you can put authentication that does not use SDK and send it directly to SQS.

• Low operating cost
• Can be implemented in any network and any environment
• Security is also decent

In order to create such a mechanism, we will realize SQS + signed URL.

Completion drawing

** The processing to be executed and the execution timing of each are as follows **

• Signed URLs expire 15 minutes after they are published and need to be updated regularly.

** The data thrown by SQS is assumed to be as follows this time **

Get SQS signed URL (run once every 15 minutes)

Request a signed URL from API Gateway [^ 1]

[^ 1]: * API Gateway authentication is omitted this time. If you want to authenticate, please put the authentication information in the header of the transmitted data.

Terminal side request

curl "https://xxxxxxxxx.execute-api.ap-northeast-1.amazonaws.com/url" -s -X POST \
-d '{"que_name":"sqs-send-request-test-0424", "patterns":["Open/Open","Close/Open","Open/Close","Close/Close"]}'

If the request is successful, you will receive the same number of signed URLs as the data pattern.

Received data on the terminal side

{
    "url": {
        "Open/Open": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxx&Action=SendMessage&MessageBody=Open%2FOpen&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx", 
        "Close/Open": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxx&Action=SendMessage&MessageBody=Close%2FOpen&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx",
        "Open/Close": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxxx&Action=SendMessage&MessageBody=Open%2FClose&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx", 
        "Close/Close": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxxx&Action=SendMessage&MessageBody=Close%2FClose&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx"
    }
}

Send terminal status to SQS (execute once every 5 seconds)

The URL to send is the data of data ["url"] [$ {state you want to send}]. In the example, "Open / Close" is sent.

Terminal side request

curl "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxxx&Action=SendMessage&MessageBody=Open%2FClose&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx"

Every time you execute it, a $ {state you want to send} message is registered in the set topic. In the example, "Open / Close" is registered in "sqs-send-request-test-0424".

Implementation

As a preliminary preparation, make a queue with SQS. Enter the name of the queue you like and click "Quick Queue Creation" at the bottom of the screen.

Also, create an IAM user dedicated to SQS. Set AmazonSQSFullAccess to the policy. No permissions other than SQS are required.

After creating the IAM user, get the access key ID and secret access key.

Implement Lambda

Create a Lambda with Python 3.8.

Set the Lambda environment variables as follows.

Paste the following source into Lambda.

Lambda

# coding: utf-8
import boto3, json, hashlib, hmac, base64, os 
from datetime import datetime, timezone, timedelta
from urllib import parse as url_encode

#Signature information (SHA256 format, signature version 2)
SIGNATURE_METHOD = "HmacSHA256"
SIGNATURE_VERSION = "2"
ENCODE = "utf-8"

#SQS method information (send message to Queue using REST API GET)
HTTP_METHOD = "GET"
SQS_METHOD = "SendMessage"
AWS_VERSION = "2012-11-05"

#Time zone
UTC_TIMEZONE = timezone(timedelta(hours = 0), 'UTC')

class Credentials:
    """
Set credentials from IAM user
    """
    @staticmethod
    def from_iam():
        instance = Credentials()
        instance.aws_access_key = os.environ.get("SQS_ACCOUNT_ID", DEFAULT.SQS_ACCOUNT_ID)
        instance.aws_secret_key = os.environ.get("SQS_SECRET_KEY", DEFAULT.SQS_SECRET_KEY)
        return instance

class Endpoint:
    """
Set SQS endpoint information
    """
    def __init__(self, topic_name):
        self.protocol = "https"
        self.host_name = "sqs.{}.amazonaws.com".format(os.environ.get("AWS_REGION", DEFAULT.AWS_REGION))
        self.url_path = "/{}/{}".format(os.environ.get("AWS_ACCOUNT_NUMBER", DEFAULT.AWS_ACCOUNT_NUMBER), topic_name)

    @property    
    def url(self):
        return f"{self.protocol}://{self.host_name}{self.url_path}"

#Create an SQS signed URL
def create_presigned_url(credential, endpoint, message):

    #Get the current date and time in UTC and convert it to a string
    current_date = datetime.now().astimezone(UTC_TIMEZONE)
    current_date_str = url_encode.quote(
        current_date.strftime('%Y-%m-%dT%H:%M:%S')
    )

    #Create a hash with a secret access key based on the data to be sent
    return endpoint.url + "?" + create_query(SQS_METHOD, message, credential.aws_access_key, current_date_str, option = {
        #Since the hash data is a byte array, URL encode + base64 encode so that it can be sent by GET.
        "Signature" : url_encode.quote(
            base64.b64encode(
                sign(
                    #Specify the secret access key to use for signing
                    credential.aws_secret_key.encode(ENCODE), 
                    #Specify the data to be sent (the specified data is hashed with SHA256)
                    create_certificate(endpoint, SQS_METHOD, message, credential.aws_access_key, current_date_str)
                )
            )
        )
    })

#Hash with secret access key
def sign(key, msg):
    return hmac.new(key, msg.encode(ENCODE), hashlib.sha256).digest()

#Create signature data in v2 format
#See official documentation for format (https)://docs.aws.amazon.com/ja_jp/general/latest/gr/signature-version-2.html）
def create_certificate(endpoint, method, message_body, aws_access_key, current_date_str):
    return "\n".join([
        HTTP_METHOD,
        endpoint.host_name,
        endpoint.url_path,
        create_query(method, message_body, aws_access_key, current_date_str)
    ])

#Normalize query data
#See official documentation for format (https)://docs.aws.amazon.com/ja_jp/general/latest/gr/signature-version-2.html）
def create_query(method, message_body, aws_access_key, current_date_str, option = None):
    query_map = {
        "AWSAccessKeyId" : aws_access_key,
        "Action" : method,
        "MessageBody" : message_body,
        "SignatureMethod" : SIGNATURE_METHOD,
        "SignatureVersion" : SIGNATURE_VERSION,
        "Timestamp" : current_date_str,
        "Version" : AWS_VERSION
    }

    #Signature should not be included in the hash, only when sending
    #Add data to the query if Signature is specified
    if option is not None:
        query_map.update(option)

    #Convert to GET query format
    return "&".join([
        f"{key}={value}"
        for key, value in query_map.items()
    ])

#Create as many signed URLs as you need
def request(credential, endpoint, data_patterns):
    url = {}
    for pattern in data_patterns:
        url[pattern] = create_presigned_url(credential, endpoint, url_encode.quote(pattern, safe = ""))
    return {
        "url" : url
    }

#Get POSTed data from APIGateway (HTTP API) arguments (data is base64 encoded on APIGateway side)
def get_payload_from_event(event):
    payload_str = ""
    if event["isBase64Encoded"]:
        payload_str = base64.b64decode(event["body"].encode(ENCODE))
    else:
        payload_str = event["body"]
    return json.loads(payload_str)

#Lambda runtime entry point
def lambda_handler(event, context):
    payload = get_payload_from_event(event)
    return {
        'statusCode': 200,
        'body': json.dumps(request(Credentials.from_iam(), Endpoint(payload["que_name"]), payload["patterns"]))
    }

# ---------------------------------------------
#The following is for local execution
#Specify when running other than Lambda
# ---------------------------------------------

#Variables for local execution
class _Default:
    def __init__(self):
        self.SQS_ACCOUNT_ID = "" #Specify the IAM access key ID
        self.SQS_SECRET_KEY = "" #Specify the IAM secret key
        self.AWS_ACCOUNT_NUMBER = "" #Specify your AWS account ID
        self.AWS_REGION = "ap-northeast-1" #Specify the region where SQS is located
        self.SQS_QUEUE_NAME = "" #Specify the destination queue name
DEFAULT = _Default()

#Entry point when running local environment debug
if __name__ == "__main__":
    print(json.dumps(request(Credentials.from_iam(), Endpoint(DEFAULT.SQS_QUEUE_NAME), [
        "Open/Open",  "Close/Open",  "Wait/Open",
        "Open/Close", "Close/Close", "Wait/Close",
        "Open/Wait",  "Close/Wait",  "Wait/Wait"
    ])))

Set the created Lambda as the back end of API Gateway.

• The example API Gateway is created with HTTP API.

If it's OK if you move, it's okay if you read this far. After deploying API Gateway, it should work just like a "rendering".

From here on, it will be a detailed story.

Detailed story

Originally, SQS has a mechanism to send messages by GET or POST without going through AWS-SDK.

Reference: Make a query API request https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-making-api-requests.html

Anonymous user, send to public queue

The simplest request method is as follows. When you hit the URL on the reference site, a message (data) will be sent to the queue (MyQueue). [^ 2]

[^ 2]: * Version is the version of SQS. For the time being, you can send it without thinking about anything.

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue ? 
    Version = 2012-11-05 &
    Action = SendMessage &
    MessageBody = data

Note that this query will fly anonymous user messages. The default queue does not arrive with a permission error.

To receive messages in this state, you must have set them to accept public disclosure. Change the queue permissions as captured and check "Everyone".

Send to a queue that allows only specific users

If you don't want to accept the public release and want to send it to a specific user, set "Who am I?".

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue ?
    Version = 2012-11-05 &
    Action = SendMessage &
    MessageBody = data &
    AWSAccessKeyId = AKIA****************

Since I set the AWSAccessKeyID, "someone" can be communicated. However, I don't know if this is really the person. You may just impersonate yourself.

Use the secret access key to prove your identity. However, since it is not possible to send the secret access key as it is, we will send the hashed version.

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue ?
    Version = 2012-11-05 &
    Action = SendMessage &
    MessageBody = data &
    AWSAccessKeyId = AKIA**************** &
    Signature = ********************************** &
    SignatureMethod = HmacSHA256 &
    SignatureVersion = 2 &
    Timestamp = 2020-04-30T10:42:54

Signature is a hash of the data to be sent, using the secret access key as a key. The data to be sent is the entire query excluding Signature and the host information combined.

Signature Method is the algorithm used to hash Signature. Since it is HMAC-SHA256, I will tell AWS about it.

Signature Version is the signature version. This time I'm using signature version 2.

TimeStamp is the hashed date and time.

Reference: Signing version 2 signing process https://docs.aws.amazon.com/ja_jp/general/latest/gr/signature-version-2.html

By the way, what is the difference from the original signed URL?

The behavior is different compared to the "signed URL" of the original S3.

** The range in which data can be changed after issuing the URL is different **

In SQS, the signature changes depending on the content of the message you send. With S3, you can send with the same signature no matter what the content of the file you send.

To send different messages with SQS, you need to issue that many Signatures.

** The lifetime of the temporary credentials passed by STS is different **

For S3, the temporary access key ID issued by Lambda can be used until the signed URL expires. Similarly for SQS, what happens if you issue and grant a temporary access key ID and secret access key?

Since the access key ID disappears when Lambda ends, authentication will not pass when Lambda returns the URL to API Gateway.