[PYTHON] Send a request from AWS Lambda to Amazon Elasticsearch Service

Amazon Elasticsearch Service is Elasticsearch. You can use the endpoint as Elasticsearch normally, and choose the access policy from the following.

• Any --Source IP address
• IAM(STS)

By the way, I thought about a request from Lambda based on IAM to lightly parse the file placed in S3 for a little convenience and thrust it into Elasticsearch, but at present it seems that I make my own signature request and throw it in.

result. I tried it, but now it seems easier to stream it to the CloudWatch Logs stream and then use the preset functions. If boto3 resource is implemented for ES, it seems that there will be no trouble.

It worked for the time being, so make a note of the code.

Elasticsearch domain side policy

--Allow roles applied to Lambda in IAM template --ʻEs: * , ʻes: ESHttpGet, etc.

Lambda function

Aside from EventSource, just send a GET to the ES / for the time being. The following materials were helpful.

-Complete Version 4 Signing Process Example (Python)-Amazon Web Services --Bringed a function to create an Authorization header. --Javascript for LogsToElasticsearch that can be created with CloudWatch Logs --Used as a reference for headers and environment variables other than Authorization. ――It's almost like transplanting this.

Mostly the code to make the header. .. ..

function.py

import urllib2, datetime, os, sys
import hashlib, hmac

endpoint = 'search-sandbox01-xxxxxxxxxxxxxxxxxxxx.ap-northeast-1.es.amazonaws.com'
url = "https://" + endpoint
region = 'ap-northeast-1'
service = 'es'
method = 'GET'

access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
    print 'No access key is available.'
    sys.exit()

def sign(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning


def lambda_handler(event, context):
    t = datetime.datetime.utcnow()
    amzdate = t.strftime('%Y%m%dT%H%M%SZ')
    datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope
    
    canonical_uri = '/'
    canonical_querystring = ""
    canonical_headers = 'host:' + endpoint + '\n' + 'x-amz-date:' + amzdate + '\n'
    signed_headers = 'host;x-amz-date'
    payload_hash = hashlib.sha256('').hexdigest()
    canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
    
    algorithm = 'AWS4-HMAC-SHA256'
    credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
    string_to_sign = algorithm + '\n' +  amzdate + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request).hexdigest()
    
    signing_key = getSignatureKey(secret_key, datestamp, region, service)
    signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()
    
    authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' +  'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
    
    headers = {
          'Content-Type': 'application/json',
                'Host': endpoint,
                'X-Amz-Security-Token': os.environ.get('AWS_SESSION_TOKEN'),
                'X-Amz-Date': amzdate,
                'Authorization': authorization_header
    }
    
    req = urllib2.Request(url, None, headers)
    
    try:
        print urllib2.urlopen(req).read()
    except urllib2.URLError, e:
        print e.read()

Is it possible to make this header quickly by using the botocore module? While thinking.

Addendum: Click here for an example of "Make it quickly using the botocore module". [Small story] Use only the signing process of botocore's AWS API request | Developers.IO

You Know, for Search

When I try to cURL to ES from my hand, it is blocked properly.

$ curl search-sandbox01-xxxxxxxxxx.ap-northeast-1.es.amazonaws.com
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:ap-northeast-1:xxxxxxxxx:domain/sandbox01/"}

When I opened Lambda and Save and Test the above code, I got a reply from ES safely.

START RequestId: b6ce7399-85eb-11e5-b2c3-99a74c2c1765 Version: $LATEST
{
  "status" : 200,
  "name" : "Chief Examiner",
  "cluster_name" : "xxxxxxxxxxx:sandbox01",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

END RequestId: b6ce7399-85eb-11e5-b2c3-99a74c2c1765
REPORT RequestId: b6ce7399-85eb-11e5-b2c3-99a74c2c1765	Duration: 64.87 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 16 MB