Upload what you got in request to S3 with AWS Lambda Python

At the women's club's Mokumokukai, I decided to make a crawler using AWS Lambda, so I lifted my back. The waist is really heavy.

Speaking of Lambda, it's free 1 million times a month, so I think you can run it 23 times a minute.

Crawler / scraping made with Lambda seems to be helpful, so at first I got it with request to S3 while looking at this I decided to upload something for the time being. The language is of course Python!

`python`


import boto3
import requests

BUCKET = 'test_requests'
s3 = boto3.client('s3')

def lambda_handler(event, context):
    key = 'corp.camon.tokyo'
    target_url = 'http://corp.camon.tokyo'
    target_html = requests.get(target_url).text
    s3.put_object(Bucket=BUCKET, Key=key, Body=target_html)

As expected, AWS Lambda is an easy win! When I press the Test button while thinking ...

`python`


"errorMessage": "Unable to import module 'lambda_function'"

Requests cannot be read and an error \ (^ o ^) / When I was wondering how to pip install on AWS Lambda, I came across this article.

Deploy AWS Lambda Python with lambda-uploader

AWS CLI required

`python`


pip install lambda-uploader

I made a directory, copied the source I wrote earlier, and prepared lambda.json and requirements.txt.

`test_requests.py`


import boto3
import requests

BUCKET = 'test_requests'
s3 = boto3.client('s3')

def lambda_handler(event, context):
    key = 'corp.camon.tokyo'
    target_url = 'http://corp.camon.tokyo'
    target_html = requests.get(target_url).text
    s3.put_object(Bucket=BUCKET, Key=key, Body=target_html)

`lambda.json`


  {
    "name": "test_requests",
    "description": "test requests",
    "region": "ap-northeast-1",
    "handler": "test_requests.lambda_handler",
    "role": "arn:aws:iam::????????????:role/lambda_s3_exec_role",
    "timeout": 300,
    "memory": 128
  }

`requirements.txt`


requests

The composition is like this.

`python`


$ tree
.
├── lambda.json
├── requirements.txt
└── test_requests.py

0 directories, 3 files

Now that it's ready, execute the command.

`python`


lambda-uploader

`python`


λ Building Package
λ Uploading Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 151, in main
    _execute(args)
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 82, in _execute
    upldr.upload(pkg)
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 112, in upload
    self.version = self.upload_new(pkg)
  File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 89, in upload_new
    Publish=self._config.publish,
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 301, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 398, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (AccessDeniedException) when calling the CreateFunction operation: User: arn:aws:iam::????????????:user/WakanaYoshizawa is not authorized to perform: lambda:CreateFunction

If you don't have permission

Add AWSLambdaFullAccess and run again

error··

At this point, the dots. Women's Club's Mokumokukai was imminently provided with beer at the end of the game, so I was sick and drank. I opened about half of the beer and got drunk, so I tried it with a slapstick ...

$ lambda-uploader
λ Building Package
λ Uploading Package
λ Fin

It seemed that it took time to reflect the policy attachment. It is important to develop with a margin.

I think I can do various things with Lambda, so I'll play with more \ (^ o ^) /