[PYTHON] I managed to do it because the custom of attaching a zip with a password to an email and saying "I will send you the password separately" is troublesome.
That custom
There is a custom that a zip with a password is attached to an email and "the password will be sent separately". I don't do it myself, but it's annoying because I have to do it according to the other party.
The pros and cons of this approach do not matter here. No matter how much I preach, the situation of having this custom does not change.
And I don't think about breaking this practice. I'll leave that to something with enormous power.
The old idiot said. "Wrap it around a long one." However, I think it is better to think about how to wind it.
Think about how to wind smartly
There is only one thing I want to solve when it is rolled up. Don't be annoyed. If you make a Web system for this purpose and open a browser to do something like this, it will be overwhelming. I want to realize it as close to normal email transmission as possible.
So, after thinking about it, I tried to solve it with a serverless feeling using Amazon SES while allowing some restrictions.
specification
- Write an email normally (new, reply, forward)
- Throw the file as it is without zipping it
- Set the email address for SES to
Toand the person to whom you actually want to send the file toReply-To. - Believe in the system and press the submit button
- You and the other party will receive an email with a password-attached zip and a password notification email.
However, there are the following restrictions. Personally, it's acceptable.
--As a result, everyone reaches the other party by To. You can't Cc (I'm Bcc)
--The name of the zip file will be the date and time (yymmddHHMMSS.zip) (the file name inside remains the same)
System configuration
- Send an email to SES
- Email data is saved in S3
- Trigger it to start Lambda
- Lambda parses the email and generates a password and zip file
- Send a nice email (send it to yourself with Bcc just in case)
Implementation
Lambda It's the first time I wrote python seriously, but is it okay like this? It's about a battle between email, character encoding, and files.
# -*- coding: utf-8 -*-
import os
import sys
import string
import random
import json
import urllib.parse
import boto3
import re
import smtplib
import email
import base64
from email import encoders
from email.header import decode_header
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from datetime import datetime
sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'vendored'))
import pyminizip
s3 = boto3.client('s3')
class MailParser(object):
"""
Email parsing class
(reference) http://qiita.com/sayamada/items/a42d344fa343cd80cf86
"""
def __init__(self, email_string):
"""
Initialization
"""
self.email_message = email.message_from_string(email_string)
self.subject = None
self.from_address = None
self.reply_to_address = None
self.body = ""
self.attach_file_list = []
#Interpretation of eml
self._parse()
def get_attr_data(self):
"""
Get email data
"""
attr = {
"from": self.from_address,
"reply_to": self.reply_to_address,
"subject": self.subject,
"body": self.body,
"attach_files": self.attach_file_list
}
return attr
def _parse(self):
"""
Parsing mail files
"""
#Analysis of message header part
self.subject = self._get_decoded_header("Subject")
self.from_address = self._get_decoded_header("From")
self.reply_to_address = self._get_decoded_header("Reply-To")
#Extract only the character string of the email address
from_list = re.findall(r"<(.*@.*)>", self.from_address)
if from_list:
self.from_address = from_list[0]
reply_to_list = re.findall(r"<(.*@.*)>", self.reply_to_address)
if reply_to_list:
self.reply_to_address = ','.join(reply_to_list)
#Analysis of message body part
for part in self.email_message.walk():
#If the ContentType is multipart, the actual content is even more
#Since it is in the inside part, skip it
if part.get_content_maintype() == 'multipart':
continue
#Get file name
attach_fname = part.get_filename()
#Should be the body if there is no file name
if not attach_fname:
charset = str(part.get_content_charset())
if charset != None:
if charset == 'utf-8':
self.body += part.get_payload()
else:
self.body += part.get_payload(decode=True).decode(charset, errors="replace")
else:
self.body += part.get_payload(decode=True)
else:
#If there is a file name, it's an attachment
#Get the data
self.attach_file_list.append({
"name": attach_fname,
"data": part.get_payload(decode=True)
})
def _get_decoded_header(self, key_name):
"""
Get the decoded result from the header object
"""
ret = ""
#A key that does not have a corresponding item returns an empty string
raw_obj = self.email_message.get(key_name)
if raw_obj is None:
return ""
#Make the decoded result unicode
for fragment, encoding in decode_header(raw_obj):
if not hasattr(fragment, "decode"):
ret += fragment
continue
#UTF for the time being without encode-Decode with 8
if encoding:
ret += fragment.decode(encoding)
else:
ret += fragment.decode("UTF-8")
return ret
class MailForwarder(object):
def __init__(self, email_attr):
"""
Initialization
"""
self.email_attr = email_attr
self.encode = 'utf-8'
def send(self):
"""
Compress the attached file with a password, forward it, and send a password notification email
"""
#Password generation
password = self._generate_password()
#zip data generation
zip_name = datetime.now().strftime('%Y%m%d%H%M%S')
zip_data = self._generate_zip(zip_name, password)
#Send zip data
self._forward_with_zip(zip_name, zip_data)
#Send password
self._send_password(zip_name, password)
def _generate_password(self):
"""
Password generation
Shuffle by taking 4 letters each from symbols, letters and numbers
"""
password_chars = ''.join(random.sample(string.punctuation, 4)) + \
''.join(random.sample(string.ascii_letters, 4)) + \
''.join(random.sample(string.digits, 4))
return ''.join(random.sample(password_chars, len(password_chars)))
def _generate_zip(self, zip_name, password):
"""
Generate data for password-protected Zip file
"""
tmp_dir = "/tmp/" + zip_name
os.mkdir(tmp_dir)
#Save the file locally
for attach_file in self.email_attr['attach_files']:
f = open(tmp_dir + "/" + attach_file['name'], 'wb')
f.write(attach_file['data'])
f.flush()
f.close()
#To zip with password
dst_file_path = "/tmp/%s.zip" % zip_name
src_file_names = ["%s/%s" % (tmp_dir, name) for name in os.listdir(tmp_dir)]
pyminizip.compress_multiple(src_file_names, dst_file_path, password, 4)
# #Read the generated zip file
r = open(dst_file_path, 'rb')
zip_data = r.read()
r.close()
return zip_data
def _forward_with_zip(self, zip_name, zip_data):
"""
Generate data for password-protected Zip file
"""
self._send_message(
self.email_attr['subject'],
self.email_attr["body"].encode(self.encode),
zip_name,
zip_data
)
return
def _send_password(self, zip_name, password):
"""
Send password for zip file
"""
subject = self.email_attr['subject']
message = """
This is the password for the file you sent earlier.
[subject] {}
[file name] {}.zip
[password] {}
""".format(subject, zip_name, password)
self._send_message(
'[password]%s' % subject,
message,
None,
None
)
return
def _send_message(self, subject, message, attach_name, attach_data):
"""
send e-mail
"""
msg = MIMEMultipart()
#header
msg['Subject'] = subject
msg['From'] = self.email_attr['from']
msg['To'] = self.email_attr['reply_to']
msg['Bcc'] = self.email_attr['from']
#Text
body = MIMEText(message, 'plain', self.encode)
msg.attach(body)
#Attachment
if attach_data:
file_name = "%s.zip" % attach_name
attachment = MIMEBase('application', 'zip')
attachment.set_param('name', file_name)
attachment.set_payload(attach_data)
encoders.encode_base64(attachment)
attachment.add_header("Content-Dispositon", "attachment", filename=file_name)
msg.attach(attachment)
#Send
smtp_server = self._get_decrypted_environ("SMTP_SERVER")
smtp_port = self._get_decrypted_environ("SMTP_PORT")
smtp_user = self._get_decrypted_environ("SMTP_USER")
smtp_password = self._get_decrypted_environ("SMTP_PASSWORD")
smtp = smtplib.SMTP(smtp_server, smtp_port)
smtp.ehlo()
smtp.starttls()
smtp.ehlo()
smtp.login(smtp_user, smtp_password)
smtp.send_message(msg)
smtp.quit()
print("Successfully sent email")
return
def _get_decrypted_environ(self, key):
"""
Decrypt encrypted environment variables
"""
client = boto3.client('kms')
encrypted_data = os.environ[key]
return client.decrypt(CiphertextBlob=base64.b64decode(encrypted_data))['Plaintext'].decode('utf-8')
def lambda_handler(event, context):
#Get bucket name and key name from event
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
try:
#Read the contents of the file from S3
s3_object = s3.get_object(Bucket=bucket, Key=key)
email_string = s3_object['Body'].read().decode('utf-8')
#Analyze email
parser = MailParser(email_string)
#Email forwarding
forwarder = MailForwarder(parser.get_attr_data())
forwarder.send()
return
except Exception as e:
print(e)
raise e
pyminizip It seems that password-protected zip cannot be done with a standard library. So, I relied on an external library called pyminizip only here. However, this was a library that was built at the time of installation and made a binary, so I made a binary by setting up a Docker container of Amazon Linux locally to run it with Lambda. Is there any other good way? ..
AWS SAM By the way, I tested this locally using AWS SAM. It was good to write the SMTP server information directly and try it, but when I moved it to an environment variable, it didn't work and I was frustrated. It looks like it has been fixed but not released.
Introduction method
I will publish it because it is a big deal. Codename zaru.
Please forgive me though the setting method remains muddy. ..
https://github.com/Kta-M/zaru
I've only tried it in my environment (Mac, Thunderbird), so it may not work depending on the mailer or other environment. Please take responsibility for your actions.
SES SES is not yet available in the Tokyo region, so we will build it in the Oregon region (us-west-2).
Domain verification
First, we will verify the domain so that you can send an email to SES. There are various methods, so I will omit this area. For example, this may be helpful-> Send domain email using Amazon SES / Route53 with Rails
Rule creation
After verifying the domain, create a Rule.
From Rule Sets on the right side of the menu, click View Active Rule Set.
Click Create Rule.
Register the email address to receive. Enter the email address of the verified domain and click ʻAdd Recipient`.
Register the action when receiving an email.
Select S3 as the action type and specify the bucket to store the received mail data. At this time, if you create a bucket with Create S3 bucket, the required bucket policy will be registered automatically, which is convenient.
A policy is set that allows file uploads from SES to the bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSESPuts-XXXXXXXXXXXX",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<ses-bucket-name>/*",
"Condition": {
"StringEquals": {
"aws:Referer": "XXXXXXXXXXXX"
}
}
}
]
}
Also, the mail data saved in the bucket can be stored, so it may be better to set a life cycle so that it will be deleted after a certain period of time.
Give the rule a name. The rest is by default.
Check the registration details and register!
Lambda
Deploy
Deploy to the Oregon region as well as SES. Since CloudFormation will be used, please create an S3 bucket to upload data.
# git clone [email protected]:Kta-M/zaru.git
# cd zaru
# aws cloudformation package --template-file template.yaml --s3-bucket <cfn-bucket-name> --output-template-file packaged.yaml
# aws cloudformation deploy --template-file packaged.yaml --stack-name zaru-stack --capabilities CAPABILITY_IAM --region us-west-2
If you go to the Lambda console, the function is created.
You've also created the IAM role needed to execute this function.
Trigger setting
Set Lambda to work by triggering mail data in the bucket.
Go to the Trigger tab on the function details screen.
Click Add Trigger to create an S3 event.
The bucket where the data comes from SES, the event type is Put. Other than that, it is the default.
Bucket is
Create an encryption key
In this Lambda function, we get the SMTP related information from the encrypted environment variables. Create a key to use for that encryption.
From the IAM console, click the encryption key at the bottom left.
Change the region to Oregon and create a key.
All you have to do is set an alias of your choice, and the rest is OK by default.
Setting the number of environment variables
Go back to Lambda and set the environment variables you want to use in your function.
At the bottom of the Code tab is a form for setting environment variables.
Check Enable encryption helper and specify the encryption key you created earlier.
For environment variables, enter the variable name and value (plain text) and press the `encrypt" button. Then, it will be encrypted with the specified encryption key.
The following four environment variables are set.
| Variable name | Description | Example |
|---|---|---|
| SMTP_SERVER | smtp server | smtp.example.com |
| SMTP_PORT | smtp port | 587 |
| SMTP_USER | Username to log in to smtp server | [email protected] |
| SMTP_PASSWORD | SMTP_USER password |
Role settings
Finally, give the role that executes this Lambda function the required permissions. --Permission to retrieve data from S3 bucket to store mail data --Permission to decrypt environment variables using encryption key
First, go to the policy of the IAM console and go to Create Policy-> Create Your Own Policy to create the following two policies.
** Policy: s3-get-object-zaru **
For <ses-bucket-name>, specify the bucket name to receive mail data from SES.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1505586008000",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<ses-bucket-name>/*"
]
}
]
}
** Policy; kms-decrypt-zaru **
For <kms-arn>, specify the ARN of the encryption key.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1448696327000",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": [
"<kms-arn>"
]
}
]
}
Finally, attach these two policies to your Lambda function execution role.
First, go to the role in the IAM console, select the role, and attach it fromAttach Policy.
Operation check
It should now work.
Please set the e-mail address set for SES in To and the e-mail address of the other party in Reply-To, and attach an appropriate file and send it. How is it?
Summary
Dontokoi zip attachment!