Created a package to support AWS Lambda development in Go language
I often use Go language + AWS Lambda at work, especially developing backend processing for security monitoring related infrastructure (this, this, this).
While advancing the development, there were various tips that said "this is convenient", but since the processing was too detailed, I used it for development by copying it between projects. However, as the number of projects managed has increased, the behavior has become different, and the number of tips has accumulated to some extent, so I cut it out as a package.
https://github.com/m-mizutani/golambda
I am aware of the Powertools (Python version, Java version) officially provided by AWS, but I have not made it for the purpose of completely reproducing it. Also, not all Go + Lambda developers think "you should follow this method!" For example, Lambda called by API gateway may not benefit much from this package because various Web Application Frameworks support similar functions. So, I hope you can see it as a story like "It is convenient to put together such processing".
Basically, we assume Lambda for data processing pipeline and a little integration, and implement the following four functions.
--Event retrieval --Structured logging --Error handling --Acquisition of hidden value
Implemented features
Retrieving an event
AWS Lambda can be triggered by specifying an event source and triggering a notification from it (https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html). At this time, the Lambda function is started by passing the data structure of the event source such as SQS and SNS. Therefore, it is necessary to extract the data to be used by yourself from various structural data. If you specify callback (Handler in the following example) in the functiongolambda.Start (), the necessary information is stored in golambda.Event and can be retrieved from there.
package main
import (
"strings"
"github.com/m-mizutani/golambda"
)
type MyEvent struct {
Message string `json:"message"`
}
//Handler that concats and returns SQS messages
func Handler(event golambda.Event) (interface{}, error) {
var response []string
//Take out the SQS body
bodies, err := event.DecapSQSBody()
if err != nil {
return nil, err
}
//SQS may receive messages in batch, so treat it as multiple messages
for _, body := range bodies {
var msg MyEvent
//bind the character string of body to msg (the content is json.Unmarshal)
if err := body.Bind(&msg); err != nil {
return nil, err
}
//Store message
response = append(response, msg.Message)
}
// concat
return strings.Join(response, ":"), nil
}
func main() {
golambda.Start(Handler)
}
This sample code is located in the ./example/deployable directory where you can deploy and try it out.
Contrary to the function DecapXxx that implements the process of retrieving data, the process of embedding data is prepared as EncapXxx. This allows you to write a test for the above Lambda Function as follows:
package main_test
import (
"testing"
"github.com/m-mizutani/golambda"
"github.com/stretchr/testify/require"
main "github.com/m-mizutani/golambda/example/decapEvent"
)
func TestHandler(t *testing.T) {
var event golambda.Event
messages := []main.MyEvent{
{
Message: "blue",
},
{
Message: "orange",
},
}
//Embedding event data
require.NoError(t, event.EncapSQS(messages))
resp, err := main.Handler(event)
require.NoError(t, err)
require.Equal(t, "blue:orange", resp)
}
Currently, it supports SQS, SNS, and SNS over SQS (SQS queue that subscribes to SNS), but we plan to implement DynamoDB stream and Kinesis stream later.
Structured logging
Lambda's standard log output destination is CloudWatch Logs, but Logs or Logs viewer Insights supports JSON-formatted logs (https://aws.amazon.com/jp/about-aws/whats-new/2015/01/20/amazon-cloudwatch-logs-json-log-format-support/). Therefore, it is convenient to have a logging tool that can output in JSON format instead of using the Go language standard log package.
Logging on Lambda, including the log output format, generally has the same requirements. Many logging tools have different options for output methods and formats, but you don't often change the settings for each Lambda function. Also, in most cases, only the variables necessary for explaining the message + context are sufficient for the output content, so we have prepared a wrapper with such a simplification in golambda. The actual output part uses zerolog. Actually, it was good to expose the logger created with zerolog as it is, but I thought it would be easier for me to narrow down what I could do, so I dared to wrap it.
A global variable called Logger is exported so that messages for each log level such as Trace, Debug, Info, and Error can be output. We have Set, which allows you to permanently embed any variable, and With, which allows you to add values in a method chain.
// ------------
//With when embedding temporary variables()use
v1 := "say hello"
golambda.Logger.With("var1", v1).Info("Hello, hello, hello")
/* Output:
{
"level": "info",
"lambda.requestID": "565389dc-c13f-4fc0-b113-xxxxxxxxxxxx",
"time": "2020-12-13T02:44:30Z",
"var1": "say hello",
"message": "Hello, hello, hello"
}
*/
// ------------
//Set to embed variables that you want to output permanently, such as request ID()use
golambda.Logger.Set("myRequestID", myRequestID)
// ~~~~~~~ snip ~~~~~~
golambda.Logger.Error("oops")
/* Output:
{
"level": "error",
"lambda.requestID": "565389dc-c13f-4fc0-b113-xxxxxxxxxxxx",
"time": "2020-11-12T02:44:30Z",
"myRequestID": "xxxxxxxxxxxxxxxxx",
"message": "oops"
}
*/
In addition, CloudWatch Logs is relatively expensive for writing logs, and if you constantly output detailed logs, it will greatly affect the cost. Therefore, it is usually convenient to output only the minimum log so that detailed output can be performed only when troubleshooting or debugging. In golambda, the log output level can be tampered with externally by setting the LOG_LEVEL environment variable. (Because only environment variables can be easily changed from the AWS console etc.)
Error handling
AWS Lambda is implemented so that each function is as single as possible, and when realizing a complicated workflow, multiple Lambda are combined using SNS, SQS, Kinesis Stream, Step Functions, etc. Therefore, if an error occurs in the middle of processing, do not try to recover forcibly in the Lambda code, but return the error as straightforwardly as possible to make it easier to notice by external monitoring, or benefit from Lambda's own retry function. It will be easier to receive.
On the other hand, Lambda itself does not handle errors very carefully, so you need to prepare your own error handling. As mentioned earlier, it is convenient to configure the Lambda function so that if something happens, it will just return an error and fail. So, in most cases, if an error occurs, if the main function (Handler () in the example described later) returns an error, it will output all the information about the error, and it will be output here and there. There is no need to write a process to output a log at the location where an error occurs or to skip an error somewhere.
golambda mainly handles the following two errors called bygolambda.Start ().
- Output detailed log of the error generated by
golambda.NewErrororgolambda.WrapError - Send the error to the error monitoring service (Sentry)
I will explain each in detail.
Detailed error log output
From experience, when an error occurs, there are two main things you want to know for debugging: "where it happened" and "what kind of situation it happened".
Strategies to find out where the error occurred include adding a context using the Wrap function, or having a stack trace like the github.com/pkg/errors package. In the case of Lambda, if you implement it as simple as possible, in most cases you can find out where the error occurred and how it occurred in the stack trace.
Also, by knowing the contents of the variable that caused the error, you can understand the conditions for reproducing the error. This can be dealt with by logging the variables that are likely to be relevant each time an error occurs, but it will result in poor log visibility across multiple output lines (especially if the call is deep). Also, you have to simply write the log output code repeatedly, which makes it redundant, and it is difficult to write it simply, and it is troublesome when you want to make changes related to log output.
Therefore, for the error generated by golambda.NewError () or golambda.WrapError () [^ error-func-name], the variable related to the error can be routed by the function With (). did. The entity is simply stored in the variable map [string] interface {} in the form of key/value. When the main logic (Handler () in the example below) returns an error generated by golambda.NewError () or golambda.WrapError (), the variables stored byWith ()and the error Prints the stack trace of the generated function to CloudWatch Logs. Below is an example of the code.
package main
import (
"github.com/m-mizutani/golambda"
)
// Handler is exported for test
func Handler(event golambda.Event) (interface{}, error) {
trigger := "something wrong"
return nil, golambda.NewError("oops").With("trigger", trigger)
}
func main() {
golambda.Start(Handler)
}
Doing this will output a log with the variables stored in With in error.values and the stack trace in error.stacktrace, as shown below. The stack trace is also output as text in the % + v format of github.com/pkg/errors, but it is also a point that it supports JSON format according to the output of the structured log.
{
"level": "error",
"lambda.requestID": "565389dc-c13f-4fc0-b113-f903909dbd45",
"error.values": {
"trigger": "something wrong"
},
"error.stacktrace": [
{
"func": "main.Handler",
"file": "xxx/your/project/src/main.go",
"line": 10
},
{
"func": "github.com/m-mizutani/golambda.Start.func1",
"file": "xxx/github.com/m-mizutani/golambda/lambda.go",
"line": 127
}
],
"time": "2020-12-13T02:42:48Z",
"message": "oops"
}
Send an error to the error monitoring service (Sentry)
There is no particular reason why it should be Sentry, but it is desirable to use some kind of error monitoring service not only for API but also for Lambda function like Web application. The reasons are as follows.
--Since it is not possible to determine whether the log ended normally or abnormally from the log output to CloudWatch Logs by default, it is difficult to extract only the log of the execution that ended abnormally. --CloudWatch Logs doesn't have a function to group errors, so it's difficult to find one that has a different type of error in only one out of 100 errors.
Both can be solved to some extent by devising the error log output method, but it is recommended to use the error monitoring service obediently because you have to be careful and implement it.
golambda sends the error returned by the main logic to Sentry by specifying the Sentry's DSN (Data Source Name) as the environment variable SENTRY_DSN (Sentry + Go Details). It doesn't matter which error you send, but the errors generated by golambda.NewError or golambda.WrapError implement a function called StackTrace () that is compatible with github.com/pkg/errors. Therefore, the stack trace is also displayed on the Sentry side.
This is the same as what is output to CloudWatch Logs, but since you can also check it on the Sentry side screen, "View notification" → "View Sentry screen" → "Search logs with CloudWatch Logs and check details" You may be able to guess the error in the second step. Also, the search for CloudWatch Logs is fairly straightforward, so if you don't have to search, it's better ...
By the way, when you send an error to Sentry, the Sentry event ID is embedded in the CloudWatch Logs log as error.sentryEventID, so you can search from the Sentry error.
Get hidden value
In Lambda, parameters that change depending on the execution environment are often stored in environment variables and used. If it is an AWS account used by an individual, it is sufficient to store it in an environment variable, but in an AWS account shared by multiple people, by separating the secret value and the environment variable, Lambda information You can separate the person (or Role) who can only refer to the secret value and the person (or Role) who can also refer to the secret value. Even if it is used by an individual, if it deals with truly dangerous information, there may be cases where the authority is separated so that even if some access key is leaked, it will not die instantly.
In my case, I often use AWS Secrets Manager to separate permissions [^ use-param-store]. Retrieving the value from Secrets Manager is relatively easy by calling the API, but I got tired of writing the same process about 100 times, so I modularized it. You can get the value by adding the json meta tag to the field of the structure.
type mySecret struct {
Token string `json:"token"`
}
var secret mySecret
if err := golambda.GetSecretValues(os.Getenv("SECRET_ARN"), &secret); err != nil {
log.Fatal("Failed: ", err)
}
Features not implemented
I thought it would be useful, but I forgot to implement it.
--Execute arbitrary processing just before the timeout: Lambda will die silently after the set maximum execution time, so there is a technique that calls some processing just before the timeout to output performance analysis information. However, in my case, I had almost no experience of the Lambda function dying silently due to a timeout, so I thought it would be useful, but I didn't touch it. --Tracing: Python's Powertools provides the ability to measure performance on AWS X-Ray using annotations and more. When I tried to do this with Go, I didn't think of a way to make it easier than Use the official SDK, so I didn't do anything in particular.
Summary
So, it was a summary of my best practices for implementing Lambda in Go and an introduction to the coded version. As I wrote at the beginning, I just made what I needed, so I think it can be used by everyone, but I hope it helps.
[^ error-func-name]: I think it's customary to use these error generation methods as errors.New () and errors.Wrap (), but I personally choose which package. It's hard to tell intuitively if you're using it, so I dared to change the naming convention.
[^ use-param-store]: Another option is to put a secret value in the AWS Systems Manager Parameter Store. I think Secrets Manager, which has a rotation function such as RDS password, is more appropriate as a service concept, and I personally use it. However, the cost and API rate limit are also different, so it seems better to use them properly according to the requirements.
Recommended Posts