AWS NodeJS Instrumentation
StackState SaaS
StackState provides an out-of-the-box OpenTelemetry solution for serverless AWS Lambda applications built with NodeJS. A modified OpenTelemetry Lambda Layer, based on the officially released AWS Distro for OpenTelemetry Lambda (aws-otel.github.io), gives a plug-and-play user experience.
OpenTelemetry has the following benefits:
- Lambda functions can include the OpenTelemetry Lambda Layer solution to collect trace data without the need to change any code.
- The specific AWS Service that a Lambda communicates with is shown in StackState.
The StackState OpenTelemetry Lambda Layer can be used with AWS Lambda scripts running NodeJS 14.x (or later) and will transform the Topology in StackState as follows:
Without OpenTelemetry
With OpenTelemetry
To set up OpenTelemetry traces, you need to have:
- StackState Agent should have traces enabled. If traces are not enabled on the Agent, OpenTelemetry will not generate any data.
- AWS Lambda scripts running
NodeJS 14.x(or later)- These will be the Lambda functions you wish to add OpenTelemetry support to.
- They should be able to communicate with the StackState Agent.
- The AWS StackPack should be installed and configured in StackState. The AWS StackPack CloudFormation template will deploy the latest supported OpenTelemetry Lambda Layer, which is required for AWS OpenTelemetry functionality.
Communication from AWS Lambda to:
- AWS Lambda
- AWS SQS
- AWS SNS
- AWS S3
- AWS StepFunction
- HTTP Endpoints
Follow the steps below to set up OpenTelemetry tracing for a NodeJS AWS Lambda script.
Note that the installation steps should be completed for every Lambda function that you wish to add OpenTelemetry tracing to.
After these steps have been completed, you should be ready to send traces to your StackState Agent.
To test the configuration, execute the associated Lambda function. New Topology relations should be created and visible in the StackState UI within a minute or so. Relations will be created wherever your Lambda is communicating with any of the supported services.
Go to the AWS Lambda Layers (console.aws.amazon.com) page listing all available Lambda Layer functions.
List of Lambda Layers
Change the region in the top-right corner to the region where the AWS StackPack cloudformation template was deployed.
Change Region
Verify that there is a Lambda Layer called
stackstate-otel-nodejs.If the Lambda Layer is not present:
- The AWS StackPack CloudFormation Template installed may not be the latest one containing the Lambda Layer. Follow the AWS StackPack documentation to verify the installation of this StackPack.
- Lambda Layers are AWS REGION based. This means that the AWS StackPack CloudFormation template should also be deployed inside the same region as the targeted Lambda. If the CloudFormation template is currently deployed in another region other than where the targeted Lambda lives.
Go to the Lambda functions (console.aws.amazon.com) page and navigate to the Lambda that you wish to add OpenTelemetry support to.
Underneath the Code tab, scroll down to the Layers section and click Add a layer on the right.
Edit Lambda Layer
Select the Custom Layer radio box under Choose a layer. This will show two dropdowns at the very bottom of the page.
- In the first dropdown, select the Lambda Layer
stackstate-otel-nodejs - In the second dropdown, select the latest version number
Click Add to create the custom layer.
Select Layer Version
A requirement to allow tracing to work is something called X-Ray headers. To set this up, follow ONE of the steps below:
- X-Ray headers will be available, but X-Ray is still disabled; thus, no extra costs will be accumulated by using Pass-Through X-Ray.
- Can only be set up using a CLI command.
OR
- The more expensive option, as X-Ray will charge per Lambda execution.
Enable Active Pass-through
To enable Active Pass-through you need the AWS CLI installed. Follow the steps to set up AWS CLI on your machine (docs.aws.amazon.com).
When the AWS CLI has been installed, run the following command in your terminal to set up your AWS Credentials:
aws configure
To verify if your Lambda function is running in Active Pass-Through, run the following AWS CLI commands. Unfortunately there is no way to verify this on the Console only through a CLI command:
aws lambda get-function-configuration \
--region <CHANGE TO THE REGION OF YOUR LAMBDA FUNCTION> \
--function-name <CHANGE TO THE NAME OF YOUR LAMBDA FUNCTION> \
--query 'TracingConfig.Mode'
You should be greeted with one of the following messages:
Active- the Lambda function is running in Active Pass-Through.PassThrough- the Lambda function is running in Active Pass-Through.DisabledorNone- the function configuration needs to be updated. Run the command below to change the Lambda to Pass-Through and then verify the updated configuration by running theget-function-configurationcommand again:aws lambda update-function-configuration \--region <CHANGE TO THE REGION OF YOUR LAMBDA FUNCTION> \--function-name <CHANGE TO THE NAME OF YOUR LAMBDA FUNCTION> \--tracing-config "Mode=PassThrough"
Enable X-Ray Active Tracing
To enable Active X-Ray Tracing, go to the Configuration tab inside the Lambda and click Edit under Monitoring and operations tools.
Edit Active Tracing
Set the radio box Active tracing to
true and click Save in the bottom right corner.
Enable Active Tracing
You can verify if Tracing is enabled by looking at the Active tracing block.
Verify Active Tracing
For OpenTelemetry to start capturing traces, certain environment variables are required.
Create all of the following environment variables in the Configuration tab under Environment variables. Click the Edit button to add more.
Verify Active Tracing
- StackState Agent OpenTelemetry Information
- Trace Agent Port:
8126 - OpenTelemetry StackState Agent Path:
/open-telemetry
Key | Description | Value |
|---|---|---|
AWS_LAMBDA_EXEC_WRAPPER | Allows the Lambda to root execution through OpenTelemetry. This allows OpenTelemetry to set up everything required to capture traces when the Lambda function executes. | /opt/otel-handler |
OTEL_LOG_LEVEL | The amount of logging showed in the Lambda post-execution window. | info |
OTEL_PROPAGATORS | The OpenTelemetry propagator context. | tracecontext |
OTEL_TRACES_EXPORTER | The type of export used with OpenTelemetry. | otlp |
OTEL_TRACES_SAMPLER | When execution data should be sampled. | always_on |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | Routing information from the Lambda to the StackState Agent. This needs to point to the open telemetry endpoint of your StackState Agent. | http://stackstate-agent-ip:8126/open-telemetry |
Key | Description | Default |
|---|---|---|
HTTP_MASK_SENSITIVE_INFORMATION | If the HTTP URL traces path contains sensitive information, it can be masked by setting this environment variable to true. For example http://www.url.com/user/USERID/edit#test?query=123 will be captured as http://www.url.com effectively removing everything except the hostname | false |
HTTP_REMOVE_HASH_DATA | Remove hash information from the captured HTTP URL traces. For example http://www.url.com#test?query=123 will be captured as http://www.url.com?query=123 | false |
HTTP_REMOVE_QUERY_DATA | Remove query information from the captured HTTP URL traces. For example http://www.url.com#test?query=123 will be captured as http://www.url.com#test | false |
To upgrade the OpenTelemetry Lambda Layer to the latest version, including the Lambda function using the Lambda Layer, follow the steps described below.
Make sure the latest AWS StackPack is installed. The StackPack documentation contains the link to the latest OpenTelemetry Lambda Layer in the CloudFormation template.
After the latest StackPack CloudFormation template has been deployed, there should be a new version for the
stackstate-otel-nodejs Lambda Layer. This can be confirmed on the page AWS Lambda Layers (console.aws.amazon.com).List of Lambda Layers
Click on the layer
stackstate-otel-nodejs to enter and view the details of the layer.The Created box on the right should have a relative new time showing when it was created. Alternatively, go to any Lambda Function that is currently using the
stackstate-otel-nodejs layer and compare the version it is using to the version displayed under the Version block for the layer.
List of Lambda Layers
Go to a Lambda function using the OpenTelemetry
stackstate-otel-nodejs Lambda layer. Scroll down to the Layers section, and click the Edit button on the right.
List of Lambda Layers
Change the version for the
stackstate-otel-nodejs layer to the latest version.
List of Lambda Layers
The Lambda will now use the latest OpenTelemetry Layer
To disable OpenTelemetry tracing, go to the Lambda's configuration tab. Under the Environment variables section, remove the environment variable
AWS_LAMBDA_EXEC_WRAPPER. This will disable the code routing through the OpenTelemetry Lambda Layer and restore your Lambda to the original running state.
Disable OpenTelemetry Tracing
Last modified 10mo ago