AWS NodeJS Instrumentation

StackState SaaS

Overview

StackState provides an out-of-the-box OpenTelemetry solution for serverless AWS Lambda applications built with NodeJS. A modified OpenTelemetry Lambda Layer, based on the officially released AWS Distro for OpenTelemetry Lambda (aws-otel.github.io), gives a plug-and-play user experience.

OpenTelemetry has the following benefits:

  • Lambda functions can include the OpenTelemetry Lambda Layer solution to collect trace data without the need to change any code.

  • The specific AWS Service that a Lambda communicates with is shown in StackState.

The StackState OpenTelemetry Lambda Layer can be used with AWS Lambda scripts running NodeJS 14.x (or later) and will transform the Topology in StackState as follows:

Setup

Prerequisites

To set up OpenTelemetry traces, you need to have:

  • StackState Agent v2.16 (or later)

  • StackState Agent should have traces enabled. If traces are not enabled on the Agent, OpenTelemetry will not generate any data.

  • AWS Lambda scripts running NodeJS 14.x (or later)

    • These will be the Lambda functions you wish to add OpenTelemetry support to.

    • They should be able to communicate with the StackState Agent.

  • The AWS StackPack should be installed and configured in StackState. The AWS StackPack CloudFormation template will deploy the latest supported OpenTelemetry Lambda Layer, which is required for AWS OpenTelemetry functionality.

Supported Services

Communication from AWS Lambda to:

  • AWS Lambda

  • AWS SQS

  • AWS SNS

  • AWS S3

  • AWS StepFunction

  • HTTP Endpoints

Installation

Follow the steps below to set up OpenTelemetry tracing for a NodeJS AWS Lambda script.

Note that the installation steps should be completed for every Lambda function that you wish to add OpenTelemetry tracing to.

After these steps have been completed, you should be ready to send traces to your StackState Agent.

To test the configuration, execute the associated Lambda function. New Topology relations should be created and visible in the StackState UI within a minute or so. Relations will be created wherever your Lambda is communicating with any of the supported services.

Verify that the Lambda Layer exists

Go to the AWS Lambda Layers (console.aws.amazon.com) page listing all available Lambda Layer functions.

Change the region in the top-right corner to the region where the AWS StackPack cloudformation template was deployed.

Verify that there is a Lambda Layer called stackstate-otel-nodejs.

If the Lambda Layer is not present:

  • The AWS StackPack CloudFormation Template installed may not be the latest one containing the Lambda Layer. Follow the AWS StackPack documentation to verify the installation of this StackPack.

  • Lambda Layers are AWS REGION based. This means that the AWS StackPack CloudFormation template should also be deployed inside the same region as the targeted Lambda. If the CloudFormation template is currently deployed in another region other than where the targeted Lambda lives.

Go to the Lambda functions (console.aws.amazon.com) page and navigate to the Lambda that you wish to add OpenTelemetry support to.

Underneath the Code tab, scroll down to the Layers section and click Add a layer on the right.

Select the Custom Layer radio box under Choose a layer. This will show two dropdowns at the very bottom of the page.

  • In the first dropdown, select the Lambda Layer stackstate-otel-nodejs

  • In the second dropdown, select the latest version number

Click Add to create the custom layer.

Set up tracing

A requirement to allow tracing to work is something called X-Ray headers. To set this up, follow ONE of the steps below:

  • Enable Active Pass-through on your Lambda Function.

    • X-Ray headers will be available, but X-Ray is still disabled; thus, no extra costs will be accumulated by using Pass-Through X-Ray.

    • Can only be set up using a CLI command.

OR

Enable Active Pass-through

To enable Active Pass-through you need the AWS CLI installed. Follow the steps to set up AWS CLI on your machine (docs.aws.amazon.com).

When the AWS CLI has been installed, run the following command in your terminal to set up your AWS Credentials:

aws configure

To verify if your Lambda function is running in Active Pass-Through, run the following AWS CLI commands. Unfortunately there is no way to verify this on the Console only through a CLI command:

aws lambda get-function-configuration \
--region <CHANGE TO THE REGION OF YOUR LAMBDA FUNCTION> \
--function-name <CHANGE TO THE NAME OF YOUR LAMBDA FUNCTION> \
--query 'TracingConfig.Mode'

You should be greeted with one of the following messages:

  • Active - the Lambda function is running in Active Pass-Through.

  • PassThrough - the Lambda function is running in Active Pass-Through.

  • Disabled or None - the function configuration needs to be updated. Run the command below to change the Lambda to Pass-Through and then verify the updated configuration by running the get-function-configuration command again:

    aws lambda update-function-configuration \
    --region <CHANGE TO THE REGION OF YOUR LAMBDA FUNCTION> \
    --function-name <CHANGE TO THE NAME OF YOUR LAMBDA FUNCTION> \
    --tracing-config "Mode=PassThrough"

Enable X-Ray Active Tracing

To enable Active X-Ray Tracing, go to the Configuration tab inside the Lambda and click Edit under Monitoring and operations tools.

Set the radio box Active tracing to true and click Save in the bottom right corner.

You can verify if Tracing is enabled by looking at the Active tracing block.

Add environment variables

For OpenTelemetry to start capturing traces, certain environment variables are required.

Create all of the following environment variables in the Configuration tab under Environment variables. Click the Edit button to add more.

  • StackState Agent OpenTelemetry Information

    • Trace Agent Port: 8126

    • OpenTelemetry StackState Agent Path: /open-telemetry

Required Environment variables

Optional Environment variables

Upgrade

To upgrade the OpenTelemetry Lambda Layer to the latest version, including the Lambda function using the Lambda Layer, follow the steps described below.

Make sure the latest AWS StackPack is installed. The StackPack documentation contains the link to the latest OpenTelemetry Lambda Layer in the CloudFormation template.

After the latest StackPack CloudFormation template has been deployed, there should be a new version for the stackstate-otel-nodejs Lambda Layer. This can be confirmed on the page AWS Lambda Layers (console.aws.amazon.com).

Click on the layer stackstate-otel-nodejs to enter and view the details of the layer.

The Created box on the right should have a relative new time showing when it was created. Alternatively, go to any Lambda Function that is currently using the stackstate-otel-nodejs layer and compare the version it is using to the version displayed under the Version block for the layer.

Go to a Lambda function using the OpenTelemetry stackstate-otel-nodejs Lambda layer. Scroll down to the Layers section, and click the Edit button on the right.

Change the version for the stackstate-otel-nodejs layer to the latest version.

The Lambda will now use the latest OpenTelemetry Layer

Disable OpenTelemetry Traces

To disable OpenTelemetry tracing, go to the Lambda's configuration tab. Under the Environment variables section, remove the environment variable AWS_LAMBDA_EXEC_WRAPPER. This will disable the code routing through the OpenTelemetry Lambda Layer and restore your Lambda to the original running state.

Last updated