Auto-instrumentation of Lambdas
SUSE Observability
Introduction
This document guides you through auto-instrumenting NodeJS Lambda functions using OpenTelemetry. Auto-instrumentation simplifies the process of adding observability to your Lambda functions by automatically capturing performance metrics and tracing information.
Prerequisites
Before you begin, ensure you have the following:
AWS Lambda function: The function you want to instrument.
OpenTelemetry SDK: Installed in your Lambda function.
OpenTelemetry Collector: Deployed and configured.
SUSE Observability: An account with SUSE Observability where you'll send your telemetry data.
Memory: Enough memory to run the Lambda’s including the instrumentation.
Values supplied by the environment
OpenTelemetry relies on various configuration values to function correctly. These values control aspects like data collection, exporting, and communication with backend systems. To make your OpenTelemetry deployment flexible and adaptable to different environments, you can provide these settings through environment variables. This approach offers several benefits:
Dynamic Configuration: Easily adjust settings without code changes.
Environment-Specific Settings: Configure OpenTelemetry differently for development, testing, and production.
Secret Management: Securely store sensitive information like API keys.
For the OpenTelemetry setup described in this documentation, you'll need to define the following environment variables:
VERBOSITY: Controls the level of detail in OpenTelemetry logs.OTLP_API_KEY: Authenticates your Lambda function to send data to SUSE Observability.OTLP_ENDPOINT: Specifies the address of your SUSE Observability instance.OPENTELEMETRY_COLLECTOR_CONFIG_FILE: Points to the configuration file for the OpenTelemetry Collector.AWS_LAMBDA_EXEC_WRAPPER: Configures the Lambda execution environment to use the OpenTelemetry handler.OTLP_INSTR_LAYER_ARN: Provides the ARN (Amazon Resource Name) of the OpenTelemetry instrumentation layer, which adds the necessary components for auto-instrumentation.OTLP_COLLECTOR_LAYER_ARN: Provides the ARN of the OpenTelemetry collector layer, which is responsible for receiving, processing, and exporting telemetry data.
Important Considerations:
GRPC Endpoint: The
OTLP_ENDPOINTshould specify the gRPC endpoint of your SUSE Observability instance without anyhttporhttpsprefix. Use port 443 for secure communication.Region-Specific Layers: Lambda layers are region-bound. Ensure that the ARNs you use for
OTLP_INSTR_LAYER_ARNandOTLP_COLLECTOR_LAYER_ARNmatch the AWS region where your Lambda function is deployed.Architecture Matching: The OpenTelemetry Collector layer is architecture-specific. Choose the correct ARN for your Lambda function's architecture (e.g.,
amd64orarm64).
A complete example: be aware you need to input your own values.
The collector.yaml file
OTEL collection configuration sets up how the data collected should be distributed. This is done in the collector.yaml file placed in the src directory where the lambda files can be found. Below is an example collector.yaml file.
Be aware this collector is used to send the data over to a next collector which then is used for tail sampling, metric aggregation, etc. before sending data over to SUSE Observability. This second collector also needs to run in the customer's environment.
Depending on the desired functionality, or based upon factors such as volumes of data being generated by lambdas instrumented in this way, collectors can be set up for batching, tail-sampling, and other pre-processing techniques to reduce the impact on SUSE Observability.
Package.json
Make sure to add "@opentelemetry/auto-instrumentations-node": "^0.55.2", to package.json and execute npm install to add the auto-instrumentation client libraries to your NodeJS Lambda.
Troubleshooting
Timeouts
If the addition of the OTEL Lambda layers results in lambdas that time out (checking the logs might indicate that the collector was asked to shut down while still busy, e.g. seeing the following log entry):
shortly after receiving the instruction to shut down:
The above indicates that the allocated resources of the lambda are not sufficient to allow execution of the lambda and the additional strain added by the OTEL instrumentation. To remedy this, the memory allocation and lambda timeout settings can be adjusted as necessary to allow the lambda to finish its work, while also allowing the telemetry collection to succeed.
Try modifying the MemorySize and TimeOut properties of the lambdas that are failing:
Note the default memory allocation is 128MB
Note the memory increment is 128MB
Note Timeout is an integer value denoting seconds.
Authentication and Source IP Filtering
If you encounter error 403 Unauthorized when submitting collector data to your cluster, or to any pre-processing or proxy collector, double-check the source IP address of the VPC NAT gateway matches what is whitelisted by the collector ingress,
also double check that the chosen authentication mechanism matches source and destination, and also that credentials (secrets, etc.) are set up correctly.
References
Last updated
