LogoLogo
StackState.comDownloadSupportExplore playground
StackState v5.1
StackState v5.1
  • Welcome to the StackState docs!
  • StackState self-hosted v5.1 docs
  • Getting Started
  • 🚀Setup
    • Install StackState
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Required Permissions
        • Non-high availability setup
        • Override default configuration
        • Configure storage
        • Configure Ingress
        • Install from custom image registry
        • Migrate from Linux install
      • Linux
        • Before you install
        • Download
        • Install StackState
        • Install with production configuration
        • Install with development configuration
        • Install with POC configuration
        • Set up a reverse proxy
        • Set up TLS without reverse proxy
      • Initial run guide
      • Troubleshooting
    • Upgrade StackState
      • Steps to upgrade
      • Version specific upgrade instructions
      • StackPack versions
      • StackState release notes
    • StackState Agent
      • About StackState Agent V3
      • Docker
      • Kubernetes / OpenShift
      • Linux
      • Windows
      • Advanced Agent configuration
      • Use an HTTP/HTTPS proxy
      • Agent V1 (legacy)
      • Migrate Agent V1 to Agent V2
        • Linux
        • Docker
    • StackState CLI
      • CLI: sts
      • CLI: stac (deprecated)
      • Comparison between CLIs
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Linux backup
        • Configuration backup
      • Data retention
      • Clear stored data
  • 👤Use
    • Concepts
      • The 4T data model
      • Components
      • Relations
      • Health state
      • Layers, Domains and Environments
      • Perspectives
      • Anomaly detection
      • StackState architecture
    • StackState UI
      • Explore mode
      • Filters
      • Views
        • About views
        • Configure the view health
        • Create and edit views
        • Visualization settings
      • Perspectives
        • Topology Perspective
        • Events Perspective
        • Traces Perspective
        • Metrics Perspective
      • Timeline and time travel
      • Analytics
      • Keyboard shortcuts
    • Checks and monitors
      • Checks
      • Add a health check
      • Anomaly health checks
      • Monitors
      • Manage monitors
    • Problem analysis
      • About problems
      • Problem lifecycle
      • Investigate a problem
      • Problem notifications
    • Metrics
      • Telemetry streams
      • Golden signals
      • Top metrics
      • Add a telemetry stream
      • Browse telemetry
      • Set telemetry stream priority
    • Events
      • About events
      • Event notifications
      • Manage event handlers
    • Glossary
  • 🧩StackPacks
    • About StackPacks
    • Add-ons
      • Autonomous Anomaly Detector
      • Health Forecast
    • Integrations
      • About integrations
      • 💠StackState Agent V2
      • 💠AWS
        • AWS
        • AWS ECS
        • AWS X-ray
        • StackState/Agent IAM role: EC2
        • StackState/Agent IAM role: EKS
        • Policies for AWS
        • AWS (legacy)
        • Migrate AWS (legacy) to AWS
      • 💠Dynatrace
      • 💠Kubernetes
      • 💠OpenShift
      • 💠OpenTelemetry
        • About instrumentations
        • AWS NodeJS Instrumentation
        • Manual Instrumentation
          • Prerequisites
          • Tracer and span mappings
          • Relations between components
          • Span health state
          • Merging components
          • Code examples
      • 💠ServiceNow
      • 💠Slack
      • 💠Splunk
        • Splunk
        • Splunk Events
        • Splunk Health
        • Splunk Metrics
        • Splunk Topology
      • 💠VMWare vSphere
      • Apache Tomcat
      • Azure
      • Cloudera
      • Custom Synchronization
      • DotNet APM
      • Elasticsearch
      • Humio
      • Java APM
      • JMX
      • Logz.io
      • MySQL
      • Nagios
      • OpenMetrics
      • PostgreSQL
      • Prometheus
      • SAP
      • SCOM
      • SolarWinds
      • Static Health
      • Static Topology
      • Traefik
      • WMI
      • Zabbix
    • Develop your own StackPacks
  • 🔧Configure
    • Topology
      • Component actions
      • Identifiers
      • Topology naming guide
      • Topology sources
      • Create a topology manually
      • Configure topology synchronizations
      • Enable email event notifications
      • Send topology data over HTTP
      • Set the topology filtering limit
      • Use a proxy for event handlers
      • Use tags
      • Tune topology synchronization
      • Debug topology synchronization
    • Telemetry
      • Add telemetry during topology synchronization
      • Data sources
        • Elasticsearch
        • Prometheus mirror
      • Send events over HTTP
      • Send metrics data over HTTP
      • Set the default telemetry interval
      • Debug telemetry synchronization
    • Traces
      • Set up traces
      • Advanced configuration for traces
    • Health
      • Health synchronization
      • Send health data over HTTP
        • Send health data
        • Repeat Snapshots JSON
        • Repeat States JSON
        • Transactional Increments JSON
      • Debug health synchronization
    • Anomaly Detection
      • Export anomaly feedback
      • Scale the AAD up and down
      • The AAD status UI
    • Security
      • Authentication
        • Authentication options
        • File based
        • LDAP
        • Open ID Connect (OIDC)
        • KeyCloak
        • Service tokens
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
        • Subjects
      • Secrets management
      • Self-signed certificates
      • Set up a security backend for Linux
      • Set up a security backend for Windows
    • Logging
      • Kubernetes logs
      • Linux logs
      • Enable logging for functions
  • 📖Develop
    • Developer guides
      • Agent checks
        • About Agent checks
        • Agent check API
        • Agent check state
        • How to develop Agent checks
        • Connect an Agent check to StackState
      • Custom functions and scripts
        • StackState functions
        • Check functions
        • Component actions
        • Event handler functions
        • ID extractor functions
        • Mapping functions
        • Monitor functions
        • Propagation functions
        • Template functions
        • View health state configuration functions
      • Custom Synchronization StackPack
        • About the Custom Synchronization StackPack
        • How to customize elements created by the Custom Synchronization StackPack
        • How to configure a custom synchronization
      • Integrate external services
      • Mirroring Telemetry
      • Monitors
        • Create monitors
        • Monitor STJ file format
      • StackPack development
        • How to create a StackPack
        • Packaging
        • How to get a template file
        • How to make a multi-instance StackPack
        • Prepare a multi-instance provisioning script
        • Upload a StackPack file
        • Prepare a shared template
        • Customize a StackPack
        • Prepare instance template files
        • Prepare a StackPack provisioning script
        • Resources in a StackPack
        • StackState Common Layer
      • Synchronizations and templated files
    • Reference
      • StackState OpenAPI docs
      • StackState Template JSON (STJ)
        • Using STJ
        • Template functions
      • StackState Markup Language (STML)
        • Using STML
        • STML Tags
      • StackState Query Language (STQL)
      • StackState Scripting Language (STSL)
        • Scripting in StackState
        • Script result: Async
        • Script result: Streaming
        • Time in scripts
        • Script APIs
          • Async - script API
          • Component - script API
          • HTTP - script API
          • Prediction - script API
          • StackPack - script API
          • Telemetry - script API
          • Time - script API
          • Topology - script API
          • UI - script API
          • View - script API
    • Tutorials
      • Create a simple StackPack
      • Push data to StackState from an external system
      • Send events to StackState from an external system
      • Set up a mirror to pull telemetry data from an external system
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • Distributed traces
  • Spans
  • Parent span
  • Child span
  • The link between parent and child spans
  • Why create parent and children spans?
  • Span contents
  • OpenTelemetry instrumentation
  • In distributed traces
  • Support in StackState
  1. StackPacks
  2. Integrations
  3. OpenTelemetry

About instrumentations

StackState Self-hosted v5.1.x

PreviousOpenTelemetryNextAWS NodeJS Instrumentation

Last updated 2 years ago

Overview

To understand what OpenTelemetry instrumentation is, we first need to understand the concept of distributed tracing, how a trace looks and what the content of a trace contains. As a side note, If you are unfamiliar with OpenTelemetry you can visit their docs page to get more information about OpenTelemetry.

Distributed traces

Distributed tracing is a way of tracking code or application requests as it flows through the system as a whole. Things like latency, errors, success statuses, and more can be tracked and monitored. All the timed operations and data mentioned above will be captured and grouped within separate spans. Below is an example of a Trace that contains two spans, one Lambda parent span, and one SNS child span. (Parent and child span definitions will be explained a bit further down inside this documentation)

Spans

A span represents each unit of work that has been done within the lifespan of the trace. A request's whole lifecycle, from creation to fulfillment, is represented by a trace.

For example, in the trace captured below, three time operations occurred.

The first was the Lambda that we monitored with OpenTelemetry, as seen in the screenshot below; this is the parent span.

The second interaction inside the Lambda script was with an SQS Queue

And the last interaction before the Trace is completed is another interaction with a second SQS Queue.

As you can see from the above, within this trace context, it had three span timed operations; thus, spans is a tree of timed operations that make up a trace.

A span can be one of two types, a parent span or a child span. It can not be both, and if it's a child span, then there should be a pre-existing parent span that it can be linked to.

A span works with the concept that you open a span and then close it when it reaches a certain point in your code. This will then determine the duration of your span and, eventually, your trace.

Parent span

  • A parent span, also referred to as a root span, contains the end-to-end latency of an entire request.

  • For example, if we execute a Lambda script, The script execution will be created as the parent span. All the other things monitored and executed inside this Lambda will be captured as children span.

  • The parent span should be the last span you close. This is usually either at the end of your script execution or after you close your last child span

  • A parent span doesn't require a child span; it can be standalone if it isn't a part of anything.

Child span

  • A child span is started by a parent span and might involve calling a function, the database, another service, etc. In the example mentioned above, a child span may be a function that determines if the item is accessible. Child spans offer insight into every element of a request.

  • If we take the same example as above where the parent span is a Lambda script, then the children spans will be the SQS, SNS, etc. requests that happen inside the Lambda script.

If we combine the trace and span concept, you have a complete and working trace.

The above will look as follows:

- Trace
  - Parent Span
    - Child Span
  - Parent Span
    - Child Span
    - Child Span
  - Parent Span
  - Parent Span

The link between parent and child spans

A parent span will contain an ID, for example, span-id: id-001. When a child span's created, it can contain a key-value pair that states the following, parent-span: id-001 and its own unique span ID, for example, span-id: id-002.

One thing that makes the structure unique is that this can be a chain of infinite children; for example, the next child can have a parent-ID of a previous child, making it nested even deeper.

For example, below, we have a root parent that has a child, that child has a child, and then that child will have a final child:

Span =>
  spanId: 001 >--------|
                       |
Span =>                |
  spanId: 002 >--------|---|
  parentSpanId: 001 <--|   |
                           |
Span =>                    |
  spanId: 003 >--------|   |
  parentSpanId: 002 <--|---|
                       |
Span =>                |
  spanId: 004          |
  parentSpanId: 003 <--|

As you can see from the above, each following span parent ID has the ID from the previous span.

Why create parent and children spans?

We want to create relations between different span timed operations, how they affect each other, and their relational flow.

Span contents

Each span has a context that defines the request it's a part of. Request, error, and duration information from Spans can be utilized to troubleshoot performance and availability problems.

To give your actions additional context, you may also include span characteristics. Key-value pairs called span attributes can be used to give more context to a span about the particular action it records.

OpenTelemetry instrumentation

When OpenTelemetry captures spans, it will fall under an instrumentation. This will be the library identifier that was used to capture the span information, for example instrumentation-http for HTTP status codes and the instrumentation-aws-sdk for capturing AWS calls, will generate a span in two different instrumentations.

In distributed traces

As we mentioned above, a trace contains multiple spans for example:

Trace [
    Span #1 - AWS-SDK Lambda Script Execution
    Span #2 - AWS-SDK S3
    Span #3 - HTTP S3 Request   <Parent Id: Span #2>
    Span #4 - HTTP S3 Response  <Parent Id: Span #2>
]

As you can see from the above, HTTP and AWS-SDK calls are mixed into one list. With using OpenTelemetry it introduces the instrumentation layer, allowing users to use public OpenTelemetry libraries to capture data and or create your instrumentations using the OpenTelemetry API.

Below is an example of how the trace grouping looks after OpenTelemetry captures data:

Trace [
    AWS-SDK Instrumentation [
        Span #1 AWS-SDK Lambda Script Execution
        Span #2 AWS-SDK S3
    ]
    HTTP Instrumentation [
        Span #3 HTTP S3 Request   <Parent Id: Span #2>
        Span #4 HTTP S3 Response  <Parent Id: Span #2>
    ]
]

As you can see, the different types of spans have been grouped under what the library is called that was responsible for extracting the data.

This allows us to capture unique data for a specific type of instrumentation and look for it specifically under that instrumentation name.

Support in StackState

StackState currently supports two types of instrumentations:

  • An out-of-the-box solution specifically for AWS using a Lambda layer for all your NodeJS functions.

    • This solution doesn't require you to write any code, but only supports certain services

The above is a snippet of the within StackState.

You can head over to the page to get a full breakdown of how StackState relations work, how parent-child spans are shown on StackState, and how relations are created between components and how the health state uses these relations to propagate health.

Visit the for more information regarding the and how to install and use this Lambda layer.

Manual instrumentation using the

This gives you the ability to create and display a custom component with a health state within StackState using the .

To learn more about how to implement a manual instrumentation specifically for StackState, head over to the page

🧩
💠
Traces Perspective
manual instrumentation component relations
OpenTelemetry API
OpenTelemetry API
manual instrumentation tracer and span mappings
What is OpenTelemetry? (opentelemetry.io)
AWS OpenTelemetry integrations page
Distributed Trace Sample
Distributed Trace Parent Span
Distributed Trace Child Span
Distributed Trace Second Child Span
supported AWS services