AWS

Overview

Amazon Web Services (AWS) is a major cloud provider. This StackPack enables in-depth monitoring of AWS services.

• StackState Agent V2 collects all service responses from the target AWS account.

• Topology is updated in real time:

  • Once an hour, all services are queried to gain a full point-in-time snapshot of resources.

  • Once a minute, Cloudtrail and Eventbridge events are read to find changes to resources.

• Logs are retrieved once a minute from Cloudwatch and a central S3 bucket. These are mapped to associated components in StackState.

• Metrics are retrieved on-demand by the StackState CloudWatch plugin. These are mapped to associated components in StackState.

Setup

Prerequisites

To set up the StackState AWS integration, you need to have:

• StackState Agent V2 installed on a machine which can connect to both AWS and StackState.

• AWS CLI version 2.0.4 or later is installed on the environment where StackState is running.

• An AWS account for the StackState Agent to use when deploying resources to the target AWS accounts. It is recommended to use a separate shared account for this and not use any of the accounts that will be monitored by StackState, but this is not required.

  • If StackState Agent is running within an AWS environment: The EC2 instance must have an IAM role attached to it.

  • If StackState Agent is running outside an AWS account: An IAM user must be made available.

• The IAM user/role must have the following IAM policy. This policy grants the IAM principal permission to assume the role created in each target AWS account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/StackStateAwsIntegrationRole"
    }
  ]
}

Deploy the AWS CloudFormation stack

The StackState AWS Cloudformation stack is deployed in your AWS account. It provides the minimum level of access required for the StackState Agent to collect topology, telemetry and logs. Quick deployment links and a default StackState CloudFormation template are provided below.

• Quick deployment - Deploy all resources to a region in an account using a link.

• StackState CloudFormation template - Download the StackState CloudFormation template to integrate into your own deployment workflow.

Quick deployment

The necessary resources can be deployed for one account in a single region using an automated CloudFormation template.

The table below includes links to deploy the template in popular AWS regions. For any regions not listed, follow the steps described for the StackState template deployment.

StackState template deployment

The default StackState CloudFormation template can be used to deploy all necessary resources. It can be deployed to multiple AWS accounts and regions at once by deploying it in a CloudFormation StackSet. It is recommended to use this template as it provides an easy upgrade path for future versions and reduces the maintenance burden compared to creating a custom template.

• Download the default StackState CloudFormation template (stackstate-integrations-resources-eu-west-1.s3.eu-west-1.amazonaws.com)

The template requires the following parameters:

• MainRegion - The primary AWS region. This can be any region, as long as this region is the same for every template deployed within the AWS account. Global resources will be deployed in this region such as the IAM role and S3 bucket. Example: us-east-1.

• StsAccountId - The 12-digit AWS account ID that the StackState Agent is deployed in, or has an IAM user for the Agent in. This will be the AWS account that the IAM role can be assumed from, to perform actions on the target AWS account. Example: 0123456789012.

• ExternalId - A shared secret that the StackState Agent will present when assuming a role. Use the same value across all AWS accounts that the Agent is monitoring. Example: uniquesecret!1.

For more information on how to use StackSets, check the AWS documentation on working with AWS CloudFormation StackSets (docs.aws.amazon.com).

Install the AWS StackPack

Install the AWS StackPack from the StackState UI StackPacks > Integrations screen. You will need to provide the following parameters, these will be used by StackState to query live telemetry from the AWS account. To create topology in StackState, you must configure the AWS check on StackState Agent V2.

• Role ARN - the ARN of the IAM Role used to deploy the AWS Cloudformation stack. For example, arn:aws:iam::<account id>:role/StackStateAwsIntegrationRole where <account id> is the 12-digit AWS account ID.

• External ID - a shared secret that StackState will present when assuming a role. Use the same value across all AWS accounts. For example, uniquesecret!1

• AWS Access Key ID - The Access Key ID of the IAM user used by the StackState Agent. If the StackState instance is running within AWS, enter the value use-role and the instance will authenticate using the attached IAM role.

• AWS Secret Access Key - The Secret Access Key of the IAM user used by the StackState Agent. If the StackState instance is running within AWS, enter the value use-role and the instance will authenticate using the attached IAM role.

Configure the AWS check

To enable the AWS check and begin collecting data from AWS, add the following configuration to StackState Agent V2:

1. Edit the Agent integration configuration file /etc/stackstate-agent/conf.d/aws_topology.d/conf.yaml to include details of your AWS instances:

   Copy

   # values in init_config are used globally; these credentials will be used for all AWS accounts
   init_config:
     aws_access_key_id: '' # The AWS Access Key ID. Leave empty quotes if the Agent is running on an EC2 instance or ECS/EKS cluster with an IAM role
     aws_secret_access_key: '' # The AWS Secret Access Key. Leave empty quotes if the Agent is running on an EC2 instance or ECS/EKS cluster with an IAM role
     external_id: uniquesecret!1 # Set the same external ID when creating the CloudFormation stack in every account and region
     # full_run_interval: 3600 # Time in seconds between a full AWS topology scan. Intermediate runs only fetch events. Is not required.
   
   instances:
     - role_arn: arn:aws:iam::123456789012:role/StackStateAwsIntegrationRole # Substitute 123456789012 with the target AWS account ID to read
       regions: # The Agent will only attempt to find resources in regions specified below
         - global # global is a special "region" for global resources such as Route53
         - eu-west-1
       min_collection_interval: 60 # The amount of time in seconds between each scan. Decreasing this value will not appreciably increase topology update speed.
       # apis_to_run: # Optionally whitelist specific AWS services. It is not recommended to set this; instead rely on IAM permissions.
       #   - ec2
       # log_bucket_name: '' # The S3 bucket that the agent should read events from. This value should only be set in custom implementations.
       # tags:
       #   - foo:bar

2. Restart the StackState Agent to apply the configuration changes.

3. Once the Agent has restarted, wait for data to be collected from AWS and sent to StackState.

Use an HTTP proxy

StackState Agent V2 must have access to the internet to call AWS APIs. If the Agent cannot be given direct internet access, an HTTP proxy can be used to proxy the API calls. The AWS documentation (docs.aws.amazon.com) outlines the variables that can be set to do this. If a proxy is required, these can be set as environment variables for the Agent.

Status

To check the status of the AWS integration, run the status subcommand and look for aws_topology under Running Checks:

sudo stackstate-agent status

Integration details

Data retrieved

Events

The AWS StackPack supports the following event:

• EC2 Instance Run State: when the instance is started, stopped, or terminated. This will appear as the Run State in the EC2 instance component.

AWS events are primarily used to provide real-time updates to topology. These events are not displayed as StackState events.

Metrics

Metrics data is pulled at a configured interval directly from AWS by the StackState CloudWatch plugin. Retrieved metrics are mapped onto the associated topology component.

Topology

The following AWS service data is available in StackState as components:

• * "All Supported Resources" - relations will be made to any other resource on this list, should the resource type support it.

• ** This relation is made by finding valid URIs in the environment variables of the resource. For example, the DNS hostname of an RDS instance will create a relation.

Traces

The AWS integration does not retrieve any Traces data.

Required AWS resources

A high-level of overview of all resources necessary to run the StackState Agent with full capabilities is provided in the graph below. Users with intermediate to high level AWS skills can use these details to set up the StackState Agent resources manually. For the majority of installations, this is not the recommended approach. Use the provided StackState CloudFormation template unless there are environment-specific issues that must be worked around.

Hourly and event-based updates collect data:

• Hourly full topology updates - collected by the StackState Agent using an IAM role with access to the AWS services.

• Event-based updates for single components and relations - captured using AWS services and placed into an S3 bucket for the StackState Agent to read.

If the StackState Agent does not have permission to access a certain component, it will skip it.

StackState Agent IAM Role

The bare minimum necessary to run the StackState Agent is an IAM role with necessary permissions. The Agent will always attempt to fetch as much data as possible for the supported resources. If a permission is omitted, the Agent will attempt to create a component with the data it has.

For example, if the permission s3:GetBucketTagging is omitted, the Agent will fetch all S3 buckets and their associated configuration, but the tags section will be empty.

• IAM policy with all required permissions - JSON object

S3 Bucket

The S3 bucket is used to store all incoming events from EventBridge and other event-based sources. The Agent then reads objects from this bucket. These events are used to provide features such as real-time topology updates, and creating relations between components based on event data such as VPC FlowLogs. If the S3 bucket is not available to the Agent it will fallback to reading CloudTrail directly, which introduces a 15 minute delay in real-time updates. EventBridge events and VPC FlowLogs are only available via the S3 bucket.

EventBridge Rule

A catch-all rule for listening to all events for services supported by the AWS StackPack. All matched rules are sent to a Kinesis Firehose delivery stream.

• EventBridge Rule - JSON object

• EventBridge IAM Role - JSON - Give permission for EventBridge to send data to Kinesis Firehose

Kinesis Firehose

Kinesis Firehose is used to receive and batch events from EventBridge. This delivery stream batches events per 60 seconds and pushes an object to S3. 60 seconds is the recommended value - setting this value any higher will negligibly decrease storage costs while increasing the delay in topology updates.

The Prefix must be set to AWSLogs/${AccountId}/EventBridge/${Region}/, where ${AccountId} and ${Region} are the account ID and region, for example, eu-west-1. Files must be compressed using the GZIP option.

• Kinesis Firehose IAM Role - JSON - Gives permission for Firehose to send data to an S3 bucket.

KMS Key (Optional)

A KMS Customer Managed Key (CMK) can be used to secure data at rest in S3. The KMS key is used in the Firehose Delivery Stream. The S3 bucket also uses the KMS key as its default key.

Use of a KMS is key is not necessary for the operation of the StackPack, however as encryption at rest is a requirement in most environments, the CloudFormation template includes this by default.

• Sample KMS Key policy.

Costs

The AWS StackPack CloudFormation template contains all resources that are necessary to run the AWS check on the StackState Agent. The installed resources are kept as minimal as possible. All costs incurred are minimal but variable, with costs scaling depending on how many events are emitted in a given account. In practice, the costs created by the AWS integration will be negligible.

• Kinesis Firehose: priced by the amount of data processed. Events use very small amounts of data. Firehose pricing (aws.amazon.com)

• S3: priced by amount of data stored, and amount of data transferred. Running the Agent inside of AWS will reduce data transfer costs. S3 pricing (aws.amazon.com)

• KMS: a flat fee of $1 per month per key, with additional costs per request. KMS pricing (aws.amazon.com)

• CloudWatch metrics: priced per metric retrived. Metrics are only retrieved when viewed or when a check is configured on a CloudWatch metric. CloudWatch pricing (aws.amazon.com)

AWS views in StackState

When the AWS integration is enabled, three views will be created in StackState for each instance of the StackPack.

• AWS - [instance_name] - All - includes all resources retrieved from AWS by the StackPack instance.

• AWS - [instance_name] - Infrastructure - includes only Networking, Storage and Machines resources retrieved from AWS by the StackPack instance.

• AWS - [instance_name] - Serverless - includes only S3 buckets, lambdas and application load balancers retrieved from AWS by the StackPack instance.

AWS actions in StackState

Components retrieved from AWS will have an additional action available in the component context menu and component details pane on the right-hand side of the screen. This provides a deep link through to the relevant AWS console at the correct point.

For example, in the StackState Topology Perspective:

• Components of type aws-subnet have the action Go to Subnet console, which links directly to this component in the AWS Subnet console.

• Components of type ec2-instance have the action Go to EC2 console, which links directly to this component in the EC2 console.

Tags and labels

The AWS StackPack converts tags in AWS to labels in StackState. In addition, the following special tags are supported:

You can distinguish topology from the new and legacy AWS integrations by the labels attached:

Troubleshooting

Troubleshooting steps can be found in the StackState support Knowledge base guide to troubleshoot the StackState AWS StackPack.

Uninstall

To uninstall the StackState AWS StackPack, click the Uninstall button from the StackState UI StackPacks > Integrations > AWS screen. This will remove all AWS specific configuration in StackState.

Once the AWS StackPack has been uninstalled, you will need to delete the StackState AWS Cloudformation stack from the AWS account being monitored. This can be done using the web console or the command line.

Web console

To delete the StackState AWS Cloudformation stack from an AWS account using the web console: If the template is in the main region, the S3 bucket used by StackState must be emptied as CloudFormation can't delete an empty bucket. Follow these steps:

1. Disable the EventBridge rule. Go to EventBridge, and find the rule name starting with stackstate-resources-StsEventBridgeRule. Open this rule, and press the "Disable" button.

2. Delete all flowlogs that send to this bucket. Go to the VPC service, and select each VPC in the VPCs list. Look in the FlowLogs tab in the details section, and delete any flowlogs that are sent to the S3 bucket starting with stackstate-logs.

3. Go to the S3 service. Select (don't open) the bucket named stackstate-logs-${AccountId} where ${AccountId} is the 12-digit identifier of your AWS account.

4. Select "Empty", and follow the steps to delete all objects in the bucket.

5. Go to the CloudFormation service. Select the CloudFormation template. This will be named stackstate-resources if created via the quick deploy method, otherwise the name was user-defined.

6. In the top right of the console, select "Delete".

Command line

These steps assume you already have the AWS CLI installed and configured with access to the target account. If not, follow the AWS documentation.

1. If --region is the main region, follow these steps to delete the S3 bucket. Before emptying the bucket, disable any event sources that are sending files to the bucket. This is a versioned S3 bucket, so each object version must be deleted individually. If there are more than 1000 items in the bucket this command will fail; it's likely more convenient to perform this in the web console.

   Copy

   BUCKET=$(aws cloudformation describe-stack-resource --stack-name stackstate-resources --logical-resource-id StsLogsBucket --query "StackResourceDetail.PhysicalResourceId" --output=text)
   
   # Disable the eventbridge rule sending to Firehose
   aws events disable-rule --name $(aws cloudformation describe-stack-resource --stack-name stackstate-resources-debug --logical-resource-id StsEventBridgeRule --query "StackResourceDetail.PhysicalResourceId" --output=text)
   # Find all flowlogs sending to the bucket and delete them
   aws ec2 delete-flow-logs --flow-log-ids $(aws ec2 describe-flow-logs --query "FlowLogs[?LogDestination==$BUCKET].[FlowLogId]" --output=text | tr '\n' ' ')
   
   sleep 60 # To make sure all objects have finished writing to bucket
   aws s3api delete-objects --bucket $BUCKET \
       --delete "$(aws s3api list-object-versions \
       --bucket $BUCKET --query Account --output text) \
       --output json \
       --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

2. Delete the CloudFormation template: aws cloudformation delete-stack --stack-name stackstate-resources --region <region>.

Find out how to uninstall using a specific AWS profile or an IAM role (docs.aws.amazon.com).

Release notes

1.0.0 (2021-07-09)

AWS StackPack v1.0.1 (2021-07-23)

• Bugfix: Use proper domain

AWS StackPack v1.0.0 (2021-07-16)

• Improvement: Full rewrite of the AWS Stackpack to use the StackState Agent V2

• Improvement: Improved AWS multi-account support by using IAM roles for account access

• Improvement: Improved AWS multi-region support - each instance can create topology for multiple regions at once

• Improvement: New, refreshed icon set, using the latest AWS branding

See also

• AWS policies

• StackState AWS (Legacy) integration

• Working with AWS CloudFormation StackSets (docs.aws.amazon.com)

• Uninstall using a specific AWS profile or an IAM role (docs.aws.amazon.com)