AAD Standalone Deployment

Search…
Standalone Deployment of the Autonomous Anomaly Detector
Overview
Autonomous Anomaly Detector (AAD) is a StackState service configured and deployed as a part of standard installation. In some cases the AAD can be deployed standalone using the AAD helm chart, e.g. when StackState and the AAD are deployed in separate kubernetes clusters. The standalone AAD deployment option is recommended only for the users with advanced knowledge of Kubernetes.
The Autonomous Anomaly Detector consists of two components:
AAD Kubernetes service
AAD StackPack.
The sections below explain how to configure AAD Kubernetes service and AAD StackPack in order to perform standalone deployment. Note that a training period is required before AAD can begin to report anomalies.
Node sizing
A minimal deployment of the AAD Kubernetes service with the default options requires one of the following instance types:
Amazon EKS: 1 instance of type m4.xlarge
Azure AKS: 1 instance of type F4s v2 (Intel or AMD CPUs)
Self-hosted Kubernetes: 1 instance with 4 CPUs and 6 Gb memory
To handle more streams or to reduce detection latency, the service can be scaled. If you want to find out how to scale the service, contact StackState support.
The AAD Kubernetes service is stateless and survives restarts. It can be relocated to a different Kubernetes node or bounced. To take full advantage of this capability, it is recommended to run the service on low cost AWS Spot Instances or Azure low-priority VMs.
Installation
Standalone deployment consists of two steps - installing AAD StackPack and install AAD Kubernetes service.
Install the AAD StackPack
Install the AAD StackPack from the StackPacks page in StackState.
Install the AAD Kubernetes service
After installing the AAD StackPack, install the AAD Kubernetes service.
1. Get access to quay.io
To be able to pull the Docker image, you will need access to quay.io. Access credentials can be requested from StackState support.
2. Install Helm
1.Install Helm (version 3). See the Helm docs https://helm.sh/docs/intro/install​
2.Add the StackState Helm repo:
1
 helm repo add stackstate https://helm.stackstate.io`
Copied!
3. Get the latest AAD Kubernetes service Helm Chart
1
    helm fetch stackstate/anomaly-detection
Copied!
4. Configure AAD Kubernetes service
Create the file values.yaml file, including the configuration described below, and save it to disk:
image:
pullSecretUsername - the image registry username (from step 1).
stackstate:
instance - the StackState instance URL. This must be a StackState internal URL to keep traffic inside the Kubernetes network and namespace. e.g http://stackstate-router:8080/ or http://<releasename>-stackstate-router:8080/
ingress: - Ingress provides access to the technical interface of the AAD, this is useful for troubleshooting. The technical interface can be accessed using kube proxy command: kubectl proxy. After proxy is running the technical interface can be accessed using the path below.
1
  http://localhost:8001/api/v1/namespaces/<namespace>/services/http:<release-name>-anomaly-detection:8090/proxy/
Copied!
Optionally, the technical interface can be exposed using ingress configuration. The example below shows how to configure an nginx-ingress controller. Setting up the controller itself is beyond the scope of this document. More information about how to set up Ingress can be found at:
​AKS​
​EKS Official docs (not using nginx)
​EKS blog post (using nginx)
values.yaml
1
image:
2
    pullSecretUsername: <image registry username>
3
​
4
stackstate:
5
    # Stackstate instance URL
6
    instance: <stackstate instance url>
7
​
8
# status UI ingress (the configuration below is example for nginx ingress controller)
9
ingress:
10
    enabled: true
11
    hostname: <domain name> # e.g. spotlight.domain.com
12
    port: 8090              # status page will be available on spotlight.domain.com:8090
13
    annotations:        
14
        external-dns.alpha.kubernetes.io/hostname: <domain name> # e.g. spotlight.domain.com
15
        kubernetes.io/ingress.class: nginx
16
        nginx.ingress.kubernetes.io/ingress.class: nginx
17
    hosts:
18
        - host: <domain name>  # e.g. spotlight.domain.com
Copied!
Details of all configuration options are available in the anomaly-detection chart with the command below.
1
    helm show all stackstate/anomaly-detection
Copied!
5. Authentication with StackState
By default, AAD Kubernetes Service is configured to use kubernetes token authentication, so one does not need to configure anything additional to that AAD Kubernetes service must be installed into the same cluster and namespace as StackState. If this is is not possible there are two other options for authentication:
Stackstate Api Token authentication. One can obtain token from User Profile page.
1
    ...
2
stackstate:
3
  authType: api-token
4
  apiToken: <stackstate api token>
5
    ...
Copied!
Cookie authentication. This type of auth is not recommended and exists only for troubleshooting/testing purposes.
1
    ...
2
stackstate:
3
  authType: cookie
4
  username: <username>
5
  password: <password>
6
    ...
Copied!
6. Install the AAD Kubernetes service
Run the command below, specifying the StackState namespace and the image registry password. Note that the AAD Kubernetes service must be installed in the same namespace as StackState to be able to use default token authentication (Otherwise consider other types of authentication above).
1
    helm upgrade anomaly-detector stackstate/anomaly-detection \
2
        --install \
3
        --namespace <stackstate-namespace> \
4
        --set image.pullSecretPassword=<image registry password>
5
        --values ./values.yaml
Copied!
Training period
The AAD will need to train on your data before it can begin reporting anomalies. With data collected in 1 minute buckets, AAD requires a 3 day training period. If historic data exists for relevant metric streams, this will also be used for training the AAD. In this case, the first results can be expected within an hour.
Upgrade the standalone AAD instance
Upgrading the standalone AAD instance consists of two steps - upgrading AAD Stackpack and upgrading AAD Kubernetes Service.
Upgrade the AAD StackPack
When new version of StackPack is available you can simply click on Upgrade on the AAD StackPack page.
Upgrade the AAD Kubernetes service
The AAD Kubernetes service upgrade is driven by availability of the new version of the helm chart therefore for upgrading one can follow the steps starting from step 3 - fetching new AAD chart.
Deactivate the AAD instance
To deactivate the AAD, uninstall the AAD StackPack. The AAD Kubernetes service will continue running and reserve its compute resources, but anomaly detection will not be executed.
To re-enable the AAD Kubernetes service, you can simply install the AAD StackPack again. It is not necessary to repeat the installation of the AAD Kubernetes service.
Full uninstall
To completely remove the AAD Kubernetes service and AAD StackPack:
Uninstall the AAD Kubernetes service:
1
helm delete anomaly-detector
Copied!
Uninstall the AAD StackPack
Troubleshooting
The status UI provides details on the technical state of the AAD. You can use it to retrieve information about scheduling progress, possible errors, the ML models selected and job statistics.
To access the status UI, one can run kubectl proxy.
1
    kubectl proxy
Copied!
The UI will be accessible by URL:
1
    http://localhost:8001/api/v1/namespaces/<namespace>/services/http:<release-name>-anomaly-detection:8090/proxy/
Copied!
Optionally to access the status UI, the AAD Kubernetes service ingress can be configured for the anomaly-detection deployment (for the details see AAD Standalone Deployment).
Common questions that can be answered in the status UI:
Is the AAD Kubernetes service running?
If the status UI is accessible: The service is running.
If the status UI is not available: Either the service is not running, or the Ingress has not been configured (See the install section).
Can the AAD Kubernetes service reach StackState?
Check the status UI sections Top errors and Last stream polling results. Errors here usually indicate connection problems.
Has the AAD Kubernetes service selected metric streams for anomaly detection?
The status UI section Anomaly Detection Summary shows the total time of all registered streams, if no streams are selected it will be zero.
Is the AAD Kubernetes service detecting anomalies?
The status UI section Top Anomalous Streams shows the streams with the highest number of anomalies. No streams in this section means that no anomalies have been detected. The status UI section Anomaly Detection Summary shows other relevant metrics, such as total time of all registered streams, total checked time and total time of all anomalies detected.
Is the AAD Kubernetes service scheduling streams?
The status UI tab Job Progress shows a ranked list of streams with scheduling progress, including the last time each stream was scheduled.
See also
​Autonomous Anomaly Detector StackPack​
Migrate from Linux install
Linux install
Last modified 11d ago
Copy link
Contents
Overview
Node sizing
Installation
Install the AAD StackPack
Install the AAD Kubernetes service
Training period
Upgrade the standalone AAD instance
Upgrade the AAD StackPack
Upgrade the AAD Kubernetes service
Deactivate the AAD instance
Full uninstall
Troubleshooting