LogoLogo
StackState.comDownloadSupportExplore playground
StackState v5.1
StackState v5.1
  • Welcome to the StackState docs!
  • StackState self-hosted v5.1 docs
  • Getting Started
  • 🚀Setup
    • Install StackState
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Required Permissions
        • Non-high availability setup
        • Override default configuration
        • Configure storage
        • Configure Ingress
        • Install from custom image registry
        • Migrate from Linux install
      • Linux
        • Before you install
        • Download
        • Install StackState
        • Install with production configuration
        • Install with development configuration
        • Install with POC configuration
        • Set up a reverse proxy
        • Set up TLS without reverse proxy
      • Initial run guide
      • Troubleshooting
    • Upgrade StackState
      • Steps to upgrade
      • Version specific upgrade instructions
      • StackPack versions
      • StackState release notes
    • StackState Agent
      • About StackState Agent V3
      • Docker
      • Kubernetes / OpenShift
      • Linux
      • Windows
      • Advanced Agent configuration
      • Use an HTTP/HTTPS proxy
      • Agent V1 (legacy)
      • Migrate Agent V1 to Agent V2
        • Linux
        • Docker
    • StackState CLI
      • CLI: sts
      • CLI: stac (deprecated)
      • Comparison between CLIs
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Linux backup
        • Configuration backup
      • Data retention
      • Clear stored data
  • 👤Use
    • Concepts
      • The 4T data model
      • Components
      • Relations
      • Health state
      • Layers, Domains and Environments
      • Perspectives
      • Anomaly detection
      • StackState architecture
    • StackState UI
      • Explore mode
      • Filters
      • Views
        • About views
        • Configure the view health
        • Create and edit views
        • Visualization settings
      • Perspectives
        • Topology Perspective
        • Events Perspective
        • Traces Perspective
        • Metrics Perspective
      • Timeline and time travel
      • Analytics
      • Keyboard shortcuts
    • Checks and monitors
      • Checks
      • Add a health check
      • Anomaly health checks
      • Monitors
      • Manage monitors
    • Problem analysis
      • About problems
      • Problem lifecycle
      • Investigate a problem
      • Problem notifications
    • Metrics
      • Telemetry streams
      • Golden signals
      • Top metrics
      • Add a telemetry stream
      • Browse telemetry
      • Set telemetry stream priority
    • Events
      • About events
      • Event notifications
      • Manage event handlers
    • Glossary
  • 🧩StackPacks
    • About StackPacks
    • Add-ons
      • Autonomous Anomaly Detector
      • Health Forecast
    • Integrations
      • About integrations
      • 💠StackState Agent V2
      • 💠AWS
        • AWS
        • AWS ECS
        • AWS X-ray
        • StackState/Agent IAM role: EC2
        • StackState/Agent IAM role: EKS
        • Policies for AWS
        • AWS (legacy)
        • Migrate AWS (legacy) to AWS
      • 💠Dynatrace
      • 💠Kubernetes
      • 💠OpenShift
      • 💠OpenTelemetry
        • About instrumentations
        • AWS NodeJS Instrumentation
        • Manual Instrumentation
          • Prerequisites
          • Tracer and span mappings
          • Relations between components
          • Span health state
          • Merging components
          • Code examples
      • 💠ServiceNow
      • 💠Slack
      • 💠Splunk
        • Splunk
        • Splunk Events
        • Splunk Health
        • Splunk Metrics
        • Splunk Topology
      • 💠VMWare vSphere
      • Apache Tomcat
      • Azure
      • Cloudera
      • Custom Synchronization
      • DotNet APM
      • Elasticsearch
      • Humio
      • Java APM
      • JMX
      • Logz.io
      • MySQL
      • Nagios
      • OpenMetrics
      • PostgreSQL
      • Prometheus
      • SAP
      • SCOM
      • SolarWinds
      • Static Health
      • Static Topology
      • Traefik
      • WMI
      • Zabbix
    • Develop your own StackPacks
  • 🔧Configure
    • Topology
      • Component actions
      • Identifiers
      • Topology naming guide
      • Topology sources
      • Create a topology manually
      • Configure topology synchronizations
      • Enable email event notifications
      • Send topology data over HTTP
      • Set the topology filtering limit
      • Use a proxy for event handlers
      • Use tags
      • Tune topology synchronization
      • Debug topology synchronization
    • Telemetry
      • Add telemetry during topology synchronization
      • Data sources
        • Elasticsearch
        • Prometheus mirror
      • Send events over HTTP
      • Send metrics data over HTTP
      • Set the default telemetry interval
      • Debug telemetry synchronization
    • Traces
      • Set up traces
      • Advanced configuration for traces
    • Health
      • Health synchronization
      • Send health data over HTTP
        • Send health data
        • Repeat Snapshots JSON
        • Repeat States JSON
        • Transactional Increments JSON
      • Debug health synchronization
    • Anomaly Detection
      • Export anomaly feedback
      • Scale the AAD up and down
      • The AAD status UI
    • Security
      • Authentication
        • Authentication options
        • File based
        • LDAP
        • Open ID Connect (OIDC)
        • KeyCloak
        • Service tokens
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
        • Subjects
      • Secrets management
      • Self-signed certificates
      • Set up a security backend for Linux
      • Set up a security backend for Windows
    • Logging
      • Kubernetes logs
      • Linux logs
      • Enable logging for functions
  • 📖Develop
    • Developer guides
      • Agent checks
        • About Agent checks
        • Agent check API
        • Agent check state
        • How to develop Agent checks
        • Connect an Agent check to StackState
      • Custom functions and scripts
        • StackState functions
        • Check functions
        • Component actions
        • Event handler functions
        • ID extractor functions
        • Mapping functions
        • Monitor functions
        • Propagation functions
        • Template functions
        • View health state configuration functions
      • Custom Synchronization StackPack
        • About the Custom Synchronization StackPack
        • How to customize elements created by the Custom Synchronization StackPack
        • How to configure a custom synchronization
      • Integrate external services
      • Mirroring Telemetry
      • Monitors
        • Create monitors
        • Monitor STJ file format
      • StackPack development
        • How to create a StackPack
        • Packaging
        • How to get a template file
        • How to make a multi-instance StackPack
        • Prepare a multi-instance provisioning script
        • Upload a StackPack file
        • Prepare a shared template
        • Customize a StackPack
        • Prepare instance template files
        • Prepare a StackPack provisioning script
        • Resources in a StackPack
        • StackState Common Layer
      • Synchronizations and templated files
    • Reference
      • StackState OpenAPI docs
      • StackState Template JSON (STJ)
        • Using STJ
        • Template functions
      • StackState Markup Language (STML)
        • Using STML
        • STML Tags
      • StackState Query Language (STQL)
      • StackState Scripting Language (STSL)
        • Scripting in StackState
        • Script result: Async
        • Script result: Streaming
        • Time in scripts
        • Script APIs
          • Async - script API
          • Component - script API
          • HTTP - script API
          • Prediction - script API
          • StackPack - script API
          • Telemetry - script API
          • Time - script API
          • Topology - script API
          • UI - script API
          • View - script API
    • Tutorials
      • Create a simple StackPack
      • Push data to StackState from an external system
      • Send events to StackState from an external system
      • Set up a mirror to pull telemetry data from an external system
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • Backup scope
  • Storage options
  • Enable backups
  • AWS S3
  • Azure Blob Storage
  • Kubernetes storage
  • Configuration and topology data (StackGraph)
  • Disable scheduled backups
  • Disable restores
  • Backup schedule
  • Backup retention
  • Telemetry data (Elasticsearch)
  • Disable scheduled snapshots
  • Disable restores
  • Snapshot schedule
  • Snapshot retention
  • Snapshot indices
  • Restore backups and snapshots
  • List StackGraph backups
  • Restore a StackGraph backup
  • List Elasticsearch snapshots
  • Delete Elasticsearch indices
  • Restore an Elasticsearch snapshot
  1. Setup
  2. Data management
  3. Backup and Restore

Kubernetes backup

StackState Self-hosted v5.1.x

PreviousBackup and RestoreNextLinux backup

Last updated 2 years ago

Overview

The Kubernetes setup for StackState has a built-in backup and restore mechanism that can be configured to store backups to the local clusters, to AWS S3 or to Azure Blob Storage.

Backup scope

The following data can be automatically backed up:

  • Configuration and topology data stored in StackGraph is backed up when the Helm value backup.stackGraph.enabled is set to true.

  • Telemetry data stored in StackState's Elasticsearch instance is backed up when the Helm value backup.elasticsearch.enabled is set to true.

The following data will NOT be backed up:

  • In transit topology and telemetry updates stored in Kafka - these only have temporary value and would be of no use when a backup is restored

  • Master node negotiations state stored in ZooKeeper - this runtime state would be incorrect when restored and will be automatically determined at runtime

  • Kubernetes configuration state and raw persistent volume state - this state can be rebuilt by re-installing StackState and restoring the backups.

  • Kubernetes logs - these are ephemeral.

Storage options

StackGraph and Elasticsearch backups are sent to an instance of , which is automatically started by the stackstate Helm chart when automatic backups are enabled. MinIO is an object storage system with the same API as AWS S3. It can store its data locally or act as a gateway to , and other systems.

The built-in MinIO instance can be configured to store the backups in three locations:

Enable backups

AWS S3

Encryption

Amazon S3-managed keys (SSE-S3) should be used when encrypting S3 buckets that store the backups.

⚠️ Encryption with AWS KMS keys stored in AWS Key Management Service (SSE-KMS) isn't supported. This will result in errors such as this one in the Elasticsearch logs:

Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Unable to verify integrity of data upload. Client calculated content hash (contentMD5: ZX4D/ZDUzZWRhNDUyZTI1MTc= in base 64) didn't match hash (etag: c75faa31280154027542f6530c9e543e in hex) calculated by Amazon S3. You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@5481a656, bucketName: stackstate-elasticsearch-backup, key: tests-UG34QIV9s32tTzQWdPsZL/master.dat)",

To enable scheduled backups to AWS S3 buckets, add the following YAML fragment to the Helm values.yaml file used to install StackState:

backup:
  enabled: true
  stackGraph:
    bucketName: AWS_STACKGRAPH_BUCKET
  elasticsearch:
    bucketName: AWS_ELASTICSEARCH_BUCKET
minio:
  accessKey: YOUR_ACCESS_KEY
  secretKey: YOUR_SECRET_KEY
  s3gateway:
    enabled: true
    accessKey: AWS_ACCESS_KEY
    secretKey: AWS_SECRET_KEY

Replace the following values:

  • YOUR_ACCESS_KEY and YOUR_SECRET_KEY are the credentials that will be used to secure the MinIO system. These credentials are set on the MinIO system and used by the automatic backup jobs and the restore jobs. They're also required if you want to manually access the MinIO system.

    • YOUR_ACCESS_KEY should contain 5 to 20 alphanumerical characters.

    • YOUR_SECRET_KEY should contain 8 to 40 alphanumerical characters.

  • AWS_ACCESS_KEY and AWS_SECRET_KEY are the AWS credentials for the IAM user that has access to the S3 buckets where the backups will be stored. See below for the permission policy that needs to be attached to that user.

  • AWS_STACKGRAPH_BUCKET and AWS_ELASTICSEARCH_BUCKET are the names of the S3 buckets where the backups should be stored. Note: The names of AWS S3 buckets are global across the whole of AWS, therefore the S3 buckets with the default name (sts-elasticsearch-backup and sts-stackgraph-backup) will probably not be available.

The IAM user identified by AWS_ACCESS_KEY and AWS_SECRET_KEY must be configured with the following permission policy to access the S3 buckets:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListMinioBackupBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::AWS_STACKGRAPH_BUCKET",
                "arn:aws:s3:::AWS_ELASTICSEARCH_BUCKET"
            ]
        },
        {
            "Sid": "AllowWriteMinioBackupBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::AWS_STACKGRAPH_BUCKET/*",
                "arn:aws:s3:::AWS_ELASTICSEARCH_BUCKET/*"
            ]
        }
    ]
}

Azure Blob Storage

To enable backups to an Azure Blob Storage account, add the following YAML fragment to the Helm values.yaml file used to install StackState:

backup:
  enabled: true
minio:
  accessKey: AZURE_STORAGE_ACCOUNT_NAME
  secretKey: AZURE_STORAGE_ACCOUNT_KEY
  azuregateway:
    enabled: true

Replace the following values:

The StackGraph and Elasticsearch backups are stored in BLOB containers called sts-stackgraph-backup and sts-elasticsearch-backup respectively. These names can be changed by setting the Helm values backup.stackGraph.bucketName and backup.elasticsearch.bucketName respectively.

Kubernetes storage

If MinIO is configured to store its data in Kubernetes storage, a PersistentVolumeClaim (PVC) is used to request storage from the Kubernetes cluster. The kind of storage allocated depends on the configuration of the cluster.

It's advised to use AWS S3 for clusters running on Amazon AWS and Azure Blob Storage for clusters running on Azure for the following reasons:

  1. Kubernetes clusters running in a cloud provider usually map PVCs to block storage, such as Elastic Block Storage for AWS or Azure Block Storage. Block storage is expensive, especially for large data volumes.

  2. Persistent Volumes are destroyed when the cluster that created them is destroyed. That means an (accidental) deletion of your cluster will also destroy all backups stored in Persistent Volumes.

  3. Persistent Volumes can't be accessed from another cluster. That means that it isn't possible to restore StackState from a backup taken on another cluster.

To enable backups to cluster-local storage, enable MinIO by adding the following YAML fragment to the Helm values.yaml file used to install StackState:

backup:
  enabled: true
minio:
  accessKey: YOUR_ACCESS_KEY
  secretKey: YOUR_SECRET_KEY
  persistence:
    enabled: true

Replace the following values:

  • YOUR_ACCESS_KEY and YOUR_SECRET_KEY - the credentials that will be used to secure the MinIO system. The automatic backup jobs and the restore jobs will use them. They're also required to manually access the MinIO storage. YOUR_ACCESS_KEY should contain 5 to 20 alphanumerical characters and YOUR_SECRET_KEY should contain 8 to 40 alphanumerical characters.

Configuration and topology data (StackGraph)

Configuration and topology data (StackGraph) backups are full backups, stored in a single file with the extension .graph. Each file contains a full backup and can be moved, copied or deleted as required.

Disable scheduled backups

When backup.enabled is set to true, scheduled StackGraph backups are enabled by default. To disable scheduled StackGraph backups only, set the Helm value backup.stackGraph.scheduled.enabled to false.

Disable restores

When backup.enabled is set to true, StackGraph restores are enabled by default. To disable StackGraph restore functionality only, set the Helm value backup.stackGraph.restore.enabled to false.

Backup schedule

By default, the StackGraph backups are created daily at 03:00 AM server time.

Backup retention

By default, the StackGraph backups are kept for 30 days. As StackGraph backups are full backups, this can require a lot of storage.

Telemetry data (Elasticsearch)

The telemetry data (Elasticsearch) snapshots are incremental and stored in files with the extension .dat. The files in the Elasticsearch backup storage location should be treated as a single whole and can only be moved, copied or deleted as a whole.

Disable scheduled snapshots

When backup.enabled is set to true, scheduled Elasticsearch snapshots are enabled by default. To disable scheduled Elasticsearch snapshots only, set the Helm value backup.elasticsearch.scheduled.enabled to false.

Disable restores

When backup.enabled is set to true, Elasticsearch restores are enabled by default. To disable Elasticsearch restore functionality only, set the Helm value backup.elasticsearch.restore.enabled to false.

Snapshot schedule

By default, Elasticsearch snapshots are created daily at 03:00 AM server time.

Snapshot retention

By default, Elasticsearch snapshots are kept for 30 days, with a minimum of 5 snapshots and a maximum of 30 snapshots.

The retention time and number of snapshots kept can be configured using the following Helm values:

  • backup.elasticsearch.scheduled.snapshotRetentionMinCount

  • backup.elasticsearch.scheduled.snapshotRetentionMaxCount

Snapshot indices

By default, a snapshot is created for Elasticsearch indices with names that start with sts.

Restore backups and snapshots

Before you use the scripts, ensure that:

  • The kubectl binary is installed and configured to connect to:

    1. The Kubernetes cluster where StackState has been installed.

    2. The namespace within that cluster where StackState has been installed.

  • The following Helm values have been correctly set:

    1. backup.enabled is set to true.

    2. backup.stackGraph.restore.enabled isn't set to false (to access StackGraph backups).

    3. backup.elasticsearch.restore.enabled isn't set to false (to access Elasticsearch snapshots).

List StackGraph backups

To list the StackGraph backups, execute the following command:

./restore/list-stackgraph-backups.sh

The output should look like this:

job.batch/stackgraph-list-backups-20210222t111942 created
Waiting for job to start...
=== Listing StackGraph backups in bucket "sts-stackgraph-backup"...
sts-backup-20210215-0300.graph
sts-backup-20210216-0300.graph
sts-backup-20210217-0300.graph
sts-backup-20210218-0300.graph
sts-backup-20210219-0300.graph
sts-backup-20210220-0300.graph
sts-backup-20210221-0300.graph
sts-backup-20210222-0300.graph
===
job.batch "stackgraph-list-backups-20210222t111942" deleted

The timestamp when the backup was taken is part of the backup name.

Lines in the output that start with Error from server (BadRequest): are expected. They appear when the script is waiting for the pod to start.

Restore a StackGraph backup

To avoid the unexpected loss of existing data, a backup can only be restored on a clean environment by default. If you are completely sure that any existing data can be overwritten, you can override this safety feature by using the command -force. Only execute the restore command when you are sure that you want to restore the backup.

To restore a StackGraph backup on a clean environment, select a backup name and pass it as the first parameter in the following command:

./restore/restore-stackgraph-backup.sh sts-backup-20210216-0300.graph

To restore a StackGraph backup on an environment with existing data, select a backup name and pass it as the first parameter in the following command next to a second parameter -force:

Note that existing data will be overwritten when the backup is restored.

Only do this if you are completely sure that any existing data can be overwritten.

./restore/restore-stackgraph-backup.sh sts-backup-20210216-0300.graph -force

The output should look like this:

job.batch/stackgraph-restore-20210222t112142 created
Waiting for job to start...
=== Downloading StackGraph backup "sts-backup-20210216-0300.graph" from bucket "sts-stackgraph-backup"...
download: s3://sts-stackgraph-backup/sts-backup-20210216-1252.graph to ../../tmp/sts-backup-20210216-0300.graph
=== Importing StackGraph data from "sts-backup-20210216-0300.graph"...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/opt/docker/lib/org.codehaus.groovy.groovy-2.5.4.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
===
job.batch "stackgraph-restore-20210222t112142" deleted

In case you are running a restore command missing the -force flag on a non-empty database the output will contain an error like this:

ERROR com.stackvista.graph.migration.Restore - Restore isn't possible in a non empty.

Lines that starts with WARNING: are expected. They're generated by Groovy running in JDK 11 and can be ignored.

List Elasticsearch snapshots

To list the Elasticsearch snapshots, execute the following command:

./restore/list-elasticsearch-snapshots.sh

The output should look like this:

job.batch/elasticsearch-list-snapshots-20210224t133115 created
Waiting for job to start...
Waiting for job to start...
=== Listing Elasticsearch snapshots in snapshot repository "sts-backup" in bucket "sts-elasticsearch-backup"...
sts-backup-20210219-0300-mref7yrvrswxa02aqq213w
sts-backup-20210220-0300-yrn6qexkrdgh3pummsrj7e
sts-backup-20210221-0300-p481sih8s5jhre9zy4yw2o
sts-backup-20210222-0300-611kxendsvh4hhkoosr4b7
sts-backup-20210223-0300-ppss8nx40ykppss8nx40yk
===
job.batch "elasticsearch-list-snapshots-20210224t133115" deleted

The timestamp when the backup was taken is part of the backup name.

Delete Elasticsearch indices

To delete existing Elasticsearch indices so that a snapshot can be restored, follow these steps.

  1. Stop indexing - scale down all *2es deployments to 0:

    kubectl scale --replicas=0 deployment/e2es
    kubectl scale --replicas=0 deployment/mm2es
    kubectl scale --replicas=0 deployment/trace2es
  2. Open a port-forward to the Elasticsearch master:

    kubectl port-forward service/stackstate-elasticsearch-master 9200:9200
  3. Get a list of all indices:

    curl "http://localhost:9200/_cat/indices?v=true"

    The output should look like this:

    health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green  open   sts_internal_events-2022.09.20 fTk7iEYtQI6ruVFuwNnbPw   1   0        125            0     71.7kb         71.7kb
    green  open   .geoip_databases               GYA_c3i6QKenfFehnwBcAA   1   0         41            0     38.8mb         38.8mb
    green  open   sts_multi_metrics-2022.09.20   MT1DceQPTDWVIfBJdl7qUg   3   0       1252            0    550.1kb        550.1kb
  4. Delete an index with a following command:

    curl -X DELETE "http://localhost:9200/INDEX_NAME?pretty"

    Replace INDEX_NAME with the name of the index to delete, for example:

    curl -X DELETE "http://localhost:9200/sts_internal_events-2021.02.19?pretty"
  5. The output should be:

    {
    "acknowledged" : true
    }

Restore an Elasticsearch snapshot

To restore an Elasticsearch snapshot, select a snapshot name and pass it as the first parameter in the following command line. You can optionally specify a second parameter with a comma-separated list of the indices that should be restored. If not specified, all indices that match the Helm value backup.elasticsearch.scheduled.indices will be restored (default "sts*"):

./restore/restore-elasticsearch-snapshot.sh \
  sts-backup-20210223-0300-ppss8nx40ykppss8nx40yk \
  "<INDEX_TO_RESTORE>,<INDEX_TO_RESTORE>"

The output should look like this:

job.batch/elasticsearch-restore-20210229t152530 created
Waiting for job to start...
Waiting for job to start...
=== Restoring Elasticsearch snapshot "sts-backup-20210223-0300-ppss8nx40ykppss8nx40yk" from snapshot repository "sts-backup" in bucket "sts-elasticsearch-backup"...
{
  "snapshot" : {
    "snapshot" : "sts-backup-20210223-0300-ppss8nx40ykppss8nx40yk",
    "indices" : [
      ".slm-history-1-000001",
      "ilm-history-1-000001",
      "sts_internal_events-2021.02.19"
    ],
    "shards" : {
      "total" : 3,
      "failed" : 0,
      "successful" : 3
    }
  }
}
===
job.batch "elasticsearch-restore-20210229t152530" deleted

The indices restored are listed in the output, as well as the number of failed and successful restore actions.

After the indices have been restored, scale up all *2es deployments:

kubectl scale --replicas=1 deployment/e2es
kubectl scale --replicas=1 deployment/mm2es
kubectl scale --replicas=1 deployment/trace2es

AZURE_STORAGE_ACCOUNT_NAME - the

AZURE_STORAGE_ACCOUNT_KEY - the where the backups should be stored.

The backup schedule can be configured using the Helm value backup.stackGraph.scheduled.schedule, specified in .

The backup retention delta can be configured using the Helm value backup.stackGraph.scheduled.backupRetentionTimeDelta, specified in .

The configuration snippets provided in the section will enable daily Elasticsearch snapshots.

The backup schedule can be configured using the Helm value backup.elasticsearch.scheduled.schedule, specified in .

backup.elasticsearch.scheduled.snapshotRetentionExpireAfter, specified in .

By default, the retention task itself . If you set snapshots to expire faster than within a day, for example for testing purposes, you will need to change the schedule for the retention task.

The indices for which a snapshot is created can be configured using the Helm value backup.elasticsearch.scheduled.indices, specified in .

Scripts to list and restore backups and snapshots can be found in the . To use the scripts, download them from GitHub or checkout the repository.

When a snapshot is restored, existing indices will NOT be overwritten. Use the Elasticsearch to remove them first.

See .

🚀
Azure storage account name (learn.microsoft.com)
Azure storage account key (learn.microsoft.com)
Kubernetes cron schedule syntax (kubernetes.io)
Python timedelta format (python.org)
Elasticsearch cron schedule syntax (elastic.co)
Elasticsearch time units (elastic.co)
runs daily at 1:30 AM UTC (elastic.co)
JSON array format (w3schools.com)
restore directory of the StackState Helm chart repository (github.com)
delete index API (elastic.co)
MinIO (min.io)
AWS S3 (min.io)
Azure BLob Storage (min.io)
AWS S3
Azure Blob Storage
Kubernetes storage
enable backups
delete Elasticsearch indices