LogoLogo
StackState.comDownloadSupportExplore playground
SUSE Observability
SUSE Observability
  • SUSE Observability docs!
  • Docs for all SUSE Observability products
  • 🚀Get started
    • Quick start guide
    • SUSE Observability walk-through
    • SUSE Rancher Prime
      • Air-gapped
      • Agent Air-gapped
    • SUSE Cloud Observability
  • 🦮Guided troubleshooting
    • What is guided troubleshooting?
    • YAML Configuration
    • Changes
    • Logs
  • 🚨Monitors and alerts
    • Monitors
    • Out of the box monitors for Kubernetes
    • Notifications
      • Configure notifications
      • Notification channels
        • Slack
        • Teams
        • Webhook
        • Opsgenie
      • Troubleshooting
    • Customize
      • Add a monitor using the CLI
      • Derived State monitor
      • Dynamic Threshold monitor
      • Override monitor arguments
      • Write a remediation guide
  • 📈Metrics
    • Explore Metrics
    • Custom charts
      • Adding custom charts to components
      • Writing PromQL queries for representative charts
      • Troubleshooting custom charts
    • Advanced Metrics
      • Grafana Datasource
      • Prometheus remote_write
      • OpenMetrics
  • 📑Logs
    • Explore Logs
    • Log Shipping
  • 🔭Traces
    • Explore Traces
  • 📖Health
    • Health synchronization
    • Send health data over HTTP
      • Send health data
      • Repeat Snapshots JSON
      • Transactional Increments JSON
    • Debug health synchronization
  • 🔍Views
    • Kubernetes views
    • Custom views
    • Component views
    • Explore views
    • View structure
      • Overview perspective
      • Highlights perspective
      • Topology perspective
      • Events perspective
      • Metrics perspective
      • Traces perspective
      • Filters
      • Keyboard shortcuts
    • Timeline and time travel
  • 🕵️Agent
    • Network configuration
      • Proxy Configuration
    • Using a custom registry
    • Custom Secret Management
      • Custom Secret Management (Deprecated)
    • Request tracing
      • Certificates for sidecar injection
  • 🔭Open Telemetry
    • Overview
    • Getting started
      • Concepts
      • Kubernetes
      • Kubernetes Operator
      • Linux
      • AWS Lambda
    • Open telemetry collector
      • Sampling
      • SUSE Observability OTLP APIs
    • Instrumentation
      • Java
      • Node.js
        • Auto-instrumentation of Lambdas
      • .NET
      • SDK Exporter configuration
    • Troubleshooting
  • CLI
    • SUSE Observability CLI
  • 🚀Self-hosted setup
    • Install SUSE Observability
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Alibaba Cloud ACK install
        • Required Permissions
        • Override default configuration
        • Configure storage
        • Exposing SUSE Observability outside of the cluster
      • Initial run guide
      • Troubleshooting
        • Advanced Troubleshooting
        • Support Package (Logs)
    • Configure SUSE Observability
      • Slack notifications
      • E-mail notifications
      • Stackpacks
      • Advanced
        • Analytics
    • Release Notes
      • v2.0.0 - 11/Sep/2024
      • v2.0.1 - 18/Sep/2024
      • v2.0.2 - 01/Oct/2024
      • v2.1.0 - 29/Oct/2024
      • v2.2.0 - 09/Dec/2024
      • v2.2.1 - 10/Dec/2024
      • v2.3.0 - 30/Jan/2025
      • v2.3.1 - 17/Mar/2025
      • v2.3.2 - 22/Apr/2025
      • v2.3.3 - 07/May/2025
    • Upgrade SUSE Observability
      • Migration from StackState
      • Steps to upgrade
      • Version-specific upgrade instructions
    • Uninstall SUSE Observability
    • Air-gapped
      • SUSE Observability air-gapped
      • SUSE Observability Kubernetes Agent air-gapped
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Configuration backup
      • Data retention
      • Clear stored data
    • Security
      • Authentication
        • Authentication options
        • Single password
        • File-based
        • LDAP
        • Open ID Connect (OIDC)
          • Microsoft Entra ID
        • KeyCloak
        • Service tokens
        • Troubleshooting
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
      • Self-signed certificates
      • External secrets
  • 🔐Security
    • Service Tokens
    • API Keys
  • ☁️SaaS
    • User Management
  • Reference
    • SUSE Observability Query Language (STQL)
    • Chart units
    • Topology Identifiers
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • Retention of topology graph data
  • Retention of events and logs
  • SUSE Observability data store
  • Retention of metrics
  • Update SUSE Observability
  • Resizing storage
  • Verify the storage class supports resizing
  • Resize the volumes
  • Update values.yaml and apply the change
  1. Self-hosted setup
  2. Data management

Data retention

SUSE Observability Self-hosted

Overview

SUSE Observability imposes data retention limits to save storage space and improve performance. You can configure the data retention period to balance the amount of data stored with SUSE Observability performance and data availability.

Retention of topology graph data

By default, topology graph data will be retained for 30 days. This works in a way that the latest state of topology graph will always be retained; only history older than 30 days will be removed. In some cases, it may be useful to keep historical data for more than 30 days or to reduce it to less than 30 days to save on disk space. Topology retention can be configured through the helm chart:

stackstate:
  topology:
    # Retention set to 1 week
    retentionHours: 144

Note that by adding more time to the data retention period, the amount of data stored is also going to grow and requires more storage space. This may also affect the performance of the Views.

When lowering the retention period, it can take some time until disk space is freed up (at least 15 minutes).

Retention of events and logs

SUSE Observability data store

If you are using the event/logs store provided with SUSE Observability, your data will by default be retained for 30 days. In most cases, the default settings will be sufficient to store all indices for this amount of time.

Configure disk space for Elasticsearch

In some circumstances it may be necessary to adjust the disk space available to Elasticsearch and how it's allocated to logs and events, for example if you anticipate a lot of data to arrive for a specific data type.

Here is a snippet with the complete disk space and retention config for Elasticsearch, including the default values.

elasticsearch:
  volumeClaimTemplate:
    resources:
      requests:
        storage: 250Gi
stackstate:
  components:
    receiver:
      esDiskSpaceShare: 70
      # Number of days to keep the logs data on Es
      retention: 7
    e2es:
      esDiskSpaceShare: 30
      # Number of days to keep the events data on Es
      retention: 30

Note: this is the disk space for each instance of ElasticSearch. For non-HA this is the total available disk space, but for HA there are 3 instances and a replication factor of 1. The end result is that the total available Elasticsearch storage will be (250Gi * 3) / 2 = 375Gi. Based on the esDiskSpaceShare and retention a portion of the Elasticsearch disk space is reserved for each data type.

Retention of metrics

SUSE Observability uses VictoriaMetrics to store metrics. It's configured with a default retention of 30 days. The helm chart allocates disk space and configures the retention period for the 1 or 2 Victoria metrics instances like this:

victoria-metrics-0:
  server:
    persistentVolume:
      size: 250Gi
    retentionPeriod: 1 # month
# For HA setups:
victoria-metrics-1:
  server:
    persistentVolume:
      size: 250Gi
    retentionPeriod: 1 # month
  • The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If no suffix is set the duration is in months.

  • Minimum retention period is 24h or 1 day.

Update SUSE Observability

After making changes to the values.yaml SUSE Observability needs to be updated to apply those changes to the runtime. This may cause some short downtime while the services restart. To update SUSE Observability use the same command that was used during installation of SUSE Observability and make sure to include the same configuration files including the changes that have been made:

Resizing storage

In most clusters it's possible to resize a persistent volume after it has been created and without interrupting the operation of applications at all. However this can't be done by simply changing the configured storage size in the values.yaml of the SUSE Observability Helm chart. Instead several steps are needed:

  1. Verify the used storage class can be resized

  2. Resize the volumes

  3. Update values.yaml and apply change (optional but recommended)

The examples below use the VictoriaMetrics storage as an example. SUSE Observability is installed in the suse-observability namespace. The volume is going to be resized to 500Gi.

Verify the storage class supports resizing

Use the following kubectl commands to get the storage class used and check that the allowVolumeExpansion is set to true.

# Get the PVC's for SUSE Observability
kubectl get pvc --namespace suse-observability

# There is a storage class column in the output, copy it and use it to describe the storage class
kubectl describe storageclass <storage-class-name>

Verify that the output contains this line:

AllowVolumeExpansion:  True

If the line is absent or if it's set to False please consult with your Kubernetes administrator if resizing is supported and can be enabled.

Resize the volumes

The SUSE Observability Helm chart creates a stateful set, which has a template to create the persistent volume claim (PVC). This template is only used to create the PVC once, after that it won't be applied anymore and it's also not allowed to change it. So to make the PVC's bigger the PVC itself needs to be edited.

To change the PVC size use the following commands.

# Get the PVC's for SUSE Observability, allows us to check the current size and copy the name of the PVC to modify it with the next command
kubectl get pvc --namespace suse-observability

# Patch the PVC's specified size, change it to 500Gi
kubectl patch pvc server-volume-stackstate-victoria-metrics-0-0 -p '{"spec":{"resources": { "requests": { "storage": "500Gi" }}}}'

# Get the PVC's again to verify if it was resized, depending on the provider this can take a while
kubectl get pvc --namespace suse-observability

Update values.yaml and apply the change

The change made to the persistent volume claim (PVC) will remain for the lifetime of the PVC, but whenever a clean install is done it will be lost. More importantly however, after resizing the PVC there is now a discrepancy between the cluster state and the definition of the desired state in the values.yaml. Therefore it's recommended to update the values.yaml as well. To circumvent the fact that this change is not allowed, first remove the stateful set (but keep the pods running) to re-create it with the new settings.

This step doesn't change the size of the PVC itself, so only doing this step will result in no changes at all to the running environment.

Now remove the stateful set for the application(s) for which the storage has been changed:

# List all stateful sets, check that all are ready, if not please troubleshoot that first
kubectl get statefulset --namespace suse-observability

# Delete the 
kubectl delete statefulset --namespace suse-observability stackstate-victoria-metrics-0 --cascade=orphan
PreviousConfiguration backupNextClear stored data

Last updated 1 month ago

The disk space available for Elasticsearch is configured via the elasticsearch.volumeClaimTemplate.resources.requests.storage key. To change this value after the initial installation some .

To change the volume size after the initial installation some .

To change the retention period override both retentionPeriod keys with the same value in your custom values.yaml and :

:

First edit your values.yaml to update the volume size for the PVC's you've just resized. See the sections on or .

Finally with the new settings.

🚀
extra steps are required
extra steps are required
update SUSE Observability
Metrics
Events and Logs
update SUSE Observability
Kubernetes
OpenShift