Installation

StackState runs on the Java virtual machines and comes packaged with its own graph database called StackGraph. To run StackState you must first run StackGraph, a time traveling graph database we’ve built on top of HBase. StackState is built on a highly distributed architecture, therefore it runs as a number of Java processes that communicate with each other via TCP/IP.

Since these are the early days of StackState there are still a lot of ways to simplify our installation process that we are aware of.

Requirements

Server requirements

For detailed information please refer to sizing guide

To install the StackState server, you must meet the following requirements:

  • Operating system: Microsoft Windows (64-bit) or a Unix-family operating system running Java

  • Oracle, IBM, or Apple Java Development Kit (JDK):

    • JDK 8 patch level 102 or later
  • RAM: At least 8 GB of RAM available for StackState and an additional 6 GB of RAM available for StackGraph

  • Hard disk space: At least 100GB of hard disk space available for StackState and an additional 50GB of hard disk space available for StackGraph. The actual disk space requirements depend on the amount of data StackState will process

Networking requirements

2 node default deployment

With a 2 node default deployment StackState runs on node 1 together with Kafka, ElasticSearch, the Kafka*ToEs processes and the Receiver api. node 2 in this setup runs the database processes; namenode, datanode, hmaster, hregion, transaction service.

node 1 to private network: public ports and 9092 (kafka), 9200, 9300 (elasticsearch node 1 to public network: 7070 (StackState), 7077 (Receiver) node 1 accessible for developers (troubleshooting): 9010, 9011, 9020, 9021, 9022, 9024

node 2 to private network: 2181 (zookeeper), 16000 (hmaster), 16020 (hregion), 8020 (namenode), 50010, 50020 (both datanode), 15165 (transaction service) node 2 to public network: none node 2 accessible for developers (troubleshooting): 9001, 9002, 9003, 9004, 9005, 9006

These ports only apply for this specific 2 node setup. In case of scaling out and running processes on multiple servers it is advisable to open all bold ports in the “Per Processes” section for the machines running the specific process.

Per process

Ports in bold need to be open between machines. Ports in bold with (public) need to be open for the rest of the network.

The JMX ports can be used for monitoring, but are also useful when troubleshooting. For the later to work they need to be accessible from a developer machine as well.

StackState
  • 7070 (public): HTTP api & UI
  • 7071: Admin API for health checks and admin operations. Typically you want to use this only from localhost
  • 9020: JMX port for monitoring Java process
Receiver (agent) api
  • 7077 (public): HTTP agent api (aka intake API). When using an agent telemtry is reported to this endpoint.
  • 9024: JMX port for monitoring Java process
Kafka*ES
  • 9021: JMX port for monitoring Java process (metrics to es)
  • 9022: JMX port for monitoring Java process (events to es)
Kafka
  • 9092: Client port
  • 9010: JMX port for monitoring Java process
Elasticsearch
  • 9200: HTTP api
  • 9300: Native api
  • 9011: JMX port for monitoring Java process
Zookeeper
  • 2181: Client API
  • 2888: Zookeeper peers (general communication), only when running a cluster
  • 3888: Zookeeper peers (leader election), only when running a cluster
  • 9003: JMX port for monitoring Java process
HBase
Master
  • 16000: Master client api (needs to be open for clients)
  • 16010: Master Web UI (optional)
  • 9004: JMX port for monitoring Java process
Region server
  • 16020: Region client API (needs to be open for clients)
  • 16030: Region Web UI (optional)
  • 9005: JMX port for monitoring Java process
HDFS
NameNode
  • 8020: File system (needs to be open for clients)
  • 9001: JMX port for monitoring Java process
  • 50070: Web UI (optional)
DataNode
  • 50010: Datanode API (needs to be open for clients)
  • 50020: IPC api (communication within HDFS cluster)
  • 9002: JMX port for monitoring Java process
  • 50075: HTTP api (optional)
Transaction service
  • 15165: Client API
  • 9006: JMX port for monitoring Java process
ProcessManager
  • 5152: StackGraph ProcessManager, at the moment only from localhost
  • 5154: StackState ProcessManager, at the moment only from localhost

Client requirements

To use the StackState GUI, you must use one of the following web browsers:

  • Firefox
  • Chrome
  • Internet Explorer 11 or later
  • Edge

Packaging

To install StackState on Fedora, Red Hat or CentOS there is an RPM package available to easily install and upgrade StackState. For Debian and Ubuntu there is a DEB package available.

Installing StackState

Below are installation instructions for StackState on different platforms.

Installing StackState on Fedora, Red Hat, CentOS, Debian or Ubuntu

Installation instructions

To let yum automatically install the right java version for the RPM package use: - yum localinstall <stackstate>.rpm

Or you can use rpm, which assumes the correct java version is already installed: - rpm -i <stackstate>.rpm

To install using the DEB package, assuming the correct java version is already installed: - dpkg -i <stackstate>.deb

Alternatively use apt-get which will also install the proper Java version: - apt-get install <stackstate>.deb

The package creates a stackstate user and group as which StackState will be running. To start you first have to register StackState by running: sudo -u stackstate /opt/stackstate/bin/sts-standalone.sh register --key <license-key>.

By default, the data directory is set to /opt/stackstate/var/lib and the log directory is set to /opt/stackstate/var/log. It is possible to change the data and log directories by setting the appropriate environment variables prior to installing StackState. Set environment variable STACKSTATE_DATA_DIR to set the data directory and set the environment variable STACKSTATE_LOG_DIR to set the log directory.

Installing StackState on MacOs and other Linux flavors

StackState has been tested on recent versions of MacOs and Ubuntu.

Installation instructions

  1. Unzip the <stackstate>.zip into the directory where you want to install StackState.
  2. Create a user and group used called stackstate.
  3. Before starting StackState, it has to configure its directory structure. This is done by running a config script, which uses a directory structure configured inside the stackstate-directories.sh script. If you want to change the default structure, you can edit this file, before starting the config script. Execute the line below to run the config script, passing the user and group created in the previous step.
    • bin/config.sh stackstate stackstate

Starting StackState

To start using StackState, see usage.

Installing StackState on Windows

StackState has been tested on Windows 7 and up. A 64 bit version of Windows is required.

Installation instructions

  1. Download and install Java 8 (JRE) 64-bits

    • http://www.oracle.com/technetwork/java/javase/downloads/index.html
    • Set a global JAVA_HOME environment variable to the installation directory of Java (parent directory of bin\java.exe).
      • Note: Please make sure you point it to the 64 bits version (so not the one under *\Program Files (x86)\Java\*)
    • Java needs to be on the path: add %JAVA_HOME%\bin at the end of the definition for the PATH environment variable. see http://goo.gl/1VgVp6 for an explanation on how to add environment variables and Java to the path.
  2. Download and install the Visual C++ Redistributable for Visual Studio 2015: vc_redist.x64.exe

    • from Microsoft https://www.microsoft.com/en-us/download/details.aspx?id=48145
    • or alternatively from Softpedia http://goo.gl/KeCU1X
  3. Unzip the zipfile containing StackState and you are ready to go.
    For an overview how to start and stop stackstate and stackgraph see usage

Starting StackState

SystemD service

The RPM and DEB packages install SystemD services for StackState and StackGraph. StackState can be started with sudo systemctl start stackstate.service this will also start service StackGraph. Starting StackState can take some time.

Open your browser to: http://<stackstate_hostname>:7070.

Stopping StackState

StackState can be stopped by sudo systemctl stop stackstate.service. StackGraph is not automatically stopped when stopping StackState, StackGraph can be stopped by sudo systemctl stop stackgraph.service.

StackState Status

Checking the service status can be done with sudo systemctl status stackstate.service and sudo systemctl status stackgraph.service.

Manual

StackState and all of its dependencies need to be started from the command prompt. This can be done with a single command but if preferred StackGraph (the database used by StackState) can be started separately. The executables needed can be found in the installation directory under the bin directory or for stackgraph inside stackgraph/bin. Windows users can use the provided .bat scripts while Linux users can use the provided .sh scripts.

Single command startup for both StackState and StackGraph

StackState can be started in demo mode or normal mode. In demo mode some data will be injected into StackState for you to play around with.

  1. Starting StackState and Stackgraph:

    • Navigate to the installation directory of StackState.
    • The first time you start StackState, you have to enter the license key by executing:
      • Linux: bin/sts.sh register --key <license-key>
      • Windows: bin/sts.bat register --key <license-key>
    • Start StackState in demo mode:
      • Linux: bin/sts.sh demo
      • Windows: bin/sts.bat demo
    • Start StackState in normal mode:
      • Linux: bin/sts.sh start
      • Windows: bin/sts.bat start
    • To see the usage of the combined StackState and StackGraph script:
      • Linux: bin/sts.sh help
      • Windows: bin/sts.bat help
  2. Open your browser to: http://<stackstate_hostname>:7070

  3. Stopping StackState and StackGraph:

    • Linux: bin/sts.sh stop
    • Windows: bin/sts.bat stop To stop StackGraph gracefully do not press ctrl-c, but open another window and execute the stop command. As mentioned earlier we are working on a better experience.

Separate startup for StackState and StackGraph

  • Navigate to the installation directory of StackState.
  1. Start StackGraph (StackState’s database):

    • Linux: stackgraph/bin/sg-service.sh start (runs as a daemon)
    • Windows: stackgraph/bin/sg.bat start
    • keep the terminal open
  2. Open a second terminal, now start StackState:

    • StackState can be started in demo mode or normal mode. In demo mode some data will be injected into StackState for you to play around with.
    • The first time you start StackState, you have to enter the license key by executing:
      • Linux: bin/sts-standalone.sh register --key <license-key>
        • Windows: bin/sts-standalone.bat register --key <license-key>
    • Start StackState in demo mode:
      • Linux: bin/sts-standalone.sh demo
      • Windows: bin/sts-standalone.bat demo
    • Start StackState in normal mode:
      • Linux: bin/sts-service.sh start (note: this will run in daemon mode)
      • Windows: bin/sts-standalone.bat start
    • To see the usage of the StackState standalone script:
      • Linux: bin/sts-standalone.sh help
      • Windows: bin/sts-standalone.bat help
  3. Open your browser to: http://<stackstate_hostname>:7070

  4. Stopping StackState:

    • Linux: bin/sts-standalone.sh stop
    • Windows: bin/sts-standalone.bat stop
  5. Stopping StackGraph (StackState’s database):

    • Linux: stackgraph/bin/sg.sh stop
    • Windows: stackgraph/bin/sg.bat stop To stop StackGraph gracefully do not press ctrl-c, but open another window and execute the stop command. As mentioned earlier we are working on a better experience.

Upgrading StackState

When upgrading stackstate, the package will not automatically create a backup of StackGraph data. If you want to make a backup run the bin/sts-backup.sh script. It will create a backup and store it inside the /opt/stackstate/backups/ directory. It will also start StackState and StackGraph when the upgrade was successfull. Run the command: - rpm -U <stackstate>.rpm

Installing StackState in a distributed configuration

The default packaging assumes StackState and StackGraph run on the same server. To install StackState and StackGraph on different servers (still only 1 instance of every application) some configurations need to be changed.

On both servers install StackState with the normal RPM or Debian package. On the server where only StackGraph should be running start StackGraph with sudo systemctl start stackgraph.service. On the server where only StackState should be running the provided SystemD configuration needs to be adapted. Edit the file /lib/systemd/system/stackstate.service file:

  • Remove the stackgraph.service from the Requires and After sections
  • Define the environment variable ZOOKEEPER_QUORUM=<your-stackgraph-server> under the [Service] section, with the hostname of your StackGraph server as the value.

After these changes the stackstate.service file should look approximately like this:

[Unit]
Description=StackState System monitor Service
After=syslog.target network.target

[Service]
LimitNOFILE=131072
Type=forking
User=stackstate
Group=stackstate
PIDFile=/opt/stackstate/var/run/stackstate.pid
ExecStart=/opt/stackstate/bin/sts-service.sh start
ExecReload=/opt/stackstate/bin/sts-service.sh reload
ExecStop=/opt/stackstate/bin/sts-service.sh stop
Environment=ZOOKEEPER_NODENAME=<your-stackgraph-server>

[Install]
Alias=stackstate.service
WantedBy=multi-user.target```  

Now register StackState with sudo -u stackstate /opt/stackstate/bin/sts-standalone.sh register --key <license-key> and start it with sudo systemctl start stackstate.service.

Configuring a reverse proxy

StackState makes heavy use of Websockets, so when setting up a reverse proxy to make StackState available on, for example, port 80 or 443 (with TLS) you will also need to proxy Websocket requests. An example setup for Nginx for this looks like this (proxying port 80 to StackState running on localhost port 7070):

  http {
      map $http_upgrade $connection_upgrade {
          default upgrade;
          '' close;
      }

      server {
          listen 80;
          location / {
              proxy_pass http://127.0.0.1:7070;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection $connection_upgrade;
          }
      }
  }

Configuration import/export

StackState’s configuration can be exported and imported. The import/export functionality can be used to automate the installation process and/or for backup purposes. An export and import can be made in the settings page of StackState’s user interface by using the buttons ‘Export Model’ and ‘Import Model’.

Export

The export of the configuration can be obtained by:

$ curl -X POST -H 'Content-Type: application/json;charset=UTF-8' -d '{"allNodesOfTypes":["ComponentType","RelationType","Domain","Layer","Environment","DataSource","View","EventHandler","CheckFunction","BaselineFunction","PropagationFunction","EventHandlerFunction","TemplateFunction","MappingFunction","IdExtractorFunction","ViewHealthStateConfigurationFunction","Sync"]}' "http://<host>:7070/api/export?timeoutSeconds=300" > export.conf

<host> is the address on where StackState is reachable.

Import

Import is intented to be a one-off action, importing multiple times might result in duplicate configuration entries. It is possible to clear StackState’s configuration before an import. To clear StackState of any configuration use:

$ curl -X POST -f "http://<host>:7071/clear" 

<host> is the address on where StackState is reachable.

The StackState configuration file can be imported by:

$ curl -X POST -d @./export.conf -H 'Content-Type: application/json;charset=UTF-8' "http://<host>:7070/api/import?timeoutSeconds=15"

<host> is the address on where StackState is reachable.

Individual items

It is possible to export and import individual configuration items through the StackState user interface. For example, to export a component type go to the Settings page and click on ‘Component Types’:

Setting menu

To export an individual component type, click on ‘Export as config’:

Export individual configuration

An individual configuration item can be imported through settings button ‘Import Model’.

Troubleshooting

Issues getting StackState started

Here is a quick guide for troubleshooting the startup of StackState:

  1. Check whether systemd service StackGraph is started by sudo systemctl status stackgraph.service
  2. Check whether systemd service StackState is started by sudo systemctl status stackstate.service
  3. Check connection to StackState’s user interface, default listening on TCP port 7070.
  4. Check log files for errors, located at /opt/stackstate/var/log/

Known issues

InterruptedException on opening a view

Symptom: opening a view that is expected to contain a large topology results in Stackie and the /opt/stackstate/var/log/stackstate.log log shows an exception similar to:

... Starting ViewEventSummaryStream web socket stream failed.
com.stackvista.graph.hbase.StackHBaseException: Error while accessing HBase
...
Caused by: java.io.InterruptedIOException: Origin: InterruptedException

Cause: topology elements that are not cached are not fully retrieved from StackGraph within a certain period of time before a timeout, InterruptedException, is triggered.

Possible solution: increase the cache size by editing StackState’s configuration.

In /opt/stackstate/etc/application_stackstate.conf add the following configuration stackgraph.vertex.cache.size = <size> where <size> is the number of Graph vertices. An initial cache size can be obtained by adding:

  • number of components * 10,
  • number of relations * 10,
  • number of checks * 5.

The default cache size is set to 8191. Make sure the cache size is defined as a power of two minus one, e.g. 2^13-1 = 8191.

Make sure that StackState has enough memory available, the available memory can be configured by editing: /opt/stackstate/etc/processmanager/processmanager.conf. Under process named stackstate-server, change -Xmx1G to -Xmx<N>G where <N> is the number of desired GBs of memory. For example, change the setting to -Xmx8G to have 8 GBs of memory available to StackState.

Restart StackState, by sudo systemctl restart stackstate.service, for the changes to be effective.