Topology synchronization

Overview

StackState can synchronize topology information from different sources. This includes your own sources. This way StackState enables you to create a model of your complete landscape.

This guide explains how to send custom topology information to StackState. The following steps describe the synchronization of your custom topology data.

Topology synchronization guide

Step 1: Send topology information via StackState’s receiver API:

The data format:

JSON example:

{
   "apiKey":"5f98193e83ece68c811df22174859355",
   "collection_timestamp":1467037580.595086,
   "internalHostname":"lnx-343242.srv.stackstate.com",
   "events":{},
   "metrics":[],
   "service_checks":[],
   "topologies":[
      {
         "starts_snapshot":true,
         "stops_snapshot":true,
         "instance":{
            "type":"mesos",
            "url":"http://localhost:5050"
         },
         "components":[
            {
               "externalId":"nginx3.e5dda204-d1b2-11e6-a015-0242ac110005",
               "type":{
                  "name":"docker"
               },
               "data":{
                  "tags":[
                     "mytag"
                  ],
                  "ip_addresses":[
                     "172.17.0.8"
                  ],
                  "labels":[
                     {
                        "key":"label1"
                     }
                  ],
                  "framework_id":"fc998b77-e2d1-4be5-b15c-1af7cddabfed-0000",
                  "docker":{
                     "image":"nginx",
                     "network":"BRIDGE",
                     "port_mappings":[
                        {
                           "container_port":31945,
                           "host_port":31945,
                           "protocol":"tcp"
                        }
                     ],
                     "privileged":false
                  },
                  "task_name":"nginx3",
                  "slave_id":"fc998b77-e2d1-4be5-b15c-1af7cddabfed-S0"
               }
            }
         ],
         "relations":[
            {
               "externalId":"nginx3.e5dda204-d1b2-11e6-a015-0242ac110005->nginx3.e5dda204-d1b2-11e6-a015-0242ac110006",
               "type":{
                  "name":"uses_service"
               },
               "sourceId":"nginx3.e5dda204-d1b2-11e6-a015-0242ac110005",
               "targetId":"nginx3.e5dda204-d1b2-11e6-a015-0242ac110006",
               "data":{

               }
            }
         ]
      }
   ]
}

Field description:

  • apiKey: The key that StackState provided for your installation.
  • collection_timestamp: Collection timestamp in Epoch miliseconds.
  • internalHostname: The hostname of the collector (which sends your custom topology data).
  • topologies: A list of one or more instance types. Instance types are described by the following fields:

  • starts_snapshot: Boolean (true/false). When set to “true” this message is handled as the beginning of a snapshot. This enables StackState to diff snapshots with the previous one to delete components / relations which are not in the snapshot anymore.

  • stops_snapshot: Boolean (true/false). When set to “true” this message is handled as the end of a snapshot.

  • instance: Describes the type and unique ID (URL format) of the topology data.

  • components: A list of components. Per component you have the following fields:
  •    externalId: A unique ID for this component. This has to be unique for this instance.
  •    type: A named parameter for this type.
  •    data: A JSON blob of arbitrary data.

  • relations: A list of relations. Per relation you have the following fields:
  •    externalId: A unique ID for this relation. This has to be unique for this instance.
  •    type: A named parameter for this type.
  •    data: A JSON blob of arbitrary data.
  •    sourceId: This refers to the source component externalId.
  •    targetId: This refers to the target component externalId.

Mandatory fields (which can be empty, see other guides how to use them): events metrics service_checks

To send the topology JSON to StackState you have to send the data in the format as just described to the following url: /stsAgent/intake/?api_key=. The content type header in mandatory:

"Content-Type": "application/json"

An example curl command:

curl -X POST -H "Content-Type": "application/json" -d '{ ...the json data.. }' "/stsAgent/intake/?api_key=API_KEY"

Step 2: StackState sources

The data send to the receiver API is split in the different instances (for example Mesos production). You have to configure a StackState topology source for each instance. A StackState source can only be configured after data for this instance has been received by the API.

Step 3: Synchronizations

For each topology source, you can define one or more synchronizations.

Step 3a: Identity extraction

In this step the unique external- and internal identity and identifiers are extracted. You can define one or more custom identity extractions. When using a single identity for your components, you can use the default Sts Plugin Component Identity Extractor (out of the box).
You may use multiple identities for components. Example [applsrv-1002-srv-01,10.0.5.34:7070]. In this example you can see that both identities describe the component. This functionality is used the merge step. The result of identity extraction is the creation of the external topology per instance. This external topology is used in the mapping step. The identity extractors are defined outside of the Synchronization wizard.

Example identity extraction:

map = topologyElement.asReadonlyMap()

externalId = map["externalId"]
type = map["typeName"].toLowerCase()

return Sts.createId(externalId, new HashSet([externalId]), type)

Step 3b: Mapping

In this step the mapping is defined. Per component/ relation the source type is mapped to a target type and a template. Per type you can define the priority. More information about this is described in the “Merge strategies”

Step 3c: Merging

The mapped components are merged into the StackState topology.

Merge strategies

A synchronization task can have 2 situations in which a component may need to be merged:

New components/relations

A component that was never synchronized before by this sync: The component is new for this synchronization but a different synchronization might have already created the component. Based on the identifiers of the component a lookup is done for any existing components with one or more identifiers in common. If this exists the new component is merged with the existing component using a merging strategy. A component that was synchronized before by this sync: The sync always stores a reference between the externalId and the component in StackState so the existing StackState component is available. A merge needs to be done with the existing component using a merging strategy.

The merging strategy is defined per mapping (so effectively per external component type) in the synchronization wizard. The 4 available merging strategies are:

(Read as follow: strategy for new -> strategy for current)

Merge strategy options

Use theirs always: If a component already exists it will remain as-is. Use mine only: If a component already exists it is replaced completely with the current component. Merge, prefer theirs: If a component already exists and it was merged with:

“Use theirs always: The existing component is replaced completely with the current component. “Use mine only”: the existing component will remain as-is “Merge, prefer (theirs|mine)”: where possible values will be added to the existing set of values (for example checks, and monitor streams). If no addition possible the existing value is kept as-is (for example name, description, etc…)

Merge, prefer mine:

  • Use theirs always The existing component is replaced completely with the current component.

  • Use mine only The existing component will remain as-is

  • Merge, prefer theirs Where possible values will be added to the existing set of values (for example checks, and monitor streams). If no addition possible the existing value is replaced (for example name, description, etc…)

  • Merge, prefer mine Where possible values will be added to the existing set of values (for example checks, and monitor streams). If no addition possible there is a conflict, because both merge strategies would like to use their own value. We resolve this by keeping the existing value (for example name, description, etc…). With this choice the values will end up in a stable situation (i.e. not flipping back-and-forth).

Phrased differently:

“Use theirs” is beaten by all others “Use mine” beats all others “Merge, prefer mine” beats all except “Use mine”. Conflicts with another sync that uses “Merge, prefer mine” as well are resolved according to the first-one-wins principle. “Merge, prefer theirs” only beats “Use theirs”

Merge Priorities

Merged Components accumulate all identifiers from all synchronizations. If components are synchronized from the same synchronization have the same identifiers they are also merged. If more than two components with the same identifiers found then they are sorted by priority of the merge strategy of their mapping (highest priority first) and then this sequence of components merged starting from component with the highest priority mapping (Ties are broken based on lexicographic order of synchronization name).

The order of priority is the following (highest to lowest):

UseMine MergePreferMine MergePreferTheirs UseTheirs