Add-on Advanced Administration Dashboard: Documentation

Instructions on how to install, run, and use the Sophora Advanced Administration Dashboard

Table of Contents

The Sophora Advanced Administration Dashboard is a monitoring tool for all Sophora components.

This is a paid Sophora add-on. For details, please refer to our website.

Installing the Dashboard

The Administration Dashboard requires Java 8. To install the Dashboard, simply unpack the archive. The folder structure should look like depicted. In case the logs and persistence folders don't exist, they will be created automatically once the Dashboard is started.

Configuration

The Dashboard is designed to do most of its configuration automatically. However, some initial settings have to be provided and some optional ones can be added. All the configuration is done in the config/dashboard-config.json file. Below you can see an example configuration:

{
	httpPort: 8090,
	vmargs: "-Xmx512m",
	hostname: "test.subshell.com",
	defaultPollInterval: 10000,
	statusPollInterval: 1000,
	documentCountInterval: 3600000,
	server: {
		sophoraUsername: "admin",
		sophoraPassword: "secret",
		masterHostname: "trunk.subshell.com",
		masterHttpPort: 1196,
		contentApiPath: "/content-api",
		jolokiaUsername: "jolokia",
		jolokiaPassword: "secret",
		jolokiaPath: "/jolokia",
		solrUsername: "solr",
		solrPassword: "solr",
		solrPath: "/solr",
		tomcatJmx: {
			port: 9004,
			username: "",
			password: ""
		}
	},
	importer: {
		jolokiaUsername: "importerjmx",
		jolokiaPassword: "secret",
		jolokiaPath: "/jolokia"
	},
 	indexer: {
		jolokiaUsername: "admin",
		jolokiaPassword: "secret",
		jolokiaPath: "/jolokia"
	},
	mail: {
		recipients: [
			"recipient1@subshell.com",
			"recipient2@subshell.com"
		],
		smtpHost: "localhost",
		smtpPort: 25,
		username: "user",
		password: "secret",
		fromAddress: "sophora-dashboard@yourdomain.com",
		notificationDelay: 30
	},
	links: [ 		
		{ 	url: "http://www.subshell.com/de/index.html", 		 
			description: "Subshell" 
		}, 		
		{ 	url: "https://github.com/airbnb/javascript", 		 
			description: "JavaScript Style Guide" 		
		}
	]
}

The following list explains the configuration options:

Configuration Options
KeyExplanationExampleValue typeRequiredDefault
httpPortThe port at which the Dashboard will be reachable.8090Integerno8080
vmargsThe arguments that should be passed to the Java VM."-Xmx512m"Stringno""
hostnameHostname of the dashboard server. Use this if the automatic resolution of the hostname fails or the external host to reach the dashboard is different."test.subshell.com"Stringnodetermined by operating system
defaultPollIntervalThe Interval in which the Dashboard polls most of the values it aggregates from other components in milliseconds. Minimal value is 1000.20000Integerno10000
statusPollIntervalThe interval in which the Dashboard polls status (online/offline) information from other components in milliseconds. Minimal value is 100.2500Integerno1000
documentCountIntervalThe interval in which the Dashboard counts the documents on all the cluster nodes and staging slaves in Milliseconds. This is a very expensive operation, hence intervals smaller than 5 minutes (300000) are strongly discouraged.7200000Integerno3600000
serverA block of configuration applied to the handling of all Sophora Servers in the cluster (master, failover slaves, staging slaves). See the corresponding table below for all available options.JSON Objectyes
importerA block of configuration applied to the handling of all Sophora Importers in the cluster. See the corresponding table below for all available options.JSON Objectyes
indexerA block of configuration applied to the handling of all Sophora Indexers in the cluster. See the corresponding table below for all available options.JSON Objectyes
mailA block of configuration used to send email notifications. See the corresponding table below for all available options. If no mail block exists, the Dashboard does not send emails.JSON Objectno
linksA collection of configurable links that will be visible in a menu inside the dashboard's navigation bar. Each link is a JSON Object, that must consist of an url and a description. The latter will be used as the link name shown in the menu.JSON Arrayno
ibfEnabledActivates support for checking document counts efficiently using invertible bloom filters. Set this to true if you have activated the configuration option sophora.ibf.enabled in the sophora.properties of all Sophora servers monitored by this dashboard instance.trueBooleannofalse
Server Block
KeyExplanationExampleValue typeRequiredDefault
sophoraUsernameThe username that is used to poll values from the Content-API."admin"Stringyes-
sophoraPasswordThe password that is used to poll values from the Content-API."secret"Stringyes-
masterHostnameThe hostname of the sophora master server."sophora.example.com"Stringyes-
masterHttpPortThe HTTP port of the sophora master server.1196Integeryes-
contentApiPathThe part of the Content-API URL after the port."/content-api"Stringno"/content-api"
jolokiaUsernameThe username for the Jolokia service."jolokia"Stringno"jolokia"
jolokiaPasswordThe password for the Jolokia service."jolokia"Stringno"jolokia"
jolokiaPathThe part of the Jolokia URL after the port."/jolokia"Stringno"/jolokia"
solrUsernameThe username for the Solr service."solr"Stringno"solr"
solrPasswordThe password for the Solr service."solr"Stringno"solr"
solrPathThe part of the Solr URL after the port."/solr"Stringno"/solr"
tomcatJmxA block of configuration to access the JMX of any Tomcat server connected to the Sophora system.see rows belowJSON Objectnosee rows below
tomcatJmx > portThe port of the JMX service of the Tomcats (all Tomcats need to offer JMX on the same port).9004Integerno9004
tomcatJmx > usernameThe username used to access the JMX service of the Tomcats (all Tomcats need to be configured alike)."jmx-user"Stringno""
tomcatJmx > passwordThe password used to access the JMX service of the Tomcats (all Tomcats need to be configured alike)."password"Stringno""
Importer Block and Indexer Block
KeyExplanationExampleValue typeRequiredDefault
jolokiaUsernameThe username for the Jolokia service."jolokia"Stringno"jolokia"
jolokiaPasswordThe password for the Jolokia service."jolokia"Stringno"jolokia"
jolokiaPathThe part of the Jolokia URL after the port."/jolokia"Stringno"/jolokia"
Mail Block
KeyExplanationExampleValue typeRequiredDefault
recipientsA list of mail addresses that will be notified in case of problems detected by the Dashboard.["recipient1@example.com", "recipient2@example.com"]JSON Array of Stringsyes-
smtpHostThe hostname for the SMTP server used."smtp.example.com"Stringno"localhost"
smtpPortThe port for the SMTP server used.25Integerno25
userThe username used to login to the SMTP service."mail-user"Stringno-
passwordThe password used to login to the SMTP service."secret"Stringno-
fromAddressThe address used for the FROM field in sent e-mails."sophora-dashboard@yourdomain.com"Stringnodetermined automatically
notificationDelayThe delay before a notification mail is sent in case of a problem in seconds. No notification will be sent if the problem is fixed within the delay.60Integerno30

Ports

The following ports are in use for the communication between the Dashboard and Sophora’s components:

Dashboard – Master, Stagings

  • HTTP-port of the server. Also, the communication of the DeskClient, the content API and Solr run on this port.

Dashboard – Importer

  • Jolokia-port. Configurable via the option sophora.importer.jolokia.port in sophora-importer.properties. 1496 is set as the default.

Dashboard – Tomcats

  • JMX-port of the Tomcat. 9004 is set as the default in the DeskClient.

JMX in Tomcat

The Dashboard uses the JMX-URLs as follows:

service:jmx:rmi:///jndi/rmi://<HOSTNAME>:<PORT>/jmxrmi

The port and the authentication can be configured via dashboard-config.json.

In order to allow access through firewall and VPN tunnels you can configure the JmxRemoteLifecycleListener. For more information see JMX Remote Lifecycle Listener.

Web Application Configuration

Webapps/deliveries only appear in the Dashboard, if sophora.delivery.externalUrl is configured (up to Delivery 2.1.16, 2.2.6 and 2.3.0).
The configured URL need to point at the Tomcat context of the webapp and need to be accessible by the Dashboard.

More about configuration parameters.

Starting and Stopping the Dashboard Application

To start the Dashboard, use the start and stop script dashboard.sh that comes with the installation. In a console in the directory of the script, type ./dashboard.sh start or ./dashboard.sh stop.

User Interface

After the Dashboard application has been launched, the user interface is available through the browser under the host it is installed on and the configured port, for example http://www.example.com:8090/#/. This will lead you to the home view.

Home View

The home view shows groups for the three different types of Sophora components. The Sophora components are grouped by Cluster Servers, Staging Slaves and Tools. Each of these groups shows a list of tiles representing each component and a graph section showing relevant time series. If you click on a tile, the details view for this component will open on the right side of the screen. Clicking on its title will show the details view on the whole screen. For a description see here.

Tiles

Sophora servers have up to seven indicator symbols in the body of their tile, ordered in two rows. The top row shows indicators associated with the Sophora server itself, while the bottom row shows indicators for the Tomcat server connected to it (if any).

Sophora tools only have the heap-indicator.

Symbols in Tiles
Indicator symbolDescription
Sophora server heap (top row)This symbol depicts the heap state of the Sophora server or tool. It is usually green but turns red if the used amount of heap space reaches the maximum.
Sophora server loadThis symbol depicts the CPU load state of the Sophora server. It is usually green but turns yellow or red if the total load devided by the number of CPUs reaches 75% or 95% respectively.
SolrThis symbol depicts the reachability of Solr in the Sophora server. It is usually green but turns yellow if Solr does not respond to pings within a second of time.
Document countThis symbol depicts the synchronisation state of documents in the Sophora server in comparison to the master Sophora server. Thus, the master server does not show this symbol. It is usually green but turns red when the numbers of documents in this Sophora server differ from the one in the master server. Clicking on this symbol will take you to the document count view.
Tomcat cache queueThis symbol depicts the state of the cache queue of the Tomcat. It is usually green but turns red when the queue size exceeds the maximum defined (by default, that's 40000).
Tomcat heap (bottom row)This symbol depicts the heap state of the Tomcat server. It is usually green but turns red if the used amount of heap space reaches the maximum.
Tomcat threadsThis symbol depicts the number of threads that are currently busy in the Tomcat server. It is usually green but turns yellow if 33% or red if 66% of the available threads in the pool are busy.
Configuration DifferencesThe symbol is only available on slaves and staging slaves and depicts the synchronization state of the configuration.

Each tile has one main indicator symbol. For Sophora servers this takes the following states:

  • green if all other indicators of this tile are green.
  • yellow if at least one of the indicators of this tile is not green and nothing causes the main indicator to turn red.
  • red if the Sophora server is offline, the solr indicator is red or the JMX of the Tomcat (if any) does not respond.

For tools, this indicator is green unless the tool is offline.

Every tile has a checkbox in its top right corner that shows the color with which this component is represented in the graph section. If the box is unchecked, the graph for this server is hidden in the graph section.

Graph Section

In the graph section you can see the values of up to six different metrics for each component over the time. The metrics are:

  • Sopohra server heap usage
  • load
  • Solr queue size
  • size of the Tomcat cache queue
  • Tomcat heap usage and
  • number of busy threads in the Tomcat

You can switch between these metrics by clicking the respective button on the left of the graph section. Also, you can view different resolutions for every metric by clicking the corresponding button at the bottom of the graph section.

Graphs are only shown for those components whose checkbox is checked in their tile. Also, graphs concerning Tomcat values are of course only shown for servers that have a Tomcat connected.

Details View

The details view shows as a side panel if a tile on the home view is clicked. It expands to full screen if the title of this side panel is clicked.

All Components

For all Sophora components the details view shows a table with information on the selected component such as hostname, different ports and version.

The table is followed by three graphs on heap space, non-heap space and CPU load of that component. Just like the graphs in the graph sections of the home view, they can be switched to different resolutions, but unlike the ones in the home screen, these graphs additionally show committed- and maximum-values for the heap and non-heap metrics.

Buttons at the top of the details view make it easy to navigate to the desired section.

Sophora Servers

For all Sophora servers (i.e., master server, failover slaves and staging slaves) two additional metrics are shown as graphs:

  • the count of logged in users and
  • the size of the solr indexing queue.

Furthermore, for Sophora servers with a Tomcat connected the following Tomcat metrics are shown as graphs:

  • the cache queue size,
  • the heap usage and maximum and
  • the number of busy and maximum available Tomcat threads.

At last, there is a list of the most recent cache flushes on this server's connected Tomcat. The content of the items in this list is explained in the cache history view section.

Importers

For importers, the details view shows another graph with the number of files that are yet to be imported in all importer instances. Also for each importer instance an own section is shown.

In each of these sections, three more metrics for this importer instance are included as well as a listing for up to the last 1000 imports. This listing can be filtered by the import result and limited to a certain date so that only imports after the defined date show in the listing. Clicking on an import in the list opens a new tab with detailed information about this import.

Document Count View

The document count view shows a comprehensive table with differences in document counts between the Sophora servers. The view is reachable by clicking the document count symbol on one of the server tiles in the home view or by clicking on the Document Count item in the top menu. There are two modes of the document count view: Solr and JCR.

Solr-Based Document Count View

The Solr-based document count view shows the counts of documents of different document types in different sites on the different Sophora servers.

The table has a row for each configured site in the Sophora system and a column for each Sophora server. The cells contain the numbers of documents for each combination. When you hover the pointer over a cell, a tooltip will explain the numbers shown.

In the column of the master server, you can see the total number of documents per site and the number of published documents in round brackets. In every other column the numbers represent the count of documents relative to the master server. The document counts on failover slaves are compared against the total counts of the master, and the counts on staging slaves are compared against the online counts of the master.

If a count differs, the respective cell turns red and shows how many excess and missing documents have been counted. "+3/-1" in the image below means there are three documents on this server that are not on the master server and one document that is on the master server but missing on this server.

If a site is ignored on a certain server, the differences for that site are not being displayed, but the corresponding cell shows a gray dash instead.

Clicking on a row for a site will expand the view for this site and show detailed document counts per document type in that site. The cell in a site-row is the summary of all the type-row cells under it.

Clicking on a cell in a type-row leads you to the document delta view, which shows the actually differing documents.

This view is based on Solr queries. As these queries have a noticable impact on the Sophora server performance, the Dashboard back end queries each Sophora server only once per hour. The interval can be configured using the configuration option documentCountInterval.

JCR-Based Document Count View

The JCR-based document count view is based on "invertible bloom filters", which each Sophora server updates continously using data from the Jackrabbit document repository. The filters have the advantage of being very cheap to query and transmit, and in this mode, the Dashboard back end updates the document counts every 30 seconds. The disadvantage of this mode is that the document count view only shows the differences for each site and document type, not absolute numbers. Also, the difference count can not be shown for a server if the combined difference for all sites and document types exceeds ca. 5000 documents.

The JCR-based document count view must be enabled in the configuration of the Dashboard and the Sophora servers (see below).

Documents Counts - JCR view

Activating the JCR-based Document Count View

The invertible bloom filters supporting the JCR-based document count view must be enabled in the configuration files of the dashboard and all Sophora servers. In the dashboard-config.json, enable the configuration option ibfEnabled. In the sophora.properties of the Sophora servers, enable the configuration option sophora.ibf.enabled. You will then see a toggle button in the document count view for switching between the Solr- and JCR-based views.

The invertible bloom filters may possibly be enabled by default in future Sophora versions.

Document Delta View

Clicking on a table cell in the document count view opens the document delta view. This view lists all documents for a specific document type and site on two Sophora servers that are only present on one of the two servers. Each document is represented by a tile that shows its most important information. Clicking on one of those document tiles opens the document details view for this document.

Document Delta View

Document Details View

In the document details view the state of a single document on the different Sophora servers can be looked up. First, you will only see a search bar that takes a Sophora-ID. Upon entering a Sophora-ID, information about that document is presented in a table. Like in the document count view, every Sophora server has its own column. The row of the master server shows the most important information on the document, while the colums for the other servers merely show a green tick as long as the value for that property does not differ from the value on the master server. If it does differ, a red cross is shown and hovering the pointer over this icon brings up a tooltip with the differing value. Since the search term is reflected in the URL, the link to a specific document details view can be copied and, for instance, sent via e-mail.

Cache History View

In the cache history view you can check cache flush orders across all the Tomcats. First you will only see a search bar that takes a UUID, a flush key or a resource name. Upon entering one of the above, for each Sophora server with a connected Tomcat a listing is shown. Each of these listings contains the cache flushes that match the given search term. Since the search term is reflected in the URL, the link to a specific cache history view can be copied and, for instance, sent via e-mail.

If you open the cache history view through clicking on an entry in a server's details view, the view will be opened with a pre-defined search term. This way you can check if the cache flush for this order has been performed on all Tomcats across the Sophora system.

Email Notification Service

If the configuration contains a mail section, the Dashboard application will send mails whenever the main indicator for any tile remains red for longer than the configured time period. The mail gets sent to all configured recipients' email addresses and contains an explanation of the problem.

Configuration Differences

This view provides an overview of differences regarding the configuration between a slave and its master server. The view is accessible using the icon "Configuration Differences" in the tile of a server on the overview page. Usually, there should not be any differences. But if there are differences, they are depicted here in detail and the view provides the necessary information to correct the situation. Since each difference must be the result of a synchronization exception between the servers, the cause of the exception should be investigated.

The view provides one section for each of the configuration entities. Each difference is visualized by a side by side difference view comparing the slave's and master's current version of the entity.

A slave with configuration differences should not be propagated to the master server in a cluster. The differences must be manually resolved prior to a cluster switch. A concrete configuration difference can be resolved by editing the entity using the DeskClient. Saving the configuration entity triggers its synchronization to the slaves.

The following configuration entities are compared for the different server modes:

Master:
Is the reference for the comparison and is therefore always in sync.

None:
A server in the mode replication mode "none" is not connected to a master server. Hence, no configuration differences can be determined.

Slave:

  • CNDs
  • Default Property Configurations
  • Default Childnode Configurations
  • Node Type Configurations
  • Form Field Groups
  • Document Proposal Sections
  • Roles
  • Users
  • Structure Nodes
  • Structure Node Documents
  • All System Documents


Staging Slave:

  • Default Property Configurations
  • Default Childnode Configurations
  • Node Type Configurations
  • Form Field Groups
  • Roles