Server 4

Throttle Mode

When the Sophora Server is under high load it can throttle certain background tasks in oder to still respond quickly to client requests.

Purpose

The Sophora Server runs several background tasks that need to be dealt with while requests are still coming in from clients such as authors using the Sophora DeskClient or automatic importer processes. If the server is under high load, some of those background tasks can be throttled in order to delay client requests as less as possible. This throttle mode is explained in this page.

Indication of high load

The decision on whether to throttle background tasks or not is based on whether the system considers itself under high load. This is the case if the system's average load throughout the last minute exceeds the high load threshold. This threshold is by default:

Configuration

This threshold can be adjusted through the property highLoad of the JMX-bean com.subshell.sophora/ContentManager and through the property sophora.cpuload.throttleLoad. Changing this setting via JMX however just affects the running JVM and will not be persisted. In no case will the setting be synchronized to Sophora Replica Servers.

Monitoring

The throttling can be monitored via the JMX-bean com.subshell.sophora:Metrics/timer.throttledThreads. It exposes statistics about how often the throttling occurs and how long tasks are throttled. Additionally, the number of throttled tasks is logged by the Sophora Server every minute while throttling is active.

Throttling tasks

Background tasks commonly perform iterative tasks. When throttled, after a certain amount of iterations they pause there work for an interval between 200ms and 5s. The exact value depends on the current load - the higher the load, the longer the delay.

Background tasks using the throttle mode

Solr indexer queue

Working of the Solr indexer queue may switch to throttle mode after each handled document

Tasks for filling the Solr indexer queue

Changing select values or structure nodes might require re-indexing several documents. Background tasks determine which documents exactly are involved before adding them to the queue. These tasks check for whether switching to throttle mode after 200 handled documents.

Thumbnail pregeneration

The thumbnail pregeneration task running on Sophora Servers of type master (Primary Server) and slave (Replica Server) is queueing steps on a per document basis. The thread reading from this queue is going to throttle after every document handled.

The throttle mode however does not affect the generation of thumbnails on demand, which is performed on a per-document basis anyway.

DeletedDocumentWorker

The task for removing documents marked as deleted is throttled unter high load after each document removed from the repository.

Other tasks checking for high load

There are some background tasks that use the above indication of high load but come with their own configurable delay.

ArchiveWorker

The archive worker has a configurable delay, which is taken into account after each document added to or removed from the archive. This delay is increased if the system is under high load.

Limitations in a cloud setup

The underlying mechanics of determining the load will lead to undesired results in a cloud setup e.g. if your Sophora Servers are running in Kubernetes.

The used method OperatingSystemMXBean::getSystemLoadAverage will measure the load of the host machine on which the container is running. This will almost always make the server assume that the system is under high load, triggering the throttle mode constantly.

To prevent this, we propose to set the property sophora.cpuload.throttleLoad to some large value (e.g. 1000) whenever your Sophora Server is running in a container.

Last modified on 10/16/20

The content of this page is licensed under the CC BY 4.0 License. Code samples are licensed under the MIT License.

Icon