# Monitoring device health

This page provides insights into monitoring network, hardware and software-related statuses in the Device List section. Discover how to utilize these statuses for system configuration and maintenance purposes.

The Device List section allows you to monitor system statuses in real-time at various detail levels—system level, array level, or controller/module/processor-specific.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FrCnSJG6WJ9vx5zB7S0FL%2FControl_Maintenance_DeviceList.gif?alt=media&#x26;token=27846bc9-8741-4a72-8a22-0605dd29d345" alt=""><figcaption><p>Device List</p></figcaption></figure>

## Status breakdown

For all statuses, HOLOPLOT Control represents the severity level in 4 statuses. Here is their meaning:

<table><thead><tr><th width="148.97052001953125">Status</th><th>Arrays / Modules</th><th>Controllers / Processors</th></tr></thead><tbody><tr><td>🟢 OK</td><td>Everything is functioning correctly. Sound can be played without issues.</td><td>Everything is functioning correctly. Operation is normal.</td></tr><tr><td>🟡 Warning</td><td>Sound can be played, there might be minor issues that compromise the system’s performance. Show can go on, but should be checked later on.</td><td>Controllers/Processors still function, but minor issues are observed. The show can go on, but it should be checked later on.</td></tr><tr><td>🟠 Error</td><td>Modules can still make sound, but with severe degradation to system’s performance. Should be checked as soon as possible.</td><td>Controllers/Processors impacted, and major issues are observed. It should be checked as soon as possible.</td></tr><tr><td>🔴 Critical</td><td>Modules are severely compromised and might not make sound.</td><td>Controllers/Processors cannot function properly.</td></tr><tr><td>⚪️ Unknown</td><td>Modules are not reporting any information, or we cannot accurately report its health.</td><td>Controllers/Processors are not reporting any information, or we cannot accurately report their health.</td></tr></tbody></table>

{% hint style="info" %}
The **Unknown** status is an addition to the health statuses used when the system cannot infer the accurate status of a specific device. An example of this is when a module or controller is offline.&#x20;
{% endhint %}

***

## Module's Health

Each module component group has its own logic and calculation approaches. Each outlined component state change generates an event for the respective device.&#x20;

* **Power status**—Power statuses have no effect on the module's health (e.g., a module can be asleep and still experience a network error).
* **Network interfaces**
  * NTP
  * Ravenna and PTP (If Dante is selected as the AoIP protocol, these statuses will not be considered in the health status calculation).
* **Hardware statuses**
  * Amplifiers
  * PSUs
  * Drivers

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FZnlv2kHArxdgjKGKsAki%2FDetailModuleStatus.gif?alt=media&#x26;token=e70e9e54-fe72-4855-9a20-c01f673ee890" alt="" width="266"><figcaption><p>Module side panel</p></figcaption></figure>

### Status calculation thresholds

The all-arrays/system, individual array, and module's health statuses are derived from the aggregation of the different network and hardware statuses using the thresholds below:

<table data-full-width="false"><thead><tr><th width="95">Status</th><th width="164">% of devices ≥ 🔴</th><th width="162">% of devices ≥ 🟠</th><th width="162">% of devices ≥ 🟡</th><th>% of devices ≥ ⚪️</th></tr></thead><tbody><tr><td>🟢</td><td>–</td><td>–</td><td>–</td><td>–</td></tr><tr><td>🟡</td><td>1%</td><td>5%</td><td>5%</td><td>1%</td></tr><tr><td>🟠</td><td>10%</td><td>30%</td><td>–</td><td>30%</td></tr><tr><td>🔴</td><td>30%</td><td>50%</td><td>–</td><td>50%</td></tr></tbody></table>

#### Module's health

The module's health is evaluated based on network and hardware statuses, which are derived from the aggregation of the criteria mentioned above. The highest severity level among these statuses determines the overall health of the module's network and hardware.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FQU8CEiH0HaHOHASZ6upn%2FControl_Maintenance_Module-Health.png?alt=media&#x26;token=a8413ce8-f929-48dc-9a66-bd7b3b360ba7" alt=""><figcaption><p>Module health</p></figcaption></figure>

#### Single array's health

The single array health is an aggregation of the hardware and network statuses of the modules inside a specific array. The highest severity of its components defines the health status shown in HOLOPLOT Control.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FzndqEeeJU4opPTVddrRr%2FControl_Maintenance_Array-Health.png?alt=media&#x26;token=1d665ebc-18e2-44ab-af7f-27df7697e3df" alt="" width="375"><figcaption><p>Single Array Health</p></figcaption></figure>

#### Arrays' health or system health <a href="#arrays-health-or-system-health" id="arrays-health-or-system-health"></a>

The arrays' health is a single status derived from the health of all the system's arrays, using the thresholds mentioned above.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FH5VFiT2YlmLQp0iSn3UG%2FControl_Maintenance_System-Health.png?alt=media&#x26;token=8175473e-59a6-4cc0-889c-4d075725104b" alt=""><figcaption><p>System / Arrays' Health in the footer</p></figcaption></figure>

***

## Controller's Health

Calculating the health of controllers follows a similar approach to modules and arrays, although with different components and thresholds.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FwBxgkK6mcBESsQbxNJv6%2FControl_Maintenance_Controller-Health.png?alt=media&#x26;token=04f16dfd-39a2-4a0c-913f-fd328fe00af2" alt=""><figcaption><p>Controllers' status</p></figcaption></figure>

Each controller component group has its own logic and calculation approaches. Each outlined component state change generates an event for the respective device.&#x20;

* **Online status**
* **Network status**
  * **Connectivity**
    * HOLOPLOT Cloud
    * NTP
  * **Network**
    * Audio Module Control Network: It enables communication between the audio modules and the controllers for system management, operation, and configuration.
    * HOLOPLOT Control Network: It provides connectivity for HOLOPLOT Control and the Public System API, supporting user interaction and external system integration.
    * Audio Network: It supports Audio over IP monitoring services specifically for RAVENNA installations. It detects audio stream announcements and monitors audio streams to facilitate failover mechanisms.
    * Intercontroller Network: It facilitates the communication between HOLOPLOT controllers to synchronize their states and ensure redundancy in system operations.
* **Hardware status**
  * Drives
  * Fans
  * PSUs

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FgsqIgRyLSEw692LfN6AM%2FDetailControllerStatus.gif?alt=media&#x26;token=47fa2c0e-2b0a-45bc-8166-f42975c2cc8e" alt="" width="269"><figcaption><p>Controller side panel</p></figcaption></figure>

### Status calculation thresholds

The controller health statuses are derived from the aggregation of the different network and hardware statuses using the thresholds below:

Each module component group has its own logic and calculation approaches. Each outlined component state change generates an event for the respective device.&#x20;

<table data-full-width="false"><thead><tr><th width="95">Status</th><th width="164">% of devices ≥ 🔴</th><th width="162">% of devices ≥ 🟠</th><th width="162">% of devices ≥ 🟡</th><th>% of devices ≥ ⚪️</th></tr></thead><tbody><tr><td>🟢</td><td>–</td><td>–</td><td>–</td><td>–</td></tr><tr><td>🟡</td><td>1%</td><td>10%</td><td>10%</td><td>1%</td></tr><tr><td>🟠</td><td>10%</td><td>30%</td><td>–</td><td>30%</td></tr><tr><td>🔴</td><td>50%</td><td>80%</td><td>–</td><td>50%</td></tr></tbody></table>

{% hint style="info" %}
**Redundancy toggle**\
An additional system setting informs whether modules and controllers are expected to have redundant network connections—Primary and secondary or just primary. This information affects how modules and controllers calculate their health.

To set up redundancy, go to **Settings** › **System Settings** and select your preferred setup.
{% endhint %}

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FYb2QrqeWS36wSz40HmzX%2Fimage.png?alt=media&#x26;token=1249a444-d531-497c-8501-4364bd18f253" alt="" width="563"><figcaption><p>Redundancy settings</p></figcaption></figure>

### CONTROL Status <a href="#redundancy-toggle" id="redundancy-toggle"></a>

This status refers to the web browser's connection to the system, which is visible only in the footer.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2F3VyAEESuVc8BwgAHUI4z%2FControl_Maintenance_ControlStatus.png?alt=media&#x26;token=01034cd0-c12e-4d53-8d3c-101b8c68f1ff" alt=""><figcaption><p>CONTROL Status in the footer</p></figcaption></figure>

***

## Processor's Health

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FZkiHmCenSPhyxsQvQdmq%2FControl_Maintenance_Processor_DeviceList.png?alt=media&#x26;token=ffd3041c-7398-4ccb-b088-348f9046dea5" alt=""><figcaption></figcaption></figure>

Calculating the health of processors follows the same status calculation thresholds as that of controllers, but considers one additional status category: software.

* **Online status**
* **Software status**
  * DSP:&#x20;
    * DSP processing information
    * Beam and processor configuration status
    * Audio engine information
  * CPU Usage
* **Network status**
  * **Connectivity**
    * HOLOPLOT Cloud
    * NTP
  * **Network interfaces**
    * **Audio Module Control Network**: It enables communication between the audio modules and the controllers for system management, operation, and configuration.
    * **Audio Bridge Network Interfaces**:
      * Audio Network: It supports Audio over IP monitoring services specifically for RAVENNA installations. It detects audio stream announcements and monitors audio streams to facilitate failover mechanisms.
      * PTP information.
* **Hardware status**
  * Drives
  * Fans
  * PSUs

#### Failover

It provides information on which processor is used as the primary and backup device, ensuring that the OS can perform a Device Failover. This involves switching between two processors in a redundant pair, changing the state of an inactive processor to active, and vice versa. The failover status does not affect the overall health calculation of the processors.&#x20;

{% hint style="info" %}
**Failover**

To perform a Device Failover, a backup processor must be assigned beforehand.&#x20;

To set it up, go to **Settings** › **Processors** and select your preferred setup.
{% endhint %}

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FCv2A7GWOOQmJt09hGY6v%2FDetailProcessorStatus.gif?alt=media&#x26;token=fcc5fb1a-8fe4-449d-9e84-6305c50439c0" alt="" width="360"><figcaption></figcaption></figure>

***

## Filtering and finding issues

When navigating to the Device List, the right panel shows two collapsed cards: the filters and the issue tracker. They can be used together or independently.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2F2wOdbtUG3k8L0umPJ5du%2FControl_Maintenance_DeviceList_Filters.gif?alt=media&#x26;token=7fe9b854-e806-49c1-a56f-6e5fff25c23d" alt=""><figcaption><p>Filters &#x26; Issue Tracker</p></figcaption></figure>

### Filter

The filter has two functions:

When collapsed, it allows for monitoring of the Network and Hardware statuses.

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FsIj4A7cDyqnqExUKX7MO%2FCleanShot%202024-12-10%20at%2014.42.20%402x.png?alt=media&#x26;token=b2e6b143-111b-40c3-b259-4b7a34c59997" alt=""><figcaption><p>Collapsed Filters</p></figcaption></figure>

When expanded, it allows for filtering based on Power, Network, and Hardware statuses, enabling the quick identification of system failures during debugging.&#x20;

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2F4TB8VKV5Bdh9ARMTosLC%2FCleanShot%202024-12-10%20at%2014.51.45%402x.png?alt=media&#x26;token=df34a429-21da-4e5d-b96a-45ec37aaf01d" alt=""><figcaption><p>Expanded Filters</p></figcaption></figure>

### Issue Tracker

Whenever there's an issue affecting the network or hardware status, the specific issue will be displayed on the Issue Tracker in the following format `Type – Issue description.`

Click the issue to filter the modules with that specific issue.&#x20;

<figure><img src="https://1870956965-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FEWZvA6Ecdn48SRkbJ1C8%2Fuploads%2FQfCU43VzrdIFSnT1F6oH%2FCleanShot%202024-12-10%20at%2014.40.04%402x.png?alt=media&#x26;token=220984ee-cf23-409f-9aaa-ec8f9d54b92d" alt=""><figcaption></figcaption></figure>
