By Darren Giacomini

When it comes to networking, real-time applications like video conferencing and streaming video can present monumental challenges to the typical network infrastructure. Underlying network issues can have a devastating effect on the performance of a video management system (VMS). But how do you know if it’s the VMS, the camera(s) or if it’s the network? Network related issues often mask themselves as IP-camera, VMS, or archiving problems. This article will look at a systematic approach for determining root cause issues related to degraded video quality.

Degraded or Poor Video Quality:

Degraded video quality has a finite number of potential root causes. In almost every case, degraded video quality is directly related to resource saturation. The resources on a surveillance network consist of IP-cameras, network switches, network uplinks, viewing stations, database management, and archivers.

Each of these resources shares a common thread. And at the basic level, each of those items are nothing more than a purpose-built computer with limited CPU, memory, and network capacity. When any of these resources exceed their capacity the quality of service delivered will degrade. The following are common resource depletions that can degrade video quality.

Root Cause: IP-camera CPU utilization is in excess of eighty-five percent utilization

• All IP-cameras have a finite number of streams they are capable of producing. When there is a response to an event the operators will want access to those video streams. In a unicast environment, each operator requesting a stream from the camera depletes additional resources. As the CPU resources elevate, the video quality will deteriorate.

Determining CPU elevation in IP-Cameras:

• Almost all IP-cameras run some derivative of embedded Linux as their operating system. If you can get telnet or SSH access to the camera you can execute the “top” command. The top command in Linux will give you real-time CPU utilization statistics regarding CPU utilization on the IP-camera. Any camera consistently at eight-five percent utilization, or spiking above ninety percent utilization is cause for concern.
• Most camera vendor support Simple Network Management Protocol or (SNMP). Setting up “traps” (aka alerts) for SNMP will allow you to set utilization thresholds that are tracked and registered with a Network Management Station (NMS) that can provide with excellent insight as to what is happening in your enviroment.

Root Cause: CPU elevation in the Decoder or Workstation decoding the video

• Similar to IP-cameras, workstations and decoders also have a finite number of streams they are capable of decoding. As the number of active video streams increase, so does the workload associated with decoding. Some VMS vendors will reduce the workload in a multi-view environment by leveraging a secondary stream at a lower resolution and frame-rate. Regardless of the approach there is always a limit to the amount of decoding that can be processed. When the CPU elevates above eighty-five percent, video begins to lag, and VMS platforms can become unstable.

Determining CPU Elevation in Workstations and Decoders:

• The approach to detecting CPU elevation in workstations and decoders depends on the platform that is being utilized. For Microsoft Windows™ based machines and analysis of the task manager can quickly determine the load that is being processed by the CPU cores. In the case of a Linux-based decoder, the “top” command can be used to determine the real-time CPU utilization of the decoder.

Root Cause: Network Congestion or CPU Elevation in the Network Switch:

Once you have ruled out both the IP-camera and the viewing station as the source of the degraded video, careful analysis of the network resources should be conducted. Network resources induce degraded video when key resources are saturated. Elevation of the switch CPU at either the core of the network or the edge, link saturation, and buffer depletion can all lead to poor video quality.

Determining CPU Elevation in Core and Edge Switching:

• While the command syntax may differ greatly with each networking vendor, all enterprise class networking switches provide commands to assess the current utilization of the CPU as well as the history of CPU utilization. Switch CPUs that are constantly above seventy percent utilization, or spiking above eighty-five percent will typically produce poor video quality, stream instability, latent PTZ control, and VMS instability.
• In addition to command line interface (CLI), the SNMP “traps” can be setup on the switching platform and triggers can be set to alert network management systems when CPU resources rise above a certain threshold.

Determining Link Saturation in Core and Edge Network Switching:

• Determining link (network port) saturation is typically more difficult that CPU elevation. Link saturation occurs when traffic traversing the link exceeds the actual line rate the links (ports) support. This generally occurs due to traffic “flooding” the network, or a miscalculation of the required throughput of a solution.
• SNMP management software can be used to monitor the packet rate traversing a port at any given time via polling. An overall percentage utilization can be determined and displayed in the network management system for analysis.
• In addition to SNMP solutions, statistical analysis of the switch interface counters can determine both inbound and outbound packet drops or loss. Analyzing the switch interface counters for both in an out-packet drops can quickly help determine if links are saturated.

Now you can see that there are professional, technical, specific steps that can be implemented to determine what is causing dropped video, video degradation, or video “black outs.” The majority of the time degraded video is associated with resource depletion in one of these key components.

Instead of randomly pointing a finger at the VMS or the server it is running on, taking a systematic approach can actually save both time and effort, while limiting the time required to resolve customers issues.

If you have additional questions, leave a note below and we can walk through more of the importance of the network infrastructure.

About The Author
Darren Giacomini is the Director of Networking at BCDVideo and has over 16 years of networking experience.