Monitoring information details (getcsv)

The Server Monitoring and Statistics features are available from a Wavestore Server both locally and from a central monitoring station.  This KB article provides detailed technical information for users of raw spreadsheet data and developers of visualisation tools.

Three types of information are available:

  • System Information - updated rarely, usually when it changes
  • Current status - updated regularly (usually every minute)
  • Historical statistics

Obtaining information

Directly from command line

Connect to the Wavestore and Execute Command "getcsv" or "getcsvq" commands for the current status and historical information. Execute Command "sysinfo" for the system information.

By remote monitoring

If the Wavestore is uploading to the central monitoring server, you can connect to the central monitoring server, log in, and issue commands to obtain data either in csv or JSON format. See JSON section below for details.

Information in CSV format

The most compact format for information from the server is in CSV (comma separated variable) format which can be imported into most spreadsheets.

It consists of a title line (which is repeated occasionally, at least monthly, and always when it changes) which contains the column headings in short format.

Then there are a number of entry lines with the statistical data.

Some columns might appear or disappear, for example a column for camera 6 is only present if there is a camera 6 on the system.

All data items are numeric except for Date, Time and Disk_X (disk status letter for some disk X) which can be treated as strings.

All numeric items are non-negative except for ntp_ms which is signed. (Note also that some temperature fields might be negative in exceptional cases, but this is very rare).

Status is a status code and might need interpretation. See below for more details.

Column headings

Short name  Description A  Notes
Date Date   YYYYMMMDD (string)
Time Time   HH:MM:SS (string) in UTC
Status System Status   Server Status Code (see below)
StatusC Camera Aggregate Status   Camera Status Code (see below)
CamG Count of Good cameras   count
CamM Count of cameras in maintenance state     count
CamW Count of cameras in warning state   count
CamF Count of cameras in failed state   count
Memory      
UsedMB System Used Memory   MegaBytes
FreeMB System Free Memory   MegaBytes
CPU      
CPU% CPU % utilisation (all cores)   up to 400 for 4 cores
cpun% CPU % utilisation (for core n)   one for each core
Ntp      
ntp_ms Wavestore time offset from NTP time   signed
stratum NTP Stratum   1..16
Network interfaces (all begin e or b)
if_tx Network transmit rate for interface if   Mbit/s
if_rx Network receive rate for interface if   Mbit/s
if_err Network errors recorded for if   count
if_drop Network packets dropped for if   count
Disks
Disk_x Disk volume Status * See Disk Status Letter
Use%_x Disk volume usage   0..100%
Temp_x Disk volume temperature   Degrees C
WriteRate_x Disk volume write rate   KBytes/s
Disk_xy Disk array x disk y Status * See Disk Status Letter  
Diskxysa Disk xy Smart attribute a value * normalised (low is worse)
Diskxywa Disk xy Smart attribute a worst   normalised (low is worse)
Temp_xy Disk array x disk y Temperature   Degrees C
Cameras
C<n> Camera <n> status   Camera Status Code (see below)
C<n>rx Camera <n> receive rate   Mbit/s
C<n>rj Camera <n> realtime jitter    
C<n>ij Camera <n> image jitter    
Cameras - Tracks per camera
Cnttb Camera n track t record bitrate   KByte/s
Cnttd Camera n track t daily bitrate   MByte/day
Cnttr Camera n track t recording duration * days
Sensors (see below for details)
FAN... FAN sensor * RPM
PSU... PSU sensor * usually just ok or fail
.../C Sensor in degrees_C * Degrees C
.../V Sensor in Volts * Volts
.../W Sensor in Watts * Watts
... other Sensor *  

Column A is set to * if an "alert" can be marked using a suffix * (star character) on this data item. Alerts are SMART failure for disk, Sensor out of range for Sensor, or not recording for desired period for recordings.  For example, if a 5V voltage rail has failed and is showing 2.3V, we will see 2.3* in that data item, meaning a value of 2.3 and an alert flag.  This allows a reporting device to flag out-of-spec values without needing to know valid ranges for every value.

Information in this table is correct for Wavestore 6.40. A number of additions have been made since 6.30.

Server Status Code

  • 0 Unlicensed
  • 1 Awaiting restart
  • 2 Licence valid but not recording (eg not enabled)
  • 3 Recording (Licence valid and in recording mode)
  • 4 (unused)
  • 5 (unused)
  • 6 Fault
  • 7 Warning
  • 8 Standby
  • 9 Standby2
  • 10 Starting
  • 99 Failed 

You might also see (from the central monitoring server):

  • 20 Maintenance / No servers in region
  • 98 Missing (no recent contact from Server)

Usually these are represented by colours: 6, 98, 99  = RED;  3, 8, 9 = GREEN;  20=BLUE;  others YELLOW.  10 Starting state was added in 6.34.

Disk Status Letter

  • F = formatted (legacy)
  • N = not formatted
  • G = good
  • L = good, read only
  • R = rebuilding (disk in array)
  • S = spare (hot spare)
  • E = foreign
  • M = missing
  • U = unlicensed
  • X = failed

If the disk is spun down to save power, it will be in lower case and for good states these are usually shown with a light-green colour.

Usually G,F,L are GREEN, g,f,l are LIGHT-GREEN;  X,M,x,m are RED,  and others are BLUE.

Disk Smart Attributes

Smart attributes are numbered 1 to 255 and the meanings might vary. Some have conventional names. The details of SMART attributes are outside the scope of this document.  See also the appendix below.

Values are normalised to 100 being normal (usually 100, sometimes 200) and lower values being worse.  The current and worst value recorded are available. Thresholds for these values are available in the system info (see below).

Camera Status Code

  • OK = 0
  • RECORDING = 1
  • MAINTENANCE = 2
  • NOT-ENABLED = 4
  • DARKENING = 16
  • CAMERAMOVE = 17
  • UNLICENSED = 21
  • VIDEOLOSS = 22
  • INITIALIZING = 27
  • UNAUTHORIZED = 28
  • NOTCONNECTED = 29
  • FAILED = 30
  • SERVER-AWAITING-RESTART = 31
  • CHECK-SYSTEM-LOG = 32
  • NETWORK-PROBLEM = 33
  • CONFIGURING-CAMERA = 34
  • CONNECTING = 35
  • CONNECTED-STARTING-STREAM = 36
  • NO-LIVE-STREAM = 37

Categories:

  • OK and RECORDING are categorised as GREEN for OK.
  • DARKENING/CAMERAMOVE are shown as YELLOW and counts as camera in Warning state
  • INITIALIZING/CONFIGURING/CONNECTING/CONNECTED-STARTING-STREAM are usually YELLOW for Warning as well
  • MAINTENANCE is BLUE
  • others are RED for Fault state.

NOT-ENABLED and unknown status are sometimes shown as grey or white, although cameras which are not enabled do not appear in the list so you should never see code 4 in monitoring information.  

NTP Stratum

  • 1..14 - good (usually show as GREEN on monitoring): show ntp_ms for time offset, eg "+3 ms"
  • 15 - manually set clock (non-NTP) (show as AMBER on monitoring)
  • 16 - unsynchronised (show as RED on monitoring)

Note: if stratum is in range 1..14 but time offset is over 5000 or under -5000 then show as amber.

Sensors

  • Sensors use names provided by IPMI or LMSensors or SES, possibly renamed.
  • Sensors are suffixed /C for degrees_C, /V for Volts, and /W for Watts
  • PSU sensors will usually be named starting PSU, many will simply indicate ok or fail
  • FAN sensors will usually be named starting FAN, will show an RPM and indicate ok or fail

Interpreting the data in each column

Most items are non-negative numeric values; a few require special handling

  • Date and Time should be handled specially
  • Status is a Server Status code
  • ntp_ms is a signed numeric parameter.
  • Disk_<l>[n]: prefix "Disk_" with text after: Disk Status letter
  • C<n>: C followed by number alone (no subsequent letter): Camera Status code
  • Any other case: numeric (non-negative)

 

System information format

Certain items of system information are required by the remote monitoring application.  These are now transferred in JSON format.

System information is provided to the cloud monitoring service periodically, this is refreshed on first connection and after any software upgrade or if any of the parameters change.

Items of system information transferred are:

Short name  Description Notes
SysVer Server venusd version and build  eg, 6.28.152
Name Server Name Configured server name
Distro Distro string eg, Wavestore-el7a-6.28.17
Model Server Model eg X-Series
MB Motherboard name  
IP IP addresses eg, 10.1.9.105 [fd01::9105] 10.2.1.2
MAC MAC addresses eg, aa:bb:cc:dd:11:22 aa:bb:cc:dd:11:23
Net Detailed network information  See below
Disk Detailed disk information See below
Camera Detailed camera information See below
MID
Machine ID
as used for licensing
LicFlags Licence flags eg, e0 i200 f9EFF v6
RMon Remote monitoring N or time interval (minutes)
  • All items are strings; history is not recorded for these items.
  • There was also a suggested Serial number ("Build ID") but that's not available to software
  • There was also Licence Status but it's unclear what is needed there and it might be better as a regularly updated field than in this static system info.
  • Note that more entries might be added in future

These are distributed as JSON and a number of new items have been added in 6.40

Detailed Network information

The above IP and MAC are not very useful if there are multiple interfaces. The "Net" object contains more detail. "Net" contains a JSON object with one ore more (interface name) objects each containing:

Field Description Notes
ip ip address(es) for this interface     omitted if slave
mac mac address for this interface
master   master interface name only if slave

 

Detailed Disk information

The "Disk" JSON object contains a lot of disk information is available, indexed by disk volume name (eg, D) or disk name (eg, D1).

Also the Enclosure geometry is shown as "Enc1" for the first enclosure.   Slots in the first enclosure are numbered s01 to s24 (for a 24 disk enclosure) and read from right to left like words on a page.   For the second enclosure (if any, rare) the slots have an e2 prefix, eg s2s01.  Sometimes there is no enclosure information so the Enc1 entry will be omitted.

For Enclosures:

Field Description
Encn     Physical disk layout for enclosure n, if available.  This is of form Height x Width, eg, 6x4

 

For each disk slot (eg s01) (if available, from V6.46):

Field Description
Family Family name (via lookup from model information)
Model      Disk Model from disk information
Serial Disk Serial number from disk information

 

For disk volume (eg D)

Field Description
GB size of disk volume in gigabytes
Geom     geometry of disk array as N+P with groups separated with comma, eg, 2+1 or 1,1,1

 

For specific disk (eg, D1)

Field Description
GB size of disk in gigabytes
Slot        disk slot name or other disk identifier (eg, s01)
ta Disk SMART attribute a threshold (normalised)

 

Example:
{
   "Enc1": "6x4",
   "s01": {"Family":"Seagate-Skyhawk","Model":"ST4000VX007-2DT166","Serial":"WDH26HB7"},
   "D":  {"GB":"8001","Geom":"2+2"},
   "D1": {"Slot":"s01","GB":"4000","t1":"44","t3":"0","t4":"20"}
}

Detailed Camera information

Field Description
Max Maximum camera number
Cn Name of camera n
Example:
{
   "Max":  "2",
   "C1":   "Camera 1 Restream",
   "C2":   "Axis Camera"
}

Note that if only two cameras, camera 4 and camera 5, are enabled then "Max" value would be 5.  It is the maximum camera number in use, not the number of enabled cameras.

Appendix: CSV Information in JSON format

A variety of methods are available to access the information in JSON format.  The full documentation for this is elsewhere.

getdata1 / getdata: All data in one column and no categories:

The same information which is available in CSV format is available in JSON format from cloud services. Each entry contains a heading and a list of values which is equivalent to the spreadsheet column.

{
  "Date": [ "20201201", "20201201" ],
  "Time": [ "00:00:00", "00:15:00" ],
  "bond0_rx": [ 176.875, 185.03 ],
  ...
}

For items where no value is available at a specific time, null is used.

{
  "bond0_rx": [ null, 185.03 ]
}

All items are numeric except Date, Time, and Disk status characters, and items with "*" suffix alert. A column with no entry is omitted, a column with some entries will always have a Date and Time but might have null for some values.

{
  "FAN1": [ 200, "25*" ]
}

See above for details of the heading names; they are the same in JSON and CSV.

getdata1x / getdatax: one item of data for each entry, and divided into categories (Category names are the bold headings in the Column Names table above: Memory CPU Ntp Network Disks Sensors):

{
  "Date": "20201201",
  "Time": "00:00:00",
  "Network" {
       "bond0_rx": "176.875",
       ...
   }
   ...
}

Appendix: SMART attribute names

The SMART attribute names are not well defined, but there are some conventional names which are listed here, with a brief description intended to fit in a small 10-character column.

1: ReadErrors
2: Performanc
3: SpinUpTime
4: StartCount
5: ReallocSec
6: ReadMargin
7: SeekErrors
8: Seek-Time 
9: PowerOnHrs
10: Spin-Retry
11: CalibRetry
12: PowrCycles
13: RdSoftErrs
173:LifeLeftTo
177:LifeLeftSa
184:End2EndErr
187:UnfixedErr
188:CmdTimeout
189:HighFlyWri
190:AirTempera
191:SenseError
192:Retracts  
193:LoadCycles
194:Temperatur
195:ECCRecover
196:ReallocEvt
197:BadPending
198:BadSectors
199:CRC-Errors
200:WriteError
202:DataAMkErr
231:LifeOrTemp
233:LifeRemain
254:FreeFallEv