Monitoring information details (getcsv)
The Server Monitoring and Statistics features are available from a Wavestore Server both locally and from a central monitoring station. This KB article provides detailed technical information for users of raw spreadsheet data and developers of visualisation tools.
Three types of information are available:
- System Information - updated rarely, usually when it changes
- Current status - updated regularly (usually every minute)
- Historical statistics
Obtaining information
Directly from command line
Connect to the Wavestore and Execute Command "getcsv" or "getcsvq" commands for the current status and historical information. Execute Command "sysinfo" for the system information.
By remote monitoring
If the Wavestore is uploading to the central monitoring server, you can connect to the central monitoring server, log in, and issue commands to obtain data either in csv or JSON format. See JSON section below for details.
Information in CSV format
The most compact format for information from the server is in CSV (comma separated variable) format which can be imported into most spreadsheets.
It consists of a title line (which is repeated occasionally, at least monthly, and always when it changes) which contains the column headings in short format.
Then there are a number of entry lines with the statistical data.
Some columns might appear or disappear, for example a column for camera 6 is only present if there is a camera 6 on the system.
All data items are numeric except for Date, Time and Disk_X (disk status letter for some disk X) which can be treated as strings.
All numeric items are non-negative except for ntp_ms which is signed. (Note also that some temperature fields might be negative in exceptional cases, but this is very rare).
Status is a status code and might need interpretation. See below for more details.
Column headings
| Short name | Description | A | Notes | |
|---|---|---|---|---|
| Date | Date | YYYYMMMDD (string) | ||
| Time | Time | HH:MM:SS (string) in UTC | ||
| Status | System Status | Server Status Code (see below) | ||
| StatusC | Camera Aggregate Status | Camera Status Code (see below) | ||
| CamG | Count of Good cameras | count | ||
| CamM | Count of cameras in maintenance state | count | ||
| CamW | Count of cameras in warning state | count | ||
| CamF | Count of cameras in failed state | count | ||
| Memory | ||||
| UsedMB | System Used Memory | MegaBytes | ||
| FreeMB | System Free Memory | MegaBytes | ||
| CPU | ||||
| CPU% | CPU % utilisation (all cores) | up to 400 for 4 cores | ||
| cpun% | CPU % utilisation (for core n) | one for each core | ||
| Ntp | ||||
| ntp_ms | Wavestore time offset from NTP time | signed | ||
| stratum | NTP Stratum | 1..16 | ||
| Network interfaces (all begin e or b) | ||||
| if_tx | Network transmit rate for interface if | Mbit/s | ||
| if_rx | Network receive rate for interface if | Mbit/s | ||
| if_err | Network errors recorded for if | count | ||
| if_drop | Network packets dropped for if | count | ||
| Disks | ||||
| Disk_x | Disk volume Status | * | See Disk Status Letter | |
| Use%_x | Disk volume usage | 0..100% | ||
| Temp_x | Disk volume temperature | Degrees C | ||
| WriteRate_x | Disk volume write rate | KBytes/s | ||
| Disk_xy | Disk array x disk y Status | * | See Disk Status Letter | |
| Diskxysa | Disk xy Smart attribute a value | * | normalised (low is worse) | |
| Diskxywa | Disk xy Smart attribute a worst | normalised (low is worse) | ||
| Temp_xy | Disk array x disk y Temperature | Degrees C | ||
| Cameras | ||||
| C<n> | Camera <n> status | Camera Status Code (see below) | ||
| C<n>rx | Camera <n> receive rate | Mbit/s | ||
| C<n>rj | Camera <n> realtime jitter | |||
| C<n>ij | Camera <n> image jitter | |||
| Cameras - Tracks per camera | ||||
| Cnttb | Camera n track t record bitrate | KByte/s | ||
| Cnttd | Camera n track t daily bitrate | MByte/day | ||
| Cnttr | Camera n track t recording duration | * | days | |
| Sensors (see below for details) | ||||
| FAN... | FAN sensor | * | RPM | |
| PSU... | PSU sensor | * | usually just ok or fail | |
| .../C | Sensor in degrees_C | * | Degrees C | |
| .../V | Sensor in Volts | * | Volts | |
| .../W | Sensor in Watts | * | Watts | |
| ... | other Sensor | * | ||
Column A is set to * if an "alert" can be marked using a suffix * (star character) on this data item. Alerts are SMART failure for disk, Sensor out of range for Sensor, or not recording for desired period for recordings. For example, if a 5V voltage rail has failed and is showing 2.3V, we will see 2.3* in that data item, meaning a value of 2.3 and an alert flag. This allows a reporting device to flag out-of-spec values without needing to know valid ranges for every value.
Information in this table is correct for Wavestore 6.40. A number of additions have been made since 6.30.
Server Status Code
- 0 Unlicensed
- 1 Awaiting restart
- 2 Licence valid but not recording (eg not enabled)
- 3 Recording (Licence valid and in recording mode)
- 4 (unused)
- 5 (unused)
- 6 Fault
- 7 Warning
- 8 Standby
- 9 Standby2
- 10 Starting
- 99 Failed
You might also see (from the central monitoring server):
- 20 Maintenance / No servers in region
- 98 Missing (no recent contact from Server)
Usually these are represented by colours: 6, 98, 99 = RED; 3, 8, 9 = GREEN; 20=BLUE; others YELLOW. 10 Starting state was added in 6.34.
Disk Status Letter
- F = formatted (legacy)
- N = not formatted
- G = good
- L = good, read only
- R = rebuilding (disk in array)
- S = spare (hot spare)
- E = foreign
- M = missing
- U = unlicensed
- X = failed
If the disk is spun down to save power, it will be in lower case and for good states these are usually shown with a light-green colour.
Usually G,F,L are GREEN, g,f,l are LIGHT-GREEN; X,M,x,m are RED, and others are BLUE.
Disk Smart Attributes
Smart attributes are numbered 1 to 255 and the meanings might vary. Some have conventional names. The details of SMART attributes are outside the scope of this document. See also the appendix below.
Values are normalised to 100 being normal (usually 100, sometimes 200) and lower values being worse. The current and worst value recorded are available. Thresholds for these values are available in the system info (see below).
Camera Status Code
- OK = 0
- RECORDING = 1
- MAINTENANCE = 2
NOT-ENABLED = 4- DARKENING = 16
- CAMERAMOVE = 17
- UNLICENSED = 21
- VIDEOLOSS = 22
- INITIALIZING = 27
- UNAUTHORIZED = 28
- NOTCONNECTED = 29
- FAILED = 30
- SERVER-AWAITING-RESTART = 31
- CHECK-SYSTEM-LOG = 32
- NETWORK-PROBLEM = 33
- CONFIGURING-CAMERA = 34
- CONNECTING = 35
- CONNECTED-STARTING-STREAM = 36
- NO-LIVE-STREAM = 37
Categories:
- OK and RECORDING are categorised as GREEN for OK.
- DARKENING/CAMERAMOVE are shown as YELLOW and counts as camera in Warning state
- INITIALIZING/CONFIGURING/CONNECTING/CONNECTED-STARTING-STREAM are usually YELLOW for Warning as well
- MAINTENANCE is BLUE
- others are RED for Fault state.
NOT-ENABLED and unknown status are sometimes shown as grey or white, although cameras which are not enabled do not appear in the list so you should never see code 4 in monitoring information.
NTP Stratum
- 1..14 - good (usually show as GREEN on monitoring): show ntp_ms for time offset, eg "+3 ms"
- 15 - manually set clock (non-NTP) (show as AMBER on monitoring)
- 16 - unsynchronised (show as RED on monitoring)
Note: if stratum is in range 1..14 but time offset is over 5000 or under -5000 then show as amber.
Sensors
- Sensors use names provided by IPMI or LMSensors or SES, possibly renamed.
- Sensors are suffixed /C for degrees_C, /V for Volts, and /W for Watts
- PSU sensors will usually be named starting PSU, many will simply indicate ok or fail
- FAN sensors will usually be named starting FAN, will show an RPM and indicate ok or fail
Interpreting the data in each column
Most items are non-negative numeric values; a few require special handling
- Date and Time should be handled specially
- Status is a Server Status code
- ntp_ms is a signed numeric parameter.
- Disk_<l>[n]: prefix "Disk_" with text after: Disk Status letter
- C<n>: C followed by number alone (no subsequent letter): Camera Status code
- Any other case: numeric (non-negative)
System information format
Certain items of system information are required by the remote monitoring application. These are now transferred in JSON format.
System information is provided to the cloud monitoring service periodically, this is refreshed on first connection and after any software upgrade or if any of the parameters change.
Items of system information transferred are:
| Short name | Description | Notes | |
|---|---|---|---|
| SysVer | Server venusd version and build | eg, 6.28.152 | |
| Name | Server Name | Configured server name | |
| Distro | Distro string | eg, Wavestore-el7a-6.28.17 | |
| Model | Server Model | eg X-Series | |
| MB | Motherboard name | ||
| IP | IP addresses | eg, 10.1.9.105 [fd01::9105] 10.2.1.2 | |
| MAC | MAC addresses | eg, aa:bb:cc:dd:11:22 aa:bb:cc:dd:11:23 | |
| Net | Detailed network information | See below | |
| Disk | Detailed disk information | See below | |
| Camera | Detailed camera information | See below | |
| MID |
|
as used for licensing | |
| LicFlags | Licence flags | eg, e0 i200 f9EFF v6 | |
| RMon | Remote monitoring | N or time interval (minutes) |
- All items are strings; history is not recorded for these items.
- There was also a suggested Serial number ("Build ID") but that's not available to software
- There was also Licence Status but it's unclear what is needed there and it might be better as a regularly updated field than in this static system info.
- Note that more entries might be added in future
These are distributed as JSON and a number of new items have been added in 6.40
Detailed Network information
The above IP and MAC are not very useful if there are multiple interfaces. The "Net" object contains more detail. "Net" contains a JSON object with one ore more (interface name) objects each containing:
| Field | Description | Notes |
|---|---|---|
| ip | ip address(es) for this interface | omitted if slave |
| mac | mac address for this interface | |
| master | master interface name | only if slave |
Detailed Disk information
The "Disk" JSON object contains a lot of disk information is available, indexed by disk volume name (eg, D) or disk name (eg, D1).
Also the Enclosure geometry is shown as "Enc1" for the first enclosure. Slots in the first enclosure are numbered s01 to s24 (for a 24 disk enclosure) and read from right to left like words on a page. For the second enclosure (if any, rare) the slots have an e2 prefix, eg s2s01. Sometimes there is no enclosure information so the Enc1 entry will be omitted.
For Enclosures:
| Field | Description |
|---|---|
| Encn | Physical disk layout for enclosure n, if available. This is of form Height x Width, eg, 6x4 |
For each disk slot (eg s01) (if available, from V6.46):
| Field | Description |
|---|---|
| Family | Family name (via lookup from model information) |
| Model | Disk Model from disk information |
| Serial | Disk Serial number from disk information |
For disk volume (eg D)
| Field | Description |
|---|---|
| GB | size of disk volume in gigabytes |
| Geom | geometry of disk array as N+P with groups separated with comma, eg, 2+1 or 1,1,1 |
For specific disk (eg, D1)
| Field | Description |
|---|---|
| GB | size of disk in gigabytes |
| Slot | disk slot name or other disk identifier (eg, s01) |
| ta | Disk SMART attribute a threshold (normalised) |
Example:
{
"Enc1": "6x4",
"s01": {"Family":"Seagate-Skyhawk","Model":"ST4000VX007-2DT166","Serial":"WDH26HB7"},
"D": {"GB":"8001","Geom":"2+2"},
"D1": {"Slot":"s01","GB":"4000","t1":"44","t3":"0","t4":"20"}
}
Detailed Camera information
| Field | Description |
|---|---|
| Max | Maximum camera number |
| Cn | Name of camera n |
Example:
{
"Max": "2",
"C1": "Camera 1 Restream",
"C2": "Axis Camera"
}
Note that if only two cameras, camera 4 and camera 5, are enabled then "Max" value would be 5. It is the maximum camera number in use, not the number of enabled cameras.
Appendix: CSV Information in JSON format
A variety of methods are available to access the information in JSON format. The full documentation for this is elsewhere.
getdata1 / getdata: All data in one column and no categories:
The same information which is available in CSV format is available in JSON format from cloud services. Each entry contains a heading and a list of values which is equivalent to the spreadsheet column.
{
"Date": [ "20201201", "20201201" ],
"Time": [ "00:00:00", "00:15:00" ],
"bond0_rx": [ 176.875, 185.03 ],
...
}
For items where no value is available at a specific time, null is used.
{
"bond0_rx": [ null, 185.03 ]
}
All items are numeric except Date, Time, and Disk status characters, and items with "*" suffix alert. A column with no entry is omitted, a column with some entries will always have a Date and Time but might have null for some values.
{
"FAN1": [ 200, "25*" ]
}
See above for details of the heading names; they are the same in JSON and CSV.
getdata1x / getdatax: one item of data for each entry, and divided into categories (Category names are the bold headings in the Column Names table above: Memory CPU Ntp Network Disks Sensors):
{
"Date": "20201201",
"Time": "00:00:00",
"Network" {
"bond0_rx": "176.875",
...
}
...
}
Appendix: SMART attribute names
The SMART attribute names are not well defined, but there are some conventional names which are listed here, with a brief description intended to fit in a small 10-character column.
1: ReadErrors
2: Performanc
3: SpinUpTime
4: StartCount
5: ReallocSec
6: ReadMargin
7: SeekErrors
8: Seek-Time
9: PowerOnHrs
10: Spin-Retry
11: CalibRetry
12: PowrCycles
13: RdSoftErrs
173:LifeLeftTo
177:LifeLeftSa
184:End2EndErr
187:UnfixedErr
188:CmdTimeout
189:HighFlyWri
190:AirTempera
191:SenseError
192:Retracts
193:LoadCycles
194:Temperatur
195:ECCRecover
196:ReallocEvt
197:BadPending
198:BadSectors
199:CRC-Errors
200:WriteError
202:DataAMkErr
231:LifeOrTemp
233:LifeRemain
254:FreeFallEv