Recording to AWS S3 storage

To record from the Wavestore to Amazon AWS S3 storage, you will need a Wavestore and an Amazon AWS account.  Amazon S3 (simple storage service) is a simple "blob" based storage service which is one of the more cost effective options for storing VMS data.

You can configure the AWS S3 cloud storage like a disk and give it a disk letter, then choose what is to be recorded to the cloud and what is to be recorded locally simply by selecting appropriate disk letter.

Steps to set up:

First, create the AWS Bucket:

  1. Log into your AWS account using a browser
  2. Select the S3 service
  3. Create the S3 storage bucket by selecting Create Bucket
  4. Select a region and give the bucket a name
  5. Disable public access on the bucket
  6. Select IAM to create an access token
  7. Create a user with AmazonS3FullAccess
  8. Under Security Credentials tab, Create Access Key
  9. Copy the Access Key and Secret to use when you log into the bucket

Now, configure the Wavestore to access the bucket:

  1. Log in using WaveView
  2. Select Setup > Server > Storage screen
  3. Select Add Network Storage
  4. Select type: AWS
  5. Choose how large the storage should be, eg, 500GB
  6. Enter the Region and Bucket name you used when creating the bucket
  7. Enter the Access Key and Secret security credentials
  8. Save
  9. Assign a disk letter to the cloud storage
  10. Save

Now your AWS S3 Bucket acts as a disk and you can read and write to it from the one Wavestore.

Note: Ensure only one Wavestore can access the Bucket at any one time or the indexes will be overwritten.

If you have multiple Wavestores, create multiple buckets, one for each Wavestore.  Multiple buckets can be created on a single AWS account.

It's acceptable to disable a cloud disk "bucket" from one Wavestore (remove the disk letter) and then add the same bucket to another Wavestore.  Use the same bucket size, name, region, key and secret and the old footage will then be accessible on the new Wavestore and new footage can be written just as if it was a physical disk which was moved over between the two machines.

Note that the region field can contain just an AWS region, in which case .amazonaws.com is appended automatically, or it can contain a .domain.com suffix as well in which case that domain is used instead of .amazonaws.com.

 

How it works

The S3 blob storage is treated by LASS just like a disk.  Like a disk which has a master index, some low level indexes, and data areas where the video (and other) footage is written, blobs are created for these.  LASS packs data into "disk blocks" as if it was writing to disk, then uploads to S3 as a blob.  These blocks are cached in the local server cache (and the oldest ones freed when the cache is full) so we do not need to access the blob storage to get recently uploaded blobs.

Like a disk, indexes are written and updated regularly.  These are small compared with the data, but the writes are fairly frequent (every 2 seconds for the master index).  Like a disk, low level indexes which are not up to date (incomplete) on startup are completed automatically by checking the actual footage recorded.

Blob storage behaves just like a disk when reading, too.  If the block is not in the cache, it is read into the cache and video streamed as usual.  The blob read can be fast if the blob storage supports fast block reads: usually this is less than 100ms for AWS S3.  Usually the block read size is 256KB, just like a disk.

 

Q & A

  • Q: Does Wavestore natively support S3-compatible or other object storage as a video archive target, or is a separate gateway, appliance, or third-party integration required?

This is a native implementation within the Wavestore Server.

  • Q: Which public cloud and object-storage platforms are officially supported (Azure, AWS, IBM, etc)?

AWS and Azure are officially supported, but many third party devices are AWS S3 compatible and can be configured as AWS.

  • What is the typical or configurable video segment/blob size?

The blobs are typically 1MB.  This is a good balance between uploading regularly (so as not to lose data if an outage occurs) and not incurring too much upload cost.

  • Q: is the Blob upload block/chunk size configurable?

Currently (in V6.52) it is not user configurable.

  • Q: Does Wavestore use Put Blob or multipart/block uploads using Put Block and Put Block List?

The server uploads single blocks, not multipart.  With multipart, an outage (which prevents all parts being uploaded) loses the entire blob.

  • Q: When archived video is requested, is it streamed directly from object storage, downloaded to a local cache first, or restored in full before playback?

The S3 storage is treated exactly like a disk.  Just like a disk, data is read into the local cache and played from there.

  • Q: how much cache capacity is recommended?

The disk cache uses a third of the total memory for Wavestore Servers with 16GB or more RAM.  This is sufficient, and is not typically user configurable.

  • Q: Can playback begin before the entire segment is downloaded?

Yes.  Just like reading a disk, the data is read in small blocks and playback appears to be (almost) instant.

  • Q: What storage transactions does Wavestore typically generate beyond video writes?  For example, read, list, metadata, or property operations?

The data is written with a blob write, read with a blob read, and deleted with a blob delete operation when it is no longer needed.  LASS indexes contain information on which blobs are associated with which channel/time so the correct blob is read without performing list operations.

  • How is missing data handled?

Missing or corrupt data is handled in the same way as LASS handles corrupt data on a disk - missing images are skipped and warning messages produced in the disk log.

  • Cloud storage is a costly option.  How can we minimize the amount of data hosted in the cloud while still maintaining 30 days of retention?

The Wavestore allows you to configure a track for recording to local disk, and a separate track with different parameters for cloud recording.  This will allow you to adjust parameters as you require without affecting local recording.

  • For S3, which endpoint is used and is it configurable?

It is configurable.  The region field can contain just an AWS region, in which case .amazonaws.com is appended automatically, or it can contain a <region>.domain.com in which case that domain is used instead of .amazonaws.com.

  • For the Azure cloud, which endpoint is used and is it configurable?

The endpoint <storageaccount>.blob.core.windows.net is used and this is not configurable.

 

 

See Also

Recording to Azure Storage Container