When a file is stored to Object Storage, the segment size can impact how the file's content is populated. For large files, if the segment size is configured, the file is broken up into multiple segments and each segment is stored as an object in Object Storage. Segment size should be configured if very large files exist, since different Object Storage providers or protocols could place a limit on the maximum object size it can store. For example, Lattus 3.0.0 has a limit of 16 TiB for each object. If no segment size is configured, uploading a file whose size is larger than 16 TiB to Lattus Object Storage (version 3.0.0) will fail. Similarly, AWS S3 Multipart Upload and Microsoft Azure Blob protocols have a maximum object size of 5 TiB and 4.77 TiB, respectively.
Storage Manager uses the MED_SEG_OVER_LATTUS system parameter to control the
segment size for files targeted for storage to LATTUS media. The default
size is set at 128 GiB (137,438,953,472 bytes). Storage Manager will segment
files greater than MED_SEG_OVER_LATTUS, when storing to LATTUS media.
The default segment size values for other Object Storage media types are set as follows:
- 128 GiB (137,438,953,472 bytes) for S3 media using
MED_SEG_OVER_S3system parameter, which applies to Lattus S3 and Q-Cloud Archive S3 media. - 5 TiB (5,497,558,138,880 bytes) for Q-Cloud Vault, S3COMPAT, and AWS media
using
MED_SEG_OVER_QVAULT,MED_SEG_OVER_S3COMPAT, andMED_SEG_OVER_AWSsystem parameters respectively. - 4.77 TiB (5,242,880,000,000 bytes) for AZURE media using
MED_SEG_OVER_AZUREsystem parameter.
There is a trade-off to selecting a larger or smaller segment size. Below are some of the advantages and disadvantages for different segment sizes. The selection relies on the system configuration, workload characteristics, application requirements, and other parameters.
- A larger segment size reduces the number of segments for a large file. An object ID is assigned for each segment and an entry for each object ID is added to the file's metadata or copy information. Therefore, a larger segment size will reduce the number of metadata entries, thus reducing the size of metadata consumed. It also reduces the space to store and transfer such metadata which is contained in the manifest file that snbackup saves to Object Storage. Additionally, it could be easier for third-party applications to operate if there are fewer objects per file (assuming the object IDs are also exported to the third-party).
- A small segment size, on the other hand, might be of benefit if the network or storage system is unreliable since this will result in reduced overhead on retries of failed transfers.
- A smaller segment size could also reduce the overall storage and retrieval time, because Storage Manager can store and retrieve segments concurrently using multiple data moving streams.
