• GuidelinesForNewDatamodel
• PatternLanguage
• Data_Shards

Data Shards ***

Context

When creating a new collection choices will often have to be made in regards to the actual structure of the data, it would for instance be technically demanding, beyond reason, to store a continuous stream of television data, it must be broken into shards in order to allow for efficient data storage and handling.

Description of pattern

This pattern describes the "low level" construction of a shard of data, it is generalised in order to span the multiple possible application of the shard data model in the DOMS.

Problem description

The major problem is the generality of the sharded data, this is also the strength of the shard pattern, shards are general enough to apply to most situations where data is split up, and can be used to model data-objects that are naturally granulated, think data that originates from pages, as well as data that is originally more complex, think of the video formats stored from a DVB-T stream.

Solution

The shards must have a sufficiently general structure to accommodate, at least, the current granulated objects as well as those envisioned for the future.

Use the shards as elements in the documents they form part of.

Consider next

The actual mapping from a shard to the file (part) in the file system, also consider the possible requirement of a way to combine shard-data into a complete file/document.