Differences between revisions 23 and 90 (spanning 67 versions)

DON'T PANIC

A definition: A datamodel describes the content of a collection. A content model describes the content of a data object. So, a datamodel is a set of content models, that together describe the collection.

The DOMS datamodel describes how the Type system underlying DOMS is realized in Fedora 3.

The entire DOMS datamodel is, in its entirety, a complex system. For proper understanding, the various components have been detailed in seperate documents. Firstly, it consist of a number of extensions to the Fedora system. Secondly, it consist of a number of predefined objects. These objects make use of the extensions to Fedora. Thirdly, it consist of a number of policies for how certain tasks are achived. And fourthly, it consist of a number of API interfaces.

Fedora and DOMS are big on namespaces. To ease writing the documentation, a namespace document, DomsNameSpacesAndSchemas, have been written. All namespaces should be defined there, and all shorthands refer to the namespaces defined therein.

Content Models in general

Fedora provides a repository for digital objects. All objects in the repository can, in principle, be unique, but Fedora provides a way of specifying that an object has a given type. Unfortunately, the type-definitions in Fedora, called Content Models, are rather simplistic by default. We use them as the basis of our type system, with certain enhancements.

For our purposes, there are two kinds of digital objects in Fedora

Data objects
Content Model objects

The Content Model object, as used in DOMS, describes the compulsory and legal content of an object of this type. It contains the information nessesary to verify if the given object is indeed of this type. For more detail on this, see FedoraOntology and FedoraTypeChecking

A data object can specify the Content Model describing its contents, via a fedora-model:hasModel relation, and in DOMS we require it to be present. A data object will be said to "subcribe" to a Content Model. Content Model inheritance, as specified in FedoraOntology, will be used.

The special Content Model object "doms:ContentModel_DOMS" is the root object. All Content Models must have an "doms-relations:extendsModel" relation to this object, possibly through a number of other Content Models. The complete description of a data object is defined as the set of the descriptions in the Content Model specified with "fedora-model:hasModel" and all Content Models that can be reached from this, by following "doms-relations:extendsModel" relations.

A Content Model can "extend" more than one other Content Model. There is no overriding of Content Models, a subscribing object must be valid in regards to all the Content Models in the inheritance tree.

Content Models have two datastreams in particular that are interesting. These are the ONTOLOGY and DS-COMPOSITE. The Ontology defines the the allowed relations in subscribing objects, and the DS-COMPOSITE defines the required datastreams and any restrictions they must adhere to.

Fedora extensions

FedoraOntology - Detailing the Ontology lanquage
FedoraTypeChecking - Detailing the extensions to the DS-COMPOSITE stream
FedoraViewBlobs - Detailing the View system, which allows you to view many objects as one
FedoraObjectTemplates - Detailing the prototype system for content models
FedoraImportExport - Detailing use of Import and Export of objects using content models
FedoraLicensePolicies - Detailing the License system, and how it interacts with the Search system
FedoraState - Detailing the use of object states to allow controlling validity, availability and deletions.

Predefined objects

DomsPredefinedObjects - The predefined content models and other objects in the doms system

Doms policies

DomsFileHandling - Detailing how we expect to handle file objects in Fedora.
DomsCollections - Detailing the use of collection objects in DOMS
DomsAuditTrail - Detailing how DOMS logs changes and new versions

API documentation

Refer to the overall Documentation page

Working with the Data model

Doms contains a number of content models. These are meant to serve as the basic buildingblocks for data models for new collections. A datamodel is, of course, not restricted to use only these content models, it can, and should, define it's own. All new content models, should extend doms:ContentModel_DOMS, and all objects that need to reference files outside Fedora should have a content model that derive from doms:ContentModel_File and so on. The content models that provide extra meaning are optional to use, and should at least be extended for the relevant collection.

Most data models are structured around some realworld concept, like a CD, modelled as a digital object. This object will be described by a content model that is totally collection specific, only extending doms:ContentModel_DOMS. It will probably have relations to digital objects, like tracks. These will be described by a content model that extends doms:ContentModel_Audio. Each of these will tracks must then reference a audio preservation file object, or some subtype of this. This is the best practice for constructing data models.

-  ⇤ ← Revision 23 as of 2008-07-01 11:00:24 → 
  Size: 17671
  Editor: abr
  Comment:
+   ← Revision 90 as of 2010-03-17 13:13:00 → ⇥
  Size: 5322
  Editor: localhost
  Comment: converted to 1.6 markup
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-= DOMS Data model =
+= DON'T PANIC =
 Line 3:
-[[ImageLink(http://merkur/viewvc/trunk/docs/datamodel/fig/DOMSBaseCollection.png?root=doms&view=co,alt=DOMS base collection,width=1024)]]
+'''A definition: A ''datamodel'' describes the content of a collection. A ''content model'' describes the content of a data object. So, a datamodel is a set of content models, that together describe the collection.'''
 Line 5:
-The DOMS datamodel describes how the Type system underlying DOMS is realised in Fedora 3. The figure above will serve as a guide through the following sections.
+The DOMS datamodel describes how the Type system underlying DOMS is realized in Fedora 3.

The entire DOMS datamodel is, in its entirety, a complex system. For proper understanding, the various components have been detailed in seperate documents. Firstly, it consist of a number of extensions to the Fedora system. Secondly, it consist of a number of predefined objects. These objects make use of the extensions to Fedora. Thirdly, it consist of a number of policies for how certain tasks are achived. And fourthly, it consist of a number of API interfaces.

Fedora and DOMS are big on namespaces. To ease writing the documentation, a namespace document, DomsNameSpacesAndSchemas, have been written. All namespaces should be defined there, and all shorthands refer to the namespaces defined therein.
-Line 8:
+Line 12:
-== DOMS Content Models ==
+== Content Models in general ==
-Line 16:
+Line 20:
-The Content Model object, as used in DOMS, describes the compulsary and legal content of an object of this type. It contains the information nessesary to verify if the given object is indeed of this type.
+The Content Model object, as used in DOMS, describes the compulsory and legal content of an object of this type. It contains the information nessesary to verify if the given object is indeed of this type. For more detail on this, see FedoraOntology and FedoraTypeChecking
-Line 18:
+Line 22:
-A data object can specify the Content Model describing its contents, via a fedora-model:hasModel relation. It can only have one such relation, and in DOMS we require it to be present.
+A data object can specify the Content Model describing its contents, via a fedora-model:hasModel relation, and in DOMS we require it to be present. A data object will be said to "subcribe" to a Content Model. Content Model inheritance, as specified in FedoraOntology, will be used.
-Line 20:
+Line 24:
-=== Inheritance ===
A type system that does not allow for inheritance, will have limited use. In spite of this, the Fedora Content Models do not provide this functionality. We have built our own inheritance system for Content Models to compensate for this lack.
+The special Content Model object "doms:!ContentModel_DOMS" is the root object. All Content Models must have an "doms-relations:extendsModel" relation to this object, possibly through a number of other Content Models. The complete description of a data object is defined as the set of the descriptions in the Content Model specified with "fedora-model:hasModel" and all Content Models that can be reached from this, by following "doms-relations:extendsModel" relations.
-Line 23:
+Line 26:
-The special Content Model object "!ContentModel_DOMS" is the root object. All Content Models must have an "extendsModel" relation to this object, possibly through a number of other Content Models. The complete description of a data object is defined as the set of the descriptions in the Content Model specified with "hasModel" and all Content Models that can be reached from this, by following "extendsModel" relations. 

A Content Model can "extend" more than one other Content Model. When determining the inheritance tree, and which Content Models that override each others, questions can arise. From the starting Content Model, perform a Breadth-first search of the inheritance tree. For datastream described by the Content Models, use only the first description you find. All the others count as overridden. 

=== Predefined Content Models ===

Shorthand: 
 * myObject.myDatastream means the Datastream myDatastream in the Object myObject.
 * $variable introduces a variable.
+A Content Model can "extend" more than one other Content Model. There is no overriding of Content Models, a subscribing object must be valid in regards to all the Content Models in the inheritance tree.
-Line 34:
+Line 29:
-==== ContentModel_DOMS ====

The following variables are used:
 * $Title: A human readable title
 * $PID: A generated PID for this object
 * $!ContentModel: The content model for this object, must derive from !ContentModel_DOMS
 * $Collection: The collection that this object belongs to. An object with a Content Model that derives from !ContentModel_Collection
 * $!LicenseObject: The License object that govern access to this object. An object with a Content Model that derives from !ContentModel_License

Requirements for objects described by !ContentModel_DOMS
 * ObjectProperties
  * External Properties
   * http://developer.statsbiblioteket.dk/doms/properties/#state : The state of the object, (Values = draft | intermediate | published)
 * Datastreams
  * DC: Fedora required datastream with the [http://dublincore.org/schemas/xmls/qdc/dc.xsd Dublin Core schema].
   * dc:title = $Title
##  * DomsDC: DC replacement, with the [http://dublincore.org/schemas/xmls/qdc/2008/02/11/dcterms.xsd qualified dublin core schema]
##   * dcterms:title = $Title
  * RELS-EXT: Fedora controlled relations between objects.
   * oai:itemID  = $PID
     (only required on harvested objects)
   * fedora:has!ContentModel -> $!ContentModel
   * doms:hasLicense -> $LicenseObject 
   * doms:isPartOfCollection -> $Collection
  * AUDIT: Fedoracontrolled audit trail
  * POLICY Fedoracontrolled datastream describing the rights and restrictions on this object
   * !ContentLocation URL =  $!LicenseObject.LICENCE
+Content Models have two datastreams in particular that are interesting. These are the ONTOLOGY and DS-COMPOSITE. The Ontology defines the the allowed relations in subscribing objects, and the DS-COMPOSITE defines the required datastreams and any restrictions they must adhere to.
-Line 63:
+Line 32:
-===== State =====
The state datastream contains xml of the form
{{{
<?xml version="1.0" encoding="UTF-8"?>
<state:state value="draft" xmlns:state="http://developer.statsbiblioteket.dk/DOMS/types/state/0/1/#"/>
}}}
+== Fedora extensions ==
 * FedoraOntology - Detailing the Ontology lanquage
 * FedoraTypeChecking - Detailing the extensions to the DS-COMPOSITE stream
 * FedoraViewBlobs - Detailing the View system, which allows you to view many objects as one
 * FedoraObjectTemplates - Detailing the prototype system for content models
 * FedoraImportExport - Detailing use of Import and Export of objects using content models
 * FedoraLicensePolicies - Detailing the License system, and how it interacts with the Search system
 * FedoraState - Detailing the use of object states to allow controlling validity, availability and deletions.

== Predefined objects ==
 * DomsPredefinedObjects - The predefined content models and other objects in the doms system

== Doms policies ==
 * DomsFileHandling - Detailing how we expect to handle file objects in Fedora.
 * DomsCollections - Detailing the use of collection objects in DOMS
 * DomsAuditTrail - Detailing how DOMS logs changes and new versions

== API documentation ==
 * Refer to the overall [[Documentation]] page
-Line 72:
+Line 54:
-==== ContentModel_Image ====
The following variables are used:
 * $TiffFile: An object with the Content Model !ContentModel_TiffFile;
-Line 76:
+Line 55:
-Requirements for objects described by !ContentModel_Image
 * Datastreams
  * RELS-EXT
   * doms:hasTiffFile -> $TiffFile
+== Working with the Data model ==
Doms contains a number of content models. These are meant to serve as the basic buildingblocks for data models for new collections. A datamodel is, of course, not restricted to use only these content models, it can, and should, define it's own. All new content models, should extend doms:!ContentModel_DOMS, and all objects that need to reference files outside Fedora should have a content model that derive from doms:!ContentModel_File and so on. The content models that provide extra meaning are optional to use, and should at least be extended for the relevant collection.
-Line 81:
+Line 58:
-==== ContentModel_Audio ====
The following variables are used:
 * $WavFile: An object with the Content Model !ContentModel_WavFile; 
 * $BwfFile: An object with the Content Model !ContentModel_BwfFile; 

Requirements for objects described by !ContentModel_Audio
 * Datastreams
  * RELS-EXT: One of the following
   * doms:hasWavFile -> $WavFile
   * doms:hasBwfFile -> $BwfFile

==== ContentModel_Video ====
The following variables are used:
 * $Mpeg1File: An object with the Content Model !ContentModel_Mpeg1File; 
 * $Mpeg2File: An object with the Content Model !ContentModel_Mpeg2File; 

Requirements for objects described by !ContentModel_Video
 * Datastreams
  * RELS-EXT: One of the following
   * doms:hasMpeg1File -> $Mpeg1File 
   * doms:hasMpeg2File -> $Mpeg2File

==== ContentModel_Text ====
The following variables are used:
 * $Utf8File: An object with the Content Model !ContentModel_Utf8File; 
 * $PdfFile: An object with the Content Model !ContentModel_PdfFile; 

Requirements for objects described by !ContentModel_Text
 * Datastreams
  * RELS-EXT: Must have utf8. May contain pdf
   * doms:hasFile -> $Utf8File
   * doms:hasFile -> $PdfFile


==== ContentModel_License ====

Requirements for objects described by !ContentModel_License
 * Datastreams
  * LICENCE: XACML describing the license. [http://www.oasis-open.org/committees/download.php/915/cs-xacml-schema-policy-01.xsd Schema]
  * DC: The DC datastream (probably the description field) is used to describe the human readable version of the license

==== ContentModel_Collection ====

Requirements for objects described by !ContentModel_Collection: None


==== ContentModel_File ====
The following variables are used:
 * $OrigFile: An object with the Content Model that derives from !ContentModel_File; 

Requirements for objects described by !ContentModel_File
 * Datastreams
  * RELS-EXT
   * (optional) doms:hasOriginal -> $OrigFile
  * CHARACTERISATION: The output of the characterisation tools. Schema attachment:Characterisation.xsd
  * CONTENTS: Datastream containing the file
   * !ContentLocation URL = The file in Bitstorage
  * ORIGIN: Metadata about the creation of the file, in the Premis [http://www.loc.gov/standards/premis/v1/Event-v1-1.xsd schema]
  * PRONOM: The pronom ID of the fileformat. Schema attachment:Pronom.xsd

The characterisation datastream could look like this.
{{{
<?xml version="1.0" encoding="UTF-8"?>
<char:characterisation xsi:schemaLocation="http://developer.statsbiblioteket.dk/DOMS/types/characterisation/0/1/# http://developer.statsbiblioteket.dk/DOMS/types/characterisation/0/1/characterisation/characterisation-0-1.xsd"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:char="http://developer.statsbiblioteket.dk/DOMS/types/characterisation/0/1/#"
  xmlns:jhove="">
  <char:characterisationRun>
    <char:tool>JHove</char:tool>
    <char:output>
      <jhove:...>
      </jhove:...>
    </char:output>
  </char:characterisationRun>
</char:characterisation>
}}}

All the predefined subtypes of File bring no new requirements.

=== Content Model implementation ===

A Content Model in DOMS must have a number of additional datastreams, in regards to the Content Model requirements defined by Fedora.

The following variables are used:
 * $PID: The PID for this object
 * $!ContentModel: The content model for this object, must derive from !ContentModel_DOMS
 * $Collection: The collection that this object belongs to. An object with a Content Model that derives from !ContentModel_Collection
 * $License: The License object that govern access to this object. An object with a Content Model that derives from !ContentModel_License


Requirements for Content Model objects (except !ContentModel_DOMS, which do not have an "extends" relation)
 * ObjectProperties
  * External Properties
   * http://developer.statsbiblioteket.dk/doms/properties/#state : The state of the object, (Values = draft | intermediate | published)
 * Datastreams
  * DC
   * dc:title = $PID
  * RELS-EXT
   * (1+) doms:extends -> $!ContentModel
   * doms:hasLicense -> $LicenseObject 
   * doms:isPartOfCollection -> $Collection
  * SCHEMABINDINGS: Described below. Schema attachment:Schemabinding.xsd
  * DS-COMPOSITE-MODEL: Fedoracontrolled structural datastream. Lists the required datastreams in the described data objects. 
  * VIEW: Described below. Schema attachment:view.xsd
  * AUDIT: Fedoracontrolled audit trail
  * POLICY Fedoracontrolled datastream describing the rights and restrictions on this object
   * !ContentLocation URL =  $LicenseObject.LICENCE
  * STATE: The state of the object, by the attachment:State.xsd
  

=== Schemabindings ===

The schemabinding datastream links datastreams in the described data objects with datastreams (containing schemas) in this object.

{{{
<?xml version="1.0" encoding="UTF-8"?>
<b:bindings xmlns:b="http://developer.statsbiblioteket.dk/DOMS/types/Schemabinding/0/1/#">
  <b:binding schema="xsd">
    <b:from name="datastream_name_in_data_object"/>
    <b:to name="datastream_with_validator_schema_in_this_object"/>
  </b:binding>
</b:bindings>
}}}

=== View ===
View datastream contain xml of the form
{{{
<?xml version="1.0" encoding="UTF-8"?>
<view:views  xmlns:view="http://developer.statsbiblioteket.dk/DOMS/types/view/0/1/#">
  <view:view name="GUI">
    <view:relations>
      <doms:hasFile xmlns:doms="http://developer.statsbiblioteket.dk/DOMS/relations/default/0/1/#"/>
    </view:relations>
    <view:inverse-relations>
      <doms:isPartOfCollection xmlns:doms="http://developer.statsbiblioteket.dk/DOMS/relations/default/0/1/#"/>
    </view:inverse-relations>
    <view:datastreams>
      <view:datastream>DomsDC</view:datastream>
    </view:datastreams>
  </view:view>
</view:views>
}}}

As can be seen, it describes all relations to be followed outwards, both directly and reverse. When including the object, only the named datastreams from the datastreams tag should be used. There can be several views, with different views in an object. The GUI should use the view with the name GUI.

== Collections ==

The DOMS system will be a system that models several collections of digital objects. Each object belongs to one or more collections. This is represented by having one or more "isPartOfCollection" relations to the parent collections. This goes for a collection object as well - it belongs to another collection. One collection has special status though: the "doms:Root_Collection" does not belong to any other collection, and is thus the bottom element for the "isPartOfCollection" relation. Every other collection has a "isPartOfCollection" relation to "doms:Root_Collection".

In addition, DOMS contains another special collection - the "doms:DOMS_Base_Collection". This collection provides objects such as content models and licenses that are utilized by (and mandatory for) the other collections in the DOMS. This collection is meant to be ingested as the first collection in the DOMS.

== File Objects ==

In DOMS, we have found it beneficial to seperate the abstract concept of "Image" or "Audio" from the concrete implementations such as "jpeg" and "mp3". The metadata about the image will be relevant no matter the manifestation of the image, and as such should not reside along with the technical metadata about the manifestation. To support this separation, we have introduced the concept of File objects.

A File object is an object, that contain a link to the file (in Bitstorage), and the technical metadata about this file. Only File objects are allowed to reference a file in Bitstorage. File objects must all have a Content Model that extends !ContentModel_File.

=== Preservation files ===
We have a special class of files in DOMS, the ones we are willing to promise to preserve. These are the eight formats
 * Tiff
 * Wav, Bwf
 * Mpeg1, Mpeg2
 * UTF8, Pdf, OfficeOpenXml
For each of these, we have defined a Content Model that extends !ContentModel_File.

Presentation files generally are dynamically generated upon request.

=== Technical Metadata ===
A file object should contain technical metadata. In this context it refers
 * A datastream with the output of the characterisation tools used on this file upon ingest, called CHARACTERISATION
 * A datastream with the metadata about the origins of the file, called Origin

In addition, it must have a relation "hasOriginal" if it was migrated from another file that exists in DOMS.


== Views ==

A view is a way of combining objects in the DOMS into a domain-relevant group. It is a way of seeing a number of objects as related - as a whole; information that can be useful for the GUI-generator when generating GUI-windows.

Those views that we imagine as being suitable for a screen or window in the GUI, are called main views. Each main view contains an object that the main view is centered around. We call this the main object, and the ID of this view is the ID of the main object. Views of other objects are simply called views.
The main object is the object that represents the main view - other objects in that view are related to the main object and would presumably be relevant to edit in the GUI at the same time. For a CD modelled in DOMS, for example, a CD object would be the main object, and objects for tracks, cover, lyrics and so on would constitute the rest of the main view.

We imagine that results appearing in searches in the GUI will all be main views. In fact every view that will be the basis for a screen/window will be a main view.

A view for an object O is represented by a Datastream VIEW on the Content Model object for O. This Datastream also mark the object as Main, if this is the case. Please note that the view is defined on Content Model level, so the same rules are used to generate the view for all objects using that Content Model.

The datastream will just contain a list of relation names and reverse relation names. Following these relations will give you the view. Views are inherited when Content Models "extends" each other, so you should generate the view for each Content Model in the inheritance tree of this Object, and remove duplicates.


Definitions:

 * 1-step relations (relations on a content model c of the form "x rel y", meaning that if an object x with content model c has relation rel to another object y, then y will be part the view too. Examples of rel for a CD modelled in DOMS could be hasTrack, hasLyrics,..)

## * x-step relations. These are relations of the form mentioned above, but they will be followed from an object an arbitrary number of times, as long as the relations match.

 * reverse relations (relations of the type "y rel x" on a content model c, where x has model c and y therefore will be included in the view for x)

In addition, we suggest to augment the 1-step approach with the idea of "includes". What this means is that when object O has a view defined by following relations from O once, and an object P is in the view of O, then the view of P will be included in the view of O.
## Note that this is different from x-step relations, where objects in the view of P would not necessarily be included in the view of O.


== Licenses ==

Licenses, in DOMS, have, as their only purpose, to restrict who can view what material. They are only a concern for people using the material in DOMS, not users working with the GUI, or otherwise administrating the contents.

Licenses are implemented by using the Fedora XACML engine. When a user authenticates with the Fedora server (or with a server passing authentication tokens to the DOMS), he gets a number of attributes. Each of these attributes name one license that the user can access material under.

Each data object in DOMS has a POLICY datastream. This datastream is just an URL, referring to a License object's LICENCE datastream. This datastream is an XACML stream, that evaluates if the user posses the attribute that specify that the user can use this License. If yes, the user is granted access to the original object, otherwise he is denied.


== Audit Trail ==

Each user that will use the GUI will need to login. They will authenticate with some external server, probably the SB LDAP server. The access control is not really a concern for the DOMS system. As such, all GUI users will have equal and full access to the DOMS repository.

Audit trails, however, are a concern. Each change to a datastream in a data object will, per default Fedora behaivour, create a new version of this datastream, marked with the creation time and the username. For this reason the Fedora operations PurgeObject and PurgeDatastream are blocked, as they destroy the audit trail. 
Real deletion of information is not possible, but both objects and datastreams can be marked as "deleted", again per standard Fedora behaviour. Any tools working with or on DOMS should respect this flag. The GUI should only concern itself with the most recent version of a datastream.
+Most data models are structured around some realworld concept, like a CD, modelled as a digital object. This object will be described by a content model that is totally collection specific, only extending doms:!ContentModel_DOMS. It will probably have relations to digital objects, like tracks. These will be described by a content model that extends doms:!ContentModel_Audio. Each of these will tracks must then reference a audio preservation file object, or some subtype of this. This is the best practice for constructing data models.