RLG/NARA (A-D) AND DRAMBORA (R)

Documentation Requirements

These documentation requirements are new, and we should probably update the WBS with new tasks.

Communicate requirements out of scope for DOMS but relevant for SB:

• A2.1. Repository staff have skills and expertise appropriate to their duties.
• A2.2. Repository has appropriate number of staff to support all functions and services designated in agreements with depositors.
• A2.3. Repository commits to professional development to keep staff expertise and skills current.
• A3.3. Repository is committed to formal, periodic review and assessment to ensure continued development.
• A4.1. Repository has a short- and long-term business planning process in place to support sustainability.
• A4.2. Repository has in place at least annual processes to review and adjust business plans as necessary.
• A4.3. Repository business planning and practices are transparent, compliant with relevant accounting standards and practices, and auditable.
• A4.4. Repository has ongoing commitment to risk, benefit, investment, and expenditure analysis and reporting (including assets, licenses, and liabilities).
• A4.5. Repository recognizes the eventual strong possibility of a gap between repository-generated funding and the funding necessary to meet the repository’s commitments to its depositors. It commits to bridging these gaps by securing funding and resource commitments specifically for that purpose; these commitments can come either from the repository itself or parent organizations, as applicable.
• A5.1 If repository manages, preserves, and/or provides access to digital materials on behalf of another organization, it has and maintains appropriate contracts or deposit agreements.
• A5.2 Repository contracts or deposit agreements must specify and/or transfer appropriate preservation rights, as necessary.
• A5.3 Repository tracks and manages copyrights and restrictions on use as required by contract or license
• A5.4 If repository ingests digital content with unclear ownership/rights, it has policies addressing liability and challenges to those rights.
• B1.2. Repository has specified all appropriate aspects of acquisition, maintenance, access, and withdrawal issues in written agreements with depositors.
• B3.11. Repository can provide evidence of the success of its preservation planning
• C1.1. Repository has a documented definition of its designated community/ies--who it consists of, its knowledge base, what levels of service it expects, etc.
• C1.2. Repository makes the definition of its Designated Communities available.
• C1.3. Repository defines, communicates, and commits to a definition of "understandability" with its Designated Community.
• C2.1. Repository articulates minimum metadata requirements to enable the Designated Community to discover and identify material of interest.
• C3.1. Repository documents and communicates to its designated community what access and delivery options are available.
• C4.1. Repository has a documented process to test ‘understandability to the Designated Community’, as previously defined, of the information content associated with the Content Information and PDI, and this includes defining the appropriate steps necessary should the agreed level of ‘understandability’ not be met.
• C4.2. Repository has verified that Content Information and PDI are understandable to Designated Community.
• D1.6. Repository reports to its administration all incidents of data corruption or loss, and steps taken to repair/replace corrupt or lost data.
• D1.8. Repository has a documented change management process that identifies changes to critical processes.
• D1.9. Repository has a process for testing the effect of critical changes to the system.
• D2.3. Repository has procedures in place for monitoring or receiving notifications about changes in the needs of its Designated Communities (e.g., surveys, formal reviews, workshops and individual interactions).
• D3.1. Repository maintains a systematic analysis of its environment: data, systems, personnel, physical plant, security needs, etc.
• D3.3. Repository staff have delineated roles, responsibilities, and authorizations.
• D3.4. Repository has written disaster preparedness and recovery plan(s) (including at least one off-site copy of all deposited data).
• D3.5. Repository tests disaster plans regularly.
• D3.6. Repository has defined processes for service continuity and disaster recovery.
• R01 Management failure
• R02 Loss of trust
• R03 Activity is overlooked or allocated insufficient resources
• R04 Business objectives not met
• R05 Repository loses mandate
• R06 Community requirements change substantially
• R07 Community requirements misunderstood or ineffectively communicated
• R08 Enforced cessation of repository operations
• R09 Community feedback not received
• R10 Community feedback not acted upon
• R13 Business policies and procedures are inefficient
• R14 Business policies and procedures are inconsistent or contradictory
• R16 Legal liability for breach of contractual responsibilities
• R17 Legal liability for breach of legislative requirements
• R18 Liability for regulatory non-compliance
• R19 Inability to evaluate repository’s successfulness
• R20 False perception of the extent of repository’s success
• R21 Loss of key member(s) of staff
• R22 Staff suffer skill loss
• R23 Staff skills become obsolete
• R24 Inability to evaluate staff effectiveness or suitability
• R25 Finances insufficient to meet repository commitments
• R26 Misallocation of finances
• R27 Liability for non-adherence to financial law or regulations
• R28 Financial shortfalls or income restrictions
• R29 Budgetary reduction
• R30 Hardware failure or incompatibility
• R31 Software failure or incompatibility
• R32 Hardware or software incapable of supporting emerging repository aims
• R33 Obsolescence of hardware or software
• R34 Media degradation or obsolescence
• R35 Exploitation of security vulnerability
• R36 Unidentified security compromise, vulnerability or information degradation
• R37 Physical intrusion of hardware storage space
• R38 Remote or local software intrusion
• R39 Local destructive or disruptive environmental phenomenon
• R40 Accidental system disruption
• R41 Deliberate system sabotage
• R42 Destruction or non-availability of repository site
• R43 Non availability of core utilities (e.g. electricity, gas, network bandwidth, water)
• R44 Loss of other third-party services
• R45 Change of terms within third-party service contracts
• R46 Destruction of primary documentation
• R47 Inability to evaluate effectiveness of technical infrastructure and security
• R57 Loss of non-repudiation of commitments
• R58 Loss of information reliability
• R72 Ambiguity of understandability definition
• R74 Non-availability of information delivery services
• R75 Authentication subsystem fails
• R76 Authorisation subsystem fails
• R78 Loss of performance or service level

Document and communicate repository principles for TDR fulfilling requirements from

• A1.1. Repository has a mission statement that reflects a commitment to the long-term retention of, management of, and access to digital information on behalf of depositors.
• A1.2. Repository has a formal succession plan, contingency plans, and/or escrow arrangements in place in case repository ceases to operate or substantially changes its scope (i.e., return with adequate prior notification of digital objects to depositors and/or trusted inheritors identified).
• A3.4. Repository has a documented history of the changes to its operations, procedures, software, hardware, traceable to its preservation strategies where appropriate.
• D1.8. Repository has a documented change management process that identifies changes to critical processes.
• R12 Business policies and procedures are unknown
• R73 Shortcomings in semantic or technical understandability of information

Need documented testing procedures:

• D1.9. Repository has a process for testing the effect of critical changes to the system.

Need documentation of software dependencies:

• D2.2. Repository has software technologies appropriate to the services it provides to its designated communities and has procedures in place to monitor and receive notifications when software technology changes are needed.

IT Maintenance Department Requirements

These requirements also need to be documented and communicated.

Requirements to IT maintenance department:

• D1.1. Repository functions on well-supported operating systems and other core infrastructural software.
• D1.2. Repository ensures that all platforms have a backup function, sufficient for the repository's services and for the data held (e.g., metadata associated with access controls, repository main content, etc.)
• D1.5. Repository has effective mechanisms to detect data corruption or loss.
• D1.6. Repository reports to its administration all incidents of data corruption or loss, and steps taken to repair/replace corrupt or lost data.
• D1.7. Repository has defined processes for storage media migration.
• D1.10. Repository has a process to stay current with the latest operating system security fixes.
• D2.1. Repository has hardware technologies appropriate to the services it provides to its designated communities and has procedures in place to monitor and receive notifications when hardware technology changes are needed.
• D3.1. Repository maintains a systematic analysis of its environment: data, systems, personnel, physical plant, security needs, etc.
• R30 Hardware failure or incompatibility
• R31 Software failure or incompatibility
• R32 Hardware or software incapable of supporting emerging repository aims
• R33 Obsolescence of hardware or software
• R34 Media degradation or obsolescence
• R35 Exploitation of security vulnerability
• R36 Unidentified security compromise, vulnerability or information degradation
• R37 Physical intrusion of hardware storage space
• R38 Remote or local software intrusion
• R39 Local destructive or disruptive environmental phenomenon
• R40 Accidental system disruption
• R41 Deliberate system sabotage
• R42 Destruction or non-availability of repository site
• R43 Non availability of core utilities (e.g. electricity, gas, network bandwidth, water)
• R44 Loss of other third-party services
• R45 Change of terms within third-party service contracts
• R46 Destruction of primary documentation
• R47 Inability to evaluate effectiveness of technical infrastructure and security
• R60 Loss or non-suitability of backups
• R74 Non-availability of information delivery services
• R75 Authentication subsystem fails
• R76 Authorisation subsystem fails

Requirements to IT maintenance department which we need to have feedback on:

• D1.3. Repository stipulates the number and location of copies of all digital objects.
• D1.4. Repository has mechanisms in place to insure any/multiple copies of digital objects are synchronized.
• D3.4. Repository has written disaster preparedness and recovery plan(s) (including at least one off-site copy of all deposited data).
• R53 Loss of availability of information and service
• R61 Inconsistency between redundant copies

Deployment and Collection Specifics

We should probably take a quick look at these soon and decide which requirements are relevant now and which are relevant later in the project.

Organizational procedures required around deployment

• A3.1. Repository has a mechanism in place for reviewing, updating, and developing comprehensive policies and procedures as repositories grow and as the community practice evolves.
• A3.2. Repository has monitoring and feedback mechanisms in place to ensure continued operation, support problem resolution, and address evolving requirements of providers and consumers.
• A3.3. Repository is committed to formal, periodic review and assessment to ensure continued development.
• A3.4. Repository has a documented history of the changes to its operations, procedures, software, hardware, traceable to its preservation strategies where appropriate.
• A3.5. Repository commits to transparency and accountability in all actions supporting the operation and management of the repository.
• A3.6. Repository commits to define, collect, track, and provide on demand, its information integrity measurements.
• A3.7. Repository commits to a regular schedule of certification and to notifying certifying bodies of operational changes that will change or nullify its certification status.
• D1.8. Repository has a documented change management process that identifies changes to critical processes.
• D2.2. Repository has software technologies appropriate to the services it provides to its designated communities and has procedures in place to monitor and receive notifications when software technology changes are needed.

Collection specific requirements we should consider to some degree in DOMS

• B1.3. Repository has an identifiable, written definition for each SIP or class of information ingested by the repository.
• B1.4. Repository has a process to ensure that the information is acquired from the expected source.
• B1.6. Repository’s ingest process verifies each SIP for completeness and correctness.
• B2.3. Repository has a definition of how AIPs are derived from SIPs.
• C2.1. Repository articulates minimum metadata requirements to enable the Designated Community to discover and identify material of interest.
• R72 Ambiguity of understandability definition

Ingest

The following should be considered when designing ingest.

File integrity checking during ingest, fail on errors:

• B1.6. Repository’s ingest process verifies each SIP for completeness and correctness.
• B1.8. Repository can demonstrate that all SIPs are either accepted as whole or part of an eventual AIP, or otherwise disposed of in a recorded fashion.
• B2.6. Repository verifies each AIP for completeness and correctness when generated.
• R48 Structural non-validity or malformation of received packages

Proper reporting required from ingest module, ensure only complete objects are ingested and reported ok, report indicates preservation status

• B1.7. Repository provides producer/depositor with appropriate responses at predefined points during the ingest processes.
• B1.8. Repository can demonstrate that all SIPs are either accepted as whole or part of an eventual AIP, or otherwise disposed of in a recorded fashion.
• B1.9. Repository can demonstrate when preservation responsibility is formally accepted for the contents of the AIP.
• R48 Structural non-validity or malformation of received packages
• R49 Incompleteness of submitted packages
• R63 Inability to validate effectiveness of ingest process

Ensure received data cannot be modified during ingest:

• R50 Externally motivated changes or maintenance to information during ingest

Preserve existing PIDs in metadata:

• B2.5. If unique identifiers are associated with SIPS before ingest, they are preserved in a way that maintains a persistent association with the resultant AIP.

Repository Contents

The following items are relevant to metadata and digital object design decisions and require documentation too and they should be considered early in the project.

Must be possible to document properties we will preserve for all digital material

• B1.1. Repository identifies properties it will preserve for each class of digital object.
• B3.5. Repository preserves the content information of AIPs.

Must have preservation strategy for certain classes of digital material

• B1.1. Repository identifies properties it will preserve for each class of digital object.
• B3.1. Repository has documented preservation strategies.
• R65 Preservation plans cannot be implemented
• R66 Preservation strategies result in information loss

Sufficient metadata about digital objects required: Technical, administrative and preservational as well as descriptive, generated or analyzed to be correct, fail on errors:

• B1.5. Repository obtains sufficient physical control over the digital objects to preserve them.
• B1.8. Repository can demonstrate that all SIPs are either accepted as whole or part of an eventual AIP, or otherwise disposed of in a recorded fashion.
• B3.4. Repository records/registers representation information [including formats] ingested
• B3.6 Repository acquires Preservation Description Information for its associated content information.
• B4.1. Repository captures or creates this minimum descriptive metadata and ensures it is associated with the AIP
• R11 Business fails to preserve essential characteristics of digital information
• R54 Loss of authenticity of information
• R62 Extent of what is within the archival object is unclear

Format definitions required, note that the intended usage of the format is relevant as well, use standard format registries:

• B2.1. Repository has an identifiable, written definition for each AIP or class of information preserved by the repository.
• B2.2. Repository has a definition of each AIP (or class) that is adequate to fit long-term preservation needs.
• B3.3 Repository uses appropriate international representation information [including format] registries
• R48 Structural non-validity or malformation of received packages
• R62 Extent of what is within the archival object is unclear
• R73 Shortcomings in semantic or technical understandability of information

Standard permanent UIDs needed, accessible from the rest of world:

• B2.4. Repository has and uses a naming convention that can be shown to generate visible,unique IDs for all AIPs.
• R64 Identifier to information referential integrity is compromised

Rights

Must be possible to document copyrights and restrictions on all digital material in a format that it is possible to act on/Require enforced access rights

• A5.3 Repository tracks and manages copyrights and restrictions on use as required by contract or license
• B5.1. Repository access management system fully implements access policy
• C3.3. Repository ensures that agreements applicable to access conditions are adhered to.
• C3.4. Repository has documented and implemented access policies (authorization rules, authentication requirements) consistent with deposit agreements for stored objects.
• D3.2. Repository has implemented mechanisms (processes) to adequately address each of the defined security needs.
• D3.3. Repository staff have delineated roles, responsibilities, and authorizations.
• R15 Legal liability for IPR infringement
• R16 Legal liability for breach of contractual responsibilities
• R17 Legal liability for breach of legislative requirements
• R18? Liability for regulatory non-compliance
• R36 Unidentified security compromise, vulnerability or information degradation
• R52 Loss of confidentiality of information
• R76 Authorisation subsystem fails

Repository/Object Integrity

The following items are mostly requirements to logging and should also be considered early in the project.

It must be possible to document how we ensure integrity

• A3.6. Repository commits to define, collect, track, and provide on demand, its information integrity measurements.

Must be able to document the data we have in the repository is derived from the data we received, and how:

• B2.3. Repository has a definition of how AIPs are derived from SIPs.
• R51 Archival information cannot be traced to a received package
• R54 Loss of authenticity of information
• R55 Loss of integrity of information
• R56 Unidentified information change
• R59 Loss of information provenance
• R68 Non-traceability of received, archived or disseminated package

Integrity audit needed - read the RLG/NARA source:

• B2.7. Repository provides an independent mechanism for audit of the integrity of the repository collection/content.
• (R55?) Loss of integrity of information
• (R56?) Unidentified information change

Referential integrity required between file storage and metadata (creation and validation):

• B4.2. Repository can demonstrate that referential integrity is created between all AIPs and associated descriptive information.
• B4.3. Repository can demonstrate that referential integrity is maintained between all AIPs and associated descriptive information.
• D1.5. Repository has effective mechanisms to detect data corruption or loss.
• D1.6. Repository reports to its administration all incidents of data corruption or loss, and steps taken to repair/replace corrupt or lost data.
• R64 Identifier to information referential integrity is compromised
• R69 Metadata to information referential integrity is compromised

Preservation Requirements

Need preservation strategy (Planets):

• B3.1. Repository has documented preservation strategies.
• B3.2. Repository implements/responds to strategies for AIP storage and migration.
• B3.5. Repository preserves the content information of AIPs.
• B3.9. Repository has mechanisms in place for monitoring and notification when format (or other representation information) obsolescence is near/or are no longer viable.
• B3.10. Repository has mechanisms to change its preservation plans as a result of its monitoring activities.
• R65 Preservation plans cannot be implemented
• R66 Preservation strategies result in information loss
• R67 Inability to validate effectiveness of preservation

Must be able to validate/evaluate results of preservation action (Planets):

• R66 Preservation strategies result in information loss
• R67 Inability to validate effectiveness of preservation

Active bit preservation required:

• B3.7. Repository actively monitors AIP integrity.
• D1.5. Repository has effective mechanisms to detect data corruption or loss.
• R61 Inconsistency between redundant copies

Audit trail required:

• B3.8. Repository has contemporaneous records of actions taken associated with ingest and archival storage processes and those administration processes which are relevant to the preservation.
• B3.11. Repository can provide evidence of the success of its preservation planning
• D1.6. Repository reports to its administration all incidents of data corruption or loss, and steps taken to repair/replace corrupt or lost data.
• R70 Documented change history incomplete or incorrect

Access

Data must be searchable/discoverable:

• R71 Non-discoverability of information objects

Dissemination must always return correct results, and reject requests otherwise :

• B5.3. Repository can demonstrate that the process that generates the DIP is complete in relation to the request.
• B5.4. Repository can demonstrate that the process that generates the DIP is correct in relation to the request.
• B5.5. Repository must demonstrate that all access requests result in a response of acceptance or rejection.
• B5.6. Repository enables the dissemination of authentic copies of the original or objects traceable to originals
• R77 Inability to validate effectiveness of dissemination mechanism

Access log required:

• B5.2. Repository logs all access management failures, and staff review inappropriate “access denial” incidents.
• C3.2. Repository has implemented a policy for recording all access actions (includes requests, orders etc.) that meet the requirements of the repository and information producers/depositors.
• R36 Unidentified security compromise, vulnerability or information degradation