Design Ingest Module, Migrate Pre-analysis Project Ingest Code

Create a design for the ingest module which takes batches of foxml objects with corresponding material. The ingester should perform validation and ingest the metadata to Fedora and the material to Bitstorage. The ingester should be transactional and handle special cases, such as network failure, full disks and Fedora breakdown.

For some projects there will only be metadata and for other projects the ingestion of metadata are handled in a project-specific way. Thus, it should not be a requirement from the ingester that it handles ingest of material.

Migrate existing ingest code

The migration of the existing ingest code should not involve major changes to the code. The primary goal it to conform to existing guidelines for code and to provide hooks for extra functionality, such af validation and transactions. Note that validation is not part of the 2008-schedule, which means that existing use of RIFF can/should be decoupled from the code.

Note: The Bitstorage is designed with transactions: Material is ingested, a checksum is calculated on Bitstorage-side and the Bitstorage awaits confirmation before the storage is made permanent.


Done (te 6, tsh 1+? md): Migrate code, update it to conform to guidelines and get unit-tests to run with the test installation of Fedora (a lot of detective work on the libraries). Making hooks for validation, data storage and transactions.

Done (te 1, abr ? md): Writing TaskA.4.1DesignDocument for the ingest module.

Missing: Follow-up on the follow-up for the review.

Note: The libraries used in the doms project do not conform fully to the guidelines. The licenses are present for all libraries, but version-numbers, javadoc and source is lacking for some of them.

ActionDesignAndSalvageIngestModule (last edited 2010-03-17 13:12:45 by localhost)