Differences between revisions 12 and 72 (spanning 60 versions)

Documentation

This page contains documentation produced in the DOMS project.

Install Guide for DOMS with Radio/TV

Install-sequence for parts of the DOMS system, install guides, and status of DOMS in stage
Ideas for improvement of DOMS system installation
Generating a rick-roll file

Design

The top level design documentation will be available here.

Packages

Bitstorage

Low level bitstorage webservice design

ECM

RELEASE NOTES
INSTALL GUIDE
?

IP-Role Mapper

Update tracker

DOMSServer

DOMS Server API

DOMSClient

Ingest

Radio-TV Ingester

DOMS-GUI

OAI-PMH handling

Summa integration

Broadcast Extraction Service

DOMS Wowza plugin

APIs

Note: APIs below here are abandoned.

The DOMS-related API documentation is a work in progress.

Data Model

The data model documentation is a work in progress.

Data Model
Data Model for De Danske Aviser in MiniDOMS
Data Model for TV2 Reklamer in DOMS
Data Model for the olde MiniDOMS is available at http://merkur/domswiki/MiniDOMSDataModel (Notice, you are now leaving the domswiki, going to the barren wastelands of the old abandoned MiniDOMS merkur-wiki.)

Data Provider

DOMS as an OAI-PMH repository

GUI Code Docs

Understanding gained from Mjølner

Fedora 3 Project

Main document for the Fedora team

Picture video for wowza

Picture video script

Deployment History

webapp

svn

date

comments

bes		2010-11-10
bes_dev	1308	2010-11-26	vlc-based transcoding
bes_exp	1308	2010-11-26

"Runtimes" for parts of deployment

Useful for those tasks that may be done for deploying DOMS with Radio/TV-recordings and metadata for 2005-2010.

As of 2011-02-11, after updating program mappings, the prod system ritzau database contains (for 2005-2011) a total of 4.883.822 programs. (These are for channels 100fm dk4 dr1 dr2 drbo drhd drk drol drp1 drp2 drp3 drp4b drp4f drp4k drp4k94 drp4mv drp4nj drp4o drp4re drp4s drp5000 drpk drram drup drx kanal4 kanal5 kanal6 novafm radio2 sbsnet tv2c tv2d tv2l tv2news tv2oj tv2radio tv2s tv2sport tv2z voice)

Examples of how long runs and batches have taken:

Run/batch/process	Size of task	Time
Clear metadata in Ritzau Database (thru psql)	4553311 objects	about 39 mins	Not to be done in prod
Clear metadata in Ritzau Database (thru psql)	123499 objects (1 month of metadata)	about 2 mins	Not to be done in prod
Update channel-mappings in Radio/TV GUI	? (Colin finds out)	45 mins for 2.300.000	To be done in prod
Fill Ritzau Database (bart_systemet_mirror.sh)	1 month of metadata	7 mins	To be done partly in prod (?)
Ingest references to recording-files in Radio/TV system (file_ingester.sh)	1 month of recording-files	1.5 mins	To be done in prod
Preingest	546105 xml-files output (21 channels (radio+tv), 2005-2010)	3-4-5 days	To be done in prod
Ingest	objects for 1 preingest-file (2-5 objects pr preingest xml-file)	about 2 secs	To be done in prod
Ingest	274670 preingest-files (about half of 2005-2010)	about 12,5 days	To be done in prod
Summa - harvest (incl. ingest)	-	Ca. 2 poster/sek	To be done in prod
Summa - indexing	-	500-1000 poster/sek	To be done in prod
Rebuild (in prod) of Triple Store & Fedora's database	572343 fedora objects ( = 270856 programs) (avg 2.1 fedora objects per program)	5 hours ( = 303.5 mins)	As done in prod after crash

-  ⇤ ← Revision 12 as of 2008-10-17 16:18:40 → 
  Size: 1211
  Editor: abr
  Comment:
+   ← Revision 72 as of 2011-02-24 15:56:00 → ⇥
  Size: 6122
  Editor: jrg
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 2:
-Line 5:
+Line 4:
+== Install Guide for DOMS with Radio/TV ==

 * [[Install-sequence]] for parts of the DOMS system, install guides, and status of DOMS in stage
 * [[Ideas for improvement of DOMS system installation]]
 * [[Generating a rick-roll file]]

== Design ==
The top level design documentation will be available here.

 * [[Deployment and Design of DOMS|The used modules, and the big lines about how they should be deployed and interact]]
 * [[HighLevelDesignDocumentation|High Level Design Documentation]]

## * [:TaskA.4.1DesignDocument: Ingest and Validation Design Document]
 * [[View design document|Design of datamodel views]]
 * [[User System|User system preliminary design]]

== Packages ==
=== Bitstorage ===
 * [[LowLevelBitStorageDesignDoc|Low level bitstorage webservice design]]

=== ECM ===
 * RELEASE NOTES
 * INSTALL GUIDE
 * ?

=== IP-Role Mapper ===
 * [[IPRoleMapperUsage|Usage]]
 * [[IPRoleMapperConfiguration|Configuration]]
 * [[IPRoleMapperInstallation|Installation]]

=== Update tracker ===
=== DOMSServer ===
[[DOMS Server API]]

=== DOMSClient ===
=== Ingest ===
==== Radio-TV Ingester ====
 * [[RadioTVIngesterUsage|Usage]]
 * [[RadioTVIngesterConfiguration|Configuration]]
 * [[RadioTVIngesterInstallation|Installation]]

=== DOMS-GUI ===
=== OAI-PMH handling ===
=== Summa integration ===
=== Broadcast Extraction Service ===
 * [[BESReleaseNotes|Release Notes]]
 * [[BESConfiguration|Configuration]]
 * [[BESUsage|Usage]]
=== DOMS Wowza plugin ===
 * [[DOMSWowzaPluginCheatSheet|Wowza Plugin Cheat Sheet]]
 * [[DOMSWowzaPluginReleaseNotes|Wowza Plugin Release Notes]]
-Line 6:
+Line 56:
+Note: APIs below here are abandoned.
-Line 10:
+Line 61:
- * [:Fedora 3.0 API:Fedora 3.0 API as used by DOMS]
-Line 12:
+Line 62:
- * [:LDAP API:The LDAP authentication API as used by DOMS]
 * [:Bitstorage API:API for bitstorage as used by DOMS]
 * [:File characterization API:API for file characterisation as used by DOMS]
 * [:PID API:API for getting a PID as used by DOMS]
 * [:Search API:API for searching, as used by DOMS]
+ * [[Fedora 3.0 API|Fedora 3.0 API as used by DOMS]]
 * [[LDAP API|The LDAP authentication API as used by DOMS]]
 * [[Bitstorage API|API for bitstorage as used by DOMS]]
 * [[File characterization API|API for file characterisation as used by DOMS]]
 * [[PID API|API for getting a PID as used by DOMS]]
 * [[Search API|API for searching, as used by DOMS]]
 * [[Risearch API|API for requests to the Resource index]]
 * [[Object creation API|API for getting a new object, cloned from a template]]
 * [[Object manipulation API|API for adding information to objects, currently files]]
-Line 19:
+Line 73:
-Line 22:
+Line 75:
- * [:DataModel: Data Model]
+ * [[DataModel|Data Model]]
 * [[DataModelForDDAIntoMiniDOMS|Data Model for De Danske Aviser in MiniDOMS]]
 * [[DataModelForTV2ReklamerIntoDOMS|Data Model for TV2 Reklamer in DOMS]]
 * Data Model for the olde MiniDOMS is available at http://merkur/domswiki/MiniDOMSDataModel (Notice, you are now leaving the domswiki, going to the barren wastelands of the old abandoned MiniDOMS merkur-wiki.)
-Line 29:
+Line 86:
+== Data Provider ==
 * [[DOMS OAI-PMH repository|DOMS as an OAI-PMH repository]]
-Line 30:
+Line 89:
-== Data Provider ==
+== GUI Code Docs ==
 * [[DOMSGUIUnderstandingFromMjoelner|Understanding gained from Mjølner]]
-Line 32:
+Line 92:
- * [:DOMS_OAI-PMH_repository: DOMS as an OAI-PMH repository]
+== Fedora 3 Project ==
 * [[Fedora3/Fedora3|Main document for the Fedora team]]
-Line 34:
+Line 95:
-== Design ==
+== Picture video for wowza ==
 * [[Wowza/PictureVideoScript|Picture video script]]
-Line 36:
+Line 98:
-The top level design documentation will be available here.
+== Deployment History ==
||<tablewidth="200px">webapp||svn||date||comments||
-Line 38:
+Line 101:
-## * [:TaskA.4.1DesignDocument: Ingest and Validation Design Document]
 * ["View_design_document"]
+||<tablewidth="200px">bes|| ||2010-11-10|| ||
||bes_dev||1308||2010-11-26||vlc-based transcoding||
||bes_exp||1308||2010-11-26|| ||
|| || || || ||
|| || || || ||
|| || || || ||
|| || || || ||
|| || || || ||
|| || || || ||

== "Runtimes" for parts of deployment ==
Useful for those tasks that may be done for deploying DOMS with Radio/TV-recordings and metadata for 2005-2010.

As of 2011-02-11, after updating program mappings, the prod system ritzau database contains (for 2005-2011) a total of 4.883.822 programs. (These are for channels ''100fm dk4 dr1 dr2 drbo drhd drk drol drp1 drp2 drp3 drp4b drp4f drp4k drp4k94 drp4mv drp4nj drp4o drp4re drp4s drp5000 drpk drram drup drx kanal4 kanal5 kanal6 novafm radio2 sbsnet tv2c tv2d tv2l tv2news tv2oj tv2radio tv2s tv2sport tv2z voice'')

'''''Examples''''' of how long runs and batches have taken:

|| '''Run/batch/process''' || '''Size of task''' || '''Time''' || ||
|| Clear metadata in Ritzau Database (thru psql) || 4553311 objects                         || about 39 mins || ''Not to be done in prod'' ||
|| Clear metadata in Ritzau Database (thru psql) || 123499 objects (1 month of metadata)    || about 2 mins  || ''Not to be done in prod'' ||
|| Update channel-mappings in Radio/TV GUI || ? (Colin finds out)                           || 45 mins for 2.300.000  || ''To be done in prod'' ||
|| Fill Ritzau Database (bart_systemet_mirror.sh) || 1 month of metadata                    || 7 mins || ''To be done partly in prod (?)'' ||
|| Ingest references to recording-files in Radio/TV system (file_ingester.sh)               || 1 month of recording-files || 1.5 mins || ''To be done in prod'' ||
|| Preingest         || 546105 xml-files output (21 channels (radio+tv), 2005-2010)         || 3-4-5 days || ''To be done in prod'' ||
|| Ingest            || objects for 1 preingest-file (2-5 objects pr preingest xml-file)    || about 2 secs || ''To be done in prod'' ||
|| Ingest            || 274670 preingest-files (about half of 2005-2010)                    || about 12,5 days || ''To be done in prod'' ||
|| Summa - harvest (incl. ingest)  || - || Ca. 2 poster/sek    || ''To be done in prod'' ||
|| Summa - indexing || - || 500-1000 poster/sek || ''To be done in prod'' ||
|| Rebuild (in prod) of Triple Store & Fedora's database || 572343 fedora objects ( = 270856 programs) (avg 2.1 fedora objects per program) || 5 hours ( = 303.5 mins) || As done in prod after crash ||