Differences between revisions 7 and 8
Revision 7 as of 2011-01-18 11:21:40
Size: 4154
Editor: jrg
Comment:
Revision 8 as of 2011-01-18 11:23:52
Size: 4158
Editor: jrg
Comment:
Deletions are marked like this. Additions are marked like this.
Line 41: Line 41:
This is the hot-folder which the ingester will scan continuously as long as it is running. It will read any {{{.xml}}} files in the directory, whenever it discovers them, and start processing and ingesting them into the DOMS. Therefore it is important that the files either are present in the directory before the ingester is launched or that they are moved into it with an atomic operation - i.e. with {{{mv}}}. Otherwise, the ingester may begin processing a pre-ingest file before it has been fully copied. At time of writing, the preingester does indeed generate files atomically in the hotfolder. ''As the hotfolder should be accessible for reading+writing by the ingester, and accessible for reading+writing by the preingester (from the digitv project), it may be preferable to let the hotfolder be a folder shared over the network.'' This is the hot-folder which the ingester will scan continuously as long as it is running. It will read any {{{.xml}}} files in the directory, whenever it discovers them, and start processing and ingesting them into the DOMS. Therefore it is important that the files either are present in the directory before the ingester is launched or that they are moved into it with an atomic operation - i.e. with {{{mv}}}. Otherwise, the ingester may begin processing a pre-ingest file before it has been fully copied. At time of writing, the preingester does indeed generate files atomically in the hotfolder. ''As the hotfolder should be accessible for reading(+writing) by the ingester, and accessible for writing(+reading) by the preingester (from the digitv project), it may be preferable to let the hotfolder be a folder shared over the network.''

Radio-TV Ingester Configuration

Assuming that you have unpacked the release bundle .zip file in the directory $BASEDIR then the directory $BASEDIR/bin should contain these files:

-rw-r--r--  1 fedora fedora 1021 Nov 25 14:31 ingest.sh
-rw-r--r--  1 fedora fedora  343 Jan 12 14:11 setenv.sh

The ingest.sh is the launch script for the ingester while setenv.sh is a helper script containing configuration details such as username, password and the web service URL of the DOMS server. Thus, you will probably want to edit the latter before launching the ingester.

Let's have a look at the contents of the setenv.sh file:

#
# Basic Parameters
#
COLDFOLDER=$BASEDIR/files/coldfolder
LUKEFOLDER=$BASEDIR/files/lukewarm
HOTFOLDER=$BASEDIR/files/hotfolder
STOPFOLDER=$BASEDIR/files/stopfolder
WSDL=http://alhena:7980/centralDomsWebservice/central/?wsdl
USERNAME=fedoraAdmin
PASSWORD=fedoraAdminSecretPass
SCHEMA=$BASEDIR/config/preingestedRadioTVProgram.xsd

The ingester launch script (ingester.sh) will substitute the $BASEDIR with the absolute file path of the installation directory of the ingester release bundle and thus the default location of the ingesters' work directories is next to the bin directory. Now, let us go through the variables and what they control.

COLDFOLDER

This variable must be assigned with a directory path to where the ingester should move the pre-ingest files to, after they have been successfully processed and ingested into the DOMS.

LUKEFOLDER

This variable must be assigned with a directory path to where the ingester should move the pre-ingest files to if any errors occur while processing and ingesting them into the DOMS. These "lukewarm files" will be moved to LUKEFOLDER. Each file will be accompanied by a file having the same file name with a .failedPIDs suffix, which contains a list of PIDs of DOMS objects that have been created, but not yet published, in the DOMS, before the failure occurred. This is useful for the following debugging and clean-up process.

HOTFOLDER

This is the hot-folder which the ingester will scan continuously as long as it is running. It will read any .xml files in the directory, whenever it discovers them, and start processing and ingesting them into the DOMS. Therefore it is important that the files either are present in the directory before the ingester is launched or that they are moved into it with an atomic operation - i.e. with mv. Otherwise, the ingester may begin processing a pre-ingest file before it has been fully copied. At time of writing, the preingester does indeed generate files atomically in the hotfolder. As the hotfolder should be accessible for reading(+writing) by the ingester, and accessible for writing(+reading) by the preingester (from the digitv project), it may be preferable to let the hotfolder be a folder shared over the network.

STOPFOLDER

The sole purpose of the stop-folder is to provide a means for terminating the ingester process gracefully. In order to do this create a file in this directory, named stoprunning, e.g. by issuing this command in the directory:

touch stoprunning

The ingester will then terminate as soon as it detects the file, however, it may take a while as it only scans for the stoprunning file for each 20th ingest.

Note: The stoprunning file is not removed automatically, so you must remember to delete it again, otherwise the ingester will terminate instantaneously when it is launched.

WSDL

This variable must be assigned with the web service endpoint URL of the DOMS server.

USERNAME / PASSWORD

These variables must be assigned with the username and password to use when accessing the DOMS web service. You will most likely want to use the default settings.

SCHEMA

This variable must be assigned with the full path to the XML schema to use for validation of the pre-ingest files. You will most likely want to use the default setting.

RadioTVIngesterConfiguration (last edited 2011-01-18 11:23:52 by jrg)