Radio-TV Ingester Usage
In this section we will use the terms HOTFOLDER, COLDFOLDER, LUKEFOLDER and STOPFOLDER defined in the Configuration section and we will assume that you have already completed the configuration of the ingester.
Ingest operations will usually take very long time to complete and thus you do not want to risk a premature termination due to a failed SSH connection or an accidental logout. To mitigate this risk you can launch the ingester with the nohup command which will ensure that the process will continue running in the event that your shell dies while it is running:
$BASEDIR/bin> nohup ingester.sh 2>&1 ingesterOutput.txt &
The above will start the ingester in the background and redirect all standard output/error output to the file ingesterOutput.txt. You can follow the progress of the ingester by using less or tail on it at any time.
The ingester will start scanning for pre-ingest files in the HOTFOLDER as soon as it has been launched, and it will scan continuously as long as it is running. It will read any .xml files in the directory, whenever it discovers them, and start processing and ingesting them into the DOMS. Therefore it is important that the files either are present in the directory before the ingester is launched or that they are moved into it with an atomic operation - i.e. with mv. Otherwise, the ingester may begin processing a pre-ingest file before it has been fully copied.
So, from here, all you need is to fill pre-ingest files into the HOTFOLDER directory in order to ingest them. Alternatively, you can replace the directory with a sym-link to the output directory of the pre-ingester. However, remember to restart the ingester if you do so, otherwise it will not discover the change and nothing will be ingested.
The ingester will move all pre-ingest files to the COLDFOLDER after they have been successfully processed and ingested and the event of any errors they will be moved to the LUKEFOLDER instead. Each of these "lukewarm files" will be accompanied by a file having the same file name with a .failedPIDs suffix, which contains a list of PIDs of, still unpublished, DOMS objects that were created before the failure occurred. This is useful for the following debugging and clean-up process.
When you want to terminate the ingester, whether it is prematurely or not, then you should do that by creating a stop file in the STOPFOLDER. It is easily done by issuing this command in the directory:
$STOPFOLDER> touch stoprunning
The ingester will then terminate as soon as it detects the file, however, it may take a while as it only scans for the stoprunning file for each 20th ingest.
Note: The stoprunning file is not removed automatically, so you must remember to delete it again, otherwise the ingester will terminate instantaneously when it is launched.