= Radio-TV Ingester Usage =

In this section we will use the terms {{{HOTFOLDER, COLDFOLDER, LUKEFOLDER}}} and {{{STOPFOLDER}}} defined in the [[RadioTVIngesterConfiguration|Configuration]] section and we will assume that you have already completed the configuration of the ingester.

Ingest operations will usually take very long time to complete and thus you do not want to risk a premature termination due to a failed SSH connection or an accidental logout. To mitigate this risk you can launch the ingester with the {{{nohup}}} command which will ensure that the process will continue running in the event that your shell dies while it is running:

{{{
$BASEDIR/bin> nohup ingester.sh 2>&1 ingesterOutput.txt &
}}}

The above will start the ingester in the background and redirect all standard output/error output to the file {{{ingesterOutput.txt}}}. You can follow the progress of the ingester by using {{{less}}} or {{{tail}}} on it at any time.

The ingester will start scanning for pre-ingest files in the {{{HOTFOLDER}}} as soon as it has been launched, and it will scan continuously as long as it is running. It will read any {{{.xml}}} files in the directory, whenever it discovers them, and start processing and ingesting them into the DOMS. Therefore it is important that the files either are present in the directory before the ingester is launched or that they are moved into it with an atomic operation - i.e. with {{{mv}}}. Otherwise, the ingester may begin processing a pre-ingest file before it has been fully copied.

So, from here, all you need is to fill pre-ingest files into the {{{HOTFOLDER}}} directory in order to ingest them. Alternatively, you can replace the directory with a sym-link to the output directory of the pre-ingester. However, remember to restart the ingester if you do so, otherwise it will not discover the change and nothing will be ingested.

The ingester will move all pre-ingest files to the {{{COLDFOLDER}}} after they have been successfully processed and ingested and the event of any errors they will be moved to the {{{LUKEFOLDER}}} instead. Each of these "lukewarm files" will be accompanied by a file having the same file name with a {{{.failedPIDs}}} suffix, which contains a list of {{{PIDs}}} of, still unpublished, DOMS objects that were created before the failure occurred. This is useful for the following debugging and clean-up process.

When you want to terminate the ingester, whether it is prematurely or not, then you should do that by creating a ''stop file'' in the {{{STOPFOLDER}}}. It is easily done by issuing this command in the directory:

{{{
$STOPFOLDER> touch stoprunning
}}}

The ingester will then terminate as soon as it detects the file, however, it may take a while as it only scans for the {{{stoprunning}}} file for each 20th ingest.

Note: The {{{stoprunning}}} file is not removed automatically, so you must remember to delete it again, otherwise the ingester will terminate instantaneously when it is launched.