Set-up and System Maintenance of DOMS Test Setup

Do not use this script at the moment. It is currently broken. Please follow the instructions in the "Automated Setup" section below.

1. Check out the DOMS project from SVN.

2. Install MySQL 5 on the machine (see below), if it is not already installed. (sudo /sbin/service mysqld status will tell you if it is installed)

3. If the MySQL-daemon is not running do sudo /sbin/service mysqld start to start it.

4. Execute domssetup.sh in test/bin with the wanted test installation folder (no spaces in the name, please) as argument. (F.x. create a test installation folder where the doms folder was created.)

It will prompt for the root password to the MySQL installation (not to be confused with the root password for the machine), create a database with the name fedora_testfoldername and install Fedora under the test folder.

If you want to install with demo-objects, add the command line switch -d. Example:

  ./domssetup.sh -d ~/mydoms

will install a test DOMS at the folder mydoms under home, with the provided demo-objects.

Controlling Fedora

The installation script installs two scripts bin/start.sh and bin/stop.sh in the the root of the installation folder. These should be used for starting and stopping the Fedora server.

When the fedora server has started, installation can be verified by visiting

http://localhost:7900/fedora/search

Overview over software components

MySQL installation

Still current

Note: This was taken verbatim from Fedora.

If you do not yet have MySQL 5 installed, you need to do this, set the password, and start it up.

For Fedora Core Linux 3 or above, or Redhat Enterprise Linux 4 or above, you should be able to install MySQL, and make it start up at system boot, with the following commands:

sudo yum install mysql-server

Continue with:

sudo /sbin/chkconfig mysqld on
sudo /sbin/service mysqld start

which enables start of MySQL server on startup.

Now you should set the root password:

/usr/bin/mysql -u root
mysql> set password for root@localhost=password('new-password');
mysql> set password for root@<<$HOSTNAME>>=password('new-password');

(Remember to replace <<$HOSTNAME>> with your host name)

Summa on a Stick

Almost current. Follow the exact commands in Iteration7ReleaseTest

1. Download the latest version from http://hera/~te/

2. Unzip it and run summa-setup.sh

3. Change the line <entry key="create_new_index">false</entry> to <entry key="create_new_index">true</entry> in summa-storage-build-4327/config/controllucene.properties

4. Add the following lines to summa-ingest-build-4327/config/targets.properties.xml:

   <!-- Properties for the DOMS target -->
       <entry key="doms">/fedora/domsTestHarvest20080123/summa/data/doms</entry>
       <entry key="doms_run">true</entry>
       <entry key="doms_DigesterClass">dk.statsbiblioteket.summa.ingest.OAI.OAIHarvestDirectoryDigester</entry>
       <entry key="doms_full_ingest">false</entry>
       <entry key="doms_encoding">UTF-8</entry>
       <entry key="doms_prefix"></entry> <!-- All ids from DOMS are already prefixed with 'doms:'. -->
       <entry key="doms_check_output">true</entry>
       <entry key="doms_base_name">doms</entry>

   • Change the line <entry key="targets">oai</entry> to <entry key="targets">doms</entry> in the same file

5. Copy DOMS-data to data/doms

6. Copy DOMS-XSLT's to summa-index-build-4327/config/targets/doms

7. In the file summa-index-build-4327/config/index.properties.xml remove all lines with OpenArchive and add lines like the following:

       <entry key="DOMS_run">true</entry>
       <entry key="DOMS_io_storage">//localhost:8500/summa_storage</entry>
       <entry key="DOMS_xslt_url">file:///home/te/projects/doms_test/summa-build-4327/summa-index-build-4327/config/targets/doms/doms_index.xsl</entry>
       <entry key="DOMS_last_id"></entry>
       <entry key="DOMS_complete_full_index">false</entry>
       <entry key="DOMS_time"></entry>

   • If the installation is considered stable, it is advisable to point to the XSLTs in CVS instead of the local file-system. Change the line with xslt-url to http://merkur/viewvc/trunk/summa/data/index/targets/doms/doms_index.xsl?root=doms&view=co.

Note: create_new_index in controllucene.properties controls whether the index is created from scratch upon each index or if the existing index should be used.

Summa harvester

The harvester is responsible for harvesting data from Fedora (or any other OAI-PMH source) and dump them as XML in a designated data dir. The dumped data can be ingested by Summa.

The harvester is located in the folder summa-harvest-build-4327 in the Summa on a Stick folder. The config.properties and the targets.xml files in the config folder contains setup-information. The detault-settings put harvested data in a data-subfolder in the summa-harvest-build-4327 folder and performs the harvest from http://alhena:7900/oaiprovider/.

The harvester is started with the harvest.sh-script.

Starting the Summa Prototype

It is strongly recommended that the following guide is executed by a user called summa, although this should not be strictly necessary.

Start rmiregistry

Run $JAVA_HOME/bin/rmiregistry 8500 &

Start summa-storage

Start Summa Storage with summa-storage-build-4327/luceneStorage.sh &

Ingest Sample Data

not nessesary for runs after the first run

Run the ingest.sh script found in summa-ingest-<version>/.

This will ingest a fair amount of records. They are taken from the data-folder. If a smaller test is wanted, remove some of the XML-files in data before running the ingester.

Build Main Index

not nessesary for runs after the first run

Run the index.sh script found in summa-index-<version>/. This will take some hours, depending on hardware.

Build Parallel Cluster Index

not nessesary for runs after the first run

Run cluster.sh in summa-cluster-<version>/. This will finish in minutes.

Build Facet Map

not nessesary for runs after the first run

Run facet.sh in summa-facets-<version>/. This will take a minute or two.

Set up Web Server

not nessesary for runs after the first run

Copy all .war files from the root installation directory to the webapps directory in an Apache Tomcat installation.

Start Tomcat, if not already done

Start the Tomcat server with at least 512 MB ram, fx from the Tomcat root directory:

JAVA_OPTS=-Xmx512m bin/startup.sh

Test

Point your favorite web browser (or MS IE if that is your only option) to

localhost:7900/simplesumma

The first search will put the server under a heavy load while the facet map is loaded, but after the initial set up, the service should run smoothly.

Note: Selecting a tag with non-ASCII characters does not work in simplesumma.

Trouble Shooting

Checking if anything is ingested (ie. stored in the postgres database) - run psql and enter

SELECT count (*) from summa_io WHERE base='OpenArchive';

You can query the search web service directly like this

http://localhost:7900/summaws/services/SummaSearch?method=simpleSearch&arg0=0&arg1=10&arg2=hello

To request the contents of a record directly enter the url

http://localhost:7900/summaws/services/SummaSearch?method=getRecord&arg0=oai:doaj-articles:ab394eb6b5edff982c802993b6981bcd

Browse the available Summa web services at

http://localhost:7900/summaws/services

Automated setup

Automated setup is not being maintained. Follow the instructions on Iteration7ReleaseTest instead.

Verify that JDK version 1.6 and ant is installed on the machine and can be invoked from the command prompt. Execute the below commands and verify that the output resembles the output shown here. The output you get may vary depending on your hardware and the current version of the software, however, as long as the version of the software is not older than the below examples then everything should be just fine.

tsh@pc214:~$ java -version
java version "1.6.0_03"
Java(TM) SE Runtime Environment (build 1.6.0_03-b05)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_03-b05, mixed mode)
tsh@pc214:~$ javac -version
javac 1.6.0_03
tsh@pc214:~$ ant -version
Apache Ant version 1.7.0 compiled on August 29 2007

Change directory to some new folder

svn co https://merkur.statsbiblioteket.dk/svn/doms/trunk doms
cd doms/test/bin
./domssetup.sh ../../../fedora_install
./oaiprovidersetup.sh ../../../fedora_install

cd ../../..
mkdir fedora_install/ingest
mkdir fedora_install/ingest/input
mkdir fedora_install/ingest/input/bmech
cp -r doms/data/doms_base_collection fedora_install/ingest/input/base
cp -r doms/test/data/iteration7_testcollection fedora_install/ingest/input/testcollection
cp doms/gentofte_objects/data/fedora_disseminator_object/Gentofte*DigitalObjectBundle.xml fedora_install/ingest/input/bmech
cd doms/ingest
ant release.bin
cp dist/doms-ingest-0.0.1.zip ../../fedora_install/ingest
cd ../../fedora_install/ingest
unzip doms-ingest-0.0.1.zip
rm -f doms-ingest-0.0.1.zip
mv doms-ingest-0.0.1/* .
rm -r doms-ingest-0.0.1

Stop here, if you need to do something before ingest Example to prevent the use of the bitstorage

mv input/testcollection/data input/testcollection/data2

./bin/ingest.sh base
./bin/ingest.sh bmech
./bin/ingest.sh testcollection
cd ../..
cd doms/disseminator_webservice
ant war
cp dist/gentofte_disseminator.war ../../fedora_install/tomcat/webapps
cd ../..
cd fedora_install
./bin/stop.sh
./bin/start.sh

This command clears all data from the bitstorage on halley. Beware, other might be using it.

ssh doms@halley "rm -rf stage/* storage/*"