Set-up and System Maintenance of DOMS Test Setup
Do not use this script at the moment. It is currently broken. Please follow the instructions in the "Automated Setup" section below.
Check out the DOMS project from SVN.
Install MySQL 5 on the machine (see below), if it is not already installed. (sudo /sbin/service mysqld status will tell you if it is installed)
If the MySQL-daemon is not running do sudo /sbin/service mysqld start to start it.
Execute domssetup.sh in test/bin with the wanted test installation folder (no spaces in the name, please) as argument. (F.x. create a test installation folder where the doms folder was created.)
It will prompt for the root password to the MySQL installation (not to be confused with the root password for the machine), create a database with the name fedora_testfoldername and install Fedora under the test folder.
If you want to install with demo-objects, add the command line switch -d. Example:
./domssetup.sh -d ~/mydoms
will install a test DOMS at the folder mydoms under home, with the provided demo-objects.
Controlling Fedora
The installation script installs two scripts bin/start.sh and bin/stop.sh in the the root of the installation folder. These should be used for starting and stopping the Fedora server.
When the fedora server has started, installation can be verified by visiting
http://localhost:7900/fedora/search
Overview over software components
MySQL installation
Still current
Note: This was taken verbatim from Fedora.
If you do not yet have MySQL 5 installed, you need to do this, set the password, and start it up.
For Fedora Core Linux 3 or above, or Redhat Enterprise Linux 4 or above, you should be able to install MySQL, and make it start up at system boot, with the following commands:
sudo yum install mysql-server
Continue with:
sudo /sbin/chkconfig mysqld on sudo /sbin/service mysqld start
which enables start of MySQL server on startup.
Now you should set the root password:
/usr/bin/mysql -u root mysql> set password for root@localhost=password('new-password'); mysql> set password for root@<<$HOSTNAME>>=password('new-password');
(Remember to replace <<$HOSTNAME>> with your host name)
Summa on a Stick
Almost current. Follow the exact commands in Iteration7ReleaseTest
Download the latest version from http://hera/~te/
Unzip it and run summa-setup.sh
Change the line <entry key="create_new_index">false</entry> to <entry key="create_new_index">true</entry> in summa-storage-build-4327/config/controllucene.properties
Add the following lines to summa-ingest-build-4327/config/targets.properties.xml:
<!-- Properties for the DOMS target --> <entry key="doms">/fedora/domsTestHarvest20080123/summa/data/doms</entry> <entry key="doms_run">true</entry> <entry key="doms_DigesterClass">dk.statsbiblioteket.summa.ingest.OAI.OAIHarvestDirectoryDigester</entry> <entry key="doms_full_ingest">false</entry> <entry key="doms_encoding">UTF-8</entry> <entry key="doms_prefix"></entry> <!-- All ids from DOMS are already prefixed with 'doms:'. --> <entry key="doms_check_output">true</entry> <entry key="doms_base_name">doms</entry>
Change the line <entry key="targets">oai</entry> to <entry key="targets">doms</entry> in the same file
Copy DOMS-data to data/doms
Copy DOMS-XSLT's to summa-index-build-4327/config/targets/doms
In the file summa-index-build-4327/config/index.properties.xml remove all lines with OpenArchive and add lines like the following:
<entry key="DOMS_run">true</entry> <entry key="DOMS_io_storage">//localhost:8500/summa_storage</entry> <entry key="DOMS_xslt_url">file:///home/te/projects/doms_test/summa-build-4327/summa-index-build-4327/config/targets/doms/doms_index.xsl</entry> <entry key="DOMS_last_id"></entry> <entry key="DOMS_complete_full_index">false</entry> <entry key="DOMS_time"></entry>
If the installation is considered stable, it is advisable to point to the XSLTs in CVS instead of the local file-system. Change the line with xslt-url to http://merkur/viewvc/trunk/summa/data/index/targets/doms/doms_index.xsl?root=doms&view=co.
Note: create_new_index in controllucene.properties controls whether the index is created from scratch upon each index or if the existing index should be used.
Summa harvester
The harvester is responsible for harvesting data from Fedora (or any other OAI-PMH source) and dump them as XML in a designated data dir. The dumped data can be ingested by Summa.
The harvester is located in the folder summa-harvest-build-4327 in the Summa on a Stick folder. The config.properties and the targets.xml files in the config folder contains setup-information. The detault-settings put harvested data in a data-subfolder in the summa-harvest-build-4327 folder and performs the harvest from http://alhena:7900/oaiprovider/.
The harvester is started with the harvest.sh-script.
Starting the Summa Prototype
It is strongly recommended that the following guide is executed by a user called summa, although this should not be strictly necessary.
Start rmiregistry
Run $JAVA_HOME/bin/rmiregistry 8500 &
Start summa-storage
Start Summa Storage with summa-storage-build-4327/luceneStorage.sh &
Ingest Sample Data
not nessesary for runs after the first run
Run the ingest.sh script found in summa-ingest-<version>/.
This will ingest a fair amount of records. They are taken from the data-folder. If a smaller test is wanted, remove some of the XML-files in data before running the ingester.
Build Main Index
not nessesary for runs after the first run
Run the index.sh script found in summa-index-<version>/. This will take some hours, depending on hardware.
Build Parallel Cluster Index
not nessesary for runs after the first run
Run cluster.sh in summa-cluster-<version>/. This will finish in minutes.
Build Facet Map
not nessesary for runs after the first run
Run facet.sh in summa-facets-<version>/. This will take a minute or two.
Set up Web Server
not nessesary for runs after the first run
Copy all .war files from the root installation directory to the webapps directory in an Apache Tomcat installation.
Start Tomcat, if not already done
Start the Tomcat server with at least 512 MB ram, fx from the Tomcat root directory:
JAVA_OPTS=-Xmx512m bin/startup.sh
Test
Point your favorite web browser (or MS IE if that is your only option) to
localhost:7900/simplesumma
The first search will put the server under a heavy load while the facet map is loaded, but after the initial set up, the service should run smoothly.
Note: Selecting a tag with non-ASCII characters does not work in simplesumma.
Trouble Shooting
Checking if anything is ingested (ie. stored in the postgres database) - run psql and enter
SELECT count (*) from summa_io WHERE base='OpenArchive';
You can query the search web service directly like this
http://localhost:7900/summaws/services/SummaSearch?method=simpleSearch&arg0=0&arg1=10&arg2=hello
To request the contents of a record directly enter the url
http://localhost:7900/summaws/services/SummaSearch?method=getRecord&arg0=oai:doaj-articles:ab394eb6b5edff982c802993b6981bcd
Browse the available Summa web services at
http://localhost:7900/summaws/services
Automated setup
Automated setup is not being maintained. Follow the instructions on Iteration7ReleaseTest instead.
Verify that JDK version 1.6 and ant is installed on the machine and can be invoked from the command prompt. Execute the below commands and verify that the output resembles the output shown here. The output you get may vary depending on your hardware and the current version of the software, however, as long as the version of the software is not older than the below examples then everything should be just fine.
tsh@pc214:~$ java -version java version "1.6.0_03" Java(TM) SE Runtime Environment (build 1.6.0_03-b05) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_03-b05, mixed mode) tsh@pc214:~$ javac -version javac 1.6.0_03 tsh@pc214:~$ ant -version Apache Ant version 1.7.0 compiled on August 29 2007
Change directory to some new folder
svn co https://merkur.statsbiblioteket.dk/svn/doms/trunk doms cd doms/test/bin ./domssetup.sh ../../../fedora_install ./oaiprovidersetup.sh ../../../fedora_install
cd ../../.. mkdir fedora_install/ingest mkdir fedora_install/ingest/input mkdir fedora_install/ingest/input/bmech cp -r doms/data/doms_base_collection fedora_install/ingest/input/base cp -r doms/test/data/iteration7_testcollection fedora_install/ingest/input/testcollection cp doms/gentofte_objects/data/fedora_disseminator_object/Gentofte*DigitalObjectBundle.xml fedora_install/ingest/input/bmech cd doms/ingest ant release.bin cp dist/doms-ingest-0.0.1.zip ../../fedora_install/ingest cd ../../fedora_install/ingest unzip doms-ingest-0.0.1.zip rm -f doms-ingest-0.0.1.zip mv doms-ingest-0.0.1/* . rm -r doms-ingest-0.0.1
Stop here, if you need to do something before ingest Example to prevent the use of the bitstorage
mv input/testcollection/data input/testcollection/data2
./bin/ingest.sh base ./bin/ingest.sh bmech ./bin/ingest.sh testcollection cd ../.. cd doms/disseminator_webservice ant war cp dist/gentofte_disseminator.war ../../fedora_install/tomcat/webapps cd ../.. cd fedora_install ./bin/stop.sh ./bin/start.sh
This command clears all data from the bitstorage on halley. Beware, other might be using it.
ssh doms@halley "rm -rf stage/* storage/*"