2009-10-08 IT Maintenance Meeting
Participants: KFC, JRG, ABR, JHLJ, TGC
Backup
The big problem in backup, is getting a consistent backup of the various parts of DOMS data.
DOMS contains:
- Data from Fedora
- Object files in small XML files
- An SQL database
- An RDF database
- Configuration files
- Applications
(and of course bit storage)
The only really important data are the Fedora Objects files. The databases can be rebuilt, the configuration files are under version control, and the application files are of course available under version control as well as as release packages.
However, rebuilding the databases is time consuming (for a large repository counted in days or weeks), so a consistent backup would be preferable. Unfortunately, Fedora does not offer a checkpoint, where object files and databases are considered consistent. This may even be a problem on non-clean shutdowns of Fedora.
Decisions:
- KFC contacts Fedora mailing list, to confirm that there is no transaction checkpoints in Fedora for backup
- If possible: We run the system on a VMWare server, and simply snapshot the entire state. (JHLJ, TGC)
- If possible: We shutdown Fedora every night, and run backup. DOMS services gracefully wait for Fedora to come up again. (KFC)
Bitstorage SSH shell script
We discussed a bit about the behaviour of the SSH scripts that are the interface to DOMS storage. Also, we requested HTTP access to files in stage and storage.
Decisions:
- Knowledge of checksums should be required, to approve a file in "stage". (JHLJ)
- Filesize should be given as extra input to upload, to ensure space is available, and gracefully fail if not. (JHLJ)
- Note that uploads may be concurrent, so the space needs to be reserved
- Bitstorage should keep to disk copies, until data is replicated to two backup tapes (JHLJ)
- If this is not done, we need a status method to tell us whether a file is replicated, so we know when we can delete it
- JHLJ will consider whether "upload" and "approve" could be made idempotent, that is they will report success if we try to upload a file already present, or approve a file already approved (only on correct checksum, of course)
- One API update has been done only on production scripts, not on test script. JHLJ fixes.
- TGC will set up an apache that delivers files in stage and storage in-house.
Configuration
Test/Production deploy