WAIS as a global information retrieval system for PGR

Handout following the ECPGR Workshop ‘Central Crop Databases on-line’, Bonn, June 8-10, 1997
WAIS as an Information Retrieval System on the Internet

Morten Hulden
Nordic Gene Bank, P.O. Box 41, 23053 Alnarp, Sweden

WAIS (Wide Area Information Servers) is an information retrieval system for on-line databases on the Internet. WAIS was originally developed by Apple Computers, Thinking Machine Inc. and Dow Jones. Since the release of the first software package WAIS has become a standard for connecting individual databases on the Internet into a network of servers. WAIS has its own protocol in the TCP/IP (Transport Control Protocol/Internet Protocol) stack. Computers running WAIS form a network on Internet similar to the WWW (World Wide Web).

WAIS is optimised for fast searches based on specific search criteria, and thus is very well suited as a complement to downloadable databases. A user does not have to go through the process of downloading a large file, extracting the database from the archive, running local database software and using a decoding table to find information, but can find the answers on-line within a few seconds by using a WAIS client (or a WWW-search form).

WAIS is not a programme. WAIS is a standard for indexing, storing and retrieving information from source documents that can be located anywhere on the Internet. Several programmes that support WAIS are available, both commercial and free software. Some programmes cover a wide range of source file formats such as text files, HTML, BibTex, Medline, as well as user-defined database structures.

A complete WAIS software package contains three pieces of software: the indexer, the server and the client. WAIS clients are available separately for people who only want to retrieve data. The indexer and (optionally) the server are needed by those who want to provide on-line WAIS databases. In some software packages the WAIS server programme can be replaced or enhanced with a CGI (Common Gateway Interface) script that provides a WWW server running on the same machine with direct access to the WAIS index files and source documents.

Pros and Cons

Advantages

Affordable system with only hardware costs (ordinary PC)
Extensive and versatile file and database format support
Possibilities to link with normal WWW server (allows free design of search and result pages)
Fast retrieval of data thanks to efficient algorithms
Multiple databases can be searched with the same query.

Disadvantages

Setting up from scratch with hardware, operating system and utilities, WAIS server & databases and WWW server, search & result pages etc., is technically demanding and time consuming and may need support from several people.
Several different pieces of software to maintain.

If the computer department at an institute already provides a server machine & utilities, the task is reduced to installing and managing WAIS databases & server programme and (optionally) the search and result pages of a WWW-server.

Example of a working WAIS system linked with the WWW

Requirements
Hardware: PC, Pentium 200MHz, 64MB RAM, 2GB disk space (but any 386+ will do)
Operating system: Linux 2.0.x, free software (GNU General Public License).
C-compiler: GCC 2.7.2, free software (GNU General Public License)
Perl interpreter: Perl 5.003, free software (GNU General Public License)
WAIS package: freeWAIS-sf 2.1.2, free software (GNU General Public License)
WWW server: Cern 3.0, free software (GNU General Public License)
Search engine: SFgate 5.x (enhanced by NGB), free software (GNU General Public License)

The freeWAIS-sf package
FreeWAIS-sf is available as C source code or as precompiled for several UNIX-like operating systems. Compilation on the local machine is recommended, since many more configuration options can be selected before compiling.

The Indexer
The indexer creates index files with entries for each word appearing within the source documents, including entries for every word (or number) within database fields. The indexer is run on the server at least once, or regularly after each update.

Support for 35 predefined source file formats, and user-defined database structures
Character set (defines word boundaries) can be defined freely at compile time
Support for dual field names (short/descriptive)
Support for repeated fields within records (multiple values)
Partial indexing of numeric fields (e.g. only two/four first digits of degree/date fields)
Stop list for words not to be indexed (can be global or database specific)
Global and local indexes (a global index combines certain fields into one searchable unit)
Support for Soundex & Phonix phonetic index types
Support for stemming of the index words

The Server

Support for database-specific single-word synonyms
Number of hits to be returned can be set at compile time
Access logging on several levels

The Clients

Both command line (waisq, waissearch) and graphic (xwais) interfaces are included in freeWAIS-sf, but any WAIS compatible client can be used to access any WAIS databases
Clients can search multiple databases (local or remote) simultaneously
Fully supported Boolean (AND/OR/NOT) search language

SFgate - CGI script to link WWW and WAIS
The CGI script SFgate, written in Perl, can be installed on a machine running a WWW server in order to link HTML search forms with the WAIS indexes and source documents. In this case, the WAIS server programme does not need to be running on the machine.

Linking WWW and WAIS through SFgate provides a very user-friendly way of searching the databases, since the user does not have to know the field names or the syntax of the query language in order to make a search; it is sufficient to fill in a HTML search form.

With a CGI script like SFgate the result of the searches can be presented as preformatted HTML pages, which in addition to the data extracted from the WAIS databases of course also can include pictures, maps and other graphics. Descriptor values that are coded in the source files can be decoded by the script before the data are displayed to the user.