Evolution of CDL-Hosted Abstracting and Indexing Databases (December 2000)

Introduction
Background and History
Recent Changes in the A&I
The RFP for the CDL-Hosted Databases
A New Strategy
Action and Timing
Appendix 1 - List of CDL-Hosted Databases
Appendix 2 - Strategy Endorsed by the University Librarians on 11/27/00
Appendix 3 - Draft Charge of the A&I Transition Steering Committee

Introduction

The recent RFP for CDL-hosted databases has provided an opportunity to re-examine strategies for providing better integration and service levels for abstracting and indexing (A&I) databases, especially the nine that are mounted at the CDL and the seventeen others that are available via the CDL interface using Z39.50.

This article provides background, summarizes what has been learned, and reports on a strategy for evolution endorsed by the University Librarians and the Systemwide Library and Scholarly Information Advisory Committee. All are agreed that the primary goal is clear: to provide as many databases under as few user interfaces as possible, integrate them with other services such as Request, and establish links to holdings and to full content.

Background and History

Local mounting of abstracting and indexing databases began in 1986 with MELVYL MEDLINE, an effort initiated by the health sciences librarians. At the time, there was a desire to provide end users with the capability to easily search databases rather than go through librarian intermediaries. Database aggregators were not well positioned to offer this service. Their user interfaces were cryptic and difficult to use, and their licensing terms were not suitable for providing unlimited access to a large user community. Nor could they provide other desirable features such as integration with campus journal holdings.

The strategy adopted by the University libraries was to load 10-15 core databases in all disciplines, and to provide searching using the same user interface that had been developed for the Melvyl Union Catalog, with integrated links to campus journal holdings.

By 1996, nine databases were available: ABI/Inform, Biosis Previews, Current Contents, Computer database, INSPEC, Magazine and Journal Articles, MEDLINE, National Newspaper Index, and PsycINFO.

However, difficulties emerged that slowed the progress of the program.

• Vendor negotiations: Some vendors, notably those providing core humanities databases, were reluctant to license databases for local mounting or else offered very unfavorable terms.
• Vendor changes: Several of the databases already mounted had to be redesigned and reloaded due to vendor changes in editorial policies and/or data structure, often with very little advance notice and incomplete documentation.
• Data standards: No standard emerged to fully accommodate the needs of sophisticated abstracting and indexing databases. The CDL had to modify the existing MARC standard developed for library catalog records, which required extensive analysis and pre-processing.
• Development time: Although some efficiencies were achieved in loading A&I databases, the aging technology used for CDL-hosted databases and lack of standard data formats limited how quickly a new database could be mounted, with nine months being the average. Competition for programming resources caused some planned databases to be delayed. For example, Compendex was scheduled to be loaded, but access was provided instead via a database aggregator until resources could be freed from other priority tasks.

Beginning in 1994, another strategy was employed to address the above difficulties and speed up the process of bringing databases under the common user interface. A standard search and retrieval protocol known as Z39.50 allows one system to query another and retrieve records for presentation. Users can initiate searches using the user interface of their own system, and the protocol provides a behind-the-scenes translation between system query structures. This method was used to bring another 17 databases under the user interface developed for CDL-hosted databases.

The Z39.50 strategy has also encountered difficulties that make it an increasingly ineffective strategy:

• Development time: Although there are savings in programming resources for database loading, this strategy requires just as much development time for the user interface aspect as does a local mount.
• Vendor support of and conformance with the standard: We have found that database providers often do not implement the full Z39.50 standard, resulting in the loss of functionality or differences between what they offer in their native interface versus through Z39.50. Other vendors choose not to support the standard at all, believing that the market for it is small and not worth the overhead of implementation.
• Service integration: Integration with campus journal holdings is now possible for the "Z" databases, but it functions differently from the other A&I databases. The Request service has not yet been extended to these databases. Although it is usually technically possible to extend most services, it requires an extra level of development and vendor negotiation. Even when successful, it often results in differences between the Z39.50 and the locally mounted databases that are difficult for users to anticipate or understand.
• Vendor changes: As with locally mounted databases, vendors will change their data structures, and in some cases, their entire Z39.50 service, requiring the CDL to re-implement the database. Again, this effort is no less than if the database were mounted locally.

Recent Changes in the A&I Database Environment

By the early 1990's the advent of the web meant that vendors' user interfaces were easier to learn and teach. While user interface evaluation continued to be an important part of database selection criteria, the database licensing program, both via CDL and at the campus level, actively added resources from many providers with many different user interfaces. At present, CDL provides access to 129 databases from external vendors; local campus offerings are likely to add several dozen more for any particular user.

The marketplace also began to demand additional integration services such as links to campus library holdings. The CDL has worked with three major database aggregators to provide such links. In addition, article-level links to publisher sites is becoming a standard service, although it is going through a shakeout period to reach sustainable methods for providing links customized to each library.

Another trend is that database producers provide some functions and content only through direct interaction with their system. PubMed is a case in point. Others include ABI/Inform and the providers of Magazine and Journal Article index that offer access to full content images through their services. In addition, some databases were always too complex or variable to justify local mounting, but are available from their producers or other aggregators. In this category are Biosis Previews backfiles and Web of Science.

The RFP for CDL-Hosted Databases

The RFP for a vendor system to replace the current technology offered vendors the opportunity to bid separately on the union catalog database server, the A&I database server, or the user interface, or they could bid on all three. Out of the seven vendor responses, five bid on the A&I database server for the nine databases locally mounted at CDL, but none had extensive experience in loading these databases. In all cases, they proposed that the CDL convert the vendor data to the MARC format prior to loading. This step represents a large part of the development effort and thus would not offer a great advantage over the current situation.

A New Strategy

Changes in technology and the vendor marketplace have resulted in a new set of tradeoffs for provision of A&I database services. Careful consideration of these changes, along with information from the RFP for CDL-hosted databases, provides an opportunity to revisit the strategy for A&I resources.

At the least, components of a new strategy focused on external provision of A&I databases should include:

• Assessment of expanded or enhanced content and increased functions in vendor systems as a motivation to use external vendor products.
• Preservation of links to campus library holdings.
• Preservation and rational management of links to article content from publishers.
• Use of a few database aggregators where possible to maintain some consistency in user interfaces.
• Integration of other key services such as Request.

Action and Timing

The University Librarians and the Systemwide Library and Scholarly Information Advisory Committee endorse the necessity and desirability of moving to external vendor interfaces for CDL-hosted A&I databases (9 locally loaded and 17 accessed via Z39.50) as part of the transition to new technologies.

UC health sciences librarians have analyzed the tradeoffs between CDL's MEDLINE/HealthStar and PubMed and are now working with NLM to consider whether PubMed can be a viable alternative. Although specific and unique to MEDLINE, their efforts can inform a more generalized process for the other locally mounted A&I databases.

Several of CDL's advisory groups have developed, tested and endorsed a set of criteria for evaluating information resources that can inform a systematic investigation of alternatives. Because the CDL and campus libraries already have experience in evaluating external vendor products, it is possible to develop an efficient process to specify and rank functional requirements and illuminate the tradeoffs in using vendor systems for databases that are currently locally mounted.

An evaluation process will begin in December 2000 to assess vendor options. An evaluation steering committee will be appointed with a representative from each campus. They will be assisted by the Resource Liaisons who will use the Resource Selection/Evaluation Criteria already established for evaluating external resources. CDL staff will provide other information for assessing vendors and working with them to ensure that links to Request, holdings and content are possible. The committee will be responsible for communication and education to the user community during the process. The committee will be charged to make a recommendation by July 2001 as to which external vendors to pursue for the nine locally loaded databases and any of the Z39.50 databases which need to transition to new vendors. Implementation should be complete by July 2002 and parallel systems will be operated through December 2002 if necessary.

Appendix 1 - List of CDL-Hosted Databases

Locally loaded at CDL

• ABI/Inform
• Biosis Previews
• Current Contents
• Computer database
• INSPEC
• Magazine and Journal Articles
• MEDLINE/healthStar
• National Newspaper Index
• PsycINFO

Available via Z39.50

• Anthropological Literature, Avery Index to Architectural Periodicals, Chicano, ESTC, History of Science, Technology, and Medicine; and RLIN Bibliographic File (@RLG);
• ArtAbstracts and Art Index Retrospective (@Wilson)
• MLA (@SilverPlatter)
• PAIS International, RILM Abstracts, and WorldCat (@OCLC)
• GeoRef (@Stanford)

Appendix 2 - Strategy Endorsed by the University Librarians on 11/27/00

A & I Database Evaluation Process

The recent RFP for CDL-hosted databases provided an opportunity to re-examine strategies for providing better integration and service levels for the full range of abstracting and indexing databases. Due to changes in the vendor marketplace and in technology, the University Librarians and the Systemwide Library and Scholarly Information Advisory Committee endorse the necessity and desirability of moving to external vendor interfaces for CDL-hosted A&I databases (9 locally loaded and 17 accessed via Z39.50) as part of the transition to new technologies. Our goals are clear: to provide as many databases under as few user interfaces as possible, integrate them with other services such as Request, and establish links to holdings and to full content. This strategy should result in more databases available via a common user interface than at present, and more uniform service for all databases.

Process and timeframe

An evaluation process will begin immediately to assess vendor options. An evaluation steering committee will be appointed with a representative from each campus. They will be assisted by the Resource Liaisons who will use the Resource Selection/Evaluation Criteria already established for evaluating external resources. CDL staff will provide other information for assessing vendors and working with them to ensure that links to Request, holdings and content are possible. The committee will be responsible for communication and education to the user community during the process. The committee will be charged to make a recommendation by July 2001 as to which external vendors to pursue for the nine locally loaded databases and any of the Z39.50 databases which need to transition to new vendors. Implementation should be complete by July 2002 and parallel systems will be operated through December 2002 if necessary.

Appendix 3 - Draft Charge of the A&I Transition Steering Committee

A CDL A&I Transition Steering Committee will be created as an advisory group to the CDL University Librarian. The committee is charged with:

• Identifying the stakeholders on campuses, notifying them of the process for migration of databases to external vendors, and providing them with an opportunity to provide comments and feedback.
• To conduct forums as necessary to achieve the above.
• Recommending a process for implementation of databases through new vendors that will focus on the informational and educational aspects of the transition.
• Recommend actions that will best achieve the above goals by July 1, 2001.

The Transition Steering Committee will be a nine-campus plus CDL committee and should comprise senior staff with strengths in public services and education, reflect the disciplinary breadth of the databases, and have links to the CDL Education Working Group, Tools & Services Working Group, Joint Steering Committee on Shared Collections, SOPAG, and the RFP Committees.

The CDL will provide staff to support the committee, to analyze and document external database functionality, to evaluate and negotiate technical requirements (links to holdings, links to full-text, Request), and to negotiate business and licensing terms. Resource Liaisons for the respective databases will assist with analytical work of specific databases/vendors.