Major features in the 3.1 release include:

• Improved schema handling for EAD finding aids.  In addition to EAD 2002 DTD,  XTF now provides support for search and display of:
  • EAD 2002 schema and EAD 2002 RelaxNG finding aids
  • Output from Archivists’ Toolkit and Archon
• Better OAI 2.0 conformance
• Dynamic site maps to support optimal search engine indexing

See the 3.1 change log (http://xtf.cdlib.org/documentation/changelog/#3.1) for further details.

XTF is a combination of Java and XSLT 2.0 that indexes, queries, and displays digital objects and is based on open source software (e.g. Lucene and Saxon).  XTF can be downloaded from the XTF website (http://xtf.cdlib.org/download/) or from the XTF Project page on SourceForge (http://sourceforge.net/projects/xtf/), where the source code can also be found. 

The XTF website also provides a self-guided tutorial and a sample of the default installation (http://xtf.cdlib.org:8080/xtf/search), demonstrating the capabilities of the tool out-of-the-box. Both of these resources provide a quick view of the capabilities of XTF prior to download.

Offering a suite of customizable features that support diverse intellectual access to content, XTF interfaces can be designed to support the distinct tools and presentations that are useful and meaningful to specific audiences.  In addition, XTF offers the following core features:

• Easy to deploy: Drops directly in to a Java application server such as Tomcat or Resin; has been tested on Solaris, Mac, Linux, and Windows operating systems.
• Easy to configure: Can create indexes on any XML element or attribute; entire presentation layer is customizable via XSLT.
• Robust: Optimized to perform well on large documents (e.g., a single text that exceeds 10MB of encoded text); scales to perform well on collections of millions of documents; provides full Unicode support.
• Extensible:
• Works well with a variety of authentication systems (e.g., IP address lists, LDAP, Shibboleth).
• Provides an interface for external data lookups to support thesaurus-based term expansion, recommender systems, etc.
• Can power other digital library services (e.g., XTF contains an OAI-PMH data provider that allows others to harvest metadata, and an SRU interface that exposes searches to federated search engines).
• Can be deployed as separate, modular pieces of a third-party system (e.g., the module that displays snippets of matching text).
• Powerful for the enduser:
• Spell checking of queries
• Faceted displays for browsing
• Dynamically updated browse lists
• Session-based bookbags

These basic features can be tuned and modified.  For instance, the same bookbag feature that allows users to store links to entire books can also store links to citable elements of an object, such as a note or other reference.

Examples of XTF-based applications both within and outside of the CDL include:

• eScholarship (http://www.escholarship.org), the University of California’s open access scholarly publishing and research platform.
• Mark Twain Project Online (http://www.marktwainproject.org), developed by the Mark Twain Papers Project, the CDL and the University of California Press.
• Calisphere (http://calisphere.universityofcalifornia.edu/), a curated collection of primary sources keyed to the curriculum standards of California’s K-12 community, developed by the CDL.
• SNAC: The Social Networks and Archival Context Project (prototype) (http://socialarchive.iath.virginia.edu/xtf/search), linking together descriptions of people from finding aids using the new standard Encoded Archival Context-Corporate Bodies, Persons, and Families (EAC-CPF), developed by IATH, University of Virginia (http://socialarchive.iath.virginia.edu/staff.html#iath ), the CDL and the UC Berkeley School of Information (http://socialarchive.iath.virginia.edu/staff.html#ucbsi).
• Various collections at the University of Sydney, Australia, including: Frontiers of Science, University of Sydney Library (http://frontiers.library.usyd.edu.au/); the Sydney College of the Arts Archive (http://va.library.usyd.edu.au).
• The Encyclopedia of Chicago (http://www.encyclopedia.chicagohistory.org/), developed by the Chicago History Museum, The Newberry Library, and Northwestern University.
• The Chymistry of Isaac Newton (http://webapp1.dlib.indiana.edu/newton/) and The Swinburne Project (http://webapp1.dlib.indiana.edu/swinburne/), Indiana University.

John Ober began his stint as CDL Assistant Director for Education and Communications in 1998 and leaves us as Manager of Infrastructure and Applications Support, having held several other positions in between including Director of Scholarly Communications.  Prior to coming to CDL, he served as Development Librarian for Electronic Resources for the Center for Science, Technology and Information Resources/Library Learning Complex, California State University(CSU), Monterey Bay; and Acting Director, Library Systems, at UC Berkeley. As an Instructor at CSU Monterey and an Assistant Professor at UC Berkeley, John taught courses on database management, networks and networked information, and the management of information technology

We will miss John’s intelligence, high standards, excellent judgment, writing skills, passion for scholarship, level-headedness, fairness, tact, moral compass and his wry sense of humor.  Many of his memos included final sections for “the insatiably curious” with tie-ins to perfectly matched and ironically titled country-western songs.  (A recent example had several with the words “share” and “point” in their titles following a memo about UCOP’s move to SharePoint.) 

John leaves CDL May 24.  He and his wife Barbara will be moving to Chico where they plan to expand their gardening space, raise chickens, become involved in the civic life of Chico, and spend time with family.  In the spirit of the C-W song, “When You Leave Walk Out Backwards, So I’ll Think You’re Walking In”.   We will miss you, John, and wish you all the best!

Margaret Low will be retiring from the CDL on June 28 after 30 years of service to UC.  Margaret has been at CDL and its predecessor organization DLA (Division of Library Automation) since the early days and has made significant contributions throughout her valued years of service.  For the past several years Margaret has been a key member of the University of California Curation Center (UC3) team, most recently holding responsibilities for infrastructure, deployment, and operational support for all UC3 service offerings, as well as being the primary point of contact with CDL’s Infrastructure and Applications Support (IAS) and the ITS Unix group.  She has always brought to these activities great professionalism, collegiality, and unflagging good humor. 

After CDL, Margaret plans to use her technical skills to host a personal website for her knitting, quilting and embroidery projects as well becoming more involved with the local craft communities.  She will be spending more time with her family in Southern California and planning long road trips. Since one of her many contributions to CDL was her organization of the coffee club, crucial to those early morning coffee drinkers, we wish you lots of good, strong coffee, Margaret!

Discovery & Delivery’s stalwart Claudia Woo is retiring on June 27.  Claudia started her career in library automation in 1981 working on the budget program.  In the late 1980s Claudia was responsible for the generation of the California Academic Library List of Serials (CALLS) on microfiche. After serials were added to the Melvyl database, and as Abstracting & Indexing (A&I) databases were added throughout the 1990s Claudia worked on her all time favorite program – SGGP, a generic table-driven program that could be configured to process a wide variety of bibliographic metadata records. The trick was to call generic routines in just the right order to accomplish almost any bibliographic metadata normalization.  With the retirement of the mainframe and the locally hosted Melvyl A&I databases Claudia played a leading role in sustaining Aleph Melvyl for nearly 10 years, as well as working on early metasearch programs like Searchlight and on VDX.

Claudia has also provided backup for many Discovery & Delivery systems. There is a growing folder on CDL’s shared server that speaks to Claudia’s contributions.

Claudia has been a wonderful colleague to us all and we admire the quiet good work she does in any community she enters.

Congratulations Claudia – we will miss you!

(Stephan Abrams, Lynne Cameron and Ellen Meltzer contributed to this piece.)

About us

We do more than put information online. At the California Digital Library (CDL), we provide technology and expertise to help the University of California collect, publish, access, and preserve its full range of information resources.  CDL takes a comprehensive approach to digital library development that balances experimentation with reliable production services strategy.  We pursue simple, economical solutions to big challenges.

We manage and continuously improve server, storage, and network infrastructure for application development, staging, and production activities through partnerships with state-of-the-art data centers, Amazon Web Services, and others.

Our open-access publishing platforms help the brightest minds easily disseminate their findings and ideas, putting technology to work for the producers and consumers of new knowledge.

We make interactive library systems work well together, so our users find and access the information they need as quickly as possible.

We are actively building services to manage and preserve digital information of all kinds—from images to raw data to the deepest websites. Our curatorial approach to digital information helps scholars build on past discoveries to create new knowledge.

Learn more at CDL’s website.

 Requirements

• BS in computer science, information management or a related field and 10+ years working in production environments with UNIX-based operating systems & software development that includes a minimum of five years in a supervisory capacity. Master’s degree preferred
• 5+ years information technology resource management experience including management of a team of high-level technical employees
• Proven experience managing & motivating skilled & knowledgeable staff in a technical environment
• Demonstrated experience negotiating arrangements leading to productive & sustainable IT “partner relationships” with data centers, vendors, and external IT infrastructure providers
• Demonstrated experience with technical project management

Preferred

• Knowledge and technical competence in the digital library and higher education domain

 More information

https://jobs.ucop.edu/applicants/Central?quickFind=55094

Our colleague, Martin Haye, a Senior Developer who is the mastermind behind some of CDL’s most important open source contributions, such as the XTF search and display framework and the eScholarship back-end systems , has engineered a user-friendly robot that pretty much puts him in our offices without physically having to be here.

Here’s what he told us…

Q: How long have you been working virtually?

A: About four years now. I’ve worked at a distance from the beginning of my time at CDL but at first it was just by email and phone.

Q: You’ve been through a couple of different iterations that I’ve seen- could you tell me how the idea for a virtual you got started?

A: Well, it started out because we moved to an office space that had a really nice counter space and I wanted to take advantage of it. …I wanted to put myself using Skype on a laptop on the counter, plus be portable enough for people to carry me to meetings. But no one actually ever did that because there were too many cords to deal with.

Q: Have you noticed a difference in how people are reacting to this iteration of virtual Martin over any of the other iterations?

A: Lots and lots of people have been talking to me about it. I really like that I’ve been able to achieve the level of personal interaction that I have dreamed of. Before people had to walk in to [my office space] to see my video self or have a specific question to ask me. But now I have a chance for hallway encounters. For example, I encountered a colleague in the hallway and had an ad-hoc conversation that led to a 15-20 minute meeting. But during that encounter laughs were exchanged and a relationship was renewed.

Q: Do you have an official name for your new virtual self?

A: Well, I’m actually just going to say that I’m Martin and if you want to call my medium something then you could say that it’s Martin’s robot. I got the librarian cataloguing stamp of approval for this naming convention. A colleague of ours offered to look up the proper naming convention for such a setup in the AACR2 (Anglo-American Cataloguing Rules, Second Edition) and we came away with the rule about spirits that channel through a medium. Since I am the spirit of this virtual setup, the main name or entry should be my own name and if someone wants to put a name to the robot medium it would just be a secondary name or entry.

Q: Can you tell me a bit about how you built this robot?

A: Well, it all started at Christmas a year ago when I got a robot kit as a Christmas gift. It had an aluminum base with a motor and wheels. But it took me over a year to design and implement the setup that you now see. I wanted the robot to be tall enough so that I could talk with people at a conference table and I wanted it to have a pretty substantial base so that it wouldn’t tip over.  To make it tall enough I created a shape like an oil derrick so that’s what you’d see if you took the jacket off.  The power system was the most challenging component and I had to experiment with a couple of different solutions.  I wanted something that would allow me to be mobile, reliable, and safe. One system design would have been too much of an electrocution hazard. I finally settled on a hefty uninterruptible power supply (ups). Next, I included a plain old monitor on the setup, and a Mac Mini to run the video software that allows you to see me. Finally, I have two cameras attached in front and back. The front camera has a fish eye lens and can see 180 degrees both horizontally and vertically. The back camera is pointed down and helps me when I want to back into my work space where I connect the plug to the electrical outlet for charging in between my little forays around the office.

Also, the video conferencing system is something that I’ve programmed myself based on open source software.

Q: What’s been the most challenging part of the robot? Have you had any accidents?

A: Maneuvering is definitely a challenge. Sometimes I can’t see very well and I bump into a lot of walls. One time I brushed a postcard off of someone’s wall when they weren’t there and I couldn’t pick it up. I drove back later to apologize and that was fun.

But for the most part it’s been an enormous success and a really great way to interact with my colleagues.

Next time you’re at the CDL offices, be sure to stop by to say hello to Martin.

Photo credit: Craig Thompson

UPDATE: This was a great presentation, particularly for those of you curious to know more about the technical details of web archiving.  Slides and video will be linked from here when available. 2/13/12

The CDL has adopted and published a set of Information Technology Key Security Guidelines & Baseline Supporting Practices, along with a companion System Access and Account Management Policy. The guidelines are available at http://www.cdlib.org/about/policies.html. Thanks are due to the CDL Tech Council and Director’s Cabinet, and the html markup of Eric Satzman’s web production group (notably Craig Thompson).

These security guidelines complement the March adoption of a privacy policy. Enumerating “departmental” level policies, guidelines, and baseline supporting practices for both privacy and IT security that complement existing Federal, State, and University policies allows the CDL to act consistently and confidently in these areas and to respond to related inquiries from the CDL’s constituencies and clients.

Continuing the trend, CDL Executive Director Laine Farley has charged a small group – John Ober, Lisa Schiff, Joan Starr, and Lena Zentall – to develop a departmental Business/Service Continuity and Disaster Recovery framework and implementation plan whose initial results should be available in February, 2012. As with the others, that effort will interpret and extend existing UC and UCOP policies and procedures (see http://www.ucop.edu/riskmgt/emergprep/).

In this great TEDx talk, two Harvard researchers (Jean-Baptiste Michel and Erez Lieberman Aiden) discuss the insights they’ve gleaned from the Google N-gram Viewer. From the creation of a metric for censorship to the orthography of frustration, the presentation introduces ideas that could spark great research in the digital humanities and elsewhere. It’s fun to watch, too.

You can also try out your own N-grams!

 Highlights from the 3.0 release include:

• Scanned book display support in default UI
• Stability improvements to index rotation support
• Globalization and RSS support
• Further Unicode improvements
• Many bug fixes

See the full change log (http://xtf.cdlib.org/documentation/changelog/) for further details.

The Publishing Group of the California Digital Library (CDL) announces the launch of the eXtensible Text Framework (XTF) website (http://xtf.cdlib.org/), supporting a robust open-source application for providing access to digital content. Developed and maintained by the CDL, XTF functions as the primary access technology for the CDL’s digital collections and similar projects worldwide.

XTF excels in supporting rapid, customized application development and deployment. Its high degree of extensibility and performance (even for large documents and large collections) frees implementers to focus on building sophisticated presentations for their digital object collections.

“It’s all about balancing flexibility and ease of use: putting infinite customization ability in the hands of curators and scholars with a driving need to provide deep access to their special collections,” says XTF lead developer Martin Haye.

XTF-based applications range from primary source image collections to publishing platforms and archival finding aid repositories at the University of California and many other institutions, including Northwestern University, the University of Sydney (Australia), Indiana University, Visual Arkiv (Sweden), Durham University (UK), and the University of Virginia.

Implementations include:

• CDL’s eScholarship (http://www.escholarship.org/), UC’s open access scholarly publishing platform, which publishes recent research from across the 10 campuses as well as nearly 40 UC-based scholarly journals
• CDL’s Online Archive of California (http://www.oac.cdlib.org/), a collection of more than 20,000 archival finding aids and 200,000 digital primary sources (images and texts) from more than 150 archives, libraries, and other institutions in the state of California
• The Encyclopedia of Chicago (http://encyclopedia.chicagohistory.org/), a collaboration between the Chicago Historical Society, Northwestern University, and the Newberry Library
• Indiana University’s The Chymistry of Isaac Newton (http://webapp1.dlib.indiana.edu/newton/), a digital repository of transcriptions of Newton’s alchemical manuscripts

The new site serves as an expanded resource for programmers, librarians, and the general public to explore and implement the Java and XSLT 2.0-based framework. Features include:

• XTF application download – full release or core updates to maintain customizations (http://xtf.cdlib.org/download/)
• Documentation, including downloadable deployment guide, programming guide, and tag reference (http://xtf.cdlib.org/documentation)
• Video tutorials focusing on basic setup and customization of XTF (http://xtf.cdlib.org/getting-started-tutorials)
• Example XTF implementations highlighting customized features. (http://xtf.cdlib.org/xtf)

For a full list of XTF’s features and benefits, as well as a technical overview, please visit http://xtf.cdlib.org/about or address queries to Martin Haye at martin.haye@ucop.edu.

By Lisa Schiff, eScholarship Publishing Program Technical Lead

The California Digital Library (CDL) is pleased to announce the availability of an extensive self-guided tutorial for its eXtensible Text Framework (XTF) application.  XTF is an open source, highly customizable piece of software supporting the search, browse, and display of heterogeneous digital content and offering efficient and practical methods for creating customized end-user interfaces for distinct digital collections.  The tutorial provides guidance for implementing and customizing XTF, from core functionality to overall look and feel.Downloads for the Mac and Windows operating systems are available from the XTF Project page on SourceForge, along with the complete distribution and documentation.

The tutorial comes with a complete XTF package that is ready to run when uncompressed; no other installation is required.  It contains nine modules spanning the most powerful and popular features, including how to:

• Add new content
• Change metadata
• Change logo and colors
• Increase significance of titles in ranking hits
• Customize and enable default status of advanced search
• Change fields displayed in search results
• Enable structural searching
• Create a hierarchical facet
• Change footnote behavior

XTF Background and Overview
Since first developing and deploying this indexing and display technology in 2005, the CDL has worked to build and maintain XTF as a highly customizable application built upon tested components already in use by the digital library and search communities – in particular the Lucene text search engine, Java, XML, and XSLT.  By coordinating these pieces in a single platform that can be used to create multiple unique applications, the CDL has succeeded in dramatically reducing the investment in infrastructure, staff training, and development for new digital content projects.

XTF offers the following core features out of the box:

• Easy to deploy: Drops directly in to a Java application server such as Tomcat or Resin; has been tested on Solaris, Mac, Linux, and Windows operating systems
• Easy to configure: Can create indexes on any XML element or attribute; entire presentation layer is customizable via XSLT
• Robust: Optimized to perform well on large documents (e.g., text that exceeds 10MB of encoded text); scales to perform well on collections of millions of documents; provides full Unicode support
• Extensible:
• Works well with a variety of authentication systems (e.g., IP address lists, LDAP, Shibboleth)
• Provides an interface for external data lookups to support thesaurus-based term expansion, recommender systems, etc.
• Can power other digital library services (e.g., XTF contains an OAI-PMH data provider that allows others to harvest metadata, and an SRU interface that exposes searches to federated search engines)
• Can be deployed as separate, modular pieces of a third-party system
• Powerful for the end user:
• Spell checking of queries
• Faceted displays for browsing
• Dynamically updated browse lists
• Session-based bookbags

A sampling of XTF-based applications include:

• Mark Twain Project Online (http://www.marktwainproject.org), developed by the Mark Twain Papers Project, the CDL and the University of California Press.
• Calisphere (http://www.calisphere.org/), developed by the CDL.
• The Encyclopedia of Chicago (http://www.encyclopedia.chicagohistory.org/)
• The Chymistry of Isaac Newton (http://webapp1.dlib.indiana.edu/newton/) and The Swinburne Project (http://webapp1.dlib.indiana.edu/swinburne/www/swinburne/)
• Finding Aids at the New York Public Library (http://labs.nypl.org/2007/10/30/extensible-text-framework-xtf/)
• EECS Technical Reports (http://sunsite2.berkeley.edu:8088/xtf/servlet/org.cdlib.xtf.crossQuery.CrossQuery?rmode=btr)

For more information, visit http://xtf.cdlib.org/ .