I’m writing to announce that in early August I will be moving on to an exciting new opportunity as the Digital Services Librarian at the University of Illinois at Chicago.

 It’s most important to convey that the California Digital Library is dedicated to making sure that this is a smooth transition for Web Archiving Service users.  I will still facilitate the upcoming WAS Users Group at the Society of American Archivists on August 10th, and I will still lead WAS workshops on August 13 and 14 at the UCLA and UC Irvine campuses. Existing members of the WAS support team and additional CDL staff will join those sessions to expand expertise and training for the service.   

You can continue to reach our WAS support team by using the “Contact WAS” link on any WAS screen, or by sending email to washelp@ucop.edu. Whether you have specific questions about a capture or general questions about the service, members of our WAS support team will respond.  If you have questions about personnel changes please contact Trisha Cruse patricia.cruse@ucop.edu

 It has been an incredible honor to work with the talented staff at CDL, to become part of the international web archiving community, and to work with the many partner organizations affiliated with WAS.  I’m very proud of the work that CDL has done to make the Web Archiving Service such an outstanding resource for librarians and archivists. And I’m very appreciative of the input that our WAS curators from so many organizations have provided every step of the way to help make the service what it is.

It’s amazing how much the web has changed in these last eight years… how much more a part of our lives it has become and how strongly it influences world events. The work that WAS curators are doing is more important than ever!

Best regards,
Tracy Seneca
Web Archiving Service Manager
California Digital Library

Service Manager
Tracy Seneca (tracy.seneca@ucop.edu) or washelp@ucop.edu.

Additional WAS Information  (training materials, videos, guides, etc.)
More information about WAS is available at http://was.cdlib.org or by sending an inquiry to washelp@ucop.edu.

WAS Activity, June 2012

• 93 archives actively collected
• 1,663 sites collected
• 1.5 TB of data collected

Web Archiving Service User’s Group Meeting Scheduled for SAA Annual Meeting
CDL will be hosting a Web Archiving Service User’s group meeting in conjunction with the upcoming Society of American Archivists Annual Meeting in San Diego. Since many WAS users from around the country will be attending SAA, this will be a great opportunity to share the work you’ve been doing, discuss best practices, and establish a more formal community of Web Archiving Service users.

You do not need to be registered for the SAA conference to attend this meeting, so I encourage WAS users from the UC campuses who may not be going to SAA to join us if you’re able to get to San Diego.

Please mark this event on your calendars if you can join:

Friday, August 10th
1pm to 4pm
Hilton San Diego Bayfront
1 Park Boulevard, San Diego, CA 92101
Room 202, Indigo A

Please send email to tracy.seneca@ucop.edu if you’d like to attend. We will be providing refreshments, so an early attendee list will help us plan for catering!

Jonathan LeBreton, Senior Associate University Librarian for the Temple University Libraries states: “We plan to use WAS to collect and preserve not only Temple University’s own websites, but also third party web sites which align with our existing strong archival collections including the Philadelphia Jewish Archives, Dance in Philadelphia, African American history, labor and civil rights organizations and contemporary culture”.

Purdue archivists also plan to use the service to harvest and preserve campus websites. In addition to being a major research library, Purdue is actively at the forefront of large-scale digital data management. Purdue is a member of DataCite and a partner in the DMPTool. With WAS, Purdue embarks on another new arena of digital preservation.

We look forward to working with both organizations as they build their archives!

UCSF Legacy Tobacco Documents Library Integrates Archived Websites

The April/May Web Archiving Service update described a forthcoming WAS metadata export feature currently being pilot tested by a number of WAS partners, including UCSF. This metadata export feature enables organizations to integrate the display of their archived websites into other discovery systems.

The first public example of this feature is live at the UCSF Legacy Tobacco Documents Library: http://legacy.library.ucsf.edu. For example, a search of the archive on “Breathe California” will produce a top result of the archived “Breathe California” website. From that link you can discover the complete archiving history of the site, all of the site’s metadata, and can display any date of capture for that site.

So in addition to the wealth of scanned materials, UCSF is now able to integrate a growing number of websites documenting both the current and past activity of tobacco companies, health advocacy groups and researchers.

Purdue joins the University of Michigan Bentley Historical Library and the Northwestern University Library as the 3rd Committee on Institutional Cooperation (CIC) member to subscribe to the Web Archiving Service.  The CIC is a consortium of the Big Ten universities plus the University of Chicago, and all CIC members now qualify for a discount on the yearly service fee for WAS.  Further details about service features and pricing are available in this brief guide to WAS. 

We look forward to working with the Purdue web archivists and we invite CIC member libraries to contact us for further information, a demo or a trial of the service.  (All other libraries are welcome to contact us as well, of course!). 

Tracy Seneca
Web Archiving Service Manager
washelp@ucop.edu

Temple University: Paley Library

Jonathan LeBreton, Senior Associate University Librarian for the Temple University Libraries states: “We plan to use WAS to collect and preserve not only Temple University’s own websites, but also third party web sites which align with our existing strong archival collections including the Philadelphia Jewish Archives, Dance in Philadelphia, African American history, labor and civil rights organizations and contemporary culture”. 

We look forward to working with archivists at Temple and to seeing the resulting archives!  For further information about the Web Archiving Service, see http://was.cdlib.org, or email washelp@ucop.edu. 

You do not need to be registered for the SAA conference to attend this meeting, so I encourage WAS users from the UC campuses who may not be going to SAA to join us if you’re able to get to San Diego. 

Please mark this on your calendars if you can join:

Friday, August 10^th:  1pm to 4pm
Hilton San Diego Bayfront
1 Park Boulevard, San Diego, CA 92101
Room 202, Indigo A

Please send email to tracy.seneca@ucop.edu if you’d like to attend.  We will be providing refreshments, so an early attendee list will help us plan for catering!

The Web Archiving Service (WAS) enables librarians, archivists and researchers to capture, curate and preserve websites and web‐published materials.   WAS makes it easy to build web archives, with scheduling and other tools to help manage your archive.  You control public access to your archives and can configure the appearance and navigation of each archive.  We also provide collection development consultation and help desk support for web archiving questions.

WAS Service Manager

Tracy Seneca tracy.seneca@ucop.edu or washelp@ucop.edu

Additional WAS Information (training materials, videos, guides, etc.) More information about WAS is available at <was.cdlib.org> or by sending an inquiry to <washelp@ucop.edu>  

WAS Harvest and Collection Activity, April – May 2012

• 102 archives actively collected

• 2,313 sites collected

• 2.9 TB of data collected

Recent Enhancements, News, and Activities

International Internet Preservation Consortium General Assembly The IIPC General Assembly took place in early May in Washington DC, with CDL participating as a long-standing member, and actively engaged in the IIPC’s Access Working Group.  The consortium is composed of a growing range of national libraries, academic institutions and other organizations engaged in web archiving.  The IIPC actively promotes the development of open source tools, standards and best practices in web archiving.

The Broad Value of Web Archives: General Use  The opening day of the IIPC assembly was devoted to an open session on talks devoted to researcher use of web archived data.  These sessions presented direct researcher use cases of data derived from web archives.  A WAS News posting on researcher Kalev Leetaru’s work on the mutability of White House Press releases on the web is available at: http://www.cdlib.org/cdlinfo/2012/05/31/11716, and further findings from these presentations will be shared in the upcoming weeks.  The slides for these

sessions are available at http://netpreserve.org/events/2012ga.php, and video interviews with the researchers who presented will be made available in the future.

2012 General Assembly Outcomes

Harvesting Working Group Tracy Seneca, CDL’s Web Archiving Service Manager will co-chair the IIPC Harvesting Working Group.  This working group has been at the heart of the IIPC’s activity since its inception in 2003.  The HWG guides the development priorities and process for Heritrix, the harvester behind the Web Archiving Service, the Internet Archive, the Archive-It service, the NetArchive Suite, the Web Curator Tool, and the majority of locally built web harvesting systems.  Heritirx currently stands at a cross-road in its development, with a need to foster more truly collaborative community development on future enhancements to the crawler.  This will be a noteworthy responsibility and highlights the significant role that CDL plays in IIPC.

Tracy will be setting down her prior role in the Access Working Group, and the IIPC Steering Committee has indicated that a curator from a UC Campus would be welcome to take up involvement in that working group as part of CDL’s membership in the organization.  We will be communicating with UC WAS administrators and curators about prospects for participation.  We would love for UC web archivists to have a chance to be involved in broader initiatives!

 Twittervane One of the initiatives of the Access Working Group via the British Library has been the Twittervane project, which aims to augment collection development for event archives by drawing site nominations based on trending Twitter topics.  This is a similar approach to that used by CDL to identify key Gulf Oil Spill websites when we pulled in RSS feeds from sites tagged in Delicious.  Selecting sites for event archives can be notoriously time-consuming, so any tools for extending and managing the site nomination process are welcome!  At this time, Twittervane has only been used in the context of the British Library’s selection process, but it will be made available to the community as the project wraps up.  We will keep you posted as Twittervane becomes more widely accessible.  Slides with further background are at: http://netpreserve.org/events/dc_ga/02_Tuesday_IIPC/Hockx-Yu.pdf   

 2012 Olympics Nominations Nominations are open for the 2012 Summer Olympics Project.  The goal of this project is to lay the groundwork for interoperability between diverse web archives.  That foundation will be a series of archives on the Olympics using a shared descriptive vocabulary.  Because each participating nation may be bound by varying rules concerning copyright and legal deposit, the collection effort will be a fully international effort.  Nearly 1000 websites have been nominated at this point, and CDL is one of the organizations participating in site harvesting.  To review the nominations or contribute your own, visit the Olympics 2012 nomination tool:  http://digital2.library.unt.edu/nomination/olympics2012/  You simply need to provide your name and email address to contribute.  You are able to provide a site name, URL, description, and to select a relevant sport, nation, language or subject.   

Columbia University Meeting On May 10th and 11th, The Columbia University Library hosted “Web Archiving Policies and Practices in the U.S.”.  This was a unique opportunity for US organizations, largely academic institutions, to share their practices, obstacles and visions.  Participants included curators using a range of tools, and included WAS partners from UC Irvine, the University of Michigan and NYU. The session’s goals, participants and program are available at https://webarch.cul.columbia.edu/.  While no formal plans were set for this group to meet again, it was agreed that U.S. academic institutions have a rich range of unique issues to share and collaborate upon outside the structure of the IIPC.  One immediate outcome for Web Archiving Service users was the proposal of a WAS User’s Group meeting in conjunction with the upcoming Society of American Archivist’s Meeting in August in San Diego.  Arrangements for this meeting are in the works, and UC WAS users who are not formally signed up for SAA will be welcome to attend this adjunct meeting.  Details will be provided in the next monthly newsletter, in the WAS News blog, and via email to the all WAS Users list.

Pilot Testing of WAS Metadata Export One forthcoming feature in WAS is an export of all curatorial metadata for each site in any given archive in XML format.  The purpose of this report is to provide a way to integrate access to archived websites with other relevant discovery systems, such as topical digital archives and library catalogs.  The record for each archived site points to a “site details” screen.  Here is a sample screen for a site in the UCSF California Tobacco Control Archive: http://webarchives.cdlib.org/site/sw1qn6068n  

We are currently working with pilot testers from UC Davis, UC San Francisco and the University of Michigan.  These organizations have varied use cases for these records so early pilot tests will provide us with strong foundation for a wide range of needs.  We’re also grateful for early input from Peter Filardo at NYU and Yvonne Wilson at UC Irvine for helping us to establish the requirements for this feature.

Terry Reese of the University of Oregon, and the developer behind the freely available MarcEdit tool, has created a custom crosswalk to transform our WAS report to MARC format records.  When the feature is released, we will provide access to that crosswalk file as well as documentation for installing it and producing records from the WAS report.  

Service Monitoring and Availability Check WAS’ system status page http://www.cdlib.org/contact/system.html

I’d like to focus a couple of upcoming blog posts on some of the most thought-provoking moments in the IIPC presentations.  University of Illinois at Urbana-Champaign research Kalev Leetaru shared findings from many years of analysis of blogs, online newspapers and government web content.  This post focuses on just one of the findings he shared of an analysis of White House press releases as distributed to the web from March to June 2003, at the advent of the Iraq War. 

Certainly web archivists are keenly aware of the value of preserving at-risk government publications at the Federal, State and local levels.  We’re aware that these may be moved or entirely deleted as organizations and administrations change.  Also at issue, though, is that documents that were immutable in print are not so on the web.  The 2008 study “Airbrushing History, American Style”   traces the changes to the March 27, 2003 White House press release listing the “Coalition of the Willing”.  Rather than issue subsequent press releases as nations joined and left the coalition, the original press release was updated, with the original date of the press release left in place. 

Overall, the range of presentations surfaced both anticipated and unexpected values of web archives for researchers.  Future postings will explore some of the other issues raised, particularly as researcher interviews are made available.

The Web Archiving Service (WAS) enables libraries and archives to build collections of archived websites.  For further information about WAS, visit http://was.cdlib.org or contact washelp@ucop.edu.

• Digital Archiving
• Collaboration & Communities
• Digital Preservation
• Web Publishing
• Improving Digital Info Flows & Access
• Leading Edge Technologies & Libraries

Get the word out about the innovative work you’ve been doing!  The call for submissions is open until March 26^th.  http://www.infotoday.com/il2012/CallForSpeakers.asp

UPDATE: Wednesday February 21, 2012

The Web Archiving Service developers have completed their work to improve the performance of the public WAS archives.  Developers are now resuming work on long-term improvements to search speed and capability.

If you have any questions or if you encounter any issues, please contact washelp@ucop.edu.

UPDATE: Friday, February 17, 2012

The Web Archiving Service developers have been working to improve the response time for search and display of the public archives.  Those of you who encountered difficulty using the archives during recent weeks should see improved response time.  Additional work is underway to further improve archive performance. 

The California Digital Library will be closed on Monday, February 20th in observance of Presidents’ Day, but we will be monitoring the service and our help desk during that time.

Again, we apologize for the recent problems with archive performance.  If you have any questions or if you encounter any issues, please contact washelp@ucop.edu.
 

Those of you who use the public web archives created with CDL’s Web Archiving Service (WAS) will have noticed some noteworthy delays when searching and browsing.  CDL’s WAS support team is working to improve the performance of the archives for the immediate short term future, and will soon be migrating to a new indexing system to dramatically improve the archives’ performance.

 We have been communicating with the archivists who build the web archives about our work on this issue, but we also want to communicate with anyone in the general public who may be searching those archives.

 The issue:
Web archives involve working with data at a very large scale.  When first released in 2009, the Web Archiving Service offered approximately 4 terabytes of content across all of the public archives.  Since then, the archives have grown to 20.8 terabytes, and in recent weeks that growth has begun to affect their performance.   The current delays in service are the result of the challenges that our old indexing system (based on Nutch) has with this larger scale of data. 

For the short term: we are taking some immediate steps to improve this performance prior to the more substantial upgrade to the service.

For the long term: we are  approaching completion on the development of a new indexing system (based on Solr) that will:

• address the greater scale of our current archives,
• enable future scaling of the archives, and
• provide greater search control.

Important:

• This issue does not in any way affect the content itself; it only impacts the speed of searching the archives.
• This issue does not affect the curatorial features of the service; archivists and librarians should encounter no difficulty in archiving new materials.

 Next steps:
The WAS developers are looking at a range of solutions to address the immediate performance of the archives.  We expect this work to continue into the week of February 20^th.  The improvements to performance may require us to temporarily disable certain features, such as the display of highlighted search terms in search results.  We will attempt to keep all features available, but our top priority is for the archives to return results in a timely manner. We will keep our Web Archiving Service partners informed of our progress.

 We sincerely apologize for this temporary problem with the performance of the public archives. 

 Please contact washelp@ucop.edu  if you have any questions.