California Digital Library » Rosalie Lack

Five things I learned at IIPC

Rosalie Lack — Thu, 16 May 2013 13:54:12 +0000

I recently attended the International Internet Preservation Consortium (IIPC) General Assembly. The IIPC is a consortium of libraries, academic institutions and other organization engaged in web archiving. The IIPC’s General Assembly included three days of member meetings and two days of meetings open to the public. The theme of the public conference day was: Scholarly Access to Web Archives: Progress, Requirements, and Challenges.

Ahmed Alsum from the Web Science and Digital Libraries Research Group at Old Dominion University posted a comprehensive summary of the GA. As you can see from his summary, there were many great presentations and discussions. It was very hard to choose just five things to share, but here they are:

1. Dark(ish) Archives

Because of copyright and privacy issues, many of the national libraries in Europe cannot provide online, public access to their web archives. They can only allow access in the library and many do not even allow printing in the library. So, how do you raise the awareness of web archiving when no one can see the archives?! There was much discussion about creating site lists/registries for the sites in these archives – some felt this would only lead to disappointment when the user finds out that they have to travel to the archive to see the materials. Sound familiar? Yes, finding aids. And YES, they are extremely useful!

Harvesting tools.
Source: WikiMedia Commons

2. Common tools support is critical

Most IIPC members are using the same suite of OS tools –Heritrix and Open Source Wayback. There was a lot of concern about the development path for these tools. At member breakout sessions where future paths were discussed and prioritized, there was a clear message that tools are important. The IIPC steering committee quickly responded by supporting tool management as a top priority for the organization. Be on the lookout for updates and concrete plans shortly.

3. National and University Library of Slovenia is innovative

Besides being wonderful conference hosts, the National and University Library of Slovenia is doing some innovative work when it comes to web archiving. They demo’d a prototype (see screenshots) of a tool for end users to engage and interact with web archives. It includes features such as: gather and save sites; annotate sites; tag sites; and crowd sourcing of metadata. The next generation of web archives is here!

4. Researchers use of web archives

There were several informative presentations by researchers about how they are using web archives. Sophie Geibeil, an historian from Aix-Marseille-Université, uses the archives to study the untold story of North African immigration. Megan Dougherty, a social scientist from Loyola University, was not as interested in site content as much as she was in taking an anthropological point of view of the sites; that is, studying the social aspects of sites – how people share sites, interact with sites, etc. Niels Brügger, from Netlab at Aarhus University, discussed their various research projects in the areas of digital humanities and internet studies, including RESAW and FUTARC. Helen Hockx-Yu, UK Web Archive at the British Library, presented UK Web Archives in the eyes of scholars. She made the case for thinking of the archives not as documents but rather as large datasets for data mining and analysis.

5. Dancing into the future?

David Rosenthal provided historical context for the early days of web crawling and also provided some future challenges. Not surprising, the web as mainly HTML links is rapidly becoming a thing of the past. Turns out all the current, problematic areas (including rich media, database driven features, dynamically generated URIs, etc.) remain challenging, and now add to that the fact that the new web is more and more JavaScript. Is there no rest for the weary? David did leave us with a light at the end of tunnel. He talked about recent work by the Institute National de L’Audiovisuel (INA) in Paris. The team there created a live archive proxy that shows great promise to enable the capture of some of the more problematic content. Also, there is Memento, which provides an aggregation of web archives each collected in slightly different ways by institutions so moving toward covering all the bases.

Fred Astaire.
Source: WikiMedia Commons

David also had a great analogy for one of the challenges of preserving the web today; he says it is like “preserving theatre or dance” because as we view the web it changes to become an individual experience based on who we are, displaying customized ads and other personalized content. As he put it: “Every performance is a unique and unrepeatable interaction between the performers, in this case a vast collection of dynamically changing databases, and the audience. Actually, it is even worse. Preserving the Web is like preserving a dance performed billions of times, each time for an audience of one, who is also the director of their individual performance.” (Source: http://blog.dshr.org/2013/04/talk-on-harvesting-future-web-at.html)

Overall, it was an excellent, thought-provoking conference. Clearly there are lots of challenges ahead for web archiving, but so many more opportunities.

WAS Service Update: April 2013

Rosalie Lack — Fri, 10 May 2013 19:29:55 +0000

Recent Enhancements, News, and Activities

WAS at Society of California Archivists (SCA), Berkeley, CA. April 11 – 13. The WAS team attended and staffed a booth. Rosalie Lack also gave a short presentation to the Online Archive of California (OAC)’s Users Group meeting. http://www.slideshare.net/rosalielack/society-of-california-archivists-sca
International Internet Preservation Consortium General Assembly (IIPC), Ljubljana, Slovenia. April 22‐ 26. The IIPC is a consortium of libraries, academic institutions and other organization engaged in web archiving. IIPC actively promotes the development of open source tools, standards and best practices in web archiving. Rosalie Lack participated as an IIPC member as well as a member of the IIPC’s Access Working Group member. The IIPC’s General Assembly includes 3 days of member meetings and 2 days open to the public. The theme of the public conference day was: Scholarly Access to Web Archives: Progress, Requirements, and Challenges.
Bug fixes. The WAS technical team (Erik Hetzner and Scott Fisher) fixed several ongoing, but intermittent bugs that were causing display issues in both the public and curator interfaces. You should see a marked improvement in display. A reminder to please report any problems that you encounter by sending an email to: washelp@ucop.edu.

WAS Activity, April 2013

113 archives actively collected
1340 sites collected
1.5 TB of data collected

WAS Service Description
The Web Archiving Service (WAS) enables librarians, archivists and researchers to capture, curate and preserve websites and web‐published materials. WAS makes it easy to build web archives, with scheduling and other tools to help manage your archive. You control public access to your archives and can configure the appearance and navigation of each archive. We also provide collection development consultation and help desk support for web archiving questions.

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

WAS Service Update: March 2013

Rosalie Lack — Thu, 18 Apr 2013 21:44:16 +0000

Recent Enhancements, News, and Activities

Upgrade to Wayback 1.7. We upgraded the version of Wayback (the tool that is used to replay web crawls). This has resulted in increased general reliability, as well as improved display.
End of Term (EOT) Archive crawls. The final crawl for the EOT 2012 was completed by Internet Archive. Planning has started to transfer content to the Library of Congress and other partners. The CDL will be working later in the year with Internet Archive to merge the 2012 records with the 2008 site (http://eotarchive.cdlib.org/index.html). Background information available here:
http://eotarchive.cdlib.org/2012.html.

WAS Activity, March 2013

95 archives actively collected
1508 sites collected
1.4 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

WAS Service Update: February 2013

Rosalie Lack — Tue, 19 Mar 2013 23:00:18 +0000

Recent Enhancements, News, and Activities

WAS 2.0 Release. Work continues on the release.
SJSU Intern. Caitlin O’Neal López has joined our team, virtually, as an intern from San Jose State University School of Library and Information Science. (Caitlin currently lives in Seoul, South Korea.) Caitlin will be working on developing new WAS tutorial videos. To help us prioritize which videos to do first, Caitlin sent a survey to current WAS subscribers. The top 5 subject areas indentified were: Analyze Capture Results; Export XML; Duplicate Reduction; Archive your domain or a large website, and Rights Management. We have many other topics on our list and will take them on as we can.
WAS Outreach Materials. We created a new 4×6 postcard (see image). If you would like us to mail some to you for an event or meeting, please contact Rosalie Lack at rosalie.lack@ucop.edu.

New WAS Institution. Read the announcement of our newest
WAS subscriber – Mount Holyoke College.
http://www.cdlib.org/cdlinfo/2013/03/03/mount-holyoke-college-signs-up-for-was/
Network Outage. On Thursday February 28 at 1:29 PM (PT), there was a general UCOP (UC Office of the President) network outage which impacted connectivity for CDL operations and services including the DMPTool. At 2:33 PM, we received word from the UCOP systems office that the UCOP network was coming back online. A second interruption occurred around 3 PM, although that outage was resolved within a few minutes. This outage did not have any effect on existing crawls. Since the crawler, Heritrix, is quite robust at dealing with sites that are unavailable, it would have paused at the time of the outage and picked back up once connectivity was reestablished. Please contact us at washelp@ucop.edu with any questions or concerns about this network outage.

New public content: University of California Office of the President (UCOP) site

UCOP archived the ucop.edu domain prior to redesigning the site; new site launched November 2012. Available here: http://webarchives.cdlib.org/a/ucoppre2012.

WAS Activity, February 2013

100 archives actively collected
1753 sites collected
1.4 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

Mount Holyoke College Signs up for WAS

Rosalie Lack — Sun, 03 Mar 2013 17:52:16 +0000

The CDL Web Archiving Service (WAS) is extremely pleased to welcome Mount Holyoke College as its newest subscriber. Leslie Fields, Head of Archives & Special Collections at Mount Holyoke, explains “after testing several options we determined that WAS was the best choice for us. We felt the interface was straightforward and easy to use; we liked the option to make collections public or to keep them private; we received very good customer service when we asked our many questions; and we liked the consortium pricing.”

From Mount Holyoke College website

Mount Holyoke’s web archiving activities are the continuation of a NHPRC electronic records grant project that they completed in 2011. The goal of the grant work was to establish procedures for ingest, processing, preservation, and providing access to campus electronic records of enduring value. During the period they worked on the grant staff realized that the News & Events portion of the project was really all about web archiving.

The result was an investigation into possible web archiving options, including creating an in-house service. They quickly realized that with their current level of staffing an in-house solution was not a viable option. After reviewing other services and consulting with a variety of library teams, they selected WAS.

Now that they are signed up for WAS, staff will first focus on administrative and academic sites at Mount Holyoke, as well as News and Events pages. Then they will address faculty and course sites, as well as sites that document student life, such as student organization pages.

Welcome Mount Holyoke College!

For further information about the Web Archiving Service, see http://was.cdlib.org, or email washelp@ucop.edu.

WAS Service Update: January 2013

Rosalie Lack — Thu, 21 Feb 2013 23:06:56 +0000

Recent Enhancements, News, and Activities

WAS 2.0 Release Work on the new release continues. Significant progress was made in January; testing will begin in early February. Based on the results of testing, we will have a release date.
VM Migration. We started the planning process for a very large, infrastructure project to migrate from Solaris physical hosts to Linux virtual machines. The migration will also include moving WAS storage to San Diego SuperComputer Center (SDSC). The migration will begin after the WAS 2.0 release.
New WAS institution. We are very happy to welcome Mount Holyoke College in Massachusetts as our newest WAS subscriber!

WAS Activity, January 2013

103 archives actively collected
1971 sites collected
1.8 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

WAS Service Update – November/December 2012

Rosalie Lack — Mon, 14 Jan 2013 22:23:22 +0000

Recent Enhancements, News, and Activities

WAS 2.0 Release re-scheduled for 2013. One of the key, new features of WAS 2.0 will be the Solr index. It will bring faster and more accurate search results, advanced search features, and a more robust indexing system. In order to implement the Solr index, the entire collection, which is now over 50 Terabytes, has to be re-indexed. This is a very complex and machine intensive process. Due to some complications encountered during the re-indexing, the WAS 2.0 release has been postponed to early 2013. All the development on the UI enhancements and new curator tools has been completed, but since these developments are inseparable from the new Solr indexing infrastructure, their release has also been postponed to 2013.
WAS public site redesign moving forward. The graphic design is completed and the project has now moved to the CDL Web Production team for implementation.
New WAS logo and tagline. In order to align the WAS logo with the other services in the UC3 portfolio, a new logo was created (see below). In addition, a new descriptive tagline now provides a clear overview of the WAS service.

‘

WAS presentation at UC3 workshops. Two 2-day workshops Nov 8-9 in Oakland and Nov 13-14 in Irvine introduced library staff to basic data curation concepts, presented the challenges and solutions involved in delivering data curation services, provided information about UC3’s tools and services, and shared campus solutions and approaches to data curation services. 100 participants attended the workshops. Rosalie Lack presented an overview of WAS (see presentation http://bit.ly/U196aC [pdf]) and discussed how institutions are currently using it and the future challenges of web archiving.
IIPC training. Rosalie Lack attended a 5-day International Internet Preservation Consortium-sponsored training, “How to Fit In? Integrating a web archiving program in your organization”. The workshop was hosted and organized by the French National Library and was attended by staff from libraries across Europe, Egypt, and the US. At the workshop, a well-rounded picture of how web archiving could be integrated into existing library systems was presented, new developments in web archiving showcased, and challenges were discussed.
Digital Library Federation (DLF) and NGTS activities. Erik Hetzner, WAS Tech Lead, attended DLF in early November. As a member of a UC Next-Generation Technical Services (NGTS) lightening team, along with other members — Todd Grappone (UCLA), Declan Fleming (UCSD), Brian Tingle (CDL) and Susan Perry (UCSC) – organized a birds of a feather session at which representatives from Hydra and Islandora discussed their services. Their lightening team (1.C) is charged with developing a model for a UC systemwide DAMS. Learn more: http://ucngts.tumblr.com/
End of Term (EOT) Archive crawls. CDL is working in collaboration with the Harvard University Library, Internet Archive, Library of Congress, University of North Texas Libraries, and the U.S. Government Printing Office on a continued partnership to archive 2012-2013 End of Term (EOT) sites. Internet Archive’s final crawl for 2012 was completed mid November; there will be additional crawls run in early January and continue through inauguration day. Background information available here: http://eotarchive.cdlib.org/2012.html

New public content: UCLA Library goes public with Occupy Web Archive

The UCLA Library Occupy Web Archive is now available for searching and browsing on the WAS site. The collection documents local Occupy movements and events on the west coast of the United States, Mexico, and Brazil. The collection provides a geographic, topic browse that makes it easy to navigate through over 30 cities, states and countries.

Available here: http://webarchives.cdlib.org/a/occupy

WAS Activity, November/December 2012

4353 archives actively collected
2773 sites collected
3.5 TB of data collected

WAS Service Description

The Web Archiving Service (WAS) enables librarians, archivists and researchers to capture, curate and preserve websites and web‐published materials. WAS makes it easy to build web archives, with scheduling and other tools to help manage your archive. You control public access to your archives and can configure the appearance and navigation of each archive. We also provide collection development consultation and help desk support for web archiving questions.

WAS Service Manager
Rosalie Lack rosalie.lack@ucop.edu or washelp@ucop.edu

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides available here: http://webarchives.cdlib.org/p/curators

Service Monitoring and Availability
Check system status page http://www.cdlib.org/contact/system.html

WAS Service Update – October 2012

Rosalie Lack — Tue, 06 Nov 2012 19:00:22 +0000

Recent Enhancements, News, and Activities

Work continues on the WAS 2.0 release and the WAS public interface redesign.
iPRES 2012: WAS Service Manager, Rosalie Lack, attended the iPRES 2012 Conference. Of the sessions dedicated to web archiving, of particular interest was a panel that included several national libraries, university libraries, the Internet Archive and Library of Congress discussing their web archiving programs and the challenges. Below is a brief summary of the challenges that were discussed. There were no real surprises, but it was interesting to learn that these challenges are common across all these institutions. They included: sheer size of the archives makes all tasks and management more difficult; many formats remain problematic for crawlers (e.g., streaming video, JavaScript, deep web, flash, Youtube); QA of crawled sites is still a very manual process; robots.txt (honor or override); collection development has not always been clearly defined (what you collect and what you don’t); there is a lack of use cases for how the archives are being used by scholars, students and the public; web archives are not well integrated with other collections; and there still remains a lack of widespread appreciation for the urgency of saving websites. International Internet Preservation Consortium (IIPC) and its members (CDL is an IIPC member) are actively working on many of these issues. Despite the challenges, there were many success stories shared of growing archives and increased interest in and use of them. More info about iPRES: https://ipres.ischool.utoronto.ca/ and about IIPC http://netpreserve.org/.

New public content: Michigan Historical Collections Web Archives

The Bentley Historical Library’s Web Archiving initiative has released over 100 sites in the Michigan Historical Collection Web Archives. The topics of the sites include historically-significant religious, governmental, educational, cultural, and activist organizations and institutions around Michigan.

Available here: http://webarchives.cdlib.org/a/MichHistColl

WAS Activity, October 2012

1791 archives actively collected
2246 sites collected
1.2 TB of data collected

WAS Service Description

WAS Service Manager

Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu

Additional WAS Information (training materials, videos, guides, etc.) More information about WAS is available at http://was.cdlib.org or by sending an inquiry to washelp@ucop.edu.

WAS Service Update – September 2012

Rosalie Lack — Tue, 16 Oct 2012 18:02:57 +0000

Recent Enhancements, News, and Activities

Fall 2012 releases update
- WAS 2.0: We are continuing the quality assurance testing for the WAS 2.0 release; among the areas of focus are the new solr index; duplicate reduction (which will substantially reduced the size of files stored); new curator QA tools; new search functionality; and XML record export
- WAS public site: We began planning for redesign of the WAS public interface.
EOT Archive: CDL is working in collaboration with the Harvard Library, Internet Archive, Library of Congress, University of North Texas Libraries, and the U.S. Government Printing Office on a continued partnership to archive 2012-2013 End of Term (EOT) sites. We have been publicizing the EOT Archive in order to encourage people to nominate sites to be included in the crawl. Learn more: http://eotarchive.cdlib.org/2012.html

New public content: UC Davis goes public with student newspaper archive

The California Aggie, an entirely student-run publication, is now live in WAS. First captured by the UC Davis Special Collections staff starting in October 2010, this archive provides a rich resource for examining daily news and events at UC Davis, along with international news making stories such as the Occupy Davis movement.

Avaialble here: http://webarchives.cdlib.org/a/AggieUCD

New public content: Tamiment Library’s new Housing archive

The Tamiment Library and Robert F. Wagner Labor Archives has released over 30 archived sites related to housing rights, movements, and policy organizations, with a special concentration on New York City. Issues include, affordable housing, rent control, public housing, housing law and legislation.

Available here: http://webarchives.cdlib.org/a/housing

WAS Activity, September 2012

1561 archives actively collected
2068 sites collected
1.4 TB of data collected

WAS Service Description

WAS Service Manager

Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu

Additional WAS Information (training materials, videos, guides, etc.) More information about WAS is available at http://was.cdlib.org or by sending an inquiry to washelp@ucop.edu.

WAS Service Update – August 2012

Rosalie Lack — Tue, 25 Sep 2012 17:48:02 +0000

Recent Enhancements, News, and Activities

WAS User’s Group meeting
On Friday, August 10 in conjunction with the Society of American Archivists meeting in San Diego, the first WAS User’s Group was held. There were 16 attendees from 10 different institutions. During the half day meeting, participants shared updates about their archives, discussed issues/challenges their institutions are facing, prioritized suggestions for WAS enhancements, and shared ideas about how to manage/engage the user group. Meeting notes available here: http://bit.ly/NkTa3T.

WAS Activity, August 2012

102 archives actively collected
2,153 sites collected
1.4 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu

Additional WAS Information (training materials, videos, guides, etc.) More information about WAS is available at http://was.cdlib.org or by sending an inquiry to washelp@ucop.edu.