California Digital Library » Web Archiving

Five things I learned at IIPC

Rosalie Lack — Thu, 16 May 2013 13:54:12 +0000

I recently attended the International Internet Preservation Consortium (IIPC) General Assembly. The IIPC is a consortium of libraries, academic institutions and other organization engaged in web archiving. The IIPC’s General Assembly included three days of member meetings and two days of meetings open to the public. The theme of the public conference day was: Scholarly Access to Web Archives: Progress, Requirements, and Challenges.

Ahmed Alsum from the Web Science and Digital Libraries Research Group at Old Dominion University posted a comprehensive summary of the GA. As you can see from his summary, there were many great presentations and discussions. It was very hard to choose just five things to share, but here they are:

1. Dark(ish) Archives

Because of copyright and privacy issues, many of the national libraries in Europe cannot provide online, public access to their web archives. They can only allow access in the library and many do not even allow printing in the library. So, how do you raise the awareness of web archiving when no one can see the archives?! There was much discussion about creating site lists/registries for the sites in these archives – some felt this would only lead to disappointment when the user finds out that they have to travel to the archive to see the materials. Sound familiar? Yes, finding aids. And YES, they are extremely useful!

Harvesting tools.
Source: WikiMedia Commons

2. Common tools support is critical

Most IIPC members are using the same suite of OS tools –Heritrix and Open Source Wayback. There was a lot of concern about the development path for these tools. At member breakout sessions where future paths were discussed and prioritized, there was a clear message that tools are important. The IIPC steering committee quickly responded by supporting tool management as a top priority for the organization. Be on the lookout for updates and concrete plans shortly.

3. National and University Library of Slovenia is innovative

Besides being wonderful conference hosts, the National and University Library of Slovenia is doing some innovative work when it comes to web archiving. They demo’d a prototype (see screenshots) of a tool for end users to engage and interact with web archives. It includes features such as: gather and save sites; annotate sites; tag sites; and crowd sourcing of metadata. The next generation of web archives is here!

4. Researchers use of web archives

There were several informative presentations by researchers about how they are using web archives. Sophie Geibeil, an historian from Aix-Marseille-Université, uses the archives to study the untold story of North African immigration. Megan Dougherty, a social scientist from Loyola University, was not as interested in site content as much as she was in taking an anthropological point of view of the sites; that is, studying the social aspects of sites – how people share sites, interact with sites, etc. Niels Brügger, from Netlab at Aarhus University, discussed their various research projects in the areas of digital humanities and internet studies, including RESAW and FUTARC. Helen Hockx-Yu, UK Web Archive at the British Library, presented UK Web Archives in the eyes of scholars. She made the case for thinking of the archives not as documents but rather as large datasets for data mining and analysis.

5. Dancing into the future?

David Rosenthal provided historical context for the early days of web crawling and also provided some future challenges. Not surprising, the web as mainly HTML links is rapidly becoming a thing of the past. Turns out all the current, problematic areas (including rich media, database driven features, dynamically generated URIs, etc.) remain challenging, and now add to that the fact that the new web is more and more JavaScript. Is there no rest for the weary? David did leave us with a light at the end of tunnel. He talked about recent work by the Institute National de L’Audiovisuel (INA) in Paris. The team there created a live archive proxy that shows great promise to enable the capture of some of the more problematic content. Also, there is Memento, which provides an aggregation of web archives each collected in slightly different ways by institutions so moving toward covering all the bases.

Fred Astaire.
Source: WikiMedia Commons

David also had a great analogy for one of the challenges of preserving the web today; he says it is like “preserving theatre or dance” because as we view the web it changes to become an individual experience based on who we are, displaying customized ads and other personalized content. As he put it: “Every performance is a unique and unrepeatable interaction between the performers, in this case a vast collection of dynamically changing databases, and the audience. Actually, it is even worse. Preserving the Web is like preserving a dance performed billions of times, each time for an audience of one, who is also the director of their individual performance.” (Source: http://blog.dshr.org/2013/04/talk-on-harvesting-future-web-at.html)

Overall, it was an excellent, thought-provoking conference. Clearly there are lots of challenges ahead for web archiving, but so many more opportunities.

WAS Service Update: April 2013

Rosalie Lack — Fri, 10 May 2013 19:29:55 +0000

Recent Enhancements, News, and Activities

WAS at Society of California Archivists (SCA), Berkeley, CA. April 11 – 13. The WAS team attended and staffed a booth. Rosalie Lack also gave a short presentation to the Online Archive of California (OAC)’s Users Group meeting. http://www.slideshare.net/rosalielack/society-of-california-archivists-sca
International Internet Preservation Consortium General Assembly (IIPC), Ljubljana, Slovenia. April 22‐ 26. The IIPC is a consortium of libraries, academic institutions and other organization engaged in web archiving. IIPC actively promotes the development of open source tools, standards and best practices in web archiving. Rosalie Lack participated as an IIPC member as well as a member of the IIPC’s Access Working Group member. The IIPC’s General Assembly includes 3 days of member meetings and 2 days open to the public. The theme of the public conference day was: Scholarly Access to Web Archives: Progress, Requirements, and Challenges.
Bug fixes. The WAS technical team (Erik Hetzner and Scott Fisher) fixed several ongoing, but intermittent bugs that were causing display issues in both the public and curator interfaces. You should see a marked improvement in display. A reminder to please report any problems that you encounter by sending an email to: washelp@ucop.edu.

WAS Activity, April 2013

113 archives actively collected
1340 sites collected
1.5 TB of data collected

WAS Service Description
The Web Archiving Service (WAS) enables librarians, archivists and researchers to capture, curate and preserve websites and web‐published materials. WAS makes it easy to build web archives, with scheduling and other tools to help manage your archive. You control public access to your archives and can configure the appearance and navigation of each archive. We also provide collection development consultation and help desk support for web archiving questions.

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

WAS Service Update: March 2013

Rosalie Lack — Thu, 18 Apr 2013 21:44:16 +0000

Recent Enhancements, News, and Activities

Upgrade to Wayback 1.7. We upgraded the version of Wayback (the tool that is used to replay web crawls). This has resulted in increased general reliability, as well as improved display.
End of Term (EOT) Archive crawls. The final crawl for the EOT 2012 was completed by Internet Archive. Planning has started to transfer content to the Library of Congress and other partners. The CDL will be working later in the year with Internet Archive to merge the 2012 records with the 2008 site (http://eotarchive.cdlib.org/index.html). Background information available here:
http://eotarchive.cdlib.org/2012.html.

WAS Activity, March 2013

95 archives actively collected
1508 sites collected
1.4 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

WAS Service Update: February 2013

Rosalie Lack — Tue, 19 Mar 2013 23:00:18 +0000

Recent Enhancements, News, and Activities

WAS 2.0 Release. Work continues on the release.
SJSU Intern. Caitlin O’Neal López has joined our team, virtually, as an intern from San Jose State University School of Library and Information Science. (Caitlin currently lives in Seoul, South Korea.) Caitlin will be working on developing new WAS tutorial videos. To help us prioritize which videos to do first, Caitlin sent a survey to current WAS subscribers. The top 5 subject areas indentified were: Analyze Capture Results; Export XML; Duplicate Reduction; Archive your domain or a large website, and Rights Management. We have many other topics on our list and will take them on as we can.
WAS Outreach Materials. We created a new 4×6 postcard (see image). If you would like us to mail some to you for an event or meeting, please contact Rosalie Lack at rosalie.lack@ucop.edu.

New WAS Institution. Read the announcement of our newest
WAS subscriber – Mount Holyoke College.
http://www.cdlib.org/cdlinfo/2013/03/03/mount-holyoke-college-signs-up-for-was/
Network Outage. On Thursday February 28 at 1:29 PM (PT), there was a general UCOP (UC Office of the President) network outage which impacted connectivity for CDL operations and services including the DMPTool. At 2:33 PM, we received word from the UCOP systems office that the UCOP network was coming back online. A second interruption occurred around 3 PM, although that outage was resolved within a few minutes. This outage did not have any effect on existing crawls. Since the crawler, Heritrix, is quite robust at dealing with sites that are unavailable, it would have paused at the time of the outage and picked back up once connectivity was reestablished. Please contact us at washelp@ucop.edu with any questions or concerns about this network outage.

New public content: University of California Office of the President (UCOP) site

UCOP archived the ucop.edu domain prior to redesigning the site; new site launched November 2012. Available here: http://webarchives.cdlib.org/a/ucoppre2012.

WAS Activity, February 2013

100 archives actively collected
1753 sites collected
1.4 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

Mount Holyoke College Signs up for WAS

Rosalie Lack — Sun, 03 Mar 2013 17:52:16 +0000

The CDL Web Archiving Service (WAS) is extremely pleased to welcome Mount Holyoke College as its newest subscriber. Leslie Fields, Head of Archives & Special Collections at Mount Holyoke, explains “after testing several options we determined that WAS was the best choice for us. We felt the interface was straightforward and easy to use; we liked the option to make collections public or to keep them private; we received very good customer service when we asked our many questions; and we liked the consortium pricing.”

From Mount Holyoke College website

Mount Holyoke’s web archiving activities are the continuation of a NHPRC electronic records grant project that they completed in 2011. The goal of the grant work was to establish procedures for ingest, processing, preservation, and providing access to campus electronic records of enduring value. During the period they worked on the grant staff realized that the News & Events portion of the project was really all about web archiving.

The result was an investigation into possible web archiving options, including creating an in-house service. They quickly realized that with their current level of staffing an in-house solution was not a viable option. After reviewing other services and consulting with a variety of library teams, they selected WAS.

Now that they are signed up for WAS, staff will first focus on administrative and academic sites at Mount Holyoke, as well as News and Events pages. Then they will address faculty and course sites, as well as sites that document student life, such as student organization pages.

Welcome Mount Holyoke College!

For further information about the Web Archiving Service, see http://was.cdlib.org, or email washelp@ucop.edu.

WAS Service Update: January 2013

Rosalie Lack — Thu, 21 Feb 2013 23:06:56 +0000

Recent Enhancements, News, and Activities

WAS 2.0 Release Work on the new release continues. Significant progress was made in January; testing will begin in early February. Based on the results of testing, we will have a release date.
VM Migration. We started the planning process for a very large, infrastructure project to migrate from Solaris physical hosts to Linux virtual machines. The migration will also include moving WAS storage to San Diego SuperComputer Center (SDSC). The migration will begin after the WAS 2.0 release.
New WAS institution. We are very happy to welcome Mount Holyoke College in Massachusetts as our newest WAS subscriber!

WAS Activity, January 2013

103 archives actively collected
1971 sites collected
1.8 TB of data collected

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

Temporary Changing of the Guard at UC3

Jayne Dickson — Thu, 31 Jan 2013 18:50:51 +0000

Patricia Cruse, CDL’s Director of Digital Preservation (UC3), will be reducing her duties at the CDL between January 14 and June 30, 2013. Trisha will be working remotely from Italy and Spain while she accompanies her husband on his sabbatical from the University of San Francisco (USF). During this time Trisha will continue working on various initiatives on a limited basis and will be available via email at patricia.cruse@ucop.edu. Please don’t hesitate to contact Trisha about work or great places to see or eat in Italy or Spain!

Rosalie Lack, who has returned to CDL to assume the duties of the Web Archiving Service (WAS) manager, will take on many of the responsibilities for the day-to-day running of UC3. These include administrative and systemwide activities as well as budgetary, communication, outreach, marketing and UC3 team management. Rosalie can be reached at rosalie.lack@ucop.edu.

The Progress of Preservation

Laine Farley — Wed, 16 Jan 2013 23:17:01 +0000

I was very pleased by the nice recognition from Library of Congress’s “Top Ten Digital Preservation Developments of 2012” of three projects CDL has been involved in:

The DataUp Project. The University of California Curation Center at the California Digital Library continued to produce useful tools and services in support of digital preservation with DataUp, “an open source tool helping researchers document, manage, and archive their tabular data… within the scientist’s workflow.”

End of Term Web Archive. The End of Term 2012 project got underway to capture U.S. Government websites between the first and second administration of President Barack Obama. Project partners include the California Digital Library, Internet Archive, Library of Congress, University of North Texas Libraries and the U.S. Government Printing Office.

We have also been participating through meetings and briefings in the development of another project on the list, the Digital Preservation Network.

The rest of this “Top Ten” list is equally impressive and it is heartening to realize that so much has been accomplished in the area of digital preservation in our community. Yet I can’t help but note that it can still be a hard sell to administrators to justify new or increased expenditures for something that seems abstract, unpredictable and never ending. This challenge was magnified for me recently when I presented a paper in December, 2012 at the 3rd conference on Cultural Heritage online – Trusted Digital Repositories & Trusted Professionals in Florence, Italy.

The conference began with a parade of local officials and cultural heritage ministers from the city and region, all extolling the importance of preserving cultural heritage in the digital age. Indeed, the commitment to preservation is everywhere in this city that was the heart of the Renaissance, its museums and public places overflowing with the riches of the past. I was particularly struck by the exhibits at the Galileo Museum

Sundials, Galileo Museum, Florence

where the artifacts and experimental documentation of not only Galileo but also his contemporaries were showcased in spectacular fashion. Even with all of the support given to cultural heritage, the Italians at the conference still feel they need to shore up digital preservation, but they were focusing on standards and compliance more than on basic funding.

My presentation, on the other hand, was about how we have shifted from talking about preservation as an ultimate and costly activity to curation as part of the ongoing process of creating and managing digital content. We must be aware of incentives—such as mandates from funders and publishers rather than government initiatives–for our researchers to practice good stewardship of their research output. Thus we have invested in tools such as the DataUP mentioned by LC as well as the Data Management Planning Tool (DMPTool which made the 2011 Top Ten list), EZID (which won the DataCite 2012 Gold Award for assigning more than 250,000 DOIs (digital object identifiers) in one year), and the Web Archiving Service to make it digital content easier to manage when it comes time to preserve it for the future. After all, today’s research can become tomorrow’s cultural heritage and think what we would be missing if we weren’t able to see what Galileo was up to.

DataUp, End of Term Web Archive Cited Among Top Ten Digital Preservation Developments of 2012

Ellen Meltzer — Wed, 16 Jan 2013 21:08:01 +0000

Two CDL projects are among a list of top ten digital preservation developments of 2012, according to Bill LeFurgy, Digital Initiatives Manager at the Library of Congress, DataUp and the End of Term Web Archive are included in this list. Stated LeFurgy, “I cast a wide net and mustered my objectivity in picking activities with the potential for broad, collaborative impact in the world-wide effort to keep digital material available and accessible over time. The resulting list covers an assortment of practical, hands-on information, as well as tools for helping with outreach, program assessment and research data management.”

See the full article at http://blogs.loc.gov/digitalpreservation/2013/01/top-10-digital-preservation-developments-of-2012/

Congratulations to our UC3 colleagues at CDL, along with our partners in these endeavors including the Gordon and Betty Moore Foundation, Microsoft Research Connections and DataONE for DataUp; and Harvard University Library, Internet Archive, Library of Congress, University of North Texas Libraries and the U.S. Government Printing Office for the End of Term Web Archive.

WAS Service Update – November/December 2012

Rosalie Lack — Mon, 14 Jan 2013 22:23:22 +0000

Recent Enhancements, News, and Activities

WAS 2.0 Release re-scheduled for 2013. One of the key, new features of WAS 2.0 will be the Solr index. It will bring faster and more accurate search results, advanced search features, and a more robust indexing system. In order to implement the Solr index, the entire collection, which is now over 50 Terabytes, has to be re-indexed. This is a very complex and machine intensive process. Due to some complications encountered during the re-indexing, the WAS 2.0 release has been postponed to early 2013. All the development on the UI enhancements and new curator tools has been completed, but since these developments are inseparable from the new Solr indexing infrastructure, their release has also been postponed to 2013.
WAS public site redesign moving forward. The graphic design is completed and the project has now moved to the CDL Web Production team for implementation.
New WAS logo and tagline. In order to align the WAS logo with the other services in the UC3 portfolio, a new logo was created (see below). In addition, a new descriptive tagline now provides a clear overview of the WAS service.

‘

WAS presentation at UC3 workshops. Two 2-day workshops Nov 8-9 in Oakland and Nov 13-14 in Irvine introduced library staff to basic data curation concepts, presented the challenges and solutions involved in delivering data curation services, provided information about UC3’s tools and services, and shared campus solutions and approaches to data curation services. 100 participants attended the workshops. Rosalie Lack presented an overview of WAS (see presentation http://bit.ly/U196aC [pdf]) and discussed how institutions are currently using it and the future challenges of web archiving.
IIPC training. Rosalie Lack attended a 5-day International Internet Preservation Consortium-sponsored training, “How to Fit In? Integrating a web archiving program in your organization”. The workshop was hosted and organized by the French National Library and was attended by staff from libraries across Europe, Egypt, and the US. At the workshop, a well-rounded picture of how web archiving could be integrated into existing library systems was presented, new developments in web archiving showcased, and challenges were discussed.
Digital Library Federation (DLF) and NGTS activities. Erik Hetzner, WAS Tech Lead, attended DLF in early November. As a member of a UC Next-Generation Technical Services (NGTS) lightening team, along with other members — Todd Grappone (UCLA), Declan Fleming (UCSD), Brian Tingle (CDL) and Susan Perry (UCSC) – organized a birds of a feather session at which representatives from Hydra and Islandora discussed their services. Their lightening team (1.C) is charged with developing a model for a UC systemwide DAMS. Learn more: http://ucngts.tumblr.com/
End of Term (EOT) Archive crawls. CDL is working in collaboration with the Harvard University Library, Internet Archive, Library of Congress, University of North Texas Libraries, and the U.S. Government Printing Office on a continued partnership to archive 2012-2013 End of Term (EOT) sites. Internet Archive’s final crawl for 2012 was completed mid November; there will be additional crawls run in early January and continue through inauguration day. Background information available here: http://eotarchive.cdlib.org/2012.html

New public content: UCLA Library goes public with Occupy Web Archive

The UCLA Library Occupy Web Archive is now available for searching and browsing on the WAS site. The collection documents local Occupy movements and events on the west coast of the United States, Mexico, and Brazil. The collection provides a geographic, topic browse that makes it easy to navigate through over 30 cities, states and countries.

Available here: http://webarchives.cdlib.org/a/occupy

WAS Activity, November/December 2012

4353 archives actively collected
2773 sites collected
3.5 TB of data collected

WAS Service Description

The Web Archiving Service (WAS) enables librarians, archivists and researchers to capture, curate and preserve websites and web‐published materials. WAS makes it easy to build web archives, with scheduling and other tools to help manage your archive. You control public access to your archives and can configure the appearance and navigation of each archive. We also provide collection development consultation and help desk support for web archiving questions.

WAS Service Manager
Rosalie Lack rosalie.lack@ucop.edu or washelp@ucop.edu

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides available here: http://webarchives.cdlib.org/p/curators

Service Monitoring and Availability
Check system status page http://www.cdlib.org/contact/system.html