California Digital Library » Digital Preservation (UC3)

Five things I learned at IIPC

Rosalie Lack — Thu, 16 May 2013 13:54:12 +0000

I recently attended the International Internet Preservation Consortium (IIPC) General Assembly. The IIPC is a consortium of libraries, academic institutions and other organization engaged in web archiving. The IIPC’s General Assembly included three days of member meetings and two days of meetings open to the public. The theme of the public conference day was: Scholarly Access to Web Archives: Progress, Requirements, and Challenges.

Ahmed Alsum from the Web Science and Digital Libraries Research Group at Old Dominion University posted a comprehensive summary of the GA. As you can see from his summary, there were many great presentations and discussions. It was very hard to choose just five things to share, but here they are:

1. Dark(ish) Archives

Because of copyright and privacy issues, many of the national libraries in Europe cannot provide online, public access to their web archives. They can only allow access in the library and many do not even allow printing in the library. So, how do you raise the awareness of web archiving when no one can see the archives?! There was much discussion about creating site lists/registries for the sites in these archives – some felt this would only lead to disappointment when the user finds out that they have to travel to the archive to see the materials. Sound familiar? Yes, finding aids. And YES, they are extremely useful!

Harvesting tools.
Source: WikiMedia Commons

2. Common tools support is critical

Most IIPC members are using the same suite of OS tools –Heritrix and Open Source Wayback. There was a lot of concern about the development path for these tools. At member breakout sessions where future paths were discussed and prioritized, there was a clear message that tools are important. The IIPC steering committee quickly responded by supporting tool management as a top priority for the organization. Be on the lookout for updates and concrete plans shortly.

3. National and University Library of Slovenia is innovative

Besides being wonderful conference hosts, the National and University Library of Slovenia is doing some innovative work when it comes to web archiving. They demo’d a prototype (see screenshots) of a tool for end users to engage and interact with web archives. It includes features such as: gather and save sites; annotate sites; tag sites; and crowd sourcing of metadata. The next generation of web archives is here!

4. Researchers use of web archives

There were several informative presentations by researchers about how they are using web archives. Sophie Geibeil, an historian from Aix-Marseille-Université, uses the archives to study the untold story of North African immigration. Megan Dougherty, a social scientist from Loyola University, was not as interested in site content as much as she was in taking an anthropological point of view of the sites; that is, studying the social aspects of sites – how people share sites, interact with sites, etc. Niels Brügger, from Netlab at Aarhus University, discussed their various research projects in the areas of digital humanities and internet studies, including RESAW and FUTARC. Helen Hockx-Yu, UK Web Archive at the British Library, presented UK Web Archives in the eyes of scholars. She made the case for thinking of the archives not as documents but rather as large datasets for data mining and analysis.

5. Dancing into the future?

David Rosenthal provided historical context for the early days of web crawling and also provided some future challenges. Not surprising, the web as mainly HTML links is rapidly becoming a thing of the past. Turns out all the current, problematic areas (including rich media, database driven features, dynamically generated URIs, etc.) remain challenging, and now add to that the fact that the new web is more and more JavaScript. Is there no rest for the weary? David did leave us with a light at the end of tunnel. He talked about recent work by the Institute National de L’Audiovisuel (INA) in Paris. The team there created a live archive proxy that shows great promise to enable the capture of some of the more problematic content. Also, there is Memento, which provides an aggregation of web archives each collected in slightly different ways by institutions so moving toward covering all the bases.

Fred Astaire.
Source: WikiMedia Commons

David also had a great analogy for one of the challenges of preserving the web today; he says it is like “preserving theatre or dance” because as we view the web it changes to become an individual experience based on who we are, displaying customized ads and other personalized content. As he put it: “Every performance is a unique and unrepeatable interaction between the performers, in this case a vast collection of dynamically changing databases, and the audience. Actually, it is even worse. Preserving the Web is like preserving a dance performed billions of times, each time for an audience of one, who is also the director of their individual performance.” (Source: http://blog.dshr.org/2013/04/talk-on-harvesting-future-web-at.html)

Overall, it was an excellent, thought-provoking conference. Clearly there are lots of challenges ahead for web archiving, but so many more opportunities.

CDL Welcomes New UC3 Staff

Ellen Meltzer — Wed, 15 May 2013 17:35:44 +0000

By Stephen Abrams, UC3 Associate Director

CDL’s UC3 program has just completed a flurry of hiring to fill four positions, two funded by the Alfred P. Sloan Foundation.

Joshua Hubbard and Bhavitavya (Bhavi) Vedula were hired to work on the DMPTool2 project, funded by the Alfred P. Sloan Foundation. Josh has jumped in head first, helping with requirements for the project. We’re looking forward to his Rails expertise going forward. Bhavi has more than five years of development experience, much of it with Ruby/Rails and JQuery-based user interfaces. Most recently she was part of a bioinformatics research group at Northwestern University tracking childhood developmental health.

Shirin Faenza has joined UC3 as developer on a one-year contract, initially working on Merritt user interface improvements. She has more than three years of web application development experience, most recently with Seedchange, a platform for crowd sourcing technology startup venture capital investing. She has also worked as a technical translator, converting the Linux Journal into her native Italian.

Ken Weiss is a production systems administrator, coordinating operational policies and procedures for UC3 services. He replaces Margaret Low, who retired in 2012. Most recently Ken worked as a technical analyst and architect as a contractor. Before that he was technical director for enterprise architecture at Charles Schwab. Long term UC staff may recall Ken’s earlier stint at the CDL in 1999-2000 when he redeployed OAC, then a research project at UCB, as a CDL production service.

Please welcome Josh, Bhavi, Shirin, and Ken!

WAS Service Update: April 2013

Rosalie Lack — Fri, 10 May 2013 19:29:55 +0000

Recent Enhancements, News, and Activities

WAS at Society of California Archivists (SCA), Berkeley, CA. April 11 – 13. The WAS team attended and staffed a booth. Rosalie Lack also gave a short presentation to the Online Archive of California (OAC)’s Users Group meeting. http://www.slideshare.net/rosalielack/society-of-california-archivists-sca
International Internet Preservation Consortium General Assembly (IIPC), Ljubljana, Slovenia. April 22‐ 26. The IIPC is a consortium of libraries, academic institutions and other organization engaged in web archiving. IIPC actively promotes the development of open source tools, standards and best practices in web archiving. Rosalie Lack participated as an IIPC member as well as a member of the IIPC’s Access Working Group member. The IIPC’s General Assembly includes 3 days of member meetings and 2 days open to the public. The theme of the public conference day was: Scholarly Access to Web Archives: Progress, Requirements, and Challenges.
Bug fixes. The WAS technical team (Erik Hetzner and Scott Fisher) fixed several ongoing, but intermittent bugs that were causing display issues in both the public and curator interfaces. You should see a marked improvement in display. A reminder to please report any problems that you encounter by sending an email to: washelp@ucop.edu.

WAS Activity, April 2013

113 archives actively collected
1340 sites collected
1.5 TB of data collected

WAS Service Description
The Web Archiving Service (WAS) enables librarians, archivists and researchers to capture, curate and preserve websites and web‐published materials. WAS makes it easy to build web archives, with scheduling and other tools to help manage your archive. You control public access to your archives and can configure the appearance and navigation of each archive. We also provide collection development consultation and help desk support for web archiving questions.

WAS Service Manager
Rosalie Lack, rosalie.lack@ucop.edu or washelp@ucop.edu.

WAS Training Materials, Guides, FAQs and Webinars
WAS training materials and guides are available here: http://webarchives.cdlib.org/p/curators.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

DataUp Service Update: April 2013

cstrasser — Fri, 10 May 2013 19:17:45 +0000

Recent Enhancements, News, and Activities

We gathered potential stakeholders of DataUp in Santa Barbara, CA for a meeting to discuss the tool’s future directions. The meeting took place April 2‐3 at the National Center for Ecological Analysis and Synthesis (NCEAS). The meeting report is available at
https://bitbucket.org/dataup/main/downloads/DataUpReport.pdf.
Strasser held a DataUp workshop at the UC Berkeley Library on April 11.
Strasser virtually presented on DataUp at the Association of College and Research Libraries (ACRL) 2013 Meeting in Indianapolis, IN. The video is available at http://vimeo.com/63614563.
Strasser presented on DataUp at a workshop for graduate students during the International Association of Landscape Ecologists 2013 Meeting in Austin, TX on April 16. Slides available at http://www.slideshare.net/carlystrasser/dataup-for-the-iale-2013-meeting.
Strasser visited the California Polytechnic State University on April 18 and presented on DataUp to the librarians and researchers. Slides available at http://www.slideshare.net/carlystrasser/dataup-presentation-at-cal-poly.

DataUp Service Description
The DataUp add-in and web application help researchers organize, manage, archive, and share their tabular data. The major functionality of both tools is to guide researchers through the following:

Carry out a best practices check to determine if there are issues present that might make use or interpretation of the data difficult
Create metadata in standardized EML (Ecological Metadata Language) format
Describe the attributes (i.e. columns) of the dataset in standard EML format
Obtain a unique identifier for the dataset
Generate a data citation
Post the data and associated metadata to a data repository

DataUp Service Manager
Carly Strasser carly.strasser@ucop.edu or uc3@ucop.edu.

DataUp Training Materials, Guides, FAQs and Webinars
The most current DataUp information is on the website (http://dataup.cdlib.org). This includes frequently asked questions and instructions on accessing and using DataUp. Other relevant sites include the code site (https://bitbucket.org/dataup/main) and the web application site (http://dataup.org).

DMPTool Service Update: April 2013

Perry Willett — Fri, 10 May 2013 19:06:47 +0000

Recent Enhancements, News, and Activities

DMPTool webinar series: The first webinar, an introduction to the DMPTool, will be May 28.

Register for the webinar: http://bit.ly/Z4g43Z
Read more about the webinar series: http://blogs.library.ucla.edu/dmptool/webinar-series/

We’ve created two advisory boards for the DMPTool, a Researcher Advisory Board and an Administrative Advisory Board. More about the advisory boards, with membership, here:
http://blogs.library.ucla.edu/dmptool/2013/04/11/advisory-boards-established/.
We’ve updated the data management requirements for the NSF Biological Sciences directorate. We contacted everyone who had customized this template for their institution. You will find more information about the changes here:
http://blogs.library.ucla.edu/dmptool/2013/05/02/nsf-bio-dmp-updated.
New team members:

We hired a software developer, Bhavi Vedula, who will work on the changes to the
DMPTool. She will be based in Oakland at CDL, and will begin May 1. She and our other
new developer Josh Hubbard will begin to implement the requirements developed by
the project team.

There were 241 data management plans created in April 2013, and 287 new users. There are a total of over 5,200 registered users from over 700 institutions, who have created over 4,500 data management plans.

Contributing Organizations
There are now 83 partner institutions that have either configured their campus single‐signon or customized the DMPTool for their users. There are participating organizations in 34 states and the District of Columbia; California leads all states with 18 participating organizations. A map of all participating organizations is available here: http://bit.ly/L85sKj.

DMPTool Service Description
The Data Management Plan Tool (DMPTool) provides an easy to use interface that:

Helps users create ready-to-use data management plans for specific funding agencies
Meets funder requirements for data management plans
Provides step-by-step instructions and guidance for how to manage data
Provides information about resources and services available at your institution to help fulfill the data management requirements of your grant

DMPTool Service Managers

Perry Willett perry.willett@ucop.edu
Carly Strasser carly.strasser@ucop.edu
Trisha Cruse patricia.cruse@ucop.edu

Please send any questions or general correspondence to uc3@ucop.edu. Note that campus library contacts have been embedded in the DMPTool and campus users will be directed to a local library contact.

DMPTool Training Materials, Guides, FAQs and Webinars
Help Guide: https://dmp.cdlib.org/help/guide.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

Merritt Service Update: April 2013

Perry Willett — Fri, 10 May 2013 18:21:36 +0000

Recent Enhancements, News, and Activities

Work is in‐progress and on track to migrate Merritt’s underlying technical infrastructure from physical servers to Virtual Machines (VM). (All UCOP services are moving to VMs.) We expect this work to be completed in June 2013.
The following related activities are also in‐progress and on track:
- migrating collections from the legacy Digital Preservation Repository (DPR) to Merritt
- transitioning storage from UCOP Data Center to San Diego Supercomputer Center’s Cloud Storage Service
For a current summary of 2013 development priorities for 2013, see the UC3 Curation Wiki: http://bit.ly/16O0hGK.

Merritt Service Description
Merritt is a production level service that provides the UC community with an easy to use tool to manage, archive, and share their content. Content can be deposited and managed via a user-interface or an API.

Merritt Service Managers

Perry Willett perry.willett@ucop.edu
Adrian Turner adrian.turner@ucop.edu

Please send any questions or general correspondence to uc3@ucop.edu.

Merritt Training Materials, Guides, FAQs and Webinars
More information about Merritt is available at http://www.cdlib.org/services/uc3/merritt or by sending an inquiry to uc3@ucop.edu.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.

EZID Service Update: April 2013

Joan Starr — Wed, 08 May 2013 17:33:14 +0000

Recent Enhancements, News, and Activities

For the 3rd month in a row, EZID Clients captured top honors when DataCite’s Managing Director Jan Brase blogged about the “most popular datasets” using DataCite Statistics. March numbers showed that 9 of the 10 most popular were EZID clients. Congratulations! http://bit.ly/ZlFYNy
On April 15th DataCite registration services were down for approximately 16 hours, starting at 8:25 AM, Pacific Time. EZID was not down during this time, but because it relies on DataCite to implement its DOI-related operations, DOI registrations through EZID were not possible during this window. DOI resolutions are handled by an entirely different system and were never affected. ARK services were not affected either. The cause of this outage has not yet been completely explained.
In response, we have developed tools to detect, recover from, and otherwise manage both client mistakes and DataCite and Handle System failures. This helps ensure more consistent system performance for all EZID clients.

EZID Partners and Clients
Our current EZID clients include academic institutions, government agencies, non-profit organizations and commercial entities.

New this month:

The Broad Institute of MIT and Harvard http://www.broadinstitute.org
The California Institute of Technology Library (on behalf of the CalTech campus) http://library.caltech.edu/

Complete list:

UC Berkeley Library (on behalf of the UC Berkeley campus)
UC Irvine Libraries (on behalf of the UC Irvine campus)
UC Los Angeles Library (on behalf of the UC LA campus)
UC Merced Library (on behalf of the UC Merced campus)
UC Riverside (on behalf of the UC Riverside campus)
UC San Diego Library (on behalf of the UC San Diego campus)
UC San Francisco Library (on behalf of the UC San Francisco campus)
UC Santa Cruz Library (on behalf of the UC Santa Cruz campus)
UC Santa Barbara Library (on behalf of the UC Santa Barbara campus)
American Astronomical Society http://www.aas.org
Bay Area Environmental Research Institute http://baeri.org/
Benchfly http://www.benchfly.com
The Broad Institute of MIT and Harvard http://www.broadinstitute.org
The California Institute of Technology Library (on behalf of the CalTech campus) http://library.caltech.edu/
Centre national de documentation pédagogique http://www.cndp.fr/
Cornell Institute for Social and Economic Research http://ciser.cornell.edu/
Cornell University Library http://www.library.cornell.edu/
Data Planet http://www.conquestsystems.com
The Digital Archaeological Record (tDAR) http://www.tdar.org
Digital-Science http://www.digital-science.com
Dryad Data Repository http://datadryad.org
Earth Science Information Partners (ESIP)’s Commons http://commons.esipfed.org/
Fred Hutchinson Cancer Research Center http://www.fhcrc.org
Harte Research Institute at Texas A&M, Corpus Christi http://www.harteresearchinstitute.org/
Johns Hopkins University Sheridan Libraries (on behalf of the JHU Data Management Services and Data Archive) http://dmp.data.jhu.edu
LabArchives http://www.labarchives.com/
Long Term Ecological Research Network (LTER) Office
National Atmospheric and Space Administration (NASA)’s Goddard Space Flight Center http://earthdata.nasa.gov/
The National Center for Atmospheric Research (NCAR) http://ncar.ucar.edu
The National Oceanic and Atmospheric Administration (NOAA) http://www.noaa.gov/
The National Snow and Ice Data Center (NSIDC) http://nsidc.org/
Oregon Health Sciences University Library (on behalf of the OHSU campus) http://www.ohsu.edu/xd/education/library/
Oregon State University Libraries (on behalf of the OSU campus) http://osulibrary.oregonstate.edu/
PeerJ Inc. http://peerj.com
Qualitative Data Repository (CQRM) http://www.maxwell.syr.edu/moynihan/cqrm/About_CQRM/
Rolling Deck to Repository (R2R) Program http://www.rvdata.us/
Rutgers University Libraries http://www.libraries.rutgers.edu/
UNAVCO http://www.unavco.org/
Université Claude Bernard Lyon Library http://www.univ-lyon1.fr/
University of Massachusetts Amherst Libraries (on behalf of the UMass Amherst campus)
University of Oregon Libraries (on behalf of the UO campus) http://libweb.uoregon.edu/
University of Tennessee Library (on behalf of the UTK campus) http://lib.utk.edu/
University of Utah Library (on behalf of the entire University of Utah campus)
http://www.lib.utah.edu/
University of Washington Libraries http://www.lib.washington.edu/
USGS/Earth Sciences Data Clearinghouse (formerly National Biological Info. Infrastructure)
http://www.nbii.gov/portal/server.pt
Virginia Tech University Libraries http://www.lib.vt.edu/
Washington State University Libraries (on behalf of the WSU campus) http://www.wsulibs.wsu.edu/

EZID Service Description
EZID (easy-eye-dee) is a production service that gives researchers the ability to create and manage long-term identifiers so that they can to track usage, get credit for their work, share their data, and have the data reused for additional research. As a result, EZID identifiers also make it possible to increase citations, to build on previous work, to conduct new research, and avoid duplicating previous efforts.

EZID Service Manager
Joan Starr joan.starr@ucop.edu or uc3@ucop.edu.

EZID Training Materials, Guides, FAQs and Webinars
More information about EZID is available by sending an inquiry to uc3@ucop.edu. See also EZID outreach and webinars: http://n2t.net/ezid/home/outreach.

Service Monitoring and Availability
Since June 25, 2010, EZID has had an uptime performance of 99.9%. Check the CDL system status page at http://www.cdlib.org/contact/system.html.

DMPTool Webinar Series

cstrasser — Mon, 29 Apr 2013 23:09:19 +0000

The California Digital Library received a grant from the Institute of Museum and Library Services (IMLS) last year to develop education and outreach materials for librarians surrounding the DMPTool, a service that helps researchers create and manage data management plans. As part of this grant, the DMPTool Partners are running a DMPTool Webinar series that will cover topics related to data curation, data management, and using the DMPTool to promote these topics.

Check here for the webinar schedule & information (updated frequently): http://blogs.library.ucla.edu/dmptool/webinar-series/

A few things to note:

The first webinar will be on Tuesday, May 28. We plan to present on Tuesdays, with current plans for ~15 webinars. The series will go into Fall 2013.
All webinars will be recorded and made available for viewing afterward.
The webinar schedule depends on presenters’ availability (hence the TBD dates).
We are always interested in new webinar ideas; please send them to Carly Strasser (carly.strasser@ucop.edu) or leave them as a comment on the webinar page.
We plan to collect these webinars and make them available as a set. We then hope to create a short course in Data Management with the DMPTool that will offer certification for librarians as “DMPTool Experts” –we are still working on the title!

Please circulate widely – the webinar series for anyone interested in promoting and/or using the DMPTool!

DataUp Service Update: March 2013

cstrasser — Fri, 19 Apr 2013 17:50:46 +0000

Recent Enhancements, News, and Activities

Strasser provided a demonstration of DataUp to attendees at the Beyond the PDF 2 Conference, held in Amsterdam.
Strasser submitted a paper on DataUp and the DMPTool to the Research Data Management
Infrastructure Meeting in Washington DC. The PDF is available here.
Strasser presented a webinar to the USGS community on DataUp; slides are available here.
Strasser attended UC Merced Library’s Research Week Events and presented on DataUp; slides are available here.
We had an intern, Sarina Sinick, from the University of Illinois working at the CDL on DataUp as part of the GSLIS Alternative Spring Break. She focused on improving the DataUp technical documentation.
We reorganized the DataUp BitBucket site to improve readability and usability.

Carry out a best practices check to determine if there are issues present that might make use or interpretation of the data difficult
Create metadata in standardized EML (Ecological Metadata Language) format
Describe the attributes (i.e. columns) of the dataset in standard EML format
Obtain a unique identifier for the dataset
Generate a data citation
Post the data and associated metadata to a data repository

DataUp Service Manager
Carly Strasser carly.strasser@ucop.edu or uc3@ucop.edu.

Merritt Service Update: March 2013

Perry Willett — Thu, 18 Apr 2013 22:32:44 +0000

Recent Enhancements, News, and Activities

An updated roadmap summarizing our 2013 development priorities for Merritt is now available on the UC3 Curation Wiki: http://bit.ly/16O0hGK
We are continuing work towards migrating the backend Merritt server environment from physical servers to Virtual Machines (VM). All UCOP services are moving to VMs, and we believe that this will result in more stable services. We expect this work to be complete in June 2013.
We are continuing to work on enhancements to Merritt’s underlying infrastructure, including:

migrating collections from the legacy Digital Preservation Repository (DPR) to Merritt
transitioning storage from UCOP Data Center to San Diego Supercomputer Center’s Cloud Storage Service

We have two new options for email notification of Merritt submissions. The first is a comma separated values (CSV) file. CSV files can be easily imported into Microsoft Excel for manipulation and analysis. The other format is a more enduser‐friendly text message. Please contact us at uc3@ucop.edu if you’re interested in more information, or want to begin receiving notifications in these new formats.
We now include a “readme.txt” file in the ZIP container file that explains where to find the data. Since we include system files and other content not originally submitted by the curator as part of the download, some users reported having trouble finding the actual data.

Merritt Service Managers

Perry Willett perry.willett@ucop.edu
Adrian Turner adrian.turner@ucop.edu

Please send any questions or general correspondence to uc3@ucop.edu.

Merritt Training Materials, Guides, FAQs and Webinars
More information about Merritt is available at http://www.cdlib.org/services/uc3/merritt or by sending an inquiry to uc3@ucop.edu.

Service Monitoring and Availability
Check CDL’s system status page at http://www.cdlib.org/contact/system.html.