Web archiving Survey - Internet Memory Foundation

Web Archiving in Europe A survey provided by the Internet Memory Foundation, 2010 Context

INTERNET MEMORY SURVEY • Sent in December 2010 • 385 questionnaires sent to 364 independent institutions • 37 European countries • Translated in 5 languages (English, German, French, Italian and Spanish) SOME SURVEYS ON WEB ARCHIVING • February 2011, Arquivo da Web Portuguesa/Portuguese Web Archive (to be published): http:// www.arquivo.pt • December 2010, University of North Texas: http:// research.library.unt.edu/eotcd/w/ images/2/29/ fdlp_survey_report_krm_14dec20 10.pdf • April 2010, National Diet Library: www.ndl.go.jp/en/cdnlao/meetings/ pdf/report_Japan1_doc3.pdf • December 2008, International Internet Preservation Consortium (IIPC): http://netpreserve.org/ publications/reports.php?id=005 • July, 2007, Koninklijke Bibliotheek/ National Library of the Netherlands: http://www.kb.nl/hrd/ dd/dd_projecten/webarchivering/ documenten/ KB_UserSurvey_Webarchive_EN.p df

In the framework of the European Research Project Living Web Archives -LiWA-, a three-year project funded by the European Commission through the Seventh Research Framework Programme under Project No 216267), the Internet Memory Foundation carried out a survey on Web archiving among European and International institutions. The aim of this survey was to draw up an inventory of Web archiving institutions and to have a clearer understanding of problems encountered in the field of Internet archiving.

Key figures of results Survey Participants A representative panel of Web archiving in Europe based on 74 answers. Mostly composed by Audiovisual archives (25,7%), National libraries (23%) and Institutional archives (12,2%).

Web archiving program, plan to have one in the next three years.

Maturity of Web archiving program

Selection policy Most of the institutions operate selective crawls (71%) and 30,8% focus on their own websites. Only 23% of this panel run domain crawls.

7% 1% 7%

8% 30%

23% 5%

19%

Yes, fully operational Yes, operational but still experimenting Yes, just starting a Web Archiving project No, but we plan to do it No, no funding No, not in our mandate No, an other institution is already in charge answered question

41 institutions have a Web archiving program: fully operational, experimenting, and starting. And 17 institutions, which do not have a current

Legal context 50% do Web archiving protected by a law enacted or passed

Management To fit with their Web archiving project, this panel develops their own strategy and skills. 64% operate their Web archive in house versus 36% which outsource the service. In majority they use Heritrix (80%) and manage their collection with a curator tool with an in house solution for 35% of them. 35% use open source solution as NAS, WCT or Pandora… Access User can browse Web archive by URL (68% ), by searching by keyword (70% ) and through thematic collection (65%).

INTERNET MEMORY FOUNDATION

WEB ARCHIVING IN EUROPE, 2010

Survey Analysis Presentation The aims of this survey were to: • Analyse the advancements made in Web archiving projects in Europe • Cover a larger group of institutions from Active to LongTerm Engaging ones • Increase the potential LiWA User Community by discovering new institutions engaged in Web archiving. • Before this survey, we had a view of Web archiving project through IIPC members, but the aim here was to reach a large number of institutions. • Demonstrate how LiWA developments can be useful for these institutions. The survey has been sent by email to 385 persons (within 364 independent institutions) on the 2nd week of December 2010. The email contained a link to the online survey (provided by SurveyMonkey). The survey was spread over 37 European countries and was offered in 5 languages (English, German, French, Italian and Spanish): • National and Regional Libraries (19%) • University or specialized Libraries (22%) • National and Regional Archives (11%) • Audiovisual and/or Broadcasting Archives (20%) • Institutional Archives, as Parliament Archive, European Institutions (8) and International Institutions (31) • Documentation or Archive Department of Museum (10%) This makes it the largest ever survey in the domain of Web Archiving. A special focus was laid on Germany in order to analyze Web archiving projects at a regional level and because nearly each of the 16 regions has its own Archive, Library, University Library and public TV/Radio Archive. The survey has been divided into several sections: • Characterization of the institution • State of the archiving project • Legal restriction • Management of Web archiving project: who is involved, how much staff, how much budget etc. • Which tools are used in the institution? • Research: is the institution involved in specific projects, do they know LIWA? • For institutions that are engaged in Web archiving: several specific questions about their future developments

The survey has been sent to 385 persons. Some effort was made to find the right department or person to answer. Only 5% of email addresses were invalid (17 out of 384 email addresses). 21% of the target audience have filled out the questionnaire (74 filled out surveys). This can be considered high enough to be a representative result.

1. Characterization of the community In Figure 1, answers are split up per institution, which gives a good idea of which institutions are interested in Web archiving. Audiovisual archives (26%) National libraries (23%) and institutional archives (12,2%) are the most representative in Web archiving projects.

Figure 1: Answers by institution type An interesting comparison is to see the survey answers broken down to determine which institutions are the most interested in Web archiving. Thus, as we can see, the most remarkable institutions are: • National and regional libraries: 19% of institutions contacted 25% of all responses • National and regional archives: 11% contacted, 15% of all responses • Audiovisual archives: from 20% contacted to 26% of all responses We have also contacted museum archives, representing 10% of the contacted audience, but we have not received any response from this group at all. Breaking down the answers by department, the majority of answers have been provided by «Archive/Documentation/



Survey Analysis Library departments» (52%) and «Digital preservation and Web archiving departments» (20,5%). Currently, 55% of institutions answered (Table 1) that they have a Web archiving project (fully operational, experimenting, starting), meaning 41 active institutions (including an American University and a company which provides Web archiving services). 23% of the institutions, which have answered that they do not have a current Web archiving program, plan to have one in the next three years.

Maturity of Web archiving project Survey Date

Active 2008

Short-Term

2010

2008 2010

National & Regional Archives

5 %

18 %

Audiovisual Archives

18 % 36 % 41 %

University Other institutions

11 % 15 % 26 % 18 % 12 %

Table 2: Evolution of respondents’ according to the maturity of the project

2. Legal Aspect Answer Options

%

Nb

Yes, it is fully operational

29,7%

22

Yes, it is operational but still experimenting

18,9%

14

Yes, we are just starting a Web Archiving project No, we do not but we plan to do it in a short-term (1 or 2 years) No, we do not but we plan to do it in a mid-term (from 3 years) No, we do not because we do not have funding for this project No, we do not because it is not in our mandate No, we do not because an other institution is already in charge No, other reason Answered question Table 1: Status of the archiving program

2010

Nb 34

Yes, law is expected

16,2%

11

Lobbying are in progress

7,4%

5

26,5%

18

12,2%

9

10,8%

8

No, we do not applicable law

5,4%

4

Answered question Table 3: Legal aspects

1,4%

1

6,8%

5

8,1%

6 74

Short-Term

Survey Date

2008

National & Regional libraries

54 % 37 % 18 % 18 %

Other libraries

31 %

3 %

% 50%

Yes, law is enacted or passed

5

Moreover, the type of institutions involved or interested in Web archiving has changed. In 2008 (survey provided by LiWA members), the most active Institutions were national libraries (58%). Three years later however, this group only represents 37%. The other highlight concerns audiovisual archives: in 2008, zero respondents declared to be active; in 2010, the amount is 18% for active respondents and 41% for short-term respondents (Table 2). Active

Legal Aspects

6,8%

It is clear that heritage institutions that do not have any current Web archiving project would like to start doing so. It is also interesting to notice that among these institutions 42% consider that Web archiving activities are or should be within the mandate of the national library.

Maturity of Web archiving project

Most of institutions have a law related to Web archiving (Table 3) and don’t need to ask permission (60,4%).

2008 2010 27 % 12 %

68

Depending on legal statutes, institutions do not provide the same access of their collections to users (Table 6). However, it is worth noticing that 41 % of the institutions provide online access to their collection. Access restriction Access is online for anyone Access is online with restrictions Access is on site for anyone Access is on site with restrictions We do not have access: contents are in a completely dark archive Answered question Table 4: Access restriction

% 41% 28% 18% 21%

Nb 25 17 11 13

21%

13 61

3. Management of Web archiving program? Institutions involved in Web archiving developed their own production process and methods: 64% of them are crawling or would like to crawl in-house while 36% choose or would choose an outsourcing solution. With reference to the staff and budget involved in Web archiving projects, staff numbers are dependent on the phase of the project. 38% of a fully operational program count more than 5 Full-Time Equivalent (FTE), 50% of experimenting program and 67% that



Survey Analysis just started a project count to 2 to 5 FTE and around 60% of the planned projects count 1 or less FTE. With reference to the budget allocated to the project, the curve follows the same evolution, 41% of fully operational institutions spend more than €200.000/year; 50,5% of experimental institutions have a budget of €10.000 to €200.000/year and more than 70% of planned project institutions do not have any budget for a program at all.

Web archiving policy Most of the institutions conduct selective crawls (71,2%) and 30,8% focus only on their own Websites (Table 5). We can also note that only national libraries (33% of them) operate global crawls on their national Top-Level Domain. On the other side of the spectrum, audio-visual archives (75% of them), are focused on their own Websites (only 38% collect external material according to Thematic or Selective crawls). Archive policy

%

Nb

Domain crawls - TLD (.uk, .eu, .com…)

23%

12

Thematic/selective crawls

71%

37

30,8%

16

Only our Websites and associated Websites Answered question

Figure 2: Main topics on which institutions are focusing Smart access: institutions are interested in improving end-users service. 55% of respondents choose to focus on this in their three main improving activities. This is confirmed by answers about Access and Exploitation (see Figure 3): 82% envisions the improvement of archive access tools as a high priority. This field includes the capacity of full-text indexing and advanced research in order to browse in their archive. Of course, rates increase if we focus on institutions who provide access online or on site to endusers.

52

Table 5: Archive policy Institutions mainly use tools developed by IIPC like Heritrix (80%), Wayback machine (67,5%) or NutchWAX (70%). To improve their methods however, they are interested in new developments. With these preliminary results, we notice that there is a real evolution of Web archiving practices. More and more institutions are already involved in and have explicitly expressed an interest in being involved in Web archiving.

Figure 3: Smart access and interpretability Moreover, compared to 2008 survey, the question of completeness remains important, especially regarding Media Content (68% of respondents consider it as a high priority), or dynamic Web (62%). There is still a shared concern that a growing range of material is out of reach to current crawlers and requests are clearly to enable to expand the range and variety of content that can be captured.

Web archiving priority The third part of the survey focuses on the priority of institutions who are already involved in Web archiving. The goal of this part was to define their needs and assess how LIWA tools or other developments could meet their requirements. Through this survey, some topics of improvement can be highlighted (see Figure 2). Figure 4: Specific content



Survey Analysis Conclusion Concerning specific content (See Figure 4), the main issue is the size of resources and of Websites. The relatively low response on Web Spam are expected as the majority of respondents are not concerned by global crawls. But for those who do large crawls, the main issue is actually Webspam detection (56%) and traps (44%). Interestingly, several respondents have added the social Web as a specific content issue.

In comparison to the 2008 survey, types of institutions, which express interest in Web Archiving, have clearly evolved. Web archiving is considered to be an important part of their mission by more and more institutions. Audiovisual institutions emerge as a new stakeholder, being more and more engaged in Web archiving. The predominance of focus crawls shows the increasing need for higher Web archive quality.

The Internet Memory Foundation is pleased to thank all respondents for their participation and interest. These results have been welcomed by to the European Commission during the review of the LiWA project. It will contribute to the process of planning future projects on Web archiving.

INTERNET MEMORY Netherlands: Keizersgracht 62-64 NL-1015CS Amsterdam contact /at/ internetmemory.org France: 45 ter rue de la Révolution F-93100 Montreuil contact /at/ internetmemory.org http://www.internetmemory.org

Internet Memory Foundation is a non-profit institution. It actively supports the preservation of Internet as a new media for heritage and cultural purpose. The foundation is currently archiving dozens of Terabytes of data per months and is developing several technologies to support the growth and use of the Internet Memory. It has developed a wide range of collaborations, both with cultural institutions and research team to fulfill its mission.

ARCHIVETHE.NET Web archiving service (shared platform) of the Internet Memory Foundation http://archivethe.net LIWA PROJECT Dr. Thomas Risse L3S Research Center Appelstrasse 9a 30167 Hannover - Germany info /at/ liwa-project.eu http://www.liwa-project.eu



Survey Results 1. Characterisation of your Institution/Department 1. Information

2. Your Institution is a :

Answer Options National Library University Library Regional Library National Archive Regional Archive Institutional Archive Broadcasting Archive Museum Archive Corporate Archive Research Library Archive of thematic domain Other Archive/Library (specify)

Response Percent 23,0 % 8,1 % 2,7 % 12,2 % 2,7 % 12,2 % 25,7 % 0,0 % 0,0 % 8,11 % 4,05 % 1,4 %

Response Count 17 6 2 9 2 9 19 0 0 6 3 1

answered question skipped question

74 0

Other: Commercial Web archiving service

3. What department do you work for? Response Percent 51,4 % 6,8 % 21,6 % 6,8 % 13,5 %

Answer Options Archive / Documentation / Library IT Departement Digital preservation and web archiving departement Information policy Department Others (specify)

Response Count 38 5 16 5 10


74 0

Others: E-Helvetica (1), Research Center (1)

4. What are currently the main activities of your Institution/Department? (max. 3 choices)

Answer Options Collecting non-digital content Collecting born-digital content Web archiving Cataloguing Digitisation Preservation of physical collections Building digital infrastructures Improving public access Development of cooperation with other institutions

Response Percent 44,4 % 54,2 % 33,3 % 31,9 % 44,4 % 30,6 % 40,3 % 30,6 % 16,7 % answered question skipped question

Response Count 32 39 24 23 32 22 29 22 12 72 1



Survey Results 5. Are you outsourcing some of these activities? Answer Options No, we are not Yes, we are outsourcing Specify which activities

Response Percent 48,6 % 51,4 %

Response Count 36 38 35

answered question

74 0

skipped question

Activities: Storage physical content (1), Technical Infrastructure, Maintenance (4), Web archiving (6), Cataloguing (2), Digitisation (14), Preservation (4), Digital infrastructures (5), Improving public access (2), Collecting digital content (1)

6. What are your strategic priorities for the next 3 years? (select a max. of 3 focus)

Answer Options Collecting non-digital content Collecting born-digital content Web archiving Cataloguing Digitisation Preservation of physical collections Building digital infrastructures Improving public access Development of cooperation with other institutions

Response Percent 18,3 % 62,0 % 40,8 % 16,9 % 46,5 % 21,1 % 56,3 % 40,8 % 21,1 %

Response Count 13 44 29 12 33 15 40 29 15


71 2

2. Web archiving project 1. Do you have a current Web archiving program in your Institution?

Answer Options Yes, it is fully operational Yes, it is operational but still experimenting Yes, we are just starting a Web Archiving project No, we do not but we plan to do it in a short-term (1 or 2 years) No, we do not but we plan to do it in a mid-term (from 3 years) No, we do not because we do not have funding for this project No, we do not because it is not in our mandate No, we do not because an other institution is already in charge No, other reason Yes, since answered question skipped question Since: 1999 (1), 2000 (1), 2002 (1), 2005 (4), 2007 (1), 2008 (1), 2009 (1), 2010 (3)

Response Percent 29,7 % 18,9 % 6,8 % 12,2 % 10,8 % 5,4 % 1,4 % 6,8 % 8,1 % 27,0 % 0

Response Count 22 14 5 9 8 4 1 5 6 20 74 0



Survey Results 3. Web Archiving (only for Institution which does not have Web archiving project) 1. Which institution is in charge of web archiving in your country? Answer Options National Library National Archive Each Institution None Don’t know

Response Count 14 1 1 3 5 24 9


2. Are you involved in this project ?

Answer Options

Yes

No

In the whole process In the selection policy In access issues In quality issues In preservation issues No, but regularly informed answered question skipped question

5 4 3 4 4 10

19 17 18 17 18 12

Rating Average (YES) 20,83 % 19,05 % 14,29 % 19,05 % 18,18 % 45,45 %

Response Count 24 21 21 21 22 22 29 4

3. Would you like to be more involved?

Answer Options

Yes

No

in the whole process in the selection policy in access issues in quality issues in preservation issues

4 11 9 8 11

15 7 8 10 8

Rating Average (YES) 21,05 % 61,11 % 52,94 % 44,44 % 57,89 %


Response Count 19 18 17 18 19 24 9

4. Legal restriction 1. Do you have any laws in your country related to Web archiving? Answer Options Yes, law is enacted or passed Yes, law is expected Lobbying are in progress No, we do not applicable law Since

Response Percent 50,0 % 16,2 % 7,4 % 26,5 % answered question skipped question

Since 1990 (1), 1997 (1), 1998 (1), 2003 (1), 2006 (8), 2007 (1), 2009 (4)

Response Count 34 11 5 18 21 68 6



Survey Results 2. Do you need specific permission to collect websites? Answer Options Yes, we do No, we do not


Response Count 21 32


53 11

3. What access restrictions do you have?

Answer Options Access is online for anyone Access is online with restrictions Access is on site for anyone Access is on site with restrictions We do not have access: contents are in a completely dark archive answered question skipped question

Response Percent 41,0 % 27,9 % 18,0 % 21,3 % 21,3 %

Response Count 25 17 11 13 13 61 13

5. Management 1. How do (would) you manage your Web archiving project? Answer Options We are crawling (would crawl) in-house We choose (would choose) an outsourcing solution: Internet Archive Foundation Internet Memory Foundation (formerly European Archive Foundation) Hanzo Archives Other company or organization (specify) answered question skipped question

Response Percent 133,3 % 75,0 % 27,8 %


44,4 %

8

0,0 % 27,8 %

0 5 50 24

Other companies: National Library (4), Commerciam Web Archiving Service (1), Partnership R&D Project (1), Regional Library (1)

2. How many people are involved in the project (Full Time Equivalent)?

Answer Options We do not have any staff 1 or less FTE 2 to 5 FTE More than 5 FTE

Response Percent 21,7 % 35,0 % 30,0 % 13,3 %

Response Count 13 21 18 8


60 14

3. Which budget is allocated for Web archiving project ? Answer Options No budget Less than 10'000€ From 10'000 to 50'000€ From 50'000 to 200'000€ More than 200'000€

Response Percent 52,7 % 5,5 % 14,5 % 10,9 % 16,4 % answered question skipped question

Response Count 29 3 8 6 9 55 19



Survey Results

4. Which Web archiving policy do (would) you follow?

Answer Options Domain crawls - TLD (.uk, .eu, .com…) Thematic/selective crawls Only our websites and associated websites




52 22

6. Web archiving Tools 1. Which tools do you use for collecting? Answer Options Heritrix Httrack Wget Other (please specify)

Response Percent 80,0 % 13,3 % 6,7 % 0,0 %



30 44

Others: Iterasi (1), Custom (Srce harvester) (1), Offline Explorer (2), CMS-Intern (1), Webcollect, Crocket (1), Hanzo Crawler (1), Internal development (4), Don’t know (3)

2. Do you use a curator tool to manage your collections?

Answer Options Yes, we developed an in-house solution Yes, we use an open source tool as NetArchiveSuite, Webcurator tool, Pandora… No, we do not use a curator tool answered question skipped question

Response Percent 34,7 %

Response Count 17

34,7 %

17

38,8 %

19 49 25

3. If access is provided, which solution do you use? (multiple-choice)

Answer Options Browsing by URL (Wayback…) Searching by Keywords (NutchWax…) Browsing thematic collections




40 34

7. Research 1. Are you involved in a Web Archiving organization? Answer Options No, we are not Yes, we are part of: International Internet Preservation Consortium (IIPC) UK Web Archive Other (please specify) answered question skipped question Others: DPC Web Archiving Taskforce (1)

Response Percent 212,5 % 150,0 % 93,8 % 31,3 % 50,0 %

Response Count 34 24 15 5 8 58 16



Survey Results EUscreen (1) AG Web Archiving ARD/ZDF (2) AK Web Archiving of AWV (formerly DFG-Project) (1)

2. Is your institution involved in European research project about Web Archiving topics?

Answer Options Yes, we participate No, we do not If Yes, specify which European project




62 12

European Projects: ARCOMEM, Collect-All Archives to COmmunity MEMories (3) EUscreen (1) LAWA, Longitudinal Analytics of Web Archive data (1) LivingKnowledge (1) LIWA, Living Web Archives (2) Preservation Projects (1) SCAPE, SCAlable Preservation Environments (3)

3. Did you know LiWA’s project (Living Web Archives) before this survey?

Answer Options Yes, we did No, we did not


Response Count 31 34


65 9

4. Are you (would you be) interested in LiWA tools?

Answer Options Yes, we do No, we are not interested Please specify which ones

Response Percent 82,5 % 17,5 % answered question skipped question

LiWA Tools: SemanticAnalyser (4) assessment-interface (4) rich-media-capture (5) temporal-coherence (3) web-archive-ui-framework (6) Archive Fidelity (1) Social Web (1) Capture (2) Missing Link Detection (1) Preservation (1) Flash Content (1), Animation (1), Streaming (1) Extraction (1)

Response Count 47 10 19 57 17



Survey Results

5. Would you like to be regularly informed about Web archiving?

Answer Options No Yes, email address


Response Count 8 52


60 14

8. Questions for institutions with an operational or experimenting web archives 1. Which topics does your Institution focus on to improve web archiving? Answer Options Content policy Specific formats and content Archive authenticity and fidelity Smart access to browse archive Preservation Other (please specify) answered question skipped question

Rate (high priority) 47,37 % 45,61 % 50,88 % 56,14 % 63,16 %

High priority

Medium priority

Low priority

27 26 29 32 36

14 14 12 9 12

5 6 7 7 6

Response Count 46 46 48 48 54 57 17

Others: Customer needs (1), Resource Discovery (1), Access to digital Media (1), Metadata (1), Cooperation with other institutions (1)

2. Specific formats

Answer Options Databases Media content Others protocols than HTTP Dynamic Web Non-standard formats Others (please specify) answered question skipped question

Rate (high priority) 37,74 % 67,92 % 16,98 % 62,26 % 20,75 %

High priority 20 36 9 33 11

Medium priority 17 7 18 10 16

Low priority 8 2 12 4 12

Response Count 45 45 39 47 39 53 21

Others: Video on streaming (1), JSP (1)

3. Specific contents

Answer Options Hidden web: login, forms, subscription forms Web-spams detection Traps Size of resources and websites Other (please specify) answered question skipped question

Rate (high priority)

High priority

Medium priority

Low priority

Response Count

40,00 %

18

13

10

41

35,56 % 35,56 % 48,89 %

16 16 22

10 13 5

14 7 8

40 36 35 3

Others: Social Network (1), Quoted Web resources (1), Web 2.0 communities (1)

45 29



Survey Results 4. Smart access and interpretability Answer Options Access to browse archive: full-text indexing, advanced research New functionalities for users: timeline, described and Customized Collections, notes Interpretability: data extraction, semantic analysis, text mining Other (please specify) answered question skipped question

Rate (high priority)

High priority

Medium priority

Low priority

Response Count

82,35 %

42

4

2

48

54,90 %

28

2

6

36

60,78 %

31

12

4

47 1 51 23

5. Preservation

Answer Options Metadata Standardization and guidelines MIME type Metrics Other (please specify) answered question skipped question

Rate (high priority) 78,85 % 57,69 % 30,77 % 21,15 %

High priority 41 30 16 11

Medium priority Low priority 7 13 19 17

3 2 6 6

Response Count 51 45 41 34 52 22

Others: Collaborative documentation on formats and software for emulation strategy

INTERNET MEMORY Netherlands: Keizersgracht 62-64 NL-1015CS Amsterdam contact /at/ internetmemory.org

ARCHIVETHE.NET France: 45 ter rue de la Révolution F-93100 Montreuil contact /at/ internetmemory.org http://internetmemory.org

Web archiving service of the Internet Memory Foundation http://archivethe.net

LIWA PROJECT info /at/ liwa-project.eu http://www.liwa-project.eu