Finding pages on the unarchived Web

Hugo C. Huurdeman, Anat Ben-David, Jaap Kamps, Thaer Samar, Arjen P. De Vries

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Web archives preserve the fast changing Web, yet are highly incomplete due to crawling restrictions, crawling depth and frequency, or restrictive selection policies - most of the Web is unarchived and therefore lost to posterity. In this paper, we propose an approach to recover significant parts of the unarchived Web, by reconstructing descriptions of these pages based on links and anchors in the set of crawled pages, and experiment with this approach on the DutchWeb archive. Our main findings are threefold. First, the crawled Web contains evidence of a remarkable number of unarchived pages and websites, potentially dramatically increasing the coverage of theWeb archive. Second, the link and anchor descriptions have a highly skewed distribution: popular pages such as home pages have more terms, but the richness tapers off quickly. Third, the succinct representation is generally rich enough to uniquely identify pages on the unarchived Web: in a known-item search setting we can retrieve these pages within the first ranks on average.

Original languageEnglish
Title of host publication2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages10
ISBN (Electronic)9781479955695
StatePublished - 1 Dec 2014
Externally publishedYes
Event2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014 - London, United Kingdom
Duration: 8 Sep 201412 Sep 2014

Publication series

NameProceedings of the ACM/IEEE Joint Conference on Digital Libraries
ISSN (Print)1552-5996


Conference2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014
Country/TerritoryUnited Kingdom

Bibliographical note

Publisher Copyright:
© 2014 IEEE.


  • Anchor text
  • Information retrieval
  • Link evidence
  • Web archives
  • Web archiving
  • Web crawlers


Dive into the research topics of 'Finding pages on the unarchived Web'. Together they form a unique fingerprint.

Cite this