Finding pages on the unarchived Web

Hugo C. Huurdeman; Anat Ben-David; Jaap Kamps; Thaer Samar; Arjen P. De Vries

doi:10.1109/JCDL.2014.6970188

Finding pages on the unarchived Web

Hugo C. Huurdeman, Anat Ben-David, Jaap Kamps, Thaer Samar, Arjen P. De Vries

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Web archives preserve the fast changing Web, yet are highly incomplete due to crawling restrictions, crawling depth and frequency, or restrictive selection policies - most of the Web is unarchived and therefore lost to posterity. In this paper, we propose an approach to recover significant parts of the unarchived Web, by reconstructing descriptions of these pages based on links and anchors in the set of crawled pages, and experiment with this approach on the DutchWeb archive. Our main findings are threefold. First, the crawled Web contains evidence of a remarkable number of unarchived pages and websites, potentially dramatically increasing the coverage of theWeb archive. Second, the link and anchor descriptions have a highly skewed distribution: popular pages such as home pages have more terms, but the richness tapers off quickly. Third, the succinct representation is generally rich enough to uniquely identify pages on the unarchived Web: in a known-item search setting we can retrieve these pages within the first ranks on average.

Original language	English
Title of host publication	2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	331-340
Number of pages	10
ISBN (Electronic)	9781479955695
DOIs	https://doi.org/10.1109/JCDL.2014.6970188
State	Published - 1 Dec 2014
Externally published	Yes
Event	2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014 - London, United Kingdom Duration: 8 Sep 2014 → 12 Sep 2014

Publication series

Name	Proceedings of the ACM/IEEE Joint Conference on Digital Libraries
ISSN (Print)	1552-5996

Conference

Conference	2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014
Country/Territory	United Kingdom
City	London
Period	8/09/14 → 12/09/14

Bibliographical note

Publisher Copyright:
© 2014 IEEE.

Keywords

Anchor text
Information retrieval
Link evidence
Web archives
Web archiving
Web crawlers

Access to Document

10.1109/JCDL.2014.6970188

Cite this

Huurdeman, H. C., Ben-David, A., Kamps, J., Samar, T., & De Vries, A. P. (2014). Finding pages on the unarchived Web. In 2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014 (pp. 331-340). Article 6970188 (Proceedings of the ACM/IEEE Joint Conference on Digital Libraries). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/JCDL.2014.6970188

@inproceedings{2f004c34c6e049d9820ffc8cbb015430,

title = "Finding pages on the unarchived Web",

abstract = "Web archives preserve the fast changing Web, yet are highly incomplete due to crawling restrictions, crawling depth and frequency, or restrictive selection policies - most of the Web is unarchived and therefore lost to posterity. In this paper, we propose an approach to recover significant parts of the unarchived Web, by reconstructing descriptions of these pages based on links and anchors in the set of crawled pages, and experiment with this approach on the DutchWeb archive. Our main findings are threefold. First, the crawled Web contains evidence of a remarkable number of unarchived pages and websites, potentially dramatically increasing the coverage of theWeb archive. Second, the link and anchor descriptions have a highly skewed distribution: popular pages such as home pages have more terms, but the richness tapers off quickly. Third, the succinct representation is generally rich enough to uniquely identify pages on the unarchived Web: in a known-item search setting we can retrieve these pages within the first ranks on average.",

keywords = "Anchor text, Information retrieval, Link evidence, Web archives, Web archiving, Web crawlers",

author = "Huurdeman, {Hugo C.} and Anat Ben-David and Jaap Kamps and Thaer Samar and {De Vries}, {Arjen P.}",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.; 2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014 ; Conference date: 08-09-2014 Through 12-09-2014",

year = "2014",

month = dec,

day = "1",

doi = "10.1109/JCDL.2014.6970188",

language = "אנגלית",

series = "Proceedings of the ACM/IEEE Joint Conference on Digital Libraries",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "331--340",

booktitle = "2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014",

address = "ארצות הברית",

}

Huurdeman, HC, Ben-David, A, Kamps, J, Samar, T & De Vries, AP 2014, Finding pages on the unarchived Web. in 2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014., 6970188, Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Institute of Electrical and Electronics Engineers Inc., pp. 331-340, 2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014, London, United Kingdom, 8/09/14. https://doi.org/10.1109/JCDL.2014.6970188

Finding pages on the unarchived Web. / Huurdeman, Hugo C.; Ben-David, Anat; Kamps, Jaap et al.
2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014. Institute of Electrical and Electronics Engineers Inc., 2014. p. 331-340 6970188 (Proceedings of the ACM/IEEE Joint Conference on Digital Libraries).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Finding pages on the unarchived Web

AU - Huurdeman, Hugo C.

AU - Ben-David, Anat

AU - Kamps, Jaap

AU - Samar, Thaer

AU - De Vries, Arjen P.

PY - 2014/12/1

Y1 - 2014/12/1

N2 - Web archives preserve the fast changing Web, yet are highly incomplete due to crawling restrictions, crawling depth and frequency, or restrictive selection policies - most of the Web is unarchived and therefore lost to posterity. In this paper, we propose an approach to recover significant parts of the unarchived Web, by reconstructing descriptions of these pages based on links and anchors in the set of crawled pages, and experiment with this approach on the DutchWeb archive. Our main findings are threefold. First, the crawled Web contains evidence of a remarkable number of unarchived pages and websites, potentially dramatically increasing the coverage of theWeb archive. Second, the link and anchor descriptions have a highly skewed distribution: popular pages such as home pages have more terms, but the richness tapers off quickly. Third, the succinct representation is generally rich enough to uniquely identify pages on the unarchived Web: in a known-item search setting we can retrieve these pages within the first ranks on average.

AB - Web archives preserve the fast changing Web, yet are highly incomplete due to crawling restrictions, crawling depth and frequency, or restrictive selection policies - most of the Web is unarchived and therefore lost to posterity. In this paper, we propose an approach to recover significant parts of the unarchived Web, by reconstructing descriptions of these pages based on links and anchors in the set of crawled pages, and experiment with this approach on the DutchWeb archive. Our main findings are threefold. First, the crawled Web contains evidence of a remarkable number of unarchived pages and websites, potentially dramatically increasing the coverage of theWeb archive. Second, the link and anchor descriptions have a highly skewed distribution: popular pages such as home pages have more terms, but the richness tapers off quickly. Third, the succinct representation is generally rich enough to uniquely identify pages on the unarchived Web: in a known-item search setting we can retrieve these pages within the first ranks on average.

KW - Anchor text

KW - Information retrieval

KW - Link evidence

KW - Web archives

KW - Web archiving

KW - Web crawlers

UR - http://www.scopus.com/inward/record.url?scp=84919382668&partnerID=8YFLogxK

U2 - 10.1109/JCDL.2014.6970188

DO - 10.1109/JCDL.2014.6970188

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:84919382668

T3 - Proceedings of the ACM/IEEE Joint Conference on Digital Libraries

SP - 331

EP - 340

BT - 2014 IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014

Y2 - 8 September 2014 through 12 September 2014

ER -

Finding pages on the unarchived Web

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this