** Apologies for multiple postings **

Title: A reference Web dataset for learning to rank using click-through
data -- help with relevance assessment

Learning to rank is a very important research field in Web search where
machine learning techniques are employed to improve the ranking of results
returned by search systems. One of the best ways of learning to rank is by
observing users behaviors, which requires usage data. The lack of a
publicly available dataset containing click-through data makes researches
in this topic (when using click-through data) limited to industrial
research groups.

TodoCL is a horizontal search engine which covers the Chilean Web, running
since April 2000. As a contribution for science, the company will make
available for research a series of data for the most intensive period of
use of the search engine. The dataset includes:

 - Documents: two Web pages collections, crawled in August 2003 and
  January 2004, respectively. Most of the Web pages are in Spanish.

 - Structure: internal in-links and out-links.

 - Usage data: query requests and click-through data for a span of 12
  months, from August 2003 to July 2004.

 - Relevance data for results of a sample of the real queries.

For composing the relevance data we have prepared a relevance assessment
experiment. We are looking for people to help with the relevance
assessment task, which consists on judging whether Web pages are relevant
for an associated query.

More information about the relevance assessment may be found here:

http://grupoweb.upf.es/~cmiddlet/todocl_assessment/

Our prevision is to make the data available on demand by August 2008.
Probably there will be a fee to fund the management of the dataset, but we
will offer the dataset for free to all the assessors who collaborate in
the relevance experiment.

Please, enter the relevance assessment page referred above to register for
a set of queries to assess. Your contribution is very important. Note that
you need to understand Spanish. In case you do not understand and want to
collaborate and have the dataset for free, tell a friend to participate by
forwarding this message, and let us know who is the person later.

Thank you in advance for any help.

Ricardo Baeza-Yates, Christian Middleton and Alvaro Pereira.
Web Research Group - grupoweb.upf.es
Universitat Pompeu Fabra, Barcelona, Spain


More information about the Chilean dataset:

The collections and query logs are provided by the Chilean search engine
TodoCL. For composing the collections, the complete list of the Chilean
Web primary domains were used to start the crawling, guaranteeing that a
set of pages under every Chilean domain (.cl) was crawled once the crawls
were pruned by depth. Domains outside the Chilean primary domain were only
crawled if their IP address was from a Chilean IP provider. This data set
has already been successfully used for research [2, 1, 3, 8, 5, 7, 6, 4].

Two document collections will be provided. The 2003 collection was crawled
in August 2003, and has 3.11 million documents, whereas the 2004
collection was crawled in January 2004, containing 3.13 million documents.
Among other attributes, the collections have the text of the documents,
their titles and URLs, their link structure, and duplication information.
The HTML of the pages does not exist. We will provide an API to access all
the data.

The query log span is 12 months, from August 2003 to July 2004. It
contains query requests and click-through data, with identification of
sessions, time stamp and (anonymous) host names for every entry. In order
to guarantee privacy, we will exclude queries performed only a few times.
The query log has approximately 10 million query requests and 1.7 million
clicks.

We intend to provide an automatic translation for the Web pages into
English, which may be useful when the users do not understand Spanish, as
well as a human translation of the queries used in the relevance
assessment.

[1] R. Baeza-Yates, A. Pereira, and N. Ziviani. Understanding content
reuse on the web: Static and dynamic analyses. In Advances in Web Mining
and Web Usage Analysis, O. Nasraoui, M. Spiliopoulou, J. Srivastava, B.
Mobasher, and B. Masand, editors, volume 4811 of LNCS/LNAI, pages
227--246. Springer, 2007.

[2] R. Baeza-Yates, A. Pereira, and N. Ziviani. Genealogical trees on the
Web: a search engine user perspective. In 17th International World Wide
Web Conference, pages 367--376, Beijing, China, April 2008.

[3] R. Baeza-Yates and B. Poblete. Dynamics of the chilean web structure.
Computer Networks, 50(10):1464--1473, 2006.

[4] Ricardo Baeza-Yates, Felipe Saint-Jean, and Carlos Castillo. Web
structure, age and page quality. In Proceedings of String Processing and
Information Retrieval Conference (SPIRE), pages 117--130, Lisbon,
Portugal, 2002. Springer LNCS.

[5] Ricardo A. Baeza-Yates, Liliana Calderon-Benavides, and Cristina N.
Gonzalez-Caro. The intention behind web queries. In Proceedings of String
Processing and Information Retrieval Conference (SPIRE), pages 98--109,
Glasgow, UK, 2006.

[6] Ricardo A. Baeza-Yates, Carlos A. Hurtado, and Marcelo Mendoza. Query
clustering for boosting web page ranking. In Second International Atlantic
Web Intelligence Conference, pages 164--175, Cancun, Mexico, 2004.

[7] Georges Dupret and Marcelo Mendoza. Recommending better queries from
click-through data. In Proceedings of String Processing and Information
Retrieval Conference (SPIRE), pages 41--44, Buenos Aires, Argentina, 2005.

[8] Georges Dupret and Marcelo Mendoza. Automatic query recommendation
using click-through data. In Professional Practice in Artificial
Intelligence, pages 303--312, Santiago, Chile, 2006.