Skip to content

Additional Data

iKAT Year 1 (2023)

File Description
2023_train_topics.json Train topics in JSON format.
2023_test_topics.json Test topics in JSON format.
2023_train_topics_psg_text.jsonl Text of provenance passages in the train topics.
2023_test_topics_psg_text.jsonl Text of provenance passages in the test topics.
2023_passages_hashes.tsv.bz2 TSV file containing MD5 hashes of passage texts. The .tsv file has this format: doc_id passage_number passage_MD5. Total download size is 2.2GB.
2023_top_1000_query_results.zip This zip file has queries from both training and testing topics, saved in queries_train.txt and queries_test.txt respectively. The results from the iKAT searcher (BM25 using manually resolved queries) are saved in the query_results_train and query_results_test folders. Each result file, with up to 1000 results, corresponds to a query based on line numbers, starting from zero. For instance, the results for the first query in queries_train.txt can be found in query_results_train/query_results_000.txt. In each result file, every line shows the ClueWeb22 ID followed by the URL.
ret_bm25_rm3--type_automatic--num_ptkb_3--k_100--num_psg_3.official.run.json Automatic Run
ret_bm25_rm3--type_manual--num_ptkb_3--k_100--num_psg_3.official.run.json Manual Run

Baseline Runs iKAT 2023

We provide two baseline runs (linked above).

  • Method.
    • BM25+RM3 (Pyserini default) as the initial retrieval (denoted by ret_bm25_rm3 in the file name) method to retrieve 100 passages per query (denoted by k_100).
    • The query in each turn was re-written using:
      1. The context, and
      2. The top-3 relevant PTKB statements (denoted by num_ptkb_3 in the file name).
  • Query re-writing. In all cases, the re-written query was construted by appending the relevant PTKB statements to the (manually or automatically) resolved query.
  • Response generation. A response was generated using the top-3 passages retrieved with the re-written query (denoted by num_psg_3 in the file name). We use the T5 model mrm8488/t5-base-finetuned-summarize-news available on HuggingFace for this purpose.
  • For automatic runs.
    • The relevant PTKB statements were determined automatically by re-ranking the statements using SentenceTransformers, specifically, the model cross-encoder/ms-marco-MiniLM-L-6-v2 available on HuggingFace.
    • The query was re-written automatically using the castorini/t5-base-canard model available on HuggingFace.
  • For manual runs.
    • The relevant PTKB statements provided in the ptkb_provenance field were used.
    • The manually re-written query provided in the resolved_utterance field was used.

Data from TREC CAsT

We provide the data from previous years' TREC CAsT below. The iKAT topics are similar, with the addition of the Personal Text Knowledge Base. For more information on TREC CAsT, see the website and read the overview papers [2019] [2020] [2021] [2022]

Note. TREC CAsT did not include a PTKB but you can be creative and modify the data according to your needs. Also, TREC CAsT used different collections (Wikipedia, KILT, MS MARCO, etc.) at different stages. iKAT is using a subset of the recently released ClueWeb22-B.

CAsT Year 4 (2022)

File Description
2022_automatic_evaluation_topics_tree_v1.0.json Contains each conversation tree (topic) with an automatic rewrite generated for each user utterance.
2022_evaluation_topics_turn_ids.json Contains each conversation tree (topic) with the resolved query for each user utterance.
2022_evaluation_topics_tree_v1.0.json Contains all ids that responses/ranked passages need to be returned for.
2022_evaluation_topics_flattened_duplicated_v1.0.json Contains all possible conversation paths across all the conversation trees.

CAsT Year 3 (2021)

File Description
2021_automatic_evaluation_topics_v1.0.json 25 primary evaluation topics in JSON format. Variant: Automatic
2021_manual_evaluation_topics_v1.0.json 25 primary evaluation topics in JSON format. Variant: Manual
2021qrels.txt Qrels file for passage ranking task.

CAsT Year 2 (2020)

File Description
2020_automatic_evaluation_topics_v1.0.json 25 primary evaluation topics in JSON format. Variant: Automatic
2020_manual_evaluation_topics_v1.0.json 25 primary evaluation topics in JSON format. Variant: Manual
2020qrels.txt Qrels file for passage ranking task.

CAsT Year 1 (2019)

File Description
train_topics_v1.0.json 30 example training topics in JSON format.
evaluation_topics_v1.0.json 50 evaluation topics in JSON format.
2019qrels.txt Official evaluation qrels file for passage ranking task.
train_qrels.txt Limited (incomplete) training judegements for 5 topics (approximately 50 turns). The judgments are graded on a three point scale (2 very relevant, 1 relevant, and 0 not relevant).