CIKM recognises the importance of publicly available scientific resources, such as datasets, evaluation benchmarks, tools or libraries as a necessary and important prerequisite for fostering research in core CIKM areas.

The Resources Track seeks submissions from both academia and industry that describe resources available to the community.  Resources include, but are not restricted to, information retrieval test collections, labelled datasets for machine learning, and software tools and services.  Furthermore, we also seek papers describing original research on building datasets and resources for broad use.

An ideal resource track paper will fit into one or more of the following categories:

  1. Describing a new and innovative dataset
    • To support research on novel application domains;
    • To support novel evaluation tasks;
    • Created using novel methods and/or algorithms;
    • Labeled using novel and well-described annotation and/or crowdsourcing approaches;
  2. Reusable research prototypes and services;
  3. Open software frameworks, tools and libraries which support tasks in data science, data engineering or information & knowledge management”.

Historically, CIKM has published research in all of these areas.  The goal of the Resource Track is to highlight artifacts of CIKM research and the process of making those artifacts available to the community.  It is expected that at the time of submission the described resource will be available under reasonably liberal terms and sufficiently well-documented such that reviewers may consult that documentation as they conduct their reviews.  (This of course means that reviews will be single-blind for this track.)

Format

Resource track papers should not exceed eight pages in the ACM conference camera ready template. Papers should include a description of the resource, an illustration of the use case(s) for the resource, explain its utility, reusability and novelty, and focus on anticipating and answering questions that CIKM reviewers and readers are likely to ask, as laid out in the review criteria below. It is expected that papers describing datasets clearly explain how the data was collected, all stages of processing from the source to the resulting dataset, and assumptions and risks presented by the collection and/or labeling strategy.
Papers should be submitted through the CIKM online submission system, using the “Resource Track”.

Publication & Presentation

Accepted resources track papers will be published within the official CIKM2020 proceedings as part of the ACM digital library. There should be at least one online registration (at a reduced fee compared to when CIKM was not online) per accepted paper. A video recording of a presentation of each accepted paper will have to be submitted and at least one of the authors will be asked to be present during two online Q&A sessions scheduled in the programme.

Ethics

Resources are expected to be available as described, where “available” means that most researchers in our community could obtain and make use of the resource without strongly limiting the research they can perform with it. Datasets are expected to be collected in accordance with institutional review board standards and ACM standards of ethics. Reviewers are instructed to not use their reviews as an advocacy platform for these issues, but to do what they can to help authors bring their resource to fruition.

Review Criteria

Novelty

  • What is new about this resource?
  • Does the resource represent an incremental advance or something more dramatic?

Availability

  • Is the resource available to the reviewer at the time of review?
  • Are there discrepancies between what is described and what is available?
  • Are the licensing/terms of use sufficiently open as to allow most academic and industry researchers access to the resource?
  • If the resource is data collected from people, do appropriate human subjects control board procedures appear to have been followed?

Utility

  • Is the resource well documented? What level of expertise do you expect is required to make use of the resource?
  • Are there tutorials or examples? Do they resemble actual uses or are they toy examples?
  • If the resource is data, are appropriate tools provided for loading that data?
  • If the resource is data, are the provenance (source, preprocessing, cleaning, aggregation) stages clearly documented?

Predicted Impact

  • What CIKM research activity is enabled by the availability of this resource?
  • Does the resource advance a well-established research area or a brand new one?
  • Do you expect that this resource will be useful for a long time, or will it need to be curated or updated? If the latter, is that planned?
  • How large is the (anticipated) research user community? Will that grow or shrink in the next few years?

Important Dates

Paper abstract submission deadline: June 1, 2020
Paper submission deadline: June 8, 2020
Paper acceptance notification: July 24, 2020
Camera ready submission deadline: August 14, 2020

Chairs

Ian Soboroff, National Institute of Standards and Technology, USA
Paolo Cremonesi, Politecnico di Milano, Italy